Information processing system and movable body, information processing method, and program

Abstract

To provide an information processing system or the like capable of autonomous movement control of a movable body while performing accurate self position estimation seamlessly even in a moving route extending over the inside and outside of a structure.SOLUTION: An information processing system of the present invention estimates the self position of a movable body comprising a receiver that acquires reception information from a satellite positioning system, and a sensor that acquires environmental information, and the information processing system comprises: a reference coordinate conversion unit that converts both a first coordinate system representing the reception information and a second coordinate system representing the environmental information into a reference coordinate system with respect to base point coordinates; and a self position estimation unit that estimates third self position information represented by the reference coordinate system, on the basis of first self position information indicated by the reception information and second self position information calculated through comparison between the environmental information and reference environmental information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a mobile object, an information processing method, and a program. [Background technology]

[0002] In recent years, autonomously controllable mobile objects, such as flying objects (hereinafter collectively referred to as "flying objects") such as drones and unmanned aerial vehicles (UAVs) and running objects such as unmanned ground vehicles (UGVs), have begun to be used in industry. In this context, Patent Document 1 discloses a system in which an flying object sequentially photographs a target at multiple waypoints set in advance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-089160 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology disclosed in Patent Document 1 uses a global navigation satellite system (GNSS) for self-position estimation outdoors and creates a movement route for a moving object based on latitude and longitude information, and a similar method cannot be used for the movement route of a moving object indoors.

[0005] Furthermore, when generating a movement path for a moving object indoors (for example, inside a structure such as a building), a method is conceivable in which, for example, using a technology such as Visual SLAM (Simultaneous Localization and Mapping), three-dimensional information of the indoor space is acquired in advance based on sensor information from a sensor mounted on a moving object whose movement is manually controlled, and the user sets the movement path based on this information. However, the methods for generating outdoor movement paths and indoor movement paths are different, and it cannot be said that the desire to generate a flight path that spans both the inside and outside of a structure has been sufficiently considered.

[0006] The present invention has been made in view of the above background, and aims to provide an information processing system etc. capable of self-estimating a movement route that extends across the inside and outside of a structure. [Means for solving the problem]

[0007] The main invention of the present invention for solving the above problem is an information processing system for estimating the self-position of a moving body, which includes a receiver that acquires received information from a satellite positioning system and a sensor that acquires environmental information, and which includes a reference coordinate conversion unit that converts both a first coordinate system that represents the received information and a second coordinate system that represents the environmental information into a reference coordinate system based on base point coordinates, and a self-position estimation unit that estimates third self-position information expressed in the reference coordinate system based on first self-position information indicated by the received information and second self-position information calculated by comparing the environmental information with reference environmental information. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an information processing system and the like that is capable of self-estimating a movement route that extends both inside and outside a structure. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a configuration of an information processing system according to an embodiment of the present invention. [Figure 2]FIG. 2 is a block diagram showing the hardware configuration of the management server of FIG. 1. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of the user terminal of FIG. 1. [Figure 4] FIG. 2 is a block diagram showing the hardware configuration of the moving body of FIG. [Figure 5] FIG. 2 is a block diagram showing the functions of each component in FIG. 1. [Figure 6] FIG. 1 is a diagram illustrating coordinate transformation according to an embodiment of the present invention. [Figure 7] FIG. 1 is a diagram illustrating coordinate transformation according to an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram illustrating coordinate transformation according to an embodiment of the present invention. [Figure 9] FIG. 2 is a diagram illustrating environmental information according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating reference environment information according to an embodiment of the present invention. [Figure 11] FIG. 10 is a diagram illustrating a comparison between environmental information and reference environmental information according to an embodiment of the present invention. [Figure 12] FIG. 1 is a diagram illustrating self-location estimation according to an embodiment of the present invention. [Figure 13] FIG. 1 is a diagram illustrating self-location estimation according to an embodiment of the present invention. [Figure 14] FIG. 1 is a diagram illustrating self-location estimation according to an embodiment of the present invention. [Figure 15] FIG. 10 is a diagram illustrating an example of estimation of third self-location information according to an embodiment of the present invention. [Figure 16] 1 is a flowchart of a travel route generation method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] The details of the embodiments of the present invention will be described below. An information processing system according to an embodiment of the present invention has the following configuration. [Item 1] An information processing system for estimating a self-position of a moving object, the information processing system including a receiver for acquiring reception information from a satellite positioning system and a sensor for acquiring environmental information, a reference coordinate conversion unit that converts both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on base point coordinates; a self-location estimation unit that estimates third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; Equipped with An information processing system comprising: [Item 2] the self-location estimation unit includes a state estimation filter that receives the first self-location information and the second self-location information and outputs the third self-location information; 2. The information processing system according to item 1, [Item 3] the self-location estimation unit estimates third self-location information expressed in the reference coordinate system based on at least one of the first self-location information and the second self-location information in accordance with a comparison result between the first sensitivity of the receiver, the second sensitivity of the sensor, and a reference sensitivity corresponding to each sensitivity; 2. The information processing system according to item 1, [Item 4] the receiver is a GPS receiver; 4. The information processing system according to any one of items 1 to 3. [Item 5] The sensor is a LiDAR sensor. 5. The information processing system according to any one of items 1 to 4. [Item 6] The sensor is a visual sensor. 5. The information processing system according to any one of items 1 to 4. [Item 7] a movement control unit that compares the third self-location information expressed in the reference coordinate system with movement path information expressed in the reference coordinate system to control movement of the moving object; 7. The information processing system according to any one of items 1 to 6, [Item 8] a movement path information correction unit that corrects the movement path information when the sensor detects an obstacle on the movement path of the moving object; 8. The information processing system according to item 7, [Item 9] An information processing system for estimating a self-position of a moving object, the information processing system including a receiver for acquiring reception information from a satellite positioning system and a sensor for acquiring environmental information, a reference coordinate conversion unit that converts both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on base point coordinates; a self-location estimation unit that estimates third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; Equipped with A moving object characterized by: [Item 10] An information processing method for estimating a self-position of a moving object including a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, comprising: converting, by a reference coordinate conversion unit, both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on base point coordinates; a step of estimating, by a self-location estimation unit, third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; An information processing method characterized by causing a computer to execute the above. [Item 11] A program that causes a computer to execute an information processing method for estimating a self-position of a moving body that includes a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, the program comprising: converting, by a reference coordinate conversion unit, both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on base point coordinates; a step of estimating, by a self-location estimation unit, third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; A program that causes the computer to execute the above.

[0011] <Details of implementation form> Hereinafter, an information processing system according to an embodiment of the present invention will be described. In the accompanying drawings, identical or similar elements are designated by identical or similar reference symbols and names, and duplicate descriptions of identical or similar elements may be omitted in the description of each embodiment. Furthermore, features shown in each embodiment may also be applied to other embodiments as long as they are not mutually inconsistent.

[0012] <Configuration> As shown in Fig. 1, the information processing system in this embodiment includes a management server 1, one or more user terminals 2, one or more moving objects 4 (e.g., flying objects, running objects, etc.), and one or more moving object storage devices 5. The management server 1, user terminal 2, moving objects 4, and moving object storage devices 5 are communicably connected to one another via a network. Note that the illustrated configuration is an example, and is not limited to this. For example, the system may be configured without the moving object storage device 5 and be carried by the user.

[0013] <Management Server 1> 2 is a diagram showing the hardware configuration of the management server 1. Note that the configuration shown in the figure is an example, and other configurations may also be used.

[0014] As shown in the figure, a management server 1 is connected to a user terminal 2, a mobile object 4, and a mobile object storage device 5, and constitutes part of this system. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0015] The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmitting / receiving unit 13, an input / output unit 14, etc., which are electrically connected to one another via a bus 15.

[0016] The processor 10 is a computing device that controls the overall operation of the management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and / or GPU (Graphics Processing Unit), and executes programs for this system that are stored in the storage 12 and deployed in the memory 11 to perform various information processing.

[0017] The memory 11 includes a main memory configured with a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory configured with a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). The memory 11 is used as a work area for the processor 10, and also stores a BIOS (Basic Input / Output System) that is executed when the management server 1 starts up, various setting information, etc.

[0018] The storage 12 stores various programs such as application programs. A database that stores data used for each process may be constructed in the storage 12.

[0019] The transmitting / receiving unit 13 connects the management server 1 to a network. The transmitting / receiving unit 13 may include a short-range communication interface for Bluetooth (registered trademark) and BLE (Bluetooth Low Energy).

[0020] The input / output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

[0021] A bus 15 is commonly connected to the above elements and transmits, for example, address signals, data signals and various control signals.

[0022] <User device 2> 3 also includes a processor 20, a memory 21, a storage 22, a transmitting / receiving unit 23, an input / output unit 24, etc., which are electrically connected to one another via a bus 25. The functions of each element can be configured in the same way as the management server 1 described above, so detailed explanations of each element will be omitted.

[0023] The user terminal 2 is, for example, an information processing device such as a personal computer or a tablet terminal, but may also be configured as a smartphone, a mobile phone, a PDA, etc. In particular, the input / output unit 24 is configured by a display, a keyboard, and a mouse when the user terminal 2 is configured as a personal computer, and is configured by a touch panel or the like when the user terminal 2 is configured as a smartphone or a tablet terminal.

[0024] <Mobile Unit 4> The mobile object 4 is a known mobile object including an air vehicle such as a drone or an unmanned aerial vehicle, or a running object such as an unmanned ground vehicle, and is particularly a mobile object that can be autonomously controlled. As a specific example of the mobile object 4, an air vehicle 4 will be described below. FIG. 4 is a block diagram showing the hardware configuration of the air vehicle 4. The flight controller 41 can have one or more processors such as a programmable processor (e.g., a central processing unit (CPU)).

[0025] The flight controller 41 also has and has access to memory 411. The memory 411 stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412, such as inertial sensors (acceleration sensors, gyro sensors), GPS sensors, and proximity sensors (e.g., lidar).

[0026] The memory 411 may include, for example, a separable medium such as an SD card or random access memory (RAM) or an external storage device. Data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image and video data captured by a camera or the like may be recorded in an internal memory or an external memory, but is not limited to this. The data may be recorded in at least one of the management server 1, the user terminal 2, and the mobile object storage device 5 from the camera / sensor 42 or the internal memory via the network NW. The camera 42 may be installed in the flying object 4 via a gimbal 43.

[0027] The flight controller 41 includes a control module (not shown) configured to control the state of the air vehicle. For example, the control module has six degrees of freedom (translational motion x, y, and z, and rotational motion θ x , θ y and θ z The control module controls the propulsion mechanism (motor 45, etc.) of the aircraft via an ESC 44 (Electric Speed Controller) to adjust the spatial arrangement, speed, and / or acceleration of the aircraft. The motor 45, powered by a battery 48, rotates a propeller 46, generating lift for the aircraft. The control module can control one or more of the states of the onboard components and sensors.

[0028] The flight controller 41 can also communicate with a transceiver 47 configured to transmit data to and / or receive data from one or more external devices (e.g., a transceiver (radio transmitter) 49, the management server 1, the user terminal 2, a display device, or other remote controllers). The transceiver 49 can use any suitable communication means, such as wired or wireless communication.

[0029] Furthermore, the flight controller 41 may not only perform the moving object state control function, such as controlling the state of the flying object, but also perform various functions related to data processing by executing application programs in response to instructions from an external device (particularly the user terminal 2), and may be capable of executing functions corresponding to the movement path generation unit 430 and movement instruction unit 440, which will be described later. Note that the flight controller 41 may be configured to perform data processing functions so that it can be used for both the moving object state control function and the data processing function, but instead may be provided with a separate processor (control unit) dedicated to the data processing function.

[0030] The transceiver 47 may utilize, for example, one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, and the like.

[0031] The transceiver unit 47 can transmit and / or receive one or more of the following: data acquired by the cameras / sensors 42, processing results generated by the flight controller 41, predetermined control data, user commands from a terminal or remote controller, etc.

[0032] The cameras / sensors 42 according to this embodiment may include inertial sensors (acceleration sensors, gyro sensors), receivers (RTK-GPS sensors) that acquire received information from a satellite positioning system, and sensors that acquire environmental information (proximity sensors (e.g., LiDAR (Light Detection And Ranging)), or visual sensors (e.g., including cameras), image sensors).

[0033] <Function of Mobile Unit 4> 5 is a block diagram illustrating functions implemented in the mobile object 4. In an embodiment of the present invention, the mobile object 4 is provided with a receiver that acquires received information from a satellite positioning system and a sensor that acquires environmental information, and the mobile object estimates its own position. The mobile object includes a reference coordinate conversion unit that converts both a first coordinate system that represents the received information and a second coordinate system that represents the environmental information into a reference coordinate system based on base point coordinates, and various functional units that estimate third self-position information expressed in the reference coordinate system based on the first self-position information indicated by the received information and the second self-position information calculated by comparing the environmental information with reference environmental information. Note that some or all of the various functional units may be implemented by an information processing device (processor, control unit) installed in at least one of the management server 1 and the user terminal 2.

[0034] In this embodiment, the moving object 4 includes a reference coordinate conversion unit 410, a self-position estimation unit 420, a movement path generation unit 430, a movement instruction unit 440, a movement path correction unit 450, and a storage unit 470. The storage unit 470 includes various databases, such as a movement information storage unit 471 and a movement path information storage unit 472.

[0035] The reference coordinate conversion unit 410 converts both a first coordinate system (for example, a latitude-longitude-altitude coordinate system (LLA coordinate system)) representing received information from a satellite positioning system acquired by a receiver mounted on the mobile object 4, and a second coordinate system (for example, a Point Cloud Map coordinate system) representing environmental information acquired by a sensor mounted on the mobile object 4, into a reference coordinate system with base point coordinates as the reference (for example, the origin). More specifically, the first coordinate system and the second coordinate system are converted into the reference coordinate system using, for example, conversion information between coordinates stored in advance (for example, conversion information T1 and T2 described below).

[0036] Here, an example of conversion information T1 that is referenced by the reference coordinate conversion unit 410 to convert each coordinate system into the reference coordinate system is shown. As illustrated in FIG. 6, the point cloud map coordinate system and the reference coordinate system exist independently. P1 to P3 in the figure represent point clouds (e.g., point clouds indicating objects such as buildings) contained in the point cloud map. The reference coordinate system is expressed as a three-dimensional coordinate system (XYZ coordinate system) with an arbitrary position as the origin O. On the right side of FIG. 6, the RTK-GPS latitude, longitude, and altitude coordinate system is expressed in the reference coordinate system based on conversion information T2 (not shown) with the reference coordinate system. On the left side of FIG. 6, the point cloud map coordinate system is expressed as a three-dimensional coordinate system (X'Y'Z' coordinate system) with an origin O' at a predetermined position (e.g., the position where the sensor is powered on, the position where a reset process is performed, or the position where the mobile object 4 starts moving). When the mobile object 4 moves while both the receiver and the sensor are operating correctly, the position of the mobile object 4 can be obtained from both. As shown in FIG. 6, a series of acquired self-position results can be plotted as a trajectory.

[0037] In the real world, the position of the moving body 4 at a certain time is determined to be unique, but the position information of the moving body 4 simultaneously acquired from the receiver and the sensor at a certain time is treated as a "pair of information indicating the same location," and the two pieces of position information are associated with the same time and stored, for example, in the movement information storage unit 471. FIG. 7 shows an example in which position information of points whose positions are particularly easy to associate with each other (for example, corners on the moving body's movement path) are stored in pairs. In FIG. 8, conversion information T1 is calculated based on the relationship between the origins in the positional relationship in which the sum of the distances between the stored pairs of position information is minimum. By using this conversion information T1 (and conversion information T2), it is possible to convert both a first coordinate system (e.g., a latitude-longitude-altitude coordinate system) representing information received from the satellite positioning system and a second coordinate system (e.g., a Point Cloud Map coordinate system) representing environmental information acquired by a sensor mounted on the mobile object 4 into a reference coordinate system based on base point coordinates. In other words, it is possible to integrate position information expressed in the first coordinate system and position information expressed in the second coordinate system into the reference coordinate system. Note that this method is not limited to this, and any method may be used as long as it is possible to integrate position information expressed in the first coordinate system and position information expressed in the second coordinate system into the reference coordinate system. Furthermore, although the configuration in which each piece of position information expressed in the first coordinate system and the second coordinate system is converted into the reference coordinate system and integrated has been described, at this time, attitude information at each position expressed in the first coordinate system and the second coordinate system is also similarly converted into the reference coordinate system and integrated.

[0038] The self-location estimation unit 420 estimates third self-location information expressed in the reference coordinate system based on the first self-location information indicated by the received information converted into the reference coordinate system and the second self-location information calculated by comparing the environmental information with reference environmental information. The first self-location information may be, for example, location information acquired by RTK-GPS. The second self-location information is location information determined by comparing environmental information acquired by a LiDAR sensor or a Visual sensor (including a camera) (for example, in FIG. 9, three-dimensional point cloud map information acquired by a LiDAR or the like) with reference environmental information acquired in advance by a sensor that acquires environmental information (for example, in FIG. 10, reference three-dimensional point cloud map information acquired in advance by a LiDAR or the like). The self-location estimation unit 420 determines from which observation position the environmental information was acquired by the comparison, and sets the observation position as the second self-location information. When the environmental information and the reference environmental information are expressed as point cloud maps, the second self-location information can be estimated by, for example, comparing the shapes of both pieces of environmental information like matching a puzzle using known techniques such as NDT registration and NDT Scan Matching (for example, FIG. 11 shows information in which point cloud map information and reference point cloud map information are superimposed), and determining the coordinate transformation that maximizes the degree of match between the two.

[0039] Furthermore, as a specific example, the self-location estimation unit 420 may include a state estimation filter such as a Kalman filter or a particle filter that estimates third self-location information using, as input, first self-location information (e.g., self-location information based on RTK-GPS) and second self-location information (e.g., self-location information based on a LiDAR sensor) converted into a reference coordinate system. Among these, the Kalman filter is a filter that has a function of estimating a plausible state quantity by integrating multiple observation values, and it is possible to use the first self-location information and second self-location information converted into the same reference coordinate system, thereby making it possible to estimate the third self-location information. In addition, if the first self-location information or the second self-location information does not have sufficient sensitivity (accuracy) to estimate the third self-location information, that is, if the first self-location information is based on, for example, RTK-GPS and the sensitivity is below a predetermined value depending on the communication conditions of the RTK-GPS and does not meet the criteria (for example, Figure 14 shows that the sensitivity of the RTK-GPS has become 0 because the moving body 4 has entered a structure), or if, in a form using reference environmental information for the second self-location information, the location is not one where sufficient reference environmental information is available for comparison and the score of the NDT registration etc. does not meet the criteria (for example, Figure 12 shows that there is little reference environmental information around the moving body 4 and the score indicating the discrepancy index is high), then the self-location information that does not have sufficient sensitivity may not be used as input.

[0040] As another specific example, when the first self-location information or the second self-location information is not sensitive enough to estimate the third self-location information, the self-location estimation unit 420 estimates the third self-location information by adopting self-location information that shows sufficient sensitivity (for example, the first self-location information in FIG. 12 and the second self-location information in FIG. 14) as described above. However, when both self-location information show sufficient sensitivity as shown in FIG. 13, for example, the self-location information to be preferentially adopted may be set in advance by user operation, or the third self-location information may be estimated by weighting the self-location information that shows a higher sensitivity based on the comparison result between one or more pre-set reference sensitivities and each sensitivity so that the self-location information that shows a higher sensitivity is deemed to be more reliable (that is, as exemplified on the left side of FIG. 15, when both sensitivities show the same degree of reliability, the central position of the first self-location information and the second self-location information is estimated as the third self-location information with equal weighting, and when one sensitivity is indicated to be highly reliable, the third self-location information may be estimated as a position with a ratio shifted toward the self-location information that shows a higher sensitivity).

[0041] Furthermore, when the self-location estimation unit 420 determines that neither the first self-location information nor the second self-location information is of sufficient sensitivity to estimate the third self-location information (for example, this may be determined based on the above-mentioned reference sensitivity), the self-location estimation unit 420 may activate an emergency stop function by transmitting a signal to the movement instruction unit 440 described below to stop the movement of the moving body 4.

[0042] In this way, it is possible to use the first self-location information and the second self-location information converted into the same reference coordinate system for multiple pieces of observation information obtained from different configurations and expressed in different coordinate systems, thereby making it possible to estimate the third self-location information with high accuracy.

[0043] The travel route generation unit 430 may, for example, set one or more waypoint information sequentially from a start point to an end point, or set arbitrary points in an arbitrary order, by a user's selection operation on three-dimensional model data displayed on the user terminal 2 (which may be, for example, reference three-dimensional point cloud map information as shown in FIG. 10), generate travel route information based on the waypoint information using a known method, and store and manage the information in the travel route information storage unit 472; or may analyze three-dimensional environmental data and calculate a travel route by setting waypoint information that can obtain information on specific or all components inside and outside a structure (for example, internal components such as interior walls, pillars, ceilings, windows, doors, stairs, and internal facilities, and external components such as exterior walls, roofs, external facilities, windows, doors, stairs, roads, railways, stations, streetlights, bus stops, bridges, tunnels, topography, vegetation, water areas, gas meters and other meters, etc.), and store and manage the information in the travel route information storage unit 472 as travel route information.

[0044] In addition, the movement route may be generated by, for example, using the position of the mobile body storage device 5 as the start position and end position of the movement, and passing through each waypoint, or conversely, it may be configured so that the start position of the movement is the position where the user carries the aircraft without having a mobile body storage device 5, and the user retrieves the aircraft at the end position of the movement, or it may be configured so that the movement route includes the position of the mobile body storage device 5 selected as the start position or end position of the movement based on information about the mobile body storage device 5 managed in the memory of the management server 1, the user terminal 2, or the mobile body 4 (for example, position information, storage status information, storage machine information, etc.).

[0045] The three-dimensional model data may utilize reference environment information such as reference three-dimensional point cloud map information, as described above, but is not limited to this. For example, the three-dimensional model data may be a model created based on data created using CAD (Computer-Aided Design) design software, such as BIM (Building Information Modeling) data or CIM (Construction Information Modeling) data, or three-dimensional model data reconstructed from CAD data or BIM data. Alternatively, the three-dimensional model data may be three-dimensional model data obtained by generating a structure having a predetermined height based on two-dimensional blueprint data. Alternatively, the three-dimensional model data may be three-dimensional city model data in formats such as CityGML (Generalized Markup Language), CityJson, or GeoTIFF, or three-dimensional city model data stored in a three-dimensional city model database external to the system. The reconstruction of the three-dimensional model data may be executed by a processor in the management server 1 or the user terminal 2, or may be executed externally to the management server 1 or the user terminal 2 and acquired internally.

[0046] Furthermore, when the reference environment information is reference 3D point cloud map information, the reference 3D point cloud map information may be information acquired in advance by a sensor such as LiDAR as described above. Alternatively, for example, 3D point cloud model data may be used in which the model surfaces inside and outside the structure of the above-mentioned 3D model data are converted into a point cloud in a processor of the management server 1 or the user terminal 2. Regarding a method for generating 3D point cloud model data, for example, 3D point cloud model data regarding components inside or outside the structure may be generated by moving a virtual mobile object 4 equipped with a virtual sensor (e.g., a virtual LiDAR) inside or outside the structure of the 3D model data. This theoretically makes it possible to generate point cloud data that is close to the point cloud sensing data obtained when the inside or outside of the structure is actually measured by the sensor of the mobile object 4. Other methods for generating 3D point cloud model data include converting the 3D model data into a point cloud by uniformly converting the 3D model data into a point cloud at a predetermined interval. If the 3D model data is polygon data, the point cloud may be generated by placing points at each vertex. Alternatively, the point cloud may be generated using a known point cloud conversion technique (technique for converting to point cloud data). The generated three-dimensional point cloud model data is stored in the storage unit 470, the management server 1, or the user terminal 2.

[0047] The movement instruction unit 440 refers to the movement route information stored in the movement route information storage unit 472 and transmits to the moving object 4 information instructing the moving object 4 to move in accordance with the coordinates indicated by the movement route information and the third self-location information estimated as described above. That is, the first self-location information, which is particularly sensitive outside a structure (e.g., a building) and does not require prior acquisition of reference information, and the second self-location information, which is sensitive even inside a structure but requires prior acquisition of reference information, are converted into reference coordinate systems, and the third self-location information expressed in the reference coordinate system estimated based on the converted self-location information is used. This makes it possible to compare the information indicated by the movement route information with the third self-location information whether inside a structure or outside a structure where reference environment information is not available, thereby enabling seamless movement instruction control of the moving object 4 even on a movement route that spans inside and outside a structure. Note that the movement route information may be generated by the movement route generation unit 430 described above, or may be generated and stored in an external system.

[0048] Here, the moving object 4 may further include a movement path correction unit 450. When a sensor for acquiring environmental information confirms the presence of an obstacle near the movement path (within a predetermined distance range of the movement path, including on the movement path), the movement path correction unit 450 corrects the movement path information referenced by the movement instruction unit 440. Note that in this system, the movement path can be expressed in a reference coordinate system spanning both the inside and outside of a structure, and in cases where there is an obstacle around the entrance or exit of a structure, for example, the movement path correction range can target the movement path spanning both the inside and outside of the structure.

[0049] The movement information storage unit 471 stores parameter information used when the movement path generation unit 430 generates a movement path, or when the movement instruction unit 440 instructs the movement of the autonomously controlled moving body 4 on the movement path, as well as information acquired during movement acquired on the movement path. Specific examples of parameters include, for example, movement speed, flight altitude (if the moving body 4 is an air vehicle), overlap rate of captured images, and information acquired during movement (for example, image information, video information, environmental information, etc.).

[0050] The travel route information storage unit 472 stores coordinate information (so-called waypoint information) on the travel route generated by the travel route generation unit 430. As described above, travel route information generated by a processor of the management server 1 or the user terminal 2, or an external system may also be stored.

[0051] <User device 2 functions> 5 is a block diagram illustrating functions implemented in the user terminal 2. Note that some or all of the various functional units may be realized by an information processing device (processor, control unit) installed in at least one of the management server 1 and the mobile object 4.

[0052] In this embodiment, the user terminal 2 includes a communication unit 210, a screen information generation unit 220, and a storage unit 270.

[0053] The communication unit 210 communicates with the management server 1, the mobile object 4, and the mobile object storage device 5. The communication unit 210 also functions as a reception unit that receives various requests, data, etc. from the management server 1, the mobile object 4, and the mobile object storage device 5.

[0054] The screen information generation unit 220 generates screen information to be displayed via the user interface of the user terminal 2. For example, the screen information generation unit 220 configures a user interface screen by arranging various images and text based on predetermined layout rules, and generates screen information for displaying various pieces of information acquired by the moving object 4 on the user interface screen.

[0055] The information processing method according to this embodiment will be described, including the operation of the information processing system according to this embodiment, with reference to Fig. 16. Fig. 16 shows a flowchart of the information processing method according to this embodiment. This flowchart shows, as an example, a configuration in which an application is launched on the user terminal 2, but this is not limiting. For example, the management server 1, the mobile object 4, and the mobile object storage device 5 may each have a processor and an input / output device capable of launching an application, and may be configured to allow various settings, etc.

[0056] First, the user starts an application on, for example, the user terminal 2 to operate the mobile object 4 and display acquired information (SQ101). This application may be stored in, for example, the user terminal 2, or may be software (so-called SaaS) provided by the management server 1, the mobile object 4, or another external server (not shown) connected via a network. If necessary, a login screen may be displayed, requesting, for example, a login ID and password.

[0057] Next, the user creates a new travel plan (SQ102). For example, the user sets the "plan name," "area name," "address," etc., and then obtains and displays three-dimensional model data of the travel target on the user terminal 2, and begins creating a new travel plan.

[0058] Next, the user generates a travel route for the movement of the moving object 4 (SQ103). For example, the user sets one or more waypoint information (for example, expressed in a latitude-longitude-height coordinate system on the user terminal 2) by a selection operation on the three-dimensional model data displayed on the user terminal 2. Then, the three-dimensional model data and waypoint information are transmitted to the moving object 4, and the moving object 4 generates travel route information based on the three-dimensional model data and waypoint information using a known method (for example, connecting the set waypoints with straight lines).

[0059] Next, the user instructs the moving object 4 to start moving (SQ104). For example, the user references the movement information storage unit 471 and the movement route information storage unit 472 to move the moving object 4 for the purpose of inspection, security, construction progress management, etc. At this time, the first self-location information acquired by the receiver of the moving object 4 and the second self-location information obtained by comparing the environmental information from the sensor with the reference environmental information are each converted into a reference coordinate system, and the movement of the moving object 4 is controlled based on the third self-location information expressed in the reference coordinate system estimated based on the converted self-location information and the waypoint information of the movement route information, etc.

[0060] Next, the user instructs the user terminal 2 to output the acquired information (SQ105). For example, route information actually traveled by the mobile object 4 may be superimposed on the three-dimensional model data displayed on the user terminal 2. In addition, acquired information (still images, moving images, audio, and other information) acquired by the mobile object 4 along the travel route may be displayed, or a mark such as a symbol may be added as a link to view the acquired information corresponding to a position (particularly, position information of a waypoint) associated with the position information of the acquired information. Then, by selecting the link on the user terminal 2, the corresponding acquired information may be displayed.

[0061] In this way, the present invention can provide an information processing system and the like that is capable of autonomously controlling the movement of the moving object 4 while seamlessly performing highly accurate self-position estimation even on a movement route that extends inside and outside a structure.

[0062] Furthermore, in the above-described embodiment, the mobile object 4 acquires information inside and outside a structure, but the mobile object 4 may also be used for inspecting the structure, and may be provided with devices, equipment, etc. used to inspect the presence or absence of a predetermined event on the interior and / or exterior walls of the structure. More specifically, any device necessary to know the state of an inspected structure having interior or exterior walls, such as an imaging device (visible light camera, infrared camera, metal detector, ultrasonic measuring device, etc.), a keystroke device, a detecting device (metal detector), a sound collecting device, an odor measuring device, a gas detector, an air pollution measuring device, or a detecting device (device for detecting cosmic rays, radiation, electromagnetic waves, etc.), may be employed.

[0063] Furthermore, an embodiment may be, for example, security or surveillance within a structure, and may include devices, equipment, etc. used for security or surveillance. More specifically, any device necessary for capturing images or detecting abnormalities or intruders in a structure to be guarded or monitored, such as an imaging device (visible light camera, infrared camera, night vision camera, metal detector, ultrasonic measuring device, etc.), or a sensor device (motion sensor, infrared sensor, etc.), may be employed.

[0064] The mobile body of the present invention can be suitably used as a mobile body for photography equipped with a camera or the like, and can also be used in various industries such as security, infrastructure monitoring, surveying, inspection of buildings and structures such as sports venues, factories, and warehouses, and disaster response.

[0065] The above-described embodiment is merely an example for facilitating understanding of the present invention, and is not intended to limit the present invention. The present invention can be modified and improved without departing from the spirit thereof, and it goes without saying that the present invention includes equivalents thereof. [Explanation of symbols]

[0066] 1 Management Server 2. User terminal 4. Mobile 5 Mobile storage device

Claims

1. An information processing system for estimating a self-position of a moving object, the information processing system including a receiver for acquiring reception information from a satellite positioning system and a sensor for acquiring environmental information, a reference coordinate conversion unit that converts both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate; a self-location estimation unit that estimates third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; Equipped with An information processing system comprising:

2. the self-location estimation unit includes a state estimation filter that receives the first self-location information and the second self-location information and outputs the third self-location information; 2. The information processing system according to claim 1, wherein:

3. the self-location estimation unit estimates third self-location information expressed in the reference coordinate system based on at least one of the first self-location information and the second self-location information in accordance with a comparison result between the first sensitivity of the receiver, the second sensitivity of the sensor, and a reference sensitivity corresponding to each sensitivity; 2. The information processing system according to claim 1, wherein:

4. the receiver is a GPS receiver; 4. The information processing system according to claim 1, wherein:

5. The sensor is a LiDAR sensor.

4. The information processing system according to claim 1, wherein:

6. The sensor is a visual sensor.

4. The information processing system according to claim 1, wherein:

7. a movement control unit that compares the third self-location information expressed in the reference coordinate system with movement path information also expressed in the reference coordinate system to control movement of the moving object; 4. The information processing system according to claim 1, wherein:

8. a movement path information correction unit that corrects the movement path information when the sensor detects an obstacle on the movement path of the moving object; 8. The information processing system according to claim 7,

9. An information processing system for estimating a self-position of a moving object, the information processing system including a receiver for acquiring reception information from a satellite positioning system and a sensor for acquiring environmental information, a reference coordinate conversion unit that converts both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate; a self-location estimation unit that estimates third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; Equipped with A moving object characterized by:

10. An information processing method for estimating a self-position of a moving object including a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, comprising: a step of converting, by a reference coordinate conversion unit, both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate; a step of estimating, by a self-location estimation unit, third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; An information processing method characterized by causing a computer to execute the above.

11. A program that causes a computer to execute an information processing method for estimating a self-position of a moving body that includes a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, the program comprising: a step of converting, by a reference coordinate conversion unit, both a first coordinate system representing the received information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate; a step of estimating, by a self-location estimation unit, third self-location information expressed in the reference coordinate system based on first self-location information indicated by the received information and second self-location information calculated by comparing the environmental information with reference environmental information; A program that causes the computer to execute the above.

Images