Information Processing System, Mobile Body, Information Processing Method, Program

Abstract

[Problem] The present invention provides an information processing system and the like which enable autonomous movement control of a movable body while seamlessly performing highly accurate self-position estimation in a movement path over the inside and the outside of a structure. [Solution] An information processing system according to the present invention is for self-position estimation of a movable body provided with a receiver for acquiring reception information from a satellite positioning system and with a sensor for acquiring environment information. The information processing system is provided with: a reference coordinate conversion unit for converting both a first coordinate system indicating the reception information and a second coordinate system indicating the environment information into a reference coordinate system based on base point coordinates; and a self-position estimation unit for estimating third self-position information represented by the reference coordinate system, on the basis of first self-position information indicated by the reception information and second self-position information that is calculated by comparing the environment information with reference environment information.

Description

Technical Field

[0001] The present invention relates to an information processing system, a moving body, an information processing method, and a program.

Background Art

[0002] In recent years, autonomous controllable moving bodies such as flying bodies (hereinafter collectively referred to as "flying bodies") such as drones and unmanned aerial vehicles (UAVs), and running bodies such as unmanned ground vehicles (UGVs) have begun to be used in the industry. Under these circumstances, Patent Document 1 discloses a system in which a flying body sequentially photographs a photographing target at a plurality of preset waypoints.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the disclosed technology of Patent Document 1 uses GNSS (global navigation satellite system) for self-position estimation outdoors and creates a moving path of a moving body based on latitude and longitude information, and the same method cannot be used for the moving path of a moving body indoors.

[0005] In addition, when generating a moving path of a moving object indoors (e.g., inside a structure such as a building), for example, using a technology such as Visual SLAM (Simultaneous Localization and Mapping), based on the sensor information of sensors mounted on a moving object that is manually controlled for movement, three-dimensional information indoors is acquired in advance, and based on this, a method in which the user performs a setting operation of the moving path can be considered. However, the method for creating an outdoor moving path and the method for creating an indoor moving path are separate, and it cannot be said that the desire to create a flight path that straddles the inside and outside of a structure has been fully studied.

[0006] The present invention has been made in view of such a background, and an object thereof is to provide an information processing system or the like that can self-estimate in a moving path that straddles the inside and outside of a structure.

Means for Solving the Problems

[0007] The main invention of the present invention for solving the above problems is an information processing system for estimating the self-position of a moving object including a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, the information processing system including: a reference coordinate conversion unit that converts both a first coordinate system representing the reception information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate; and a self-position estimation unit that estimates third self-position information represented in the reference coordinate system based on first self-position information indicated by the reception information and second self-position information calculated by comparing the environmental information with reference environmental information.

Effects of the Invention

[0008] According to the present invention, in particular, it is possible to provide an information processing system or the like that can self-estimate in a moving path that straddles the inside and outside of a structure.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0010] The content of the embodiment of the present invention will be listed and described. The information processing system etc. according to the embodiment of the present invention has the following configuration. [Item 1] An information processing system for estimating the self-position of a moving body including a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, a reference coordinate conversion unit that converts both a first coordinate system representing the reception information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate, a self-position estimation unit that estimates third self-position information represented in the reference coordinate system based on first self-position information indicated by the reception information and second self-position information calculated by comparing the environmental information with reference environmental information, comprising An information processing system characterized by the above. [Item 2] The self-position estimation unit includes a state estimation filter into which the first self-position information and the second self-position information are input and that outputs the third self-position information. The information processing system according to Item 1, characterized by the above. [Item 3] The self-position estimation unit estimates third self-position information represented in the reference coordinate system based on at least one of the first self-position information and the second self-position information according to a comparison result between a first sensitivity of the receiver, a second sensitivity of the sensor, and a reference sensitivity corresponding to each sensitivity. The information processing system according to Item 1, characterized by the above. [Item 4] The receiver is a GPS receiver. The information processing system according to any one of Items 1 to 3, characterized by the above. [Item 5] The sensor is a LiDAR sensor. The information processing system according to any one of Items 1 to 4, characterized by the above. [Item 6] The sensor is a Visual sensor. The information processing system according to any one of Items 1 to 4, characterized by the above. [Item 7] Further comprising a movement control unit that compares the third self-position information represented in the reference coordinate system with the movement path information represented in the reference coordinate system and controls the movement of the moving body. The information processing system according to any one of Items 1 to 6, characterized in that. [Item 8] Further comprising a movement path information correction unit that corrects the movement path information when an obstacle is detected on the movement path of the moving body by the sensor. The information processing system according to Item 7, characterized in that. [Item 9] An information processing system for estimating the self-position of a moving body comprising a receiver for acquiring reception information from a satellite positioning system and a sensor for acquiring environmental information, A reference coordinate conversion unit that converts both the first coordinate system representing the reception information and the second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate, A self-position estimation unit that estimates third self-position information represented in the reference coordinate system based on first self-position information indicated by the reception information and second self-position information calculated by comparing the environmental information with reference environmental information, Comprising. A moving body, characterized in that. [Item 10] An information processing method for estimating the self-position of a moving body comprising a receiver for acquiring reception information from a satellite positioning system and a sensor for acquiring environmental information, A step of converting, by a reference coordinate conversion unit, both the first coordinate system representing the reception information and the second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate, A step of estimating, by a self-position estimation unit, third self-position information represented in the reference coordinate system based on first self-position information indicated by the reception information and second self-position information calculated by comparing the environmental information with reference environmental information, An information processing method characterized by causing a computer to execute. [Item 11] A program that causes a computer to execute an information processing method for estimating the self-position of a mobile body including a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, a step of converting, by a reference coordinate conversion unit, both a first coordinate system representing the reception information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate, a step of estimating, by a self-position estimation unit, third self-position information represented by the reference coordinate system based on first self-position information indicated by the reception information and second self-position information calculated by comparing the environmental information with reference environmental information, wherein the computer is caused to execute the program.

[0011] <Details of Embodiment> Hereinafter, embodiments of an information processing system and the like according to embodiments of the present invention will be described. In the accompanying drawings, the same or similar elements are given the same or similar reference numerals and names, and duplicate descriptions regarding the same or similar elements may be omitted in the description of each embodiment. Further, the features shown in each embodiment are applicable to other embodiments as long as they do not contradict each other.

[0012] <Configuration> As shown in FIG. 1, the information processing system in the present embodiment includes a management server 1, one or more user terminals 2, one or more mobile bodies 4 (for example, aircraft, vehicles, etc.), and one or more mobile body storage devices 5. The management server 1, the user terminal 2, the mobile body 4, and the mobile body storage device 5 are communicably connected to each other via a network. Note that the illustrated configuration is an example, and the present invention is not limited thereto. For example, a configuration in which the mobile body storage device 5 is not provided and the mobile body is carried by a user may be used.

[0013] <Management Server 1> FIG. 2 is a diagram showing the hardware configuration of the management server 1. Note that the illustrated configuration is an example, and the management server 1 may have other configurations.

[0014] As shown in the figure, the management server 1 is connected to the user terminal 2, the mobile body 4, and the mobile body storage device 5 and constitutes a part of this system. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0015] The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14, etc., and these are electrically connected to each other through a bus 15.

[0016] The processor 10 is an arithmetic unit that controls the overall operation of the management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for the execution and authentication processing of applications. For example, the processor 10 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs for this system stored in the storage 12 and expanded in the memory 11 to perform each information processing.

[0017] The memory 11 includes a main memory composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory composed of a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). The memory 11 is used as a work area of the processor 10, and stores a BIOS (Basic Input / Output System) executed when the management server 1 is started, and various setting information, etc.

[0018] The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12.

[0019] The transceiver unit 13 connects the management server 1 to the network. Note that the transceiver unit 13 may be provided with a short-range communication interface for Bluetooth (registered trademark) and BLE (Bluetooth Low Energy).

[0020] The input / output unit 14 includes information input devices such as keyboards and mice, and output devices such as displays.

[0021] The bus 15 is commonly connected to each of the above elements and transmits, for example, address signals, data signals, and various control signals.

[0022] <User terminal 2> The user terminal 2 shown in FIG. 3 also includes a processor 20, a memory 21, a storage 22, a transceiver unit 23, an input / output unit 24, etc., which are electrically connected to each other through a bus 25. Since the functions of each element can be configured in the same manner as those of the management server 1 described above, detailed descriptions of each element are omitted.

[0023] The user terminal 2 is, for example, an information processing device such as a personal computer or a tablet terminal, but may be configured by a smartphone, a mobile phone, a PDA, etc. In particular, when the user terminal 2 is configured by a personal computer, the input / output unit 24 is composed of a display, a keyboard, and a mouse, and when the user terminal 2 is configured by a smartphone or a tablet terminal, it is composed of a touch panel or the like.

[0024] <Mobile body 4> The mobile body 4 is a known mobile body including a flying body such as a drone or an unmanned aerial vehicle, or a traveling body such as an unmanned ground vehicle, and is particularly a mobile body capable of autonomous control. As a specific example of the mobile body 4, the flying body 4 will be exemplified and described below. FIG. 4 is a block diagram showing the hardware configuration of the flying body 4. The flight controller 41 can have one or more processors such as a programmable processor (for example, a central processing unit (CPU)).

[0025] In addition, the flight controller 41 has a memory 411 and can access the memory. The memory 411 stores logic, code, and / or program instructions that can be executed by the flight controller to perform one or more steps. Also, the flight controller 41 may include sensors 412 such as inertial sensors (acceleration sensors, gyro sensors), GPS sensors, proximity sensors (e.g., lidar), etc.

[0026] The memory 411 may include a separable medium or an external storage device such as an SD card or a random access memory (RAM). The data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image / moving image data captured by a camera or the like may be recorded in the built-in memory or an external memory, but not limited to this, it may be recorded in at least one of the management server 1, the user terminal 2, and the mobile body storage device 5 via the network NW from the camera / sensor 42 or the built-in memory. The camera 42 may be installed on the flying object 4 via a gimbal 43.

[0027] The flight controller 41 includes a control module (not shown) configured to control the state of the flying object. For example, the control module adjusts the spatial arrangement, speed, and / or acceleration of a flying object having six degrees of freedom (translational motions x, y, and z, and rotational motions θ x , θ y and θ z ) by controlling the propulsion mechanism (such as the motor 45) of the flying object via an ESC 44 (Electric Speed Controller). The propeller 46 rotates by the motor 45 powered by the battery 48 to generate lift for the flying object. The control module can control one or more of the mounting part and the states of the sensors.

[0028] In addition, the flight controller 41 is communicable with a transmission / reception unit 47 configured to enable transmission of data to one or more external devices (e.g., a transceiver (propo) 49, a management server 1, a user terminal 2, a display device, or another remote controller) and / or reception of data from the external devices. The transceiver 49 can use any suitable communication means such as wired communication or wireless communication.

[0029] Furthermore, the flight controller 41 may not only perform a mobile body state control function such as controlling the state of the above-described flying object, but also implement various functions related to data processing by executing an application program in response to an instruction from an external device (particularly the user terminal 2). For example, it may be capable of executing functions corresponding to a movement path generation unit 430 and a movement instruction unit 440 described later. Note that the flight controller 41 may implement functions related to data processing to serve both as a mobile body state control function and a data processing function. Alternatively, a dedicated processor (control unit) for data processing may be provided separately.

[0030] The transmission / reception unit 47 can utilize one or more of, for example, a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communication, and the like.

[0031] The transmission / reception unit 47 can transmit and / or receive one or more of data acquired by the camera / sensors 42, a processing result generated by the flight controller 41, predetermined control data, a user command from a terminal or a remote controller, and the like.

[0032] The cameras / sensors 42 according to this embodiment may include an inertial sensor (an acceleration sensor, a gyro sensor), a receiver (RTK-GPS sensor) that acquires reception information from a satellite positioning system, and a sensor that acquires environmental information (a proximity sensor (e.g., LiDAR (Light Detection And Ranging), etc.) or a Visual sensor (e.g., including a camera), an image sensor).

[0033] <Functions of the moving body 4> FIG. 5 is a block diagram illustrating the functions implemented in the moving body 4. In an embodiment of the present invention, it estimates the self-position of a moving body including a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, and a reference coordinate conversion unit that converts both a first coordinate system representing the reception information and a second coordinate system representing the environmental information into a reference coordinate system based on a base point coordinate, and various functional units for estimating third self-position information represented in the reference coordinate system based on first self-position information indicated by the reception information and second self-position information calculated by comparing the environmental information with reference environmental information. Note that some or all of the various functional units may be realized by an information processing device (processor, control unit) mounted on at least one of the management server 1 and the user terminal 2.

[0034] In this embodiment, the moving body 4 includes a reference coordinate conversion unit 410, a self-position estimation unit 420, a movement route generation unit 430, a movement instruction unit 440, a movement route correction unit 450, and a storage unit 470. The storage unit 470 includes various databases such as a movement information storage unit 471 and a movement route information storage unit 472.

[0035] The reference coordinate conversion unit 410 converts both the first coordinate system (e.g., the latitude-longitude-altitude coordinate system (LLA coordinate system)) representing the reception information from the satellite positioning system acquired by the receiver mounted on the mobile body 4 and the second coordinate system (e.g., the Point Cloud Map coordinate system) representing the environmental information acquired by the sensor mounted on the mobile body 4 into a reference coordinate system with the base point coordinate as a reference (e.g., the origin). More specifically, for example, using the conversion information between coordinates (e.g., the conversion information T1, T2 described later) stored in advance, the first coordinate system and the second coordinate system are converted into the reference coordinate system.

[0036] Here, an example of the conversion information T1 referred to for converting each coordinate system into the reference coordinate system in the reference coordinate conversion unit 410 is shown. As illustrated in FIG. 6, the point cloud map coordinate system and the reference coordinate system exist independently. P1 to P3 in the figure represent the point cloud of the point cloud map (e.g., the point cloud indicating an object such as a building). The reference coordinate system is represented by a three-dimensional coordinate system (XYZ coordinate system) with an arbitrary position as the origin O. On the right side of FIG. 6, the latitude-longitude-altitude coordinate system of RTK-GPS is represented in the reference coordinate system based on the conversion information T2 (not shown) with the reference coordinate system. On the left side of FIG. 6, the point cloud map coordinate system is represented by a three-dimensional coordinate system (X'Y'Z' coordinate system) with a predetermined position (e.g., the position where the sensor is turned on, the position where the reset process is performed, the position where the mobile body 4 starts to move, etc.) as the origin O'. When the mobile body 4 moves with both the receiver and the sensor operating correctly, the position of the mobile body 4 can be obtained from both of them. As illustrated in FIG. 6, a series of acquired self-position results can be drawn as a trajectory.

[0037] In the real world, while the position of the moving object 4 at a certain moment is uniquely determined, the position information of the moving object 4 simultaneously obtained from the receiver and the sensor at a certain moment is associated with both pieces of position information at the same time as a "pair of information indicating the same location", and stored in, for example, the movement information storage unit 471. In FIG. 7, an example is shown in which position information at points where the positions to be associated with each other are particularly easy to understand (for example, a portion corresponding to a corner on the movement path of the moving object) is stored in pairs. In FIG. 8, the conversion information T1 is calculated based on the relationship between the origins in the positional relationship where the sum of the distances between the position information of each pair stored in this way is minimized. By using this conversion information T1 (and conversion information T2), the first coordinate system representing the reception information from the satellite positioning system (for example, the latitude-longitude-altitude coordinate system) and the second coordinate system representing the environmental information obtained by the sensor mounted on the moving object 4 (for example, the Point Cloud Map coordinate system) can both be converted into a reference coordinate system based on the base point coordinates, that is, the position information expressed in the first coordinate system and the position information expressed in the second coordinate system can be integrated into the reference coordinate system. Note that this is not limited to this method, and any method may be used as long as the position information expressed in the first coordinate system and the position information expressed in the second coordinate system can be integrated into the reference coordinate system. In addition, although the configuration in which each piece of position information expressed in the first coordinate system and the second coordinate system is converted into the reference coordinate system and integrated has been described, at this time, the attitude information at each position expressed in the first coordinate system and the second coordinate system is also similarly converted into the reference coordinate system and integrated.

[0038] The self-position estimation unit 420 estimates the third self-position information represented in the reference coordinate system based on the first self-position information indicated by the received information, which has been converted into the reference coordinate system, and the second self-position information calculated by comparing the environmental information with the reference environmental information. The first self-position information can be, for example, position information obtained by RTK-GPS. The second self-position information is, for example, environmental information (e.g., the three-dimensional point cloud map information obtained by LiDAR etc. in FIG. 9) obtained by sensors such as LiDAR sensors and Visual sensors (including cameras), and the reference environmental information (e.g., the reference three-dimensional point cloud map information obtained by LiDAR etc. in advance in FIG. 10) obtained by sensors that acquire environmental information in advance. It is the position information determined by comparing them with each other, determines from which observation position the environmental information is obtained by the comparison, and sets the observation position as the second self-position information. When the environmental information and the reference environmental information are represented as point cloud maps, for example, the shapes of both environmental information are compared like a puzzle using known techniques such as NDT registration and NDT Scan Matching (e.g., FIG. 11 shows the information obtained by overlapping the point cloud map information and the reference point cloud map information), and by obtaining the coordinate transformation with the highest degree of coincidence between the two, it is possible to estimate the second self-position information.

[0039] Furthermore, as a specific example, the self-position estimation unit 420 may include a state estimation filter such as a Kalman filter or a particle filter that estimates third self-position information using, as inputs, first self-position information (e.g., self-position information based on RTK-GPS) and second self-position information (e.g., self-position information based on a LiDAR sensor) that have been converted into a reference coordinate system. Among these, the Kalman filter is a filter that has a function of integrating a plurality of observation values and estimating a plausible state quantity. Since it is possible to use the first self-position information and the second self-position information that have been converted into the same reference coordinate system, it is possible to estimate the third self-position information. When the first self-position information or the second self-position information does not have sufficient sensitivity (accuracy) for estimating the third self-position information, that is, when the first self-position information is based on RTK-GPS, for example, and the sensitivity becomes less than or equal to a predetermined value depending on the communication status of RTK-GPS and does not meet the standard (e.g., in FIG. 14, it is shown that the sensitivity of RTK-GPS becomes 0 because the moving body 4 has entered the structure), or when the reference environmental information is used for the second self-position information and the reference environmental information for comparison is not prepared sufficiently at the position where the score of NDT registration or the like does not meet the standard (e.g., in FIG. 12, it is shown that there is little reference environmental information around the moving body 4 and the score indicating the index of inconsistency is high), etc., it may be possible not to adopt the self-position information for which sufficient sensitivity cannot be obtained as an input.

[0040] Also, as another specific example, when the first self-position information or the second self-position information does not have sufficient sensitivity for estimating the third self-position information, the self-position estimation unit 420 adopts self-position information (for example, the first self-position information in FIG. 12 and the second self-position information in FIG. 14) that shows sufficient sensitivity as described above to estimate the third self-position information. However, as shown in FIG. 13, when both pieces of self-position information show sufficient sensitivity, for example, the self-position information to be preferentially adopted by user operation may be set in advance, or based on the comparison result between one or more preset reference sensitivities and each sensitivity, the third self-position information may be estimated by weighting so that the self-position information showing higher sensitivity is regarded as having higher reliability (that is, as illustrated on the left side of FIG. 15, when both sensitivities show the same level of reliability, equal weighting is used and the central position of the first self-position information and the second self-position information is estimated as the third self-position information. When the reliability of one sensitivity is shown to be high, the position of the ratio shifted towards the self-position information showing high sensitivity is estimated as the third self-position information, etc.).

[0041] Further, when it is determined that neither the first self-position information nor the second self-position information has sufficient sensitivity for estimating the third self-position information (for example, it may be a determination based on the above-mentioned reference sensitivity), the self-position estimation unit 420 may activate the emergency stop function by transmitting a signal to stop the movement of the moving body 4 to the movement instruction unit 440 described later.

[0042] In this way, in a plurality of pieces of observation information obtained from different configurations and expressed in different coordinate systems, it is possible to use the first self-position information and the second self-position information converted into the same reference coordinate system, so that it is possible to estimate highly accurate third self-position information.

[0043] The movement path generation unit 430, for example, sets one or more waypoint information from the start point to the end point in sequence or sets arbitrary points in an arbitrary order by a user's selection operation on three-dimensional model data (which may be, for example, reference three-dimensional point cloud map information as shown in FIG. 10) displayed on the user terminal 2, generates movement path information by a known method based on the waypoint information, and stores and manages it in the movement path information storage unit 472. Alternatively, it analyzes the three-dimensional environment data, for example, sets waypoint information that can acquire information on specific or all components (such as internal components like inner walls, columns, ceilings, windows, doors, stairs, internal equipment, etc., and external components like outer walls, roofs, external equipment, windows, doors, stairs, roads, railways, stations, street lights, bus stops, bridges, tunnels, terrain, vegetation, water areas, gas meters and other measuring instruments, etc.) inside and outside the structure, calculates a movement path passing through each waypoint, and stores and manages this as movement path information in the movement path information storage unit 472.

[0044] Note that the movement path may be generated, for example, with the position of the mobile body storage device 5 as the movement start position and the movement end position, and a movement path passing through each waypoint. Conversely, without having the mobile body storage device 5, a configuration where the position where the user carries the aircraft is used as the movement start position or the user retrieves the aircraft at the movement end position may also be possible. Or, based on the information of the mobile body storage device 5 (such as position information, storage state information, storage mechanism information, etc.) managed in the storage units of the management server 1, the user terminal 2, and the mobile body 4, a configuration where a movement path including the position of the mobile body storage device 5 selected as the movement start position or the movement end position is generated may also be possible.

[0045] The three-dimensional model data may use reference environment information such as the reference three-dimensional point cloud map information as described above, but is not limited thereto. For example, it may be a model created based on data created by CAD (Computer-Aided Design) design software, or three-dimensional model data reconstructed from BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, BIM data, etc., or three-dimensional model data obtained by generating a structure having a predetermined height based on two-dimensional design drawing data, or three-dimensional city model data such as CityGML (Generalized Markup Language), CityJson, GeoTIFF, or three-dimensional city model data stored in a three-dimensional city model database outside this system. Note that the reconstruction, etc. of the three-dimensional model data may be executed in the processor of the management server 1 or the user terminal 2, or may be executed outside the management server 1 and the user terminal 2 and acquired internally.

[0046] Also, when the reference environmental information is the reference three-dimensional point cloud map information, the reference three-dimensional point cloud map information may be information acquired in advance by a sensor such as LiDAR as described above. For example, three-dimensional point cloud model data obtained by point cloudifying the model surface (surface) inside and outside the structure of the above-mentioned three-dimensional model data in a processor of the management server 1 or the user terminal 2 may also be used. Regarding the method for generating the three-dimensional point cloud model data, for example, a virtual moving body 4 equipped with a virtual sensor (e.g., virtual LiDAR) may be moved inside or outside the structure of the three-dimensional model data to generate three-dimensional point cloud model data regarding the components inside or outside the structure. Thereby, theoretically, point cloud data close to the point cloud sensing data in the case where the inside or outside of the structure is actually measured by the sensor of the moving body 4 can be generated. Also, for other methods of generating three-dimensional point cloud model data, the three-dimensional model data may be evenly point cloudified at a predetermined interval, or when the three-dimensional model data is polygon data, points may be arranged at each vertex for point cloudification, or known point cloudification techniques (conversion techniques to point cloud data) may be used for point cloudification. The generated three-dimensional point cloud model data is stored in a storage unit of either the storage unit 470, the management server 1, or the user terminal 2.

[0047] The movement instruction unit 440 refers to the movement route information stored in the movement route information storage unit 472, and transmits to the moving body 4 information for instructing the movement of the moving body 4 according to the coordinates indicated by the movement route information and the third self-position information estimated as described above. That is, the first self-position information that is particularly sensitive outside a structure (such as a building) and does not require prior acquisition of reference information, and the second self-position information that requires prior acquisition of reference information but is also sensitive inside the structure, are each converted into a reference coordinate system, and the third self-position information represented in the reference coordinate system estimated based on the converted self-position information is used. Thus, whether inside the structure or outside the structure without reference environment information, it is possible to compare the information indicated by the movement route information with the third self-position information. Therefore, seamless control of the movement instruction of the moving body 4 is possible even on a movement route that straddles inside and outside the structure. Note that the movement route information may be generated by the above-described movement route generation unit 430, or may be generated and stored by an external system.

[0048] Here, the moving body 4 may further include a movement route correction unit 450. When it is confirmed by a sensor for acquiring environmental information that there is an obstacle in the vicinity of the movement route (within a predetermined distance range of the movement route including the movement route), the movement route correction unit 450 corrects the movement route information referred to by the movement instruction unit 440. In this system, since the movement route can be expressed in the reference coordinate system across inside and outside the structure, for example, when there is an obstacle around the entrance and exit of the structure, the correction range of the movement route can target the movement route that straddles inside and outside the structure.

[0049] The movement information storage unit 471 stores parameter information and movement-time acquisition information obtained on the movement route, etc., which are used when the movement route generation unit 430 generates a movement route, or when the movement instruction unit 440 instructs the movement of the moving body 4 autonomously controlled on the movement route. Examples of specific parameters include, for example, movement speed, flight altitude (when the moving body 4 is an aircraft), overlap rate of captured images, movement-time acquisition information (such as image information, video information, environmental information, etc.).

[0050] The movement route information storage unit 472 stores coordinate information (so-called waypoint information) on the movement route generated by the movement route generation unit 430. As described above, it may also store movement route information generated by a processor or the like of the management server 1, the user terminal 2, or an external system.

[0051] <Functions of the User Terminal 2> FIG. 5 is a block diagram illustrating functions implemented in the user terminal 2. Note that some or all of the various functional units may be realized by an information processing device (processor, control unit) mounted on at least one of the management server 1 or the mobile body 4.

[0052] In the present embodiment, the user terminal 2 includes a communication unit 210, a screen information generation unit 220, and a storage unit 270.

[0053] The communication unit 210 communicates with the management server 1, the mobile body 4, and the mobile body storage device 5. The communication unit 210 also functions as a reception unit that receives various requests, data, etc. from the management server 1, the mobile body 4, and the mobile body storage device 5.

[0054] The screen information generation unit 220 generates screen information to be displayed via the user interface of the user terminal 2. For example, it configures a user interface screen generated by arranging various images and texts based on a predetermined layout rule, and generates screen information for displaying various information acquired by the mobile body 4 on the user interface screen.

[0055] Referring to FIG. 16, the information processing method according to this embodiment will be described, including the operation of the information processing system in this embodiment. FIG. 16 illustrates a flowchart of the information processing method according to this embodiment. In this flowchart, a configuration for starting an application on the user terminal 2 is exemplarily shown, but it is not limited thereto. For example, the management server 1, the mobile body 4, and the mobile body storage device 5 may have a processor capable of starting an application and an input / output device, and may be configured to enable various settings and the like.

[0056] First, the user starts an application for operating the mobile body 4 and displaying acquired information on, for example, the user terminal 2 (SQ101). This application may be stored in the user terminal 2, for example, or may be software (so-called SaaS) provided from the management server 1, the mobile body 4, or another external server (not shown) connected via a network. A login screen may be displayed as needed, and for example, a configuration that requests a login ID and a password may be adopted.

[0057] Next, the user creates a new movement plan (SQ102). For example, set "plan name", "area name", "address", etc., acquire and display three-dimensional model data to be moved on the user terminal 2, and start creating a new movement plan.

[0058] Next, the user generates a movement route for the movement of the mobile body 4 (SQ103). For example, one or more waypoint information (for example, expressed in a latitude / longitude / altitude coordinate system on the user terminal 2) is set by the user's selection operation on the three-dimensional model data displayed on the user terminal 2. Then, the three-dimensional model data and the waypoint information are transmitted to the mobile body 4, and movement route information is generated by a known method (for example, connecting the set waypoints with straight lines respectively) based on the three-dimensional model data and the waypoint information in the mobile body 4.

[0059] Next, the user instructs the moving body 4 to start executing the movement (SQ104). For example, with reference to the movement information storage unit 471 and the movement route information storage unit 472, the movement of the moving body 4 for purposes such as inspection, security, and construction progress management is executed. At this time, the first self-position information acquired by the receiver of the moving body 4 and the second self-position information obtained from the result of comparing the environmental information by the sensor with the reference environmental information are each converted into the reference coordinate system, and the third self-position information represented in the reference coordinate system estimated based on the converted self-position information and the movement of the moving body 4 is controlled based on the waypoint information of the movement route information and the like.

[0060] Next, the user instructs the user terminal 2 to output the acquired information (SQ105). For example, the route information of the actual movement of the moving body 4 may be superimposed on the three-dimensional model data displayed on the user terminal 2 and displayed. In addition, the acquired information (still image, moving image, sound, or other information) acquired by the moving body 4 on the movement route may be displayed, or a mark such as a symbol serving as a link for browsing the acquired information corresponding to the position information of the acquired information (particularly the position information of the waypoint) may be attached. Then, by selecting the link on the user terminal 2, the corresponding acquired information may be displayed.

[0061] In this way, the present invention can provide an information processing system or the like that can perform seamless and highly accurate self-position estimation even on a movement route straddling inside and outside a structure while enabling autonomous movement control of the moving body 4.

[0062] In addition, in the above-described embodiment, the information acquisition inside and outside the structure by the moving body 4 is taken as a specific example. However, for example, it may be an inspection of the structure, and it may be provided with devices, equipment, etc. used for inspecting the presence or absence of a predetermined event on the inner wall and / or outer wall of the structure. More specifically, imaging devices (visible light cameras, infrared cameras, metal detectors, ultrasonic measuring devices, etc.), key input devices, etc., detection devices (metal detectors), sound collection devices, odor measuring devices, gas detectors, air pollution measuring devices, detection devices (devices for detecting cosmic rays, radiation, electromagnetic waves, etc.) and all other devices necessary for knowing the state of the inspection target structure having an inner wall and an outer wall can be adopted.

[0063] Also, the embodiments may be, for example, security or surveillance within a structure, and may be provided with devices, equipment, etc. used for security or surveillance. More specifically, all devices necessary for imaging and detecting abnormalities or intruders in the security / surveillance target structure, such as imaging devices (visible light cameras, infrared cameras, night vision cameras, metal detectors, ultrasonic measuring devices, etc.) and sensor devices (motion sensors, infrared sensors, etc.), can be adopted.

[0064] The mobile body of the present invention can be suitably used as a mobile body for photography equipped with a camera or the like. In addition, it can also be used in various industries such as the security field, infrastructure monitoring, surveying, inspection of buildings and structures such as sports venues, factories, and warehouses, and disaster response.

[0065] The above-described embodiments are merely examples for facilitating the understanding of the present invention, and are not for limiting and interpreting the present invention. Needless to say, the present invention can be changed and improved without departing from its gist, and equivalents thereof are included in the present invention.

Description of Reference Numerals

[0066] 1 Management server 2 User terminal 4 Mobile body 5 Mobile body storage device

Claims

1. An information processing system for estimating the self-position of a moving body including a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, a reference coordinate conversion unit that converts both the first self-position information indicated in a first coordinate system representing the reception information and the second self-position information indicated in a second coordinate system representing the environmental information and calculated by comparing the pre-acquired reference environmental information and the environmental information into a reference coordinate system different from the first coordinate system and the second coordinate system based on a base point coordinate; a self-position estimation unit that estimates third self-position information represented in the reference coordinate system based on the first self-position information converted into the reference coordinate system and the second self-position information converted into the reference coordinate system; comprising an information processing system characterized by the above.

2. The self-position estimation unit includes a state estimation filter into which the first self-position information converted into the reference coordinate system and the second self-position information converted into the reference coordinate system are input and which outputs the third self-position information, The information processing system according to claim 1, characterized by the above.

3. The self-position estimation unit estimates third self-position information represented in the reference coordinate system based on at least one of the first self-position information and the second self-position information according to a comparison result between a first sensitivity of the receiver, a second sensitivity of the sensor, and reference sensitivities corresponding to the respective sensitivities, The information processing system according to claim 1, characterized by the above.

4. The receiver is a GPS receiver, The information processing system according to any one of claims 1 to 3, characterized by the above.

5. The sensor is a LiDAR sensor, The information processing system according to any one of claims 1 to 3, characterized by the above.

6. The sensor is a Visual sensor, The information processing system according to any one of claims 1 to 3, characterized by the above.

7. further comprising a movement control unit that compares the third self-position information represented in the reference coordinate system with movement route information represented in the reference coordinate system and controls the movement of the moving body, The information processing system according to any one of claims 1 to 3, characterized by the above.

8. further comprising a movement route information correction unit that corrects the movement route information when an obstacle is detected on the movement route of the moving body by the sensor The information processing system according to claim 7, characterized in that...

9. An information processing system for estimating the self-position of a moving body, comprising a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, a reference coordinate conversion unit that converts both the first self-position information represented in a first coordinate system representing the reception information and the second self-position information represented in a second coordinate system representing the environmental information and calculated by comparing the previously acquired reference environmental information and the environmental information into a reference coordinate system different from the first coordinate system and the second coordinate system based on the base point coordinates; a self-position estimation unit that estimates third self-position information represented in the reference coordinate system based on the first self-position information converted into the reference coordinate system and the second self-position information converted into the reference coordinate system; comprising A moving body characterized by the above.

10. An information processing method for estimating the self-position of a moving body, comprising a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, a step of converting, by a reference coordinate conversion unit, both the first self-position information represented in a first coordinate system representing the reception information and the second self-position information represented in a second coordinate system representing the environmental information and calculated by comparing the previously acquired reference environmental information and the environmental information into a reference coordinate system different from the first coordinate system and the second coordinate system based on the base point coordinates; a step of estimating, by a self-position estimation unit, third self-position information represented in the reference coordinate system based on the first self-position information converted into the reference coordinate system and the second self-position information converted into the reference coordinate system; An information processing method characterized by causing a computer to execute the above steps.

11. A program for causing a computer to execute an information processing method for estimating the self-position of a moving body, comprising a receiver that acquires reception information from a satellite positioning system and a sensor that acquires environmental information, a step of converting, by a reference coordinate conversion unit, both the first self-position information represented in a first coordinate system representing the reception information and the second self-position information represented in a second coordinate system representing the environmental information and calculated by comparing the previously acquired reference environmental information and the environmental information into a reference coordinate system different from the first coordinate system and the second coordinate system based on the base point coordinates; A step of estimating third self-position information represented in the reference coordinate system based on the first self-position information converted into the reference coordinate system and the second self-position information converted into the reference coordinate system by the self-position estimation unit; A program characterized by causing the computer to execute the above.

Images