Information processing device, information processing method, and information processing program

Abstract

This information processing device: acquires correspondence information of a key frame and a query image which have been provided in advance in map data; and synthesizes a plurality of pieces of map data on the basis of the correspondence information.

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, technologies have been proposed for acquiring surrounding information and map information by moving moving objects such as vehicles, drones, and robots. In addition, a technology has been proposed for synthesizing information acquired by each moving object (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-90239 A Summary of the Invention [Problem to be solved by the invention]

[0004] When multiple moving bodies are used, each moving body acquires surrounding information and map information independently, and the position and attitude of each moving body is estimated based on a different origin for each piece of information. This makes it difficult to grasp the distance and direction from one moving body to another, resulting in the problem that it is difficult to accurately combine multiple pieces of information.

[0005] The present technology has been made in consideration of the above points, and has an object to provide an information processing device, an information processing method, and an information processing program capable of synthesizing a plurality of map data. [Means for solving the problem]

[0006] In order to solve the above-mentioned problem, the first technique obtains correspondence information between a key frame and a query image that are arranged in advance in map data, and synthesizes multiple map data based on the correspondence information. If the map data has multiple key frames, one key frame is identified for which corresponding information is to be obtained by matching feature points between the key frame and the query image. It is an information processing device.

[0007] The second technique obtains correspondence information between a key frame and a query image that is pre-arranged in map data, and synthesizes multiple map data based on the correspondence information. If the map data has multiple key frames, one key frame is identified for which corresponding information is to be obtained by matching feature points between the key frame and the query image. It is an information processing method.

[0008] Furthermore, the third technology acquires correspondence information between key frames pre-arranged in map data and query images, and synthesizes multiple map data based on the correspondence information. If the map data has multiple key frames, one key frame is identified for which corresponding information is to be obtained by matching feature points between the key frame and the query image. It is an information processing program that causes a computer to execute the information processing method. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an information processing system 10. FIG. [Diagram 2] 1 is a block diagram showing a configuration of a moving object 100. FIG. [Diagram 3] FIG. 2 is a block diagram showing a configuration of an information processing device 200. [Figure 4] 13 is a flowchart showing a process in the moving body 100. [Diagram 5] FIG. 11 is an explanatory diagram of hash value generation. [Figure 6] FIG. 11 is an explanatory diagram of hash value generation. [Figure 7] 4 is a flowchart showing a process in the information processing device 200. [Figure 8] FIG. 2 is a diagram showing an example of first map data and second map data. [Figure 9] FIG. 4 is an explanatory diagram of a process of merging map data. [Figure 10] FIG. 4 is an explanatory diagram of a process of combining map data. [Figure 11] FIG. 4 is an explanatory diagram of a process of combining map data. [Figure 12] FIG. 4 is a diagram showing composite map data created by combining first map data and second map data. [Figure 13] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order.

[0011] <1. Preferred embodiment> [1-1. Configuration of information processing system 10] [1-2. Configuration of moving body 100] [1-3. Configuration of information processing device 200] [1-4. Processing in the Mobile Body 100] [1-5. Processing in information processing device 200] <2. Modifications>

[0012] <1. Preferred embodiment> [1-1. Configuration of information processing system 10] First, a configuration of an information processing system 10 according to an embodiment of the present technology will be described with reference to Fig. 1. The information processing system 10 is composed of a plurality of moving bodies 100 and a plurality of information processing devices 200. In this embodiment, it is assumed that the moving bodies 100 include a first moving body 100A and a second moving body 100B, and the information processing devices 200 include a first information processing device 200A and a second information processing device 200B.

[0013] The first information processing device 200A operates in a first server device 300A, and the second information processing device 200B operates in a second server device 300B. The mobile object 100 and the server device 300 including the information processing device 200 are connected via a network such as the Internet.

[0014] The mobile body 100 may be any device that moves independently, such as a drone, a robot vacuum cleaner, a pet-type robot, a humanoid robot, a self-driving car, an autonomous delivery robot, etc. The mobile body 100 can create map data of the surroundings of the mobile body 100 and estimate its own position using a SLAM (Simultaneous Localization and Mapping) function. There is no limit to the number of mobile bodies 100 that configure the information processing system 10.

[0015] The information processing device 200 holds map data created by the moving object 100 and performs a synthesis process of the map data. In this embodiment, the information processing device 200 stores a plurality of map data created by the plurality of moving objects 100 individually using SLAM.

[0016] The first moving body 100A is associated with the first information processing device 200A, and the first information processing device 200A stores the first map data created by the first moving body 100A. The second moving body 100B is associated with the second information processing device 200B, and the second information processing device 200B stores the second map data created by the second moving body 100B.

[0017] The server device 300 includes at least a control unit 301, a communication unit 302, and a storage unit 303. The information processing device 200 communicates with the mobile object 100 using the communication unit 302 included in the server device 300.

[0018] The control unit 301 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The ROM stores programs that are read and run by the CPU. The RAM is used as a working memory for the CPU. The CPU controls the entire server device and each unit by executing various processes and issuing commands according to the programs stored in the ROM.

[0019] The communication unit 302 is a communication module for communicating with mobile objects and other server devices. Communication methods include wireless LAN (Local Area Network), WAN (Wide Area Network), WiFi (Wireless Fidelity), 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), Bluetooth (registered trademark), ZigBee (registered trademark), etc.

[0020] The storage unit 303 is, for example, a large-capacity storage medium such as a hard disk or a flash memory.

[0021] The first information processing device 200A operates in a first server device 300A, and the second information processing device 200B operates in a second server device 300B.

[0022] [1-2. Configuration of moving body 100] Next, the configuration of the moving object 100 will be described with reference to Fig. 2. The moving object 100 is configured to include a control unit 101, a storage unit 102, a communication unit 103, a camera unit 104, a sensor unit 105, a SLAM processing unit 106, a drive unit 107, a power supply unit 108, and a moving object search unit 110. The moving object search unit 110 is configured to include an information acquisition unit 111, a hash generation unit 112, and a search processing unit 113.

[0023] The control unit 101 is composed of a CPU, RAM, ROM, etc. The CPU executes various processes and issues commands according to the programs stored in the ROM, thereby controlling the entire moving body 100 and each part. The control unit 101 also controls the moving speed, moving direction, etc. of the moving body 100 by supplying a control signal to the driving unit 107 that controls the output of the driving unit 107.

[0024] The storage unit 102 is, for example, a large-capacity storage medium such as a hard disk, a flash memory, etc. The storage unit 102 stores map data created by the SLAM processing unit 106, images captured by the camera unit 104, and other data and applications necessary for using the moving body 100.

[0025] The communication unit 103 is a communication module for communicating with the server device 300. The communication method may be a wireless LAN, a WAN, WiFi, 4G, 5G, Bluetooth (registered trademark), ZigBee (registered trademark), or the like.

[0026] The camera unit 104 is made up of an image sensor, an image processing engine, etc., and has a function as a camera that can capture RGB or monochrome two-dimensional still images and videos. The camera unit 104 may be provided in the moving body 100 itself, or may be configured as a device separate from the moving body 100 and may be connected to and mounted on the moving body 100.

[0027] The sensor unit 105 is a sensor such as a GPS (Global Positioning System) module that can detect the position of the mobile object 100. The GPS is a system that determines the current position by receiving signals from multiple artificial satellites positioned around the earth with a receiver.

[0028] Furthermore, the sensor unit 105 may include a sensor that detects angular velocity, such as an IMU (Inertial Measurement Unit) module. The IMU module is an inertial measurement device that detects the attitude, direction, etc. of the moving body 100 by obtaining three-dimensional angular velocity and acceleration using an acceleration sensor, an angular velocity sensor, a gyro sensor, etc. in two or three axial directions.

[0029] The SLAM processing unit 106 uses a SLAM function to create map data of the surroundings of the mobile object 100 and estimate the mobile object's own position. SLAM makes it possible to create map data and estimate the mobile object's own position in the map data by using a combination of feature points detected from an image captured by the camera unit 104, detection results from the IMU, and sensor information from various other sensors. The map data created by the SLAM processing unit 106 is transmitted to the information processing device 200 by the communication unit 103.

[0030] The driving unit 107 is a functional block that realizes the movement of the moving body according to a predetermined movement commanded by the control unit 101. The driving unit 107 realizes the movement of the moving body 100 by operating the motors and actuators that are the respective operating parts of the moving body 100. Note that the movement method of the moving body 100 differs depending on the type of the moving body, and in this embodiment, the movement method of the moving body may be any movement method. For example, the movement method may be the movement of a rotor blade if the moving body 100 is a drone, the rotation of wheels if the moving body 100 is a robot vacuum cleaner, an autonomous driving car, or an autonomous delivery robot, or the movement of feet if the moving body is a pet-type robot or a humanoid robot.

[0031] The power supply unit 108 supplies power to each electric circuit of the mobile object 100. The mobile object 100 is an autonomous driving type that uses a battery, and the power supply unit 108 is composed of, for example, a rechargeable battery and a charge / discharge control unit that manages the charge / discharge state of the rechargeable battery.

[0032] The mobile object search unit 110 is for searching for other mobile objects.

[0033] The information acquisition unit 111 acquires information used by the hash generation unit 112 to generate a hash value. In this embodiment, time information and a network ID are used to generate the hash value. The time information can be acquired from a clock function that the mobile body 100 normally has. The network ID can be acquired from a communication function by the communication unit 103 that the mobile body 100 has. Examples of network IDs include an SSID (Service Set Identifier), which is an identification name for a Wi-Fi access point, and an ID for a cellular base station. Furthermore, the information acquisition unit 111 acquires the radio wave strength of the network from which the network ID has been acquired as necessary. The radio wave strength may be acquired from the communication function by the communication unit 103, or a publicly known application dedicated to acquiring radio wave strength may be used.

[0034] The hash generation unit 112 generates a hash value from the time information and the network ID. The hash value is used by the search processing unit 113 in the mobile object search process.

[0035] The search processing unit 113 searches for other moving bodies existing in the vicinity of the moving body 100 using the hash value generated by the hash generation unit 112. In this embodiment, the map data generated by the moving body 100 and the map data generated by other moving bodies existing in the vicinity are synthesized by the information processing device 200. Note that the vicinity range is not limited to a specific range, but is determined depending on the range indicated by the map data and the extent to which other moving bodies are searched for.

[0036] When the moving body 100 discovers another moving body, it transmits an image captured by the camera unit 104 as a query image to the information processing device 200, inquires about the shooting position and shooting posture of the query image on the map data, and instructs the information processing device 200 to combine the map data created by itself with the map data created by the other moving body.

[0037] The mobile object search unit 110 is realized by executing a program, and the program may be installed in the mobile object in advance, or may be distributed by download or storage medium, etc., and installed by the user. Furthermore, the mobile object search unit 110 may not only be realized by a program, but may also be realized by combining a dedicated device, circuit, etc., made of hardware having the function.

[0038] [1-3. Configuration of information processing device 200] Next, the configuration of the information processing device 200 will be described with reference to Fig. 3. The information processing device 200 is configured to include a map storage unit 201, a key frame search unit 202, a feature point matching unit 203, a corresponding information estimation unit 204, and a map synthesis unit 205. The key frame search unit 202, the feature point matching unit 203, and the corresponding information estimation unit 204 perform localization processing.

[0039] The map storage unit 201 is a memory unit that stores and preserves map data. The map data may be generated by the moving body 100 using SLAM, or may be an existing map database or map data provided by a map service. When the information processing device 200 performs a map synthesis process, the map data is read out from the map storage unit 201.

[0040] It is assumed that an origin O and x, y and z axes are set in advance in the map data at an arbitrary position. The positions of the query image and key frame on the map data are specified based on the displacement from the origin O, and the displacement and rotation between the query image and the key frame are calculated based on the positions. In addition, when combining multiple map data, it is necessary to match the x, y and z axes. The origin O is an arbitrary point on the map data, and the x, y and z axes are, for example, based on the compass, with the east-west direction being the x direction, the north-south direction being the x axis direction, and height being the z axis direction.

[0041] Key frames are arranged in advance in the map data. A key frame is a representative image that represents the environment in the map data, and includes information on the position and posture in the map data. One or more key frames may be arranged in one map data. There is no limit to the number of key frames. The number of subjects in the key frames and the number of key frames may be determined according to the size of the map data and the objects in the real space corresponding to the map data. The more key frames there are, the more the features in the real space corresponding to the map data can be grasped, and therefore the accuracy of the map data synthesis can be improved. For example, the key frames are photographed in advance by the user who uses the moving body 100 and the information processing device 200 and associated with the map data. Key frames may be added while the moving body 100 is executing the SLAM process.

[0042] A keyframe contains information about feature points. A feature point is a feature of a keyframe that changes little even when environmental conditions such as time and weather change, such as an object in the keyframe. Feature points can be detected from a keyframe using a conventional algorithm. For example, there are corner detection methods (Harris algorithm, FAST (Features from Accelerated Segment Test)), feature point detection methods using neural networks, feature point detection using deep learning, and brightness gradients. One or more keyframes are arranged in advance in the map data, and feature points are detected in each keyframe. In addition to publicly known algorithms, keyframes and feature points may be set directly by the user.

[0043] The keyframe search unit 202 searches for a keyframe for which feature point matching processing with a query image is performed from among a plurality of keyframes arranged in the map data based on the feature amount in the keyframe. The keyframe search unit 202 is supplied with keyframe feature amounts from the map storage unit 201. All keyframes may be compared with the query image in order, or keyframes for which comparison processing is performed may be selected based on the features in the keyframe, and only the selected keyframes may be compared with the query image. Information on the keyframe for which feature point matching processing is performed is supplied to the feature point matching unit 203. In the following description, a keyframe that performs feature point matching processing with the query image and serves as a reference for identifying the position of the query image on the map data is referred to as a specific keyframe.

[0044] The feature point matching unit 203 performs feature point matching processing between the specific key frame searched by the key frame search unit 202 and the query image. By comparing the feature points of the specific key frame with those of the query image, matching feature points are extracted, enabling the correspondence information estimation unit to estimate correspondence information between the specific key frame and the query image. The feature points of the specific key frame are supplied to the feature point matching unit 203 from the map storage unit 201. The feature point matching processing can be performed by a method such as SIFT (Scale Invariant Feature Transform) or SURF (Speed ​​Upped Robust Feature). The result of the feature point matching processing is supplied to the correspondence information estimation unit 204.

[0045] The correspondence information estimation unit 204 estimates correspondence information between the specific key frame and the query image, and the position and orientation of the query image on the map data, based on the result of the feature point matching process and the position and orientation information of the specific key frame. The correspondence information estimation unit 204 is supplied with the position and orientation information of the specific key frame from the map storage unit 201.

[0046] The correspondence information is the displacement (amount of movement) from the position in the map data of a specific key frame to the position in the map data of the query image, and the rotation from the orientation in the map data of a specific key frame to the orientation in the map data of the query image. The correspondence information is obtained by grasping the positions, displacement, rotation, etc. of objects in the key frame and the query image from the matches between feature points in a specific key frame and feature points in the query image. The correspondence information can be obtained by taking the displacement and rotation from the position and orientation indicated by the key frame to the position and orientation indicated by the query image, based on the origin O and the xyz axes set in the map data.

[0047] The mobile object 100 can estimate its own position in the map data by SLAM. Therefore, the mobile object 100 may transmit information on its own position in the map data that coincides with the shooting position of the query image together with the query image to the information processing device 200, and the information processing device 200 may use the information on its own position to identify the position of the query image in the map data or create a map.

[0048] The corresponding information estimation unit 204 outputs the estimated position and orientation of the query image on the map data as position and orientation information. This position and orientation information of the query image is transmitted to the mobile object 100 that transmitted the query image.

[0049] The map synthesis unit 205 performs a process of synthesizing a plurality of map data based on correspondence information between the plurality of map data and the query image in each of the plurality of map data.

[0050] The information processing device 200 is configured as described above. The information processing device 200 is realized by executing a program, which may be installed in a server device or the like in advance, or may be distributed by download or storage medium, and installed by the user. Furthermore, the information processing device 200 may not only be realized by a program, but may also be realized by combining a dedicated device, circuit, or the like, which is hardware having the function.

[0051] [1-4. Processing in the Mobile Body 100] Next, the processing in the moving body 100 will be described with reference to the flowchart in Fig. 4. Here, the description will be made on the assumption that the first moving body 100A finds the second moving body 100B by searching, and the first information processing device 200A associated with the first moving body 100A combines the first map data and the second map data. Therefore, the following description will be made on the processing in the first moving body 100A.

[0052] In the process in the first moving object 100A, first, in step S101, the information acquisition unit 111 acquires the current time information and the network ID.

[0053] Next, in step S102, the hash generation unit 112 generates a hash value based on the time information and the network ID.

[0054] Here, the creation of a hash value will be described with reference to Fig. 5 and Fig. 6. As an example, it is assumed that there are access points for networks A, B and C, as shown in Fig. 5.

[0055] The first mobile unit 100A acquires the network IDs of Network A, Network B, and Network C, and further acquires the radio wave strength of each network. For ease of explanation, the network ID of Network A is assumed to be "AAA", the network ID of Network B is assumed to be "BBB", and the network ID of Network C is assumed to be "CCC".

[0056] Next, the first mobile unit 100A sorts the networks in order of radio wave strength. Since the access point of Network A is the closest to the first mobile unit 100A, Network A has the strongest radio wave strength, followed by Network B and Network C.

[0057] Furthermore, the acquired time information is rounded off, and a hash value is generated from the time information and the sorted network IDs using a predetermined hash function. Rounding is performed in 10-minute increments, for example, and in that case, if the time is 9:04 or 9:08, the result of rounding is 9:00. Since the condition for merging map data is that the hash value calculated for each moving body 100 is the same, even a slight difference in the time information will result in a different hash value, making it impossible to merge the map data. Therefore, rounding can provide flexibility to the conditions for merging map data, and the timing of hash value generation for each moving body 100 does not need to be exactly the same.

[0058] Since the mobile unit 100 is often in a moving state, if a hash value is generated from old time information and network ID, the hash value may deviate from the real-time location of the mobile unit 100. Therefore, it is advisable to generate hash values ​​and search for the mobile unit 100 at relatively short time intervals (for example, 10 minutes).

[0059] The hash value in the first moving body 100A is calculated from the time information and the sorted network IDs as shown in Fig. 6. The second moving body 100B generates a hash value in the same way. For ease of explanation, it is further assumed that a third moving body 100C exists, and the third moving body 100C generates a hash value in the same way.

[0060] For both the first moving body 100A and the second moving body 100B, the access points are closest in the order of network A, network B, and network C, so the sorting order based on the radio wave strength of the network IDs is the same as that shown in Fig. 5. However, for the third moving body 100C, the access point of network C is closest, so the radio wave strength of network C is the highest, followed by network B and network A. Therefore, the sorting order of the network IDs for the third moving body 100C is different from the sorting order of the network IDs for the first moving body 100A and the second moving body 100B. Note that the sorting is not limited to the order of radio wave strength, and may be in another order, such as dictionary order.

[0061] Similarly, the hash values ​​in the second moving body 100B and the third moving body 100C are calculated from the time and the sorted network IDs as shown in Fig. 6. Note that the hash values ​​shown in Fig. 6 are assumed as examples for the sake of convenience, and are not actual hash values.

[0062] The first moving body 100A and the second moving body 100B have the same sorting order of the time information and the radio wave strength of the network ID, so the hash values ​​are the same. However, the sorting order of the network ID in the third moving body 100C is different from that of the first moving body 100A and the second moving body 100B, so the hash value of the third moving body 100C will not be the same as the hash values ​​of the first moving body 100A and the second moving body 100B.

[0063] Returning to the explanation of the flowchart, next, in step S103, the mobile object search unit 110 searches for other mobile objects existing in the vicinity of the first mobile object 100A based on the hash value.

[0064] The mobile object search unit 110 searches for other mobile objects by broadcasting the hash value generated by the hash generation unit 112 to other mobile objects around the first mobile object 100A and inquiring whether the other mobile objects have the same hash value as the first mobile object 100A. The other mobile objects that receive the inquiry transmit a response to the first mobile object 100A indicating whether the received hash value is the same as the hash value that they generated. Therefore, the other mobile objects must also generate hash values ​​using the same method.

[0065] If there is a management server that centrally manages hash values ​​transmitted from all mobile bodies 100 under its management together with the identification information of the transmitting mobile bodies 100, the search for the mobile bodies 100 can also be performed in other ways. The first mobile body 100A can also search for other mobile bodies by transmitting a hash value it generates to the management server and inquiring whether there is a mobile body that has transmitted the same hash value to the management server. The management server stores all hash values ​​transmitted together with the inquiry and uses them to check the presence or absence of hash values ​​in response to subsequent inquiries from other mobile bodies.

[0066] Next, if the search for other moving bodies is successful in step S104, that is, if a moving body having the same hash value exists, the process proceeds to step S105 (Yes in step S104). On the other hand, if the search is not successful, that is, if a moving body having the same hash value does not exist, the process proceeds to step S101, and steps S101 to S104 are repeated (No in step S104). Here, it is assumed that a second moving body 100B has been found as another moving body.

[0067] Next, in step S105, the first moving body 100A establishes communication with another moving body, the second moving body 100B, and acquires identification information of the second server device 300B (the second information processing device 200B) that stores the second map data created by the second moving body 100B. This enables the first moving body 100A to transmit a query image to the second information processing device 200B.

[0068] Next, in step S106, the first moving body 100A transmits a query image to the first information processing device 200A and inquires about the position and orientation information of the query image in the first map data. The first moving body 100A also transmits the query image to the second information processing device 200B corresponding to the second moving body 100B and inquires about the position and orientation of the query image in the second map data.

[0069] Furthermore, the first mobile body 100A transmits to the first information processing device 200A an instruction to combine the first map data stored in the first information processing device 200A and the second map data stored in the second information processing device 200B. At this time, the first mobile body 100A transmits to the first information processing device 200A information indicating that the second map data is held by the second information processing device 200B associated with the second mobile body 100B. This enables the first information processing device 200A to establish communication with the second information processing device 200B and transmit and receive information required for combining the map data.

[0070] Processing in the mobile body 100 is performed in the above manner. According to the present technology, since the network ID is used to search for other mobile bodies, but only the hash value is made public, there is no risk of the location information of the mobile body 100 leaking from the network ID to the outside. Even if the hash value is intercepted, a huge amount of calculation is required to find the network ID from the hash value, so the network ID is unlikely to be leaked. Also, the hash value can be sent to a key-value store service on the network to search for other mobile bodies.

[0071] [1-5. Processing in information processing device] Next, the processing in the information processing device 200 will be described with reference to the flowchart in Fig. 7. The above-mentioned processing in the moving body 100 is a processing up to confirming that the second moving body 100B exists in the vicinity of the first moving body 100A and issuing an instruction to transmit a query image and combine map data. At this stage, it is unknown whether the first map data created by the first moving body 100A and the second map data created by the second moving body 100B can be combined. The information processing device 200 determines whether the first map data and the second map data can be combined, and if they can, combines them into one map data.

[0072] First, in step S201, the first information processing device 200A receives a query image transmitted from the first moving object 100A, and accepts an inquiry about the position and orientation information of the query image in the first map data and an instruction to combine map data.

[0073] Next, in step S202, the first information processing device 200A performs localization processing on the query image and the first map data transmitted from the first moving body 100A. This allows the position and orientation of the query image in the first map data and correspondence information of the query image to a specific key frame to be acquired. Similarly, the second information processing device 200B also performs localization processing on the query image and the second map data transmitted from the first moving body 100A. If the localization process is successful, the process proceeds to step S204 (Yes in step S203), and if the localization process is not successful, the process returns to step S201 (No in step S203).

[0074] Next, in step S204, the first information processing device 200A transmits the estimated position and orientation information of the query image to the first moving body 100A that has transmitted the query image and made the inquiry. This enables the first moving body 100A to recognize which map data, at which position, and in what orientation the query image was captured by the first moving body 100A.

[0075] Next, in step S205, the first information processing device 200A transmits a communication request to the second information processing device 200B, and establishes communication with the second information processing device 200B.

[0076] Next, in step S206, the first information processing device 200A checks whether communication has been established with the second information processing device 200B having the second map data. Steps S205 and S206 are continued until communication with the second information processing device 200B is established.

[0077] Next, in step S207, the first information processing device 200A checks whether the localization process has been successful in itself and in the second information processing device 200B based on the same query image. To do this, the first information processing device 200A needs to receive a notification from the second information processing device 200B as to whether the localization process has been successful. Since the first information processing device 200A synthesizes a plurality of map data based on a query image, in order to synthesize the map data, it is necessary that the localization process is successful in the first map data and the second map data based on the same query image. If the localization process is successful in the first map data and the second map data based on the same query image, the first information processing device 200A determines that the first map data and the second map data can be synthesized, and the process proceeds to step S207 (Yes in step S207).

[0078] Next, in step S208, the first information processing device 200A receives correspondence information of the query image in the second map data from the second information processing device 200B. Note that the correspondence information of the query image in the second map data may be transmitted and received by communication via the first mobile body 100A, instead of direct communication between the server devices. However, from the viewpoint of communication efficiency and security, direct communication between the server devices is considered to be better.

[0079] Then, in step S209, the first information processing device 200A performs a process of merging the first map data and the second map data for which the localization process has been successful.

[0080] The map data merging process will now be described with reference to Figures 8 to 12. The description will be made on the assumption that the first map data is as shown in Figure 8A, and the second map data is as shown in Figure 8B. The first map data and the second map data are associated with different origins O and x, y and z axes, respectively.

[0081] As shown in Fig. 8A, it is necessary to arrange a plurality of key frames in advance in the first map data, and as shown in Fig. 8B, it is also necessary to arrange a plurality of key frames in advance in the second map data.

[0082] Then, as shown in Fig. 9A, when the position of the query image in the first map data is identified by the localization process, correspondence information between the specific key frame in the first map data and the query image can be obtained. The specific key frame in the first map data is referred to as a first specific key frame.

[0083] Similarly, as shown in Fig. 9B, when the position of the query image in the second map data is identified by the localization process, correspondence information between the specific key frame in the second map data and the query image can be obtained. The specific key frame in the second map data is referred to as the second specific key frame.

[0084] 10, the correspondence information in the first map data is the displacement L1 from the position in the map data indicated by the first specific key frame to the position in the map data indicated by the query image, and the rotation (angle R1) from the attitude in the map data indicated by the first specific key frame to the attitude in the map data indicated by the query image. The angle R1 is calculated based on, for example, any of the x, y and z axes preset in the first map data.

[0085] 10, the correspondence information in the second map data is the displacement L2 from the position in the map data indicated by the second specific key frame to the position in the map data indicated by the query image, and the rotation (angle R2) from the attitude in the map data indicated by the second specific key frame to the attitude in the map data indicated by the query image. Like angle R1, angle R2 is calculated based on any one of the x, y and z axes preset in the second map data.

[0086] 10 and 11, the displacement L and angle R are calculated based on the approximate centers of the first specific key frame, the second specific key frame, and the query image, respectively, but this is set for convenience of illustration and explanation. In reality, the displacement L and angle R are calculated based on the positions on the map data indicated by the first specific key frame, the second specific key frame, and the query image, respectively.

[0087] The query image arranged in the first map data and the query image arranged in the second map data are the same and represent the same position and orientation in real space. Therefore, when the correspondence information between the first map data and the query image and the correspondence information between the second map data and the query image can be acquired by the localization process, the correspondence relationship (relative relationship) between the first map data and the second map data can be acquired based on the positions of the query image on the first map data and the second map data.

[0088] The corresponding relationship (relative relationship) between the first map data and the second map data is a displacement L3 from a first specific key frame of the first map data to a second specific key frame of the second map data, and a rotation (angle R3) from the first specific key frame of the first map data to the second specific key frame of the second map data, as shown in Figure 11. This makes it possible to generate one composite map data by combining the first map data and the second map data, as shown in Figure 12.

[0089] The correspondence information between the query image and the key frame can be expressed by three-dimensional coordinates based on the position of the key frame. Also, the orientation (direction) from the key frame to the query image can be expressed by a matrix.

[0090] Although the present embodiment has been described with reference to the synthesis of two-dimensional map data, the present technology can also be applied to the synthesis of three-dimensional map data.

[0091] The processing in the information processing device 200 is performed as described above. According to the present technology, larger map data can be created by composing multiple map data. Furthermore, since multiple map data are composed based on one query image, the composition of map data can be easily performed. By composing map data, the mobile body 100 can use map data created by other mobile bodies, and the mobile body 100 can also allow other mobile bodies to use the map data it created. The mobile body 100 itself can also obtain map data for areas that it has not yet explored. Furthermore, since the positions of multiple mobile bodies can be estimated on one map data after composition, the multiple mobile bodies can exchange position information with each other and operate in cooperation with each other.

[0092] <2. Modifications> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical ideas of the present technology are possible.

[0093] In the embodiment, the combination of two map data stored in two information processing devices 200 has been described as an example, but the number and combination of map data are not limited thereto. For example, as shown in FIG. 13, assume that there are three information processing devices, namely, a first information processing device 200A, a second information processing device 200B, and a third information processing device 200C, and the third information processing device 200C further stores two map data. In this case, the mobile object 100 transmits a query image to all three information processing devices 200 to instruct them to combine the map data, and for example, the third information processing device 300C combines the third map data and the fourth map data, and further, one of the information processing devices 200 combines the first map data, the second map data, and the third and fourth combined map data to create one combined map data consisting of a total of four map data.

[0094] In the embodiment, the first information processing device 200A stores the first map data, and the second information processing device 200B stores the second map data, but one information processing device 200 may store a plurality of map data. When one information processing device 200 stores a plurality of map data, localization processing is performed on the plurality of map data in turn to select the map data to be combined.

[0095] In the embodiment, the first moving body 100A transmits a query image to the first information processing device 200A and the second information processing device 200B to instruct them to combine map data, but in addition to the first moving body 100A, the second moving body 100B may also transmit a query image to the first information processing device 200A and the second information processing device 200B. By performing localization processing using the query image from the first moving body 100A and the query image from the second moving body 100B, respectively, it is possible to improve the accuracy of the map data combination.

[0096] Alternatively, the first moving body 100A itself may function as the first information processing device 200A and store the first map data, and the second moving body 100B itself may function as the second information processing device 200B and store the second map data, without providing the server device 300. In this case, the moving body 100 that transmits the query image needs to perform a localization process between the map data stored therein and the query image.

[0097] In addition, similar to the embodiment, map data synthesis can be performed for multiple map data stored in one information processing device 200 associated with one moving object 100. Furthermore, similar to the embodiment, map data synthesis can be performed for multiple map data stored in one moving object 100 having the function of the information processing device 200. In other words, as long as there are multiple map data, the number of moving objects and the number of information processing devices do not matter.

[0098] The information processing device 200 may incorporate a query image into the map data to expand the map data.

[0099] The information processing device 200 may operate in a mobile object 100, in addition to the server device 300 described in the embodiment, or in a terminal device such as a personal computer, a smartphone, or a tablet terminal, or may operate in the cloud.

[0100] There may be a plurality of mobile objects 100, and a plurality of map data created by the plurality of mobile objects 100 may be stored in one common information processing device 200.

[0101] Alternatively, the information processing device 200 itself may have a communication function and communicate with the mobile object 100 without going through another device such as the server device 300 .

[0102] The query image does not necessarily have to be captured by a camera provided in the moving body 100 for which the map data was created. For example, an image captured by a user using a terminal device or the like may be transmitted to the information processing device 200 as a query image.

[0103] Since the user can specify at least two pieces of map data and prepare a query image to have the information processing device 200 synthesize the map data, the presence of the moving object 100 is not essential for synthesizing the map data.

[0104] In the embodiment, the first map data and the second map data are described as being combined, but the number of map data to be combined is not limited to two, and may be three or more. Therefore, the number of mobile objects, information processing devices, and server devices may also be three or more.

[0105] The present technology can also be configured as follows. (1) Acquire correspondence information between a key frame pre-arranged in the map data and the query image; The map data is synthesized based on the correspondence information. Information processing device. (2) The information processing device according to (1), wherein the key frame is an image captured of a real space represented by the map data. (3) The information processing device according to (1) or (2), wherein the correspondence information is a displacement from a position in the map data indicated by the key frame to a position in the map data indicated by the query image. (4) The information processing device according to any one of (1) to (3), wherein the correspondence information is a rotation from an orientation in the map data shown in the key frame to an orientation in the map data shown in the query image. (5) An information processing device according to any one of (1) to (4), wherein, when the map data includes a plurality of key frames, one of the key frames is identified for obtaining the corresponding information by matching feature points between the key frame and the query image. (6) 6. The information processing device according to claim 1, further comprising: a first information processing unit that receives the first and second map data from the first and second map data; (7) The information processing device according to any one of (1) to (6), wherein the map data is acquired by SLAM based on the movement of a moving object. (8) The information processing device according to (7), wherein the plurality of map data are synthesized when the plurality of map data are acquired by the plurality of moving bodies existing within a vicinity range. (9) The information processing device according to (8), wherein whether or not a plurality of the moving objects are present within the vicinity range is determined based on a hash value. (10) The information processing device according to (9), wherein if a hash value generated by one of the moving bodies is identical to a hash value generated by another of the moving bodies, the one of the moving bodies and the other of the moving bodies are determined to be in a nearby range. (11) The information processing device according to (9), wherein the hash value is generated based on time information and a network ID. (12) The information processing device according to (7), wherein the query image is transmitted from the moving object. (13) The information processing device according to (12), wherein the query image is transmitted from the mobile object together with an inquiry about location information of the query image in the map data. (14) The information processing device according to (13), further comprising: transmitting position information of the query image on the map to the mobile body that transmitted the query image. (15) Acquire correspondence information between a key frame pre-arranged in the map data and the query image; The information processing method further comprises: synthesizing a plurality of the map data based on the correspondence information. (16) Acquire correspondence information between a key frame pre-arranged in the map data and the query image; An information processing program that causes a computer to execute an information processing method for synthesizing a plurality of the map data based on the correspondence information. [Explanation of symbols]

[0106] 100...Mobile 200...Information processing device.

Claims

1. Acquire correspondence information between a key frame pre-arranged in the map data and the query image; A plurality of the map data are synthesized based on the correspondence information. When the map data includes a plurality of key frames, one of the key frames from which the corresponding information is obtained is identified by matching feature points between the key frame and the query image. Information processing device.

2. The key frame is an image captured of the real space represented by the map data. The information processing device according to claim 1 .

3. The correspondence information is a displacement from a position in the map data indicated by the key frame to a position in the map data indicated by the query image. The information processing device according to claim 1 .

4. The correspondence information is a rotation from a posture in the map data shown in the key frame to a posture in the map data shown in the query image. The information processing device according to claim 1 .

5. The plurality of map data are synthesized based on a relative relationship of feature information in each of the plurality of map data. The information processing device according to claim 1 .

6. The map data is acquired by SLAM based on the movement of a mobile object. The information processing device according to claim 1 .

7. When the plurality of map data are acquired by the plurality of moving bodies existing within a vicinity range, the plurality of map data are synthesized. The information processing device according to claim 6.

8. Whether or not a plurality of the moving objects are present within the vicinity range is determined based on the hash value. The information processing device according to claim 7.

9. If the hash value generated by one of the moving bodies is the same as the hash value generated by the other of the moving bodies, it is determined that the one of the moving bodies and the other of the moving bodies are in a nearby range. The information processing device according to claim 8.

10. The hash value is generated based on the time information and the network ID. The information processing device according to claim 8.

11. The query image is transmitted from the mobile object. The information processing device according to claim 6.

12. The query image is transmitted from the mobile body together with a query for location information of the query image in the map data. The information processing device according to claim 11.

13. The location information of the query image in the map data is transmitted to the mobile unit that transmitted the query image. The information processing device according to claim 12.

14. Acquire correspondence information between a key frame pre-arranged in the map data and the query image; A plurality of the map data are synthesized based on the correspondence information. When the map data includes a plurality of key frames, one of the key frames from which the corresponding information is obtained is identified by matching feature points between the key frame and the query image. Information processing methods.

15. Acquire correspondence information between a key frame pre-arranged in the map data and the query image; A plurality of the map data are synthesized based on the correspondence information. When the map data includes a plurality of key frames, one of the key frames from which the corresponding information is obtained is identified by matching feature points between the key frame and the query image. An information processing program that causes a computer to execute an information processing method.

Images