Information processing method, information processing device, terminal device, and system

Abstract

An information processing method, wherein an information processing device executes processing for: providing one or more terminal devices with other partial map information generated by another terminal device different from the terminal device itself; acquiring, from each terminal device, partial map information generated by the device itself and information relating to a matching result obtained as a result of the terminal device matching a relative positional relationship between trajectory position information and the other partial map information on the basis of the other partial map information and the trajectory position information, which indicates a position on a trajectory along which the device itself moves through a region included in the partial map information; and performing optimization processing for correcting an error in overall trajectory position information constructed by combining the trajectory position information of the terminal device on the basis of the information relating to the matching result.

Description

Information processing method, information processing device, terminal device, and system

[0001] The present disclosure relates to an information processing method, an information processing device, a terminal device, and a system.

[0002] A technique called SLAM (Simultaneous Localization and Mapping) is known, which simultaneously estimates a vehicle's own position and creates an environmental map.

[0003] For example, in Patent Document 1, in a robot system including a robot and a computer (server device), the computer generates and updates map data based on sensor data collected by each robot, and each robot estimates its own position based on the map data obtained from the computer.

[0004] JP 2012-256344 A

[0005] However, in a SLAM system in which multiple robots operate simultaneously, the above-described conventional technology does not necessarily make it possible to appropriately reduce the processing load on the server device that is required to perform SLAM calculations for multiple robots.

[0006] For example, the above-mentioned conventional technology is intended to reduce the computational load on a server device involved in movement control calculations for multiple robots in a SLAM system in which multiple robots operate simultaneously, but does not take into consideration reducing the computational load on a server device involved in SLAM calculations for multiple robots.

[0007] Therefore, this disclosure proposes an information processing method, information processing device, terminal device, and system that are capable of appropriately reducing the processing load on a server device for SLAM calculations targeting multiple robots in a SLAM system in which multiple robots operate simultaneously.

[0008] According to the present disclosure, there is provided an information processing method in which an information processing device provides one or more terminal devices with other partial map information generated by other terminal devices different from the terminal device, and the terminal device acquires from the terminal device information and the other partial map information, information on a matching result obtained by matching a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device, and the other partial map information, and performs an optimization process to correct errors in overall trajectory position information constructed by combining the trajectory position information of the terminal devices based on the matching result information.

[0009] Furthermore, according to the present disclosure, there is provided an information processing method in which a terminal device acquires other partial map information generated by another terminal device different from the terminal device itself from an information processing device on the cloud side, and performs processing to match the relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating the position on the trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device itself, point cloud information indicating the position of an object within the partial map information, and the other partial map information, and uploads information on the matching result of the relative positional relationship between the trajectory position information and the other partial map information, and the partial map information, to the information processing device that generates an environmental map corresponding to the partial map.

[0010] As described above, according to the present disclosure, in a SLAM system in which multiple robots operate simultaneously, it is possible to appropriately reduce the processing load on a server device for SLAM calculations targeting multiple robots. Note that the effects described here are not necessarily limited to those described herein, and may be any of the effects described in the present disclosure.

[0011] 1 is a diagram illustrating an overview and an example configuration of a multi-robot SLAM system 1 according to an embodiment of the present disclosure. FIG. 1 is a functional block diagram illustrating an example of the functional configuration of a sensing camera 20 according to an embodiment. FIG. 2 is a diagram illustrating the process of superimposing sensor data on a submap. FIG. 3 is a diagram illustrating an example of a matching process between a self-trajectory and a self submap by a sensing camera 20-1. FIG. 4 is a diagram illustrating an overview of a matching process between a self-trajectory and another submap. FIG. 5 is a diagram illustrating a specific example of multi-robot SLAM processing. FIG. 6 is a functional block diagram illustrating an example of a conventional functional configuration of the multi-robot SLAM system 1. FIG. 7 is a functional block diagram illustrating an example of the functional configuration of a proposed method in the multi-robot SLAM system 1. FIG. 8 is a flowchart illustrating an upload processing procedure executed by the sensing camera 20 in the proposed method. FIG. 9 is a flowchart illustrating a scan matching detection procedure executed by the sensing camera 20 in the proposed method. FIG. 10 is a flowchart illustrating a scan matching processing procedure executed by the sensing camera 20 in the proposed method. FIG. 11 is a flowchart illustrating an optimization processing procedure executed by a server device 100 in the proposed method. FIG. 12 is a functional block diagram illustrating an example of the functional configuration of a first modified example of the multi-robot SLAM system 1. FIG. 13 is a flowchart illustrating a preprocessing procedure related to a request for matching processing. FIG. 14 is a flowchart illustrating a request processing procedure for requesting matching processing. Fig. 1 is a flowchart showing a matching processing procedure executed by the server device 100 in response to a request for matching processing. Fig. 2 is a functional block diagram showing an example functional configuration of a modified example 2 in the multiple-robot SLAM system 1. Fig. 3 is a flowchart showing a request processing procedure for requesting matching processing. Fig. 4 is a block diagram showing an example hardware configuration of a computer corresponding to an apparatus according to an embodiment of the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects.

[0014] The present disclosure will be described in the following order: 1. Introduction 2. Embodiment 2-1. Example of a System Schematic Configuration 2-2. Functional Configuration of a Sensing Camera 2-3. Overview of Multiple-Robot SLAM 2-4. Superimposition of Sensor Data 2-5. Matching Processing within a Single Device 2-6. Matching Processing Between Multiple Devices 2-7. Specific Example of Multiple-Robot SLAM Processing 2-8. Example of a Detailed System Configuration (Conventional Method) 2-8-1. Functional Configuration of a Terminal Device 2-8-2. Functional Configuration of an Information Processing Device 2-9. Example of a Detailed System Configuration (Proposed Method) 2-9-1. Functional Configuration of a Terminal Device 2-9-2. Functional Configuration of an Information Processing Device 2-10. Operation Procedure 2-10-1. Upload Procedure of a Terminal Device 2-10-2. Scan Matching Detection Procedure of a Terminal Device 2-10-3. Scan Matching Processing Procedure of a Terminal Device 2-10-4. Optimization Processing Procedure of an Information Processing Device 3. 3. Modification 1 3-1. Example of detailed configuration of system in modification 1 3-1-1. Functional configuration of terminal device 3-1-2. Functional configuration of information processing device 3-2. Operation procedure 3-2-1. Request pre-processing procedure 3-2-2. Request processing procedure 3-2-3. Scan matching processing procedure of information processing device 4. Modification 2 4-1. Example of detailed configuration of system in modification 2 4-1-1. Functional configuration of terminal device 4-1-2. Functional configuration of information processing device 4-2. Operation procedure 5. Hardware configuration 6. Summary

[0015] [1. Introduction] A SLAM system that realizes SLAM is composed of a front-end processing section that focuses on sensor signal processing, and a back-end processing section that focuses on pose graph optimization.

[0016] In a SLAM (multi-robot SLAM) system in which multiple robots operate simultaneously, the front-end processing unit may be located on each edge robot, and the back-end processing unit may be located on a cloud server. In such a configuration, map data is integrated on the server side, allowing each robot to use the same environmental map and reducing the processing load on the robots.

[0017] On the other hand, in a multi-robot SLAM system, as the number of robots increases, the load of the matching process performed by the server device between the robot's sensor data and map data increases at a rate that is squared with the number of robots.

[0018] Therefore, this disclosure proposes a method for suppressing an increase in the processing load on a server device when the number of robots increases by managing map data on a server device on the cloud side and using computing resources on the edge side for matching processing between map data and sensor data.

[0019] For example, according to the proposed method of the present disclosure, an edge-side terminal device uploads map data to a cloud-side server device, but does not upload sensor data, instead performing matching processing within the device itself. For example, the terminal device performs matching processing between the sensor data acquired by the device itself and the map data generated by the device itself. The terminal device also performs matching processing between the sensor data acquired by the device itself and the map data generated by other terminal devices. This processing is performed by each terminal device included in the multi-robot SLAM, and each terminal device transmits the matching results to the server device.

[0020] The server device generates a more accurate map by optimizing the map data acquired from each terminal device based on the matching results acquired from each server device. Each terminal device can then use this accurate map to estimate its own location.

[0021] 2. Embodiments Hereinafter, information processing according to an embodiment of the present disclosure will be described in detail with reference to the drawings. The information processing according to an embodiment of the present disclosure is realized in a SLAM system in which multiple robots operate simultaneously.

[0022] 1 is a diagram illustrating an overview and configuration example of a multi-robot SLAM system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the multi-robot SLAM system 1 according to this embodiment includes a mobile object 10, a sensing camera 20, and a server device 100.

[0023] The mobile body 10 is an autonomously moving robot equipped with a sensing camera 20. Examples of the mobile body 10 include a robot vacuum cleaner and a drone. As shown in FIG. 1 , the multi-robot SLAM system 1 includes multiple mobile bodies 10, such as mobile body 10-1 to mobile body 10-n. Note that the number of mobile bodies 10 included in the multi-robot SLAM system 1 is not limited.

[0024] The sensing camera 20 is an example of a terminal device according to the embodiment, and is responsible for front-end processing in the multi-robot SLAM system 1, which is divided into a front-end and a back-end. Therefore, the sensing camera 20 can be said to be an edge-side terminal device.

[0025] As shown in FIG. 1 , the multi-robot SLAM system 1 includes multiple sensing cameras 20, namely, sensing camera 20-1 to sensing camera 20-n, and each sensing camera 20 is mounted on a moving body 10. FIG. 1 shows an example in which sensing camera 20-1 is mounted on moving body 10-1 and sensing camera 20-n is mounted on moving body 10-n. Note that the sensing camera 20 may be implemented as an external module attached to the moving body 10, or may be built into the moving body 10 during manufacturing. Therefore, in the following embodiments, the processing described as being performed by the sensing camera 20 can be rephrased as processing performed by the moving body 10.

[0026] In the following embodiments, the expression "plurality of robots" can be rephrased as "plurality of sensing cameras 20," and the expression "own device" can include both the meaning of "the sensing camera 20 itself" and "the mobile body 10 on which the sensing camera 20 is mounted." Furthermore, the position and orientation of the sensing camera 20 essentially indicate the position and orientation of the mobile body 10.

[0027] As shown in FIG. 1 , the sensing camera 20 executes front-end SLAM processing using sensor data sensed by its own sensor function (sensor unit 21 in FIG. 2 ), and transmits the results of the SLAM processing to the server device. Specifically, the sensing camera 20 observes sensor data of objects (obstacles, people, etc.) present in the environment as the mobile object 10 moves through the environment. The sensing camera 20 generates map data relating to an environmental map based on the sensor data. The sensing camera 20 uploads the generated map data to the server device 100. Furthermore, each sensing camera 20 receives map data from the server device 100, which has been error-corrected based on the map data from the multiple sensing cameras 20, and estimates its own position based on the received map data.

[0028] The server device 100 is an example of an information processing device in the embodiment, and is responsible for back-end processing in the multi-robot SLAM system 1, which is divided into a front-end and a back-end. Therefore, in the present embodiment, the server device 100 is a cloud-side computer. However, the server device 100 may also be a computer on a local network. In other words, the server device 100 is not limited to being either a cloud or a local computer, as long as it can be accessed via a network.

[0029] As shown in Fig. 1 , the server device 100 generates an integrated map by integrating the map data by performing back-end SLAM processing using the map data acquired from each sensing camera 20. Specifically, the server device 100 acquires map data related to an environmental map from each of the multiple sensing cameras 20. The server device 100 performs a matching process to match the map data acquired from each of the sensing cameras 20 with each other. The server device 100 also performs an error correction process on the map data based on the result of the matching process and the relative positional relationship between the map data. The server device 100 then distributes the error-corrected map data downstream to each of the multiple sensing cameras 20.

[0030] 1, the server device 100 may also perform navigation control of the mobile object 10, which is a robot, based on the integrated map, i.e., error-corrected map data. As a result, each sensing camera 20 can move the mobile object 10 mounted on the sensing camera 20 in accordance with the content of the navigation control.

[0031] 2 is a functional block diagram showing an example of the functional configuration of the sensing camera 20 according to the embodiment. The sensing camera 20 has a sensor unit 21.

[0032] Also, according to the example of FIG. 2 , the sensor unit 21 includes a ranging sensor 211a and a ranging sensor signal processor 211b. The sensor unit 21 includes an RGB camera 212a and an RGB camera signal processor 212b. The sensor unit 21 includes an IMU 213a and an IMU signal processor 213b. Although not shown in FIG. 2 , the sensor unit 21 may further include an odometry sensor. The configuration of the sensor unit 21 is not limited to the example of FIG. 2 . For example, in SLAM processing, the ranging sensor 211a and the RGB camera 212a are required as external sensors (sensors that observe the outside) for creating an environmental map, while internal sensors for measuring the internal state of the robot may be treated as optional sensors. In the example of FIG. 2 , the internal sensors are the IMU 213a and the odometry sensor, and may be selectively introduced, for example, to improve the accuracy of SLAM processing.

[0033] Furthermore, according to the example of FIG. 2 , the sensing camera 20 has a SLAM recognition processing unit 214 , an obstacle recognition processing unit 215 , and a person / object recognition processing unit 216 .

[0034] The ranging sensor 211a may include an ultrasonic sensor (Sound Navigation and Ranging: SONAR), a Time of Flight (ToF) sensor, a Light Detection and Ranging (LiDAR) sensor, etc. For example, the ranging sensor 211a acquires information regarding the distance and direction to a measurement point as sensor data.

[0035] The ranging sensor signal processing unit 211b converts the sensor data acquired by the ranging sensor 211a into signal data that can be used by each recognition processing unit (SLAM recognition processing unit 214, obstacle recognition processing unit 215, and person / object recognition processing unit 216). For example, the SLAM recognition processing unit 214 converts measurement points into points in a local coordinate system based on the signal data acquired from the ranging sensor signal processing unit 211b. The SLAM recognition processing unit 214 then outputs coordinate information of the multiple points (point cloud) obtained by the conversion as point cloud information.

[0036] The obstacle recognition processing unit 215 recognizes obstacles present in the environment in which the mobile body 10 moves, based on the signal data acquired from the distance measurement sensor signal processing unit 211b. The person / object recognition processing unit 216 recognizes people and objects present in the environment in which the mobile body 10 moves, based on the signal data acquired from the distance measurement sensor signal processing unit 211b.

[0037] The RGB camera 212a captures, as sensor data, image data that captures the state of the environment in which the mobile object 10 on which the device is mounted moves.

[0038] The RGB camera signal processing unit 212b converts the sensor data acquired by the RGB camera 212a into signal data that can be used by each recognition processing unit (SLAM recognition processing unit 214, obstacle recognition processing unit 215, and person / object recognition processing unit 216). For example, the SLAM recognition processing unit 214 estimates the position and attitude of the mobile object 10 on which its own device is mounted based on the signal data acquired by the RGB camera signal processing unit 212b, and estimates the trajectory of the mobile object 10 moving through the environment from the estimation result.

[0039] The obstacle recognition processing unit 215 recognizes obstacles present in the environment in which the mobile body 10 is moving, based on the signal data acquired from the RGB camera signal processing unit 212b. The person / object recognition processing unit 216 recognizes people and objects present in the environment in which the mobile body 10 is moving, based on the signal data acquired from the RGB camera signal processing unit 212b.

[0040] The IMU 213a acquires, as sensor data, angles and accelerations of three axes that govern the motion of the moving body 10 on which the IMU 213a is mounted. Specifically, the IMU outputs data including "time," "frequency," "orientation," "angular velocity," and "acceleration (linear acceleration)." "Time" indicates the time when the output data was generated. "Frequency" indicates the output frequency of the IMU. "Orientation" indicates the orientation of the moving body 10. "Angular velocity" indicates the angular velocity of the moving body 10. "Acceleration (linear acceleration)" indicates the acceleration of the moving body 10. Note that the data output by the IMU 213a may differ depending on the type of the IMU 213a. Therefore, the data output by the IMU 213a does not necessarily need to include a combination of "time," "frequency," "orientation," "each speed," and "acceleration."

[0041] The IMU signal processing unit 213b converts the sensor data acquired by the IMU 213a into signal data that can be used by each recognition processing unit (the SLAM recognition processing unit 214, the obstacle recognition processing unit 215, and the person / object recognition processing unit 216). For example, the SLAM recognition processing unit 214 generates self-position data of the mobile body 10 on which the device is mounted, based on the signal data acquired by the IMU signal processing unit 213b. For example, the SLAM recognition processing unit 214 generates self-position data in a local coordinate system.

[0042] The person / object recognition processing unit 216 recognizes people and objects present in the environment in which the moving body 10 moves, based on the signal data acquired from the IMU signal processing unit 213b.

[0043] The sensor unit 21 does not necessarily have to include all of the SLAM recognition processing unit 214, the obstacle recognition processing unit 215, and the person / object recognition processing unit 216. For example, an external sensor such as the distance measurement sensor 211a or the RGB camera 212a can substitute for the function of the obstacle recognition processing unit 215, so the sensor unit 21 does not necessarily have to include the obstacle recognition processing unit 215.

[0044] [2-3. Overview of Multi-Robot SLAM] Next, an overview of multi-robot SLAM will be described. In multi-robot SLAM, to achieve matching processing, each sensing camera 20 sequentially generates a partial SLAM map. A partial map is called a submap, and is, for example, a lattice map (also called a grid map) that is two-dimensional matrix data.

[0045] The submap may be three-dimensional map data. For example, three-dimensional map data includes a voxel map and a point cloud map with added height information. Specifically, in a grid map, the probability of an obstacle being present is set high for squares on the grid map corresponding to positions where an obstacle is observed by a sensor mounted on a mobile object, and the probability of an obstacle being present is set low for squares where no obstacle is observed.

[0046] For example, the sensing camera 20 sequentially generates submaps at any time using sensor data at any time so that matching processing can be performed. The sensing camera 20 also superimposes on the submap representative points (trajectory nodes) of the movement path (trajectory) of the moving object 10 while it moves through an area included in the submap, and sensor data observed at the positions of the trajectory nodes.

[0047] 3 is a diagram illustrating the process of overlaying sensor data onto a submap. When the sensing camera 20 generates the submap sbm, the sensing camera 20 overlays, onto the submap sbm, the sensor data SD observed at a trajectory node tND on the movement path TR of the moving body 10 (the moving body 10 on which the sensing camera 20 is mounted) that has moved through an area included in the submap sbm.

[0048] 3 shows an example in which nine trajectory nodes tND are generated on the travel route TR by the sensing camera 20. In this example, the sensing camera 20 superimposes, for each of the nine trajectory nodes tND, a point cloud of sensor data SD observed at the trajectory node tND on the sub-map sbm.

[0049] [2-5. Matching Process Within a Single Device] Each sensing camera 20 performs a matching process between the trajectory nodes generated by the sensing camera itself and a submap as a front-end SLAM process. Specifically, the sensing camera 20 performs a matching process between the trajectory nodes generated by the sensing camera itself and its own submap, which is a submap generated by the sensing camera itself based on the trajectory nodes. Hereinafter, this matching process will be referred to as a "matching process between own trajectory and own submap."

[0050] The sensing camera 20 also performs a matching process between the trajectory nodes generated by the sensing camera 20 itself and other submaps that are submaps generated by other sensing cameras 20. Hereinafter, this matching process will be referred to as a "matching process between the sensing camera's own trajectory and other submaps."

[0051] An example of the matching process between the own trajectory and the own submap will be described with reference to Fig. 4. Fig. 4 shows an example of the matching process between the own trajectory and the own submap by the sensing camera 20-1. For example, the sensing camera 20-1 performs a matching process between the sensor data SD observed at time t1 at a trajectory node tND on the movement route TR of the own device and the submap smb generated by the own device based on the sensor data SD for an observation time range including time t1 (for example, times t1 to t3).

[0052] In this way, in the matching process between the vehicle's own trajectory and the vehicle's own submap, the sensor data on the trajectory is data from a certain moment, so matching must be performed according to the time of the sensor data on the trajectory. This also applies to the matching process between the vehicle's own trajectory and other submaps.

[0053] 4, the sensing camera 20-1 superimposes the sensor data SD on a submap smb that it has generated based on the sensor data SD. Then, the sensing camera 20-1 matches the relative positional relationship between a submap node sND, which is a node indicating the reference point of the submap sbm, and a trajectory node tND, and rotates or translates the submap sbm and the trajectory node tND relative to each other so that the overall relationship is consistent (so that the position of the obstacle in the submap sbm matches the position of the sensor data SD).

[0054] Furthermore, when the sensing camera 20-1 executes the matching process as described above, it generates, based on the matching results, a constraint condition between its own trajectory and its own submap COT1 as a constraint condition indicating the relative positional relationship between the trajectory node tND and the submap sbm. In this way, the sensing camera 20-1 matches the relative positional relationship between its own trajectory position information and its own partial map information, thereby generating a constraint condition (self-trajectory-self submap constraint condition COT1) indicating this relative positional relationship.

[0055] Next, the sensing camera 20-1 uploads the submap data, which is information about the submap sbm generated by the sensing camera 20-1, and the constraint condition COT1 between the self trajectory and the self submap, to the server device 100. On the other hand, the sensing camera 20-1 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND including the sensor data SD, to the server device 100, but keeps it internally.

[0056] Up to this point, the self-trajectory-self sub-map matching process has been explained using the sensing camera 20-1 as an example, but in a multiple-robot SLAM, the other sensing cameras 20 also perform the self-trajectory-self sub-map matching process in a similar manner.

[0057] For example, the sensing camera 20-2 performs a matching process between sensor data SD observed at time t1 at a trajectory node tND on the movement route TR of the own device and a submap smb generated by the own device for an observation time range including time t1 (for example, times t1 to t3) based on the sensor data SD. Then, the sensing camera 20-2 generates a self-trajectory-self-submap constraint condition COT2 as a constraint condition indicating the relative positional relationship between the trajectory node tND and the submap sbm.

[0058] Next, the sensing camera 20-2 uploads the submap data, which is information about the submap sbm generated by the sensing camera 20-2, and the constraint condition COT2 between the self trajectory and the self submap, to the server device 100. On the other hand, the sensing camera 20-2 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND including the sensor data SD, to the server device 100, but keeps it internally.

[0059] 4 has described the self-trajectory-self submap matching process, which is completed by each sensing camera 20 using only the information generated by that camera. However, as described above, each sensing camera 20 also performs self-trajectory-other submap matching process, which matches the trajectory nodes generated by that camera with other submaps generated by other sensing cameras 20. Therefore, an overview of the self-trajectory-other submap matching process will be described in FIG. 5.

[0060] According to Figure 5, in multi-robot SLAM, the matching process within a single device (matching process between self-trajectory and self-submap) described in Figure 4 is expanded to matching process between multiple devices (matching process between self-trajectory and other submaps).

[0061] The left side of Fig. 5 shows a conceptual diagram of the self-trajectory-self submap matching process described in Fig. 4 using the sensing camera 20-1. Meanwhile, the right side of Fig. 5 shows a conceptual diagram of the self-trajectory-other submap matching process extended between the sensing camera 20-1 and the sensing camera 20-2. According to this example, the sensing camera 20-1 performs a matching process between sensor data SD observed at time t1 and a submap smb2 generated by the sensing camera 20-2 in an observation time range including time t1 (for example, time t2) based on the sensor data SD at a trajectory node tND1 on the movement route TR1 of the own device.

[0062] For example, the sensing camera 20-1 superimposes its own sensor data SD on the submap smb2. Then, the sensing camera 20-1 matches the relative positional relationship between the submap node sND2 indicating the reference point of the submap sbm2 and the trajectory node tND1, and rotates or translates the submap sbm2 and the trajectory node tND1 relative to each other so that the data is consistent overall (so that the position of the obstacle in the submap sbm2 matches the position of the sensor data SD).

[0063] Furthermore, based on the matching results of the extended matching process, the sensing camera 20-1 generates a constraint condition COT12 between its own trajectory and another submap as a constraint condition indicating the relative positional relationship between the trajectory node tND1 and the submap sbm2. In this way, the sensing camera 20-1 matches the relative positional relationship between its own trajectory position information and other partial map information, thereby generating a constraint condition (constraint condition COT12 between its own trajectory and another submap) indicating these relative positional relationships.

[0064] Next, the sensing camera 20-1 uploads the submap data, which is information about the submap sbm1 generated by the sensing camera 20-1, and the constraint condition COT12 between the sensing camera's own trajectory and other submaps, to the server device 100. On the other hand, the sensing camera 20-1 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND1 including the sensor data SD, to the server device 100, but instead stores the data within the sensing camera 20-1.

[0065] In a multiple robot SLAM system, as shown in FIG. 5, the same processing is performed on the sensing camera 20-2 side as on the sensing camera 20-1.

[0066] For example, the sensing camera 20-2 performs a matching process between the sensor data SD observed at time t1 at the trajectory node tND2 on the movement route TR2 of the device itself and the submap smb1 generated by the sensing camera 20-1 based on the sensor data SD in an observation time range including time t1 (for example, time t2).

[0067] For example, the sensing camera 20-2 superimposes its own sensor data SD on the submap smb1. Then, the sensing camera 20-2 matches the relative positional relationship between the submap node sND1 indicating the reference point of the submap sbm1 and the trajectory node tND2, and rotates or translates the submap sbm1 and the trajectory node tND2 relative to each other so that the data is consistent overall (so that the position of the obstacle in the submap sbm1 matches the position of the sensor data SD).

[0068] Furthermore, based on the matching results of the extended matching process, the sensing camera 20-2 generates a constraint condition COT21 between its own trajectory and another submap as a constraint condition indicating the relative positional relationship between the trajectory node tND2 and the submap sbm1. In this way, the sensing camera 20-2 matches the relative positional relationship between its own trajectory position information and other partial map information, thereby generating a constraint condition (constraint condition COT21 between its own trajectory and another submap) indicating these relative positional relationships.

[0069] Next, the sensing camera 20-2 uploads the submap data, which is information about the submap sbm2 generated by the sensing camera 20-2, and the constraint condition COT21 between the own trajectory and other submaps, to the server device 100. On the other hand, the sensing camera 20-2 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND2 including the sensor data SD, to the server device 100, but keeps it internally.

[0070] 2-7. Specific Example of Multiple Robot SLAM Processing] Fig. 6 is a diagram showing a specific example of multiple robot SLAM processing. Fig. 6 shows a specific example of matching processing according to the embodiment and optimization processing based on the results of the matching processing. Note that the matching processing according to the embodiment is a general term for matching processing between the user's own trajectory and the user's own submap and matching processing between the user's own trajectory and another submap.

[0071] 6 also shows an example in which the sensing camera 20-1 performs self-trajectory-self submap matching processing between the submap sbmA1 generated at time t1, the submap sbmA2 generated at time t2, and the submap sbmA3 generated at time t3. Furthermore, FIG. 6 also shows an example in which the sensing camera 20-1 performs self-trajectory-other submap matching processing between the submap sbmA1, submap sbmA2, and submap sbm3 and the submap sbmB2 generated by the sensing camera 20-2 at time t2.

[0072] Submap node sNDA1 is the center point defined for submap sbmA1, submap node sNDA2 is the center point defined for submap sbmA2, submap node sNDA3 is the center point defined for submap sbmA3, while submap node sNDB2 is the center point defined for submap sbmB2.

[0073] In this state, the sensing camera 20-1 can estimate the position and orientation tND and the movement trajectory TR of the device based on the sensor data SD of the device, as shown in Figure 6 (a), and therefore determines some representative points of the position and orientation tND as trajectory nodes tND and constructs a pose graph PG for each of the submaps sbmA1, sbmA2, and sbmA3.

[0074] Similarly, the sensing camera 20-2 can estimate the position and orientation tND and the movement trajectory TR of its own device based on its own sensor data SD. For this reason, the sensing camera 20-2 also determines some representative points of the position and orientation tND as trajectory nodes tND and constructs a pose graph PG for the submap sbmB2.

[0075] The pose graph PG is a graph representation of the movement trajectory TR, with the position and orientation of the object (in the example of FIG. 6, the sensing camera 20-1 and the sensing camera 20-2) represented as trajectory nodes tND and the relative positions between the nodes tND represented as edges. For example, when the same landmark is observed in image data acquired at different times by the RGB camera 212a of the sensing camera 20-1 (similar to the sensing camera 20-2), an edge can be created between the nodes tND that observed this landmark.

[0076] In this state, as shown in FIG. 6B, the sensing camera 20-1 and the sensing camera 20-2 each perform the matching process according to the embodiment.

[0077] First, the self-trajectory-self sub-map matching process will be described. According to the example of FIG. 6B, the sensing camera 20-1 performs the self-trajectory-self sub-map matching process, which matches the relative positional relationship between each trajectory node tND included in the sub-map sbmA1 and the sub-map node sNDA1 of the sub-map sbmA1. Based on the sensing results, the sensing camera 20-1 then generates the self-trajectory-self sub-map constraint condition COT, which indicates the relative positional relationship between each trajectory node tND included in the sub-map sbmA1 and the sub-map sbmA1.

[0078] Similarly, the sensing camera 20-1 performs a self-trajectory-self submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap node sNDA2 of the adjacent submap sbmA2. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-self submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap sbmA2.

[0079] The sensing camera 20-1 also performs a self-locus-self submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA2 and the submap node sNDA2 of the submap sbmA2. Based on the sensing results, the sensing camera 20-1 then generates a self-locus-self submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA2 and the submap sbmA2.

[0080] Similarly, the sensing camera 20-1 performs a self-trajectory-self submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA2 and the submap node sNDA3 of the adjacent submap sbmA3. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-self submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA3 and the submap sbmA3.

[0081] 6(b), the sensing camera 20-2 performs a self-trajectory-self submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmB2 and the submap node sNDA2 of the submap sbmB2. Then, based on the sensing results, the sensing camera 20-2 generates a self-trajectory-self submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmB2 and the submap sbmB2.

[0082] Next, the matching process between the sensing camera's own trajectory and other submaps will be described. In the example of Fig. 6(b), the sensing camera 20-1 can use the submaps sbmA1 and sbmA3 of its own adjacent devices for the submap sbmB2 generated by the sensing camera 20-2.

[0083] For example, the sensing camera 20-1 performs a self-trajectory-to-other-submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap node sNDB2 of the adjacent submap sbmB2. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-to-other-submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap sbmB2.

[0084] The sensing camera 20-1 also performs a self-trajectory-to-other-submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA3 and the submap node sNDB2 of the adjacent submap sbmB2. Based on the sensing results, the sensing camera 20-1 then generates a self-trajectory-to-other-submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA3 and the submap sbmB2.

[0085] 6(a), the sensing camera 20-1 uploads the pose graph PG generated by itself, the constraint condition COT generated by itself, and the submap sbm (A1, A2, A3) generated by itself to the server device 100. On the other hand, the sensing camera 20-1 does not upload the sensor data SD associated with the trajectory node tND determined by itself to the server device 100.

[0086] Furthermore, the sensing camera 20-2 uploads the pose graph PG generated by itself, the constraint condition COT generated by itself, and the submap sbmB2 generated by itself to the server device 100. On the other hand, the sensing camera 20-2 does not upload to the server device 100 the sensor data SD associated with the trajectory node tND determined by itself.

[0087] 6(c), the server device 100 executes an optimization process to correct errors between the submaps using the information uploaded from the sensing camera 20-1 and the sensing camera 20-2. In the optimization process, errors in the positions of all nodes are corrected so that the relative positional relationships between all submap nodes sND and trajectory nodes tND become consistent.

[0088] Specifically, the server device 100 corrects errors between the submaps sbm (A1, A2, A3, B2) based on the submaps sbm (A1, A2, A3, B2) and the constraint condition COT. In this way, the server device 100 corrects errors in the overall map information constructed by combining the partial map information generated by each of the sensing cameras 20 based on the constraint condition COT.

[0089] Furthermore, based on the constraint COT, the server device 100 corrects errors in the overall pose graph that combines the pose graphs PG constructed for the submaps sbmA1, sbmA2, and sbmA3 with the pose graph PG constructed for the submap sbmB2. In this way, based on the constraint COT, the server device 100 corrects errors in the overall trajectory position information that is constructed by combining the trajectory position information generated by each of the sensing cameras 20.

[0090] As a result of the above-described optimization process, the server device 100 can generate a single environmental map (an accurate map) in which the submaps sbmA, sbmA, sbmA3, and sbmB2 are integrated so as to correct the errors. Fig. 6C shows the environmental map MP in which the submaps sbmA, sbmA, sbmA3, and sbmB2 are integrated through the optimization process.

[0091] [2-8. Detailed System Configuration Example (Conventional Method)] Here, a conventional functional configuration example of a multi-robot SLAM system 1 will be described in detail using Fig. 7. Fig. 7 is a functional block diagram showing a conventional functional configuration example of a multi-robot SLAM system 1. The multi-robot SLAM system 1 shown in Fig. 7 includes a mobile body 10-1 (robot A) equipped with a sensing camera 20-1, a mobile body 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100.

[0092] 7, the functional configuration of the sensing camera 20 will be described focusing on sensing camera 20-1, but sensing camera 20-2 may have a similar functional configuration. Furthermore, in addition to sensing cameras 20-1 and 20-2, the multi-robot SLAM system 1 may further include mobile bodies 10 equipped with other sensing cameras 20, and all of the sensing cameras 20 may have the same functional configuration as sensing camera 20-1.

[0093] [2-8-1. Functional Configuration of Terminal Device] First, an example of the functional configuration of a sensing camera 20-1 (terminal device) according to a conventional method will be described. As shown in Fig. 7, the sensing camera 20-1 has a sensor unit 21, a pre-processing unit 22, and a SLAM processing unit 23. Although not described here, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1.

[0094] (Sensor Unit 21) The sensor unit 21 has the various sensor functions described in FIG. 2. For example, the sensor unit 21 includes a distance measurement sensor 211a, a distance measurement sensor signal processing unit 211b, an RGB camera 212a, an RGB camera signal processing unit 212b, an IMU 213a, and an IMU signal processing unit 213b. The sensor unit 21 also includes a SLAM recognition processing unit 214, an obstacle recognition processing unit 215, and a person / object recognition processing unit 216. The sensor unit 21 may further include an odometry sensor. As described in FIG. 2, the sensor unit 21 does not necessarily have to include all of the distance measurement sensor 211a, the RGB camera 212a, and the IMU 213a. The sensor unit 21 does not necessarily have to include all of the SLAM recognition processing unit 214, the obstacle recognition processing unit 215, and the person / object recognition processing unit 216.

[0095] (Preprocessing Unit 22) The preprocessing unit 22 performs preprocessing for the SLAM processing performed by the SLAM processing unit 23. For example, the preprocessing unit 22 acquires sensor data SD from the sensor unit 21 and performs preprocessing for the SLAM processing based on the acquired sensor data SD. For example, the preprocessing unit 22 acquires output data from an IMU (e.g., acceleration data), output data from an odometry sensor, and output data from a LiDAR (e.g., point cloud). Note that a point cloud indicates surrounding two-dimensional or three-dimensional point cloud data (coordinates of each point) and data on the time the point cloud data was generated. For example, the preprocessing may include processes for removing unnecessary features and noise from the point cloud, as well as segmentation and downsampling.

[0096] (SLAM Processing Unit 23) The SLAM processing unit 23 corresponds to a front-end processing unit in SLAM. For example, the SLAM processing unit 23 generates a pose graph PG based on the sensor data SD acquired from the sensor unit 21, and uploads the pose graph data of the generated pose graph PG to the server device 100.

[0097] Note that the matching process requires a pose graph PG, but in the conventional method shown in FIG. 7 , the matching process is performed on the server device 100 side. Therefore, for example, the SLAM processing unit 23 of the sensing camera 20-1 does not need the pose graph PG generated by the sensing camera 20-2 in the SLAM process. Therefore, the optimization unit 130 of the server device 100 only needs to transmit to the sensing camera 20-1 the portion of the pose graph PG corrected in the optimization process that corresponds to the sensing camera 20-1, and does not need to transmit the portion corresponding to the sensing camera 20-2. Furthermore, in this example, the SLAM processing unit 23 of the sensing camera 20-1 synchronizes the pose graph PG generated by its own device with the pose graph PG corrected on the server device 100 side.

[0098] The same can be said for sensing camera 20-2. For example, the SLAM processing unit 23 of sensing camera 20-2 does not need the pose graph PG generated by sensing camera 20-2 in the SLAM process. Therefore, the optimization unit 130 of server device 100 only needs to transmit to sensing camera 20-2 the portion of the pose graph PG corrected in the optimization process that corresponds to sensing camera 20-2, and does not need to transmit the portion corresponding to sensing camera 20-1. Furthermore, in this example, the SLAM processing unit 23 of sensing camera 20-2 synchronizes the pose graph PG generated by its own device with the pose graph PG corrected on the server device 100 side.

[0099] The SLAM processing unit 23 also estimates the trajectory of its own device based on the sensor data SD acquired from the sensor unit 21. For example, the SLAM processing unit 23 generates trajectory nodes TG based on output data from the IMU 213 and output data from the odometry sensor, and uploads the trajectory node data of the generated trajectory nodes TG to the server device 100. Specifically, the SLAM processing unit 23 uploads the sensor data SD corresponding to the trajectory nodes TG to the server device 100.

[0100] In addition, the SLAM processing unit 23 generates a submap sbm based on the sensor data SD acquired from the sensor unit 21.

[0101] For example, the SLAM processing unit 23 generates a submap sbm based on output data from the RGB camera 212a or output data from LiDAR, and uploads the submap data of the generated submap sbm to the server device 100.

[0102] 7, the server device 100 includes a detection unit 110, a matching unit 120, an optimization unit 130, and a visualization unit 140.

[0103] (Detection Unit 110) The detection unit 110 detects combinations of sub-maps sbm to be matched and trajectory nodes tNG based on the pose graphs PG acquired from the sensing cameras 20.

[0104] 7 , in the example of a multi-robot SLAM system 1 including sensing cameras 20-1 and 20-2, the detection unit 110 detects a combination of matching targets between the submap sbm1 generated by the sensing camera 20-1 and the trajectory node tND1. The detection unit 110 also detects a combination of matching targets between the submap sbm2 generated by the sensing camera 20-2 and the trajectory node tND2.

[0105] The detection unit 110 also detects a combination of matching targets between the submap sbm1 generated by the sensing camera 20-1 and the trajectory node tND2 generated by the sensing camera 20-2. Similarly, the detection unit 110 detects a combination of matching targets between the submap sbm2 generated by the sensing camera 20-2 and the trajectory node tND1 generated by the sensing camera 20-1.

[0106] The detection unit 110 registers the detected combinations in a queue and causes the matching unit 120 to perform scan matching for each combination.

[0107] (Matching unit 120) The matching unit 120 performs scan matching. For example, the matching unit 120 performs scan matching based on a loop-closing query. For example, the matching unit 120 performs scan matching based on point cloud information of trajectory node data (sensor data SD) acquired from each sensing camera 20 and the submap sbm acquired from each sensing camera 20. For example, the matching unit 120 superimposes point cloud information from which unnecessary features and noise have been removed by preprocessing such as downsampling onto the submap sbm.

[0108] Furthermore, the matching unit 120 generates constraint conditions indicating the relative positional relationship with the submap sbm as a result of the scan matching.

[0109] In the example of Figure 7, the matching unit 120 matches the relative positional relationship between the trajectory node tND1 generated by the sensing camera 20-1 and the submap node sND1 of the submap sbm1 generated by the sensing camera 20-1, and generates a constraint condition COT1 that indicates this relative positional relationship.

[0110] In addition, the matching unit 120 matches the relative positional relationship between the trajectory node tND2 generated by the sensing camera 20-2 and the submap node sND2 of the submap sbm2 generated by the sensing camera 20-2, and generates a constraint condition COT2 that indicates this relative positional relationship.

[0111] The matching unit 120 also matches the relative positional relationship between the trajectory node tND1 generated by the sensing camera 20-1 and the submap node sND2 of the submap sbm2 generated by the sensing camera 20-2, and generates a constraint condition COT12 that indicates this relative positional relationship. Similarly, the matching unit 120 matches the relative positional relationship between the trajectory node tND2 generated by the sensing camera 20-2 and the submap node sND1 of the submap sbm1 generated by the sensing camera 20-1, and generates a constraint condition COT21 that indicates this relative positional relationship.

[0112] (Optimization Unit 130) The optimization unit 130 corrects errors in the map data based on each constraint condition COT generated by the matching unit 120. For example, the optimization unit 130 corrects errors in an overall submap sbm constructed by combining the submaps sbm generated by each sensing camera 20. For example, the optimization unit 130 corrects errors in an overall pose graph PG constructed by combining the pose graphs PG generated by each sensing camera 20 based on the constraint condition COT. According to the above example, the matching unit 120 corrects errors based on the constraint condition COT1, the constraint condition COT2, the constraint condition COT12, and the constraint condition COT21, thereby generating a single environmental map MP in which errors have been corrected.

[0113] The optimization unit 130 also transmits the corrected overall pose graph PG (the overall pose graph PG reflecting the constraint COT) to each sensing camera 20. In the example of FIG. 7 , the optimization unit 130 transmits only the portion of the corrected pose graph PG that corresponds to the pose graph PG acquired from sensing camera 20-1 to the pose graph PG of sensing camera 20-1, thereby synchronizing the two pose graphs PG. As described above, this processing is performed because, in the conventional method shown in FIG. 7 , sensing camera 20-1 does not perform matching processing and does not need to use the pose graph PG of sensing camera 20-2.

[0114] 7, the optimization unit 130 synchronizes the two pose graphs PG by transmitting only the portion of the corrected pose graph PG that corresponds to the pose graph PG acquired from the sensing camera 20-2 to the pose graph PG of the sensing camera 20-2. This processing is performed because, in the conventional method shown in FIG. 7, the sensing camera 20-2 does not perform matching processing and does not need to use the pose graph PG of the sensing camera 20-1.

[0115] (Visualization Unit 140) The visualization unit 140 generates a rendering map using the map data that has been error-corrected by the optimization unit 130. For example, the visualization unit 140 performs rendering processing on the submaps sbm that have been error-corrected by the optimization unit 130 to generate a single environmental map MP in which the submaps sbm are integrated. The visualization unit 140 may then output the environmental map MP to each sensing camera 20.

[0116] 2-9. Detailed System Configuration Example (Proposed Method) Next, an example of the functional configuration of a multi-robot SLAM system 1 using the proposed method according to the present disclosure will be described in detail using FIG. 8 . FIG. 8 is a functional block diagram showing an example of the functional configuration of a multi-robot SLAM system 1 using the proposed method. Similar to FIG. 7 , the multi-robot SLAM system 1 shown in FIG. 8 includes a mobile object 10-1 (robot A) equipped with a sensing camera 20-1, a mobile object 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100. Hereinafter, the multi-robot SLAM system 1 according to the proposed method will be referred to as the "multi-robot SLAM system 1A."

[0117] In the multi-robot SLAM system 1A according to the proposed method of the present disclosure, matching processing is performed on the sensing camera 20 side. Specifically, the sensing camera 20 performs matching processing between its own sensor data SD and its own submap sbm. The sensing camera 20 also performs matching processing between its own sensor data SD and the submaps sbm of other devices. That is, the matching processing described as being performed by the server device 100 in the conventional method of FIG. 7 is performed by each sensing camera 20. In this way, the proposed method employs a configuration in which matching processing is offloaded from the cloud server device 100 to the edge sensing cameras 20. As a result, it is believed that the processing load on the server device due to SLAM calculations can be appropriately reduced.

[0118] [2-9-1. Functional Configuration of Terminal Device] First, an example of the functional configuration of the sensing camera 20-1 (terminal device) according to the proposed method will be described. As shown in FIG. 8, the sensing camera 20-1 has a sensor unit 21, a pre-processing unit 22, and a SLAM processing unit 23, just like the conventional method, but also has a detection unit 24 and a matching unit 25. Although not explained further, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1. Also, in FIG. 8, dotted arrows indicate a new information processing flow compared to the conventional method of FIG. 7.

[0119] (SLAM Processing Unit 23) As in the conventional method, the SLAM processing unit 23 generates a pose graph PG based on the sensor data SD and uploads the pose graph data of the generated pose graph PG to the server device 100. The SLAM processing unit 23 also generates a submap sbm based on the sensor data SD and uploads the submap data of the generated submap sbm to the server device 100. On the other hand, in the proposed method, the SLAM processing unit 23 generates trajectory nodes TG based on the sensor data SD, but does not upload the trajectory node data of the generated trajectory nodes TG to the server device 100. Specifically, the SLAM processing unit does not upload the sensor data SD corresponding to the trajectory nodes TG to the server device 100.

[0120] In the proposed method shown in FIG. 8 , the matching process is performed on the sensing camera 20 side. Therefore, for example, the SLAM processing unit 23 of sensing camera 20-1 needs not only the pose graph PG generated by itself but also the pose graph PG generated by sensing camera 20-2 in the SLAM process. Therefore, the optimization unit 130 of the server device 100 transmits to sensing camera 20-1 not only the portion of the pose graph PG corrected in the optimization process that corresponds to sensing camera 20-1, but also the portion that corresponds to sensing camera 20-2. Furthermore, in this example, the SLAM processing unit 23 of sensing camera 20-1 synchronizes not only the pose graph PG generated by itself but also the pose graphs PG generated by all of the sensing cameras 20 included in the multi-robot SLAM system 1A.

[0121] The same can be said for sensing camera 20-2. For example, in SLAM processing, the SLAM processing unit 23 of sensing camera 20-2 needs not only the pose graph PG generated by itself but also the pose graph PG generated by sensing camera 20-1. Therefore, the optimization unit 130 of server device 100 transmits to sensing camera 20-2 not only the portion of the pose graph PG corrected in the optimization process that corresponds to sensing camera 20-2, but also the portion that corresponds to sensing camera 20-1. Furthermore, in this example, the SLAM processing unit 23 of sensing camera 20-2 synchronizes not only the pose graph PG generated by itself, but also the pose graphs PG generated by all of the sensing cameras 20 included in the multi-robot SLAM system 1A.

[0122] (Detection Unit 24) As shown in FIG. 8 , the detection unit 24 detects, based on the pose graph PG, a submap sbm to be registered in a queue as a matching query and a trajectory node tNG to be registered in a queue as a matching query. For example, the detection unit 24 detects, based on the pose graph PG, a combination of a submap sbm to be matched and a trajectory node tNG. The pose graph PG used here may include a pose graph PG transmitted from the optimization unit 130. Taking the sensing camera 20-1 as an example, the pose graph PG transmitted from the optimization unit 130 may include not only a pose graph PG generated by the sensing camera itself, but also a pose graph PG generated by the sensing camera 20-2.

[0123] For example, the detection unit 24 detects a combination of matching targets between the trajectory node tND1 generated by the own device and the submap sbm1 generated by the own device. The detection unit 24 also detects a combination of matching targets between the trajectory node tND1 generated by the own device and the submap sbm2 generated by the sensing camera 20-2.

[0124] The detection unit 24 registers the detected combinations in a queue and causes the matching unit 25 to perform scan matching for each combination.

[0125] (Matching unit 25) The matching unit 25 performs matching processing according to the embodiment. The matching unit 25 sequentially performs matching processing for each combination of a trajectory node tNG and a submap sbm that are registered in the queue as a matching query. Specifically, the matching unit 25 performs matching processing between the own trajectory and the own submap, and matching processing between the own trajectory and another submap.

[0126] When performing the matching process between the self-trajectory and another submap, the matching unit 25 acquires a submap sbm generated by another sensing camera 20 different from the self-device from the server device 100. For example, the matching unit 25 of the sensing camera 20-1 acquires the submap sbm2 generated by the sensing camera 20-2 from the server device 100. On the other hand, the matching unit 25 of the sensing camera 20-2 acquires the submap sbm1 generated by the sensing camera 20-1 from the server device 100.

[0127] In this state, in the self-trajectory-self submap matching process, the matching unit 25 of the sensing camera 20-1 matches the relative positional relationship between the trajectory node tND1 generated by its own device and the submap node sND1 of the submap sbm1 generated by its own device, and generates a self-trajectory-self submap constraint condition COT1 that indicates this relative positional relationship. The matching unit 25 also matches the relative positional relationship between the trajectory node tND1 generated by its own device and the submap node sND2 of the submap sbm2 generated by the sensing camera 20-2, and generates a self-trajectory-other submap constraint condition COT12 that indicates this relative positional relationship.

[0128] Similar processing is also performed by the matching unit 25 of the sensing camera 20-2, and as a result, a self-trajectory-to-self submap constraint condition COT2 is generated by the self-trajectory-to-self submap matching processing, and a self-trajectory-to-other submap constraint condition COT21 is generated by the self-trajectory-to-other submap matching processing.

[0129] Furthermore, the matching unit 25 uploads the constraint conditions COT generated by its own device to the server device 100. According to the above example, the matching unit 25 of the sensing camera 20-1 uploads the constraint conditions COT1 and COT12 to the server device 100. Furthermore, the matching unit 25 of the sensing camera 20-2 uploads the constraint conditions COT2 and COT21 to the server device 100.

[0130] [2-9-2. Functional Configuration of Information Processing Device] Next, an example of the functional configuration of the server device 100 (information processing device) according to the proposed method will be described. In the proposed method, the matching process is performed by each sensing camera 20. For this reason, the server device 100 does not have the detection unit 110 and matching unit 120 that are responsible for the matching process in the conventional method, as shown in FIG. 8 .

[0131] (Optimization Unit 130) The optimization unit 130 corrects errors in the map data based on the constraint conditions COT generated by each sensing camera 20. The correction process may be the same as a conventional method. That is, the optimization unit 130 corrects errors in the overall submap sbm constructed by combining the submaps sbm generated by each sensing camera 20. For example, the optimization unit 130 corrects errors in the overall pose graph PG constructed by combining the pose graphs PG generated by each sensing camera 20 based on the constraint conditions COT. In the example of FIG. 8 , the optimization unit 130 corrects errors based on the constraint conditions COT1, COT2, COT12, and COT21, thereby generating a single error-corrected environmental map MP.

[0132] The optimization unit 130 also transmits the corrected overall pose graph PG (the overall pose graph PG reflecting the constraint COT) to each sensing camera 20. In the example of FIG. 8 , the optimization unit 130 transmits not only the portion of the corrected pose graph PG corresponding to sensing camera 20-1 but also the portion corresponding to sensing camera 20-2 to sensing camera 20-1. In this way, the optimization unit 130 causes the SLAM processing unit 23 of sensing camera 20-1 to synchronize the pose graphs PG generated by all of the sensing cameras 20 included in the multi-robot SLAM system 1A. As described above, this processing is performed because, in the proposed method shown in FIG. 8 , sensing camera 20-1 must also use the pose graph PG of sensing camera 20-2 in the matching process.

[0133] 8, the optimization unit 130 transmits to the sensing camera 20-2 not only the portion of the corrected pose graph PG corresponding to the sensing camera 20-2, but also the portion corresponding to the sensing camera 20-1. As a result, the optimization unit 130 causes the SLAM processing unit 23 of the sensing camera 20-2 to synchronize the pose graphs PG generated by all of the sensing cameras 20 included in the multi-robot SLAM system 1A. As described above, this processing is performed because, in the proposed method shown in FIG. 8, the sensing camera 20-2 must also use the pose graph PG of the sensing camera 20-1 in the matching process.

[0134] (Visualization Unit 140) The visualization unit 140 generates a rendering map using the map data that has been error-corrected by the optimization unit 130. Then, the visualization unit 140 may output the environmental map MP generated by rendering to each sensing camera 20.

[0135] [2-10. Operational Procedure] Next, we will explain an example of the operational procedure implemented by the multi-robot SLAM system 1A according to the proposed method explained in Fig. 8. Specifically, we will explain the operational procedure for each scene in the multi-robot SLAM system 1A using Figs. 9 to 12.

[0136] [2-10-1. Upload Procedure of Terminal Device] First, the upload procedure of the sensing camera 20 to the server device 100 will be described using Fig. 9. Fig. 9 is a flowchart showing the upload processing procedure executed by the sensing camera 20 in the proposed method. Note that the upload procedure shown in Fig. 9 is performed individually by each sensing camera 20 using only its own device information.

[0137] The SLAM processing unit 23 determines whether new sensor data SD has been acquired from the sensor unit 21 (step S901). If new sensor data SD has not been acquired from the sensor unit 21 (step S901; No), the SLAM processing unit 23 waits until new sensor data SD can be acquired.

[0138] If the SLAM processing unit 23 has acquired new sensor data SD from the sensor unit 21 (step S901; Yes), it determines the type of the acquired sensor data SD (step S902). For example, if the SLAM processing unit 23 determines in step S902 that the acquired sensor data SD is output data from an IMU or an odometry sensor, it calculates the amount of movement of the device in the short period of time (step S903) and returns to step S901.

[0139] If the SLAM processing unit 23 determines in step S902 that the acquired sensor data SD is camera image data, it matches (superimposes) the sensor data SD (point cloud information of the sensor data SD) on the sub-map sbm being created based on the amount of movement over the small amount of time calculated in step S903 (step S904).

[0140] Furthermore, the SLAM processing unit 23 generates a trajectory node TG based on the sensor data SD, and adds the generated trajectory node TG to the pose graph PG (step S905).

[0141] Next, the SLAM processing unit 23 determines whether the update of the submap sbm being created has been completed (step S906). If the update of the submap sbm has not been completed (step S906; No), the SLAM processing unit 23 returns to step S901.

[0142] When the updating of the submap sbm is completed (step S906; Yes), the SLAM processing unit 23 adds the updated submap sbm to the pose graph PG (step S907).

[0143] Next, the SLAM processing unit 23 uploads the pose graph data and submap data to the server device 100 while retaining the trajectory node data within the SLAM processing unit 23 (step S908).

[0144] [2-10-2. Scan matching detection procedure of terminal device] Next, the scan matching detection procedure by the sensing camera 20 will be described using Fig. 10. Fig. 10 is a flowchart showing the scan matching detection procedure executed by the sensing camera 20 in the proposed method. Note that the scan matching detection procedure shown in Fig. 10 is performed individually by each sensing camera 20.

[0145] First, the detection unit 24 determines whether data generated by the device itself has been added to the pose graph PG of the device itself (step S1001). If the data generated by the device itself has not been added to the pose graph PG of the device itself (step S1001; No), the detection unit 24 waits until the data is added.

[0146] On the other hand, if data generated by the sensing camera 20 itself has been added to the pose graph PG of the sensing camera 20 itself (step S1001; Yes), the detection unit 24 determines the type of the added data (step S1002). For example, if the added data is trajectory node data in step S1002, the detection unit 24 detects submaps sbm that are to be matched with the trajectory node TG generated by the sensing camera 20 itself, from among the submaps sbm previously generated by the sensing camera 20, including the sensing camera 20 itself (step S1003).

[0147] Then, the detection unit 24 lists combinations of the trajectory nodes TG and the submaps sbm to be matched (step S1004), and adds the listed combinations to the queue of the device (step S1005). For example, the detection unit 24 may list combinations of adjacent nodes in step S1004.

[0148] In step S1002, if the added data is submap data, the detection unit 24 detects, from among the trajectory nodes TG previously generated by the device, trajectory nodes TG that are candidates for matching with the submap sbm generated by the device itself (step S1006).

[0149] Then, the detection unit 24 lists combinations of the trajectory nodes TG and the submaps sbm to be matched (step S1007), and adds the listed combinations to the queue of the device (step S1008). For example, in step S1007, the detection unit 24 may list combinations of adjacent nodes.

[0150] [2-10-3. Scan Matching Processing Procedure of Terminal Device] Next, the scan matching processing procedure by the sensing camera 20 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the scan matching processing procedure executed by the sensing camera 20 in the proposed method. Note that the scan matching processing procedure shown in Fig. 11 is performed individually by each sensing camera 20.

[0151] First, the matching unit 25 determines whether an item for scan matching (a list of combinations to be matched) has been added to the queue of its own device (step S1101). If an item for scan matching has not been added to the queue (step S1101; No), the matching unit 25 waits until an item for scan matching is added to the queue.

[0152] On the other hand, when an item for scan matching is added to the queue (step S1101; Yes), the matching unit 25 determines whether or not it has actual data for a submap sbm among the submaps sbm listed as matching targets that was generated by another sensing camera 20 other than the device itself (step S1102).

[0153] If the matching unit 25 does not have actual data of the submap sbm generated by the other sensing camera 20 (step S1102; No), it obtains the actual data of the submap sbm generated by the other sensing camera from the server device 100 (step S1103).

[0154] Then, the matching unit 25 performs scan matching (step S1104). Specifically, the matching unit 25 performs a matching process between the self trajectory and the self submap, and a matching process between the self trajectory and another submap. Note that if the matching unit 25 has actual data of a submap sbm generated by another sensing camera 20 (step S1102; Yes), it may skip step S1103.

[0155] The matching unit 25 generates constraint conditions COT indicating the relative positional relationship between the trajectory node tND and the submap sbm based on the matching result (step S1105), and then uploads the constraint conditions COT to the server device 100 (step S1106).

[0156] 2-10-4. Optimization Processing Procedure of Information Processing Device Next, the optimization processing procedure by the server device 100 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the optimization processing procedure executed by the server device 100 in the proposed method.

[0157] The optimization unit 130 determines whether or not a sufficient amount of constraint conditions COT has been accumulated (step S1201) in response to the constraint conditions COT being uploaded from each sensing camera 20. If the amount of accumulated constraint conditions COT is insufficient (step S1201; No), the optimization unit 130 waits until a sufficient amount of constraint conditions COT has been accumulated.

[0158] On the other hand, if the accumulated amount of constraint conditions COT is sufficient (step S1201; Yes), the optimization unit 130 executes an optimization process to correct the error between the sub-maps sbm uploaded from each sensing camera 20 based on the constraint conditions COT (step S1202).

[0159] Next, the optimization unit 130 reflects the results of the optimization process in the pose graph PG (step S1203). Specifically, the optimization unit 130 transmits to each sensing camera 20 a corrected overall pose graph PG in which errors in the overall pose graph PG constructed by combining the pose graphs PG generated by each sensing camera 20 have been corrected. As a result, each sensing camera 20 reflects the pose graphs PG generated by the other sensing cameras 20 in its own pose graph PG. In other words, each sensing camera 20 synchronizes the pose graph PG generated by itself with the pose graphs PG generated by the other sensing cameras 20.

[0160] The optimization unit 130 then generates a rendering map (environment map) using the map data that has been error-corrected in the optimization process (step S1204). The optimization unit 130 may output the rendering map to each sensing camera 20.

[0161] [3. Variation 1] Next, we will explain a variation of the multi-robot SLAM system 1. We have explained that the multi-robot SLAM system 1A according to the proposed method of the present disclosure employs a configuration in which matching processing is offloaded from the cloud server device 100 to the edge sensing camera 20.

[0162] However, the sensing cameras 20 often have inferior processing capabilities compared to the server device 100, and there may also be differences in processing capabilities between sensing cameras 20 depending on the type. Thus, if the sensing camera 20 has low processing capabilities, for example, it may be unable to complete all matching processes within a certain period of time, resulting in a longer time required to obtain the self-position estimation results. Therefore, in Modification 1 according to the proposed method of the present disclosure, a request unit 26 is newly introduced into each sensing camera 20 as a new component to solve this problem. Therefore, the functional configuration of Modification 1 according to the proposed method of the present disclosure will first be described below.

[0163] 3-1. Detailed System Configuration Example for Modification 1 FIG. 13 is a functional block diagram showing an example of the functional configuration of a multi-robot SLAM system 1 according to Modification 1. Similar to FIG. 8, the multi-robot SLAM system 1 shown in FIG. 13 includes a mobile object 10-1 (robot A) equipped with a sensing camera 20-1, a mobile object 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100. Hereinafter, the multi-robot SLAM system 1 according to Modification 1 of the proposed method will be referred to as a "multi-robot SLAM system 1B."

[0164] Similar to the multi-robot SLAM system 1A, the multi-robot SLAM system 1B according to the first variation of the proposed method also performs the matching process on the sensing camera 20 side. Therefore, the multi-robot SLAM system 1B not only appropriately reduces the processing load on the server device for SLAM calculations, but also produces a new effect. Specifically, the multi-robot SLAM system 1B can complete the matching process within the allowed time even when a sensing camera 20 with a performance that makes it impossible to complete the matching process within the allowed time is installed.

[0165] [3-1-1. Functional Configuration of Terminal Device] First, an example of the functional configuration of the sensing camera 20-1 (terminal device) according to Modification 1 will be described. As shown in FIG. 13, the sensing camera 20-1, like the proposed method, has a sensor unit 21, a preprocessing unit 22, a SLAM processing unit 23, a detection unit 24, and a matching unit 25, but also has a request unit 26. Although not described here, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1. Also, in FIG. 13, dotted arrows indicate a new information processing flow compared to the proposed method of FIG. 8.

[0166] (Request unit 26) The request unit 26 performs process dispatcher processing to allocate (request) matching processing between the device itself and an external device. Specifically, when an unprocessed matching processing occurs among the matching processing for matching relative positional relationships that is predicted to not be calculated within a predetermined time, the request unit 26 requests an external device different from the device itself to perform the unprocessed matching processing.

[0167] For example, when the detection unit 24 registers a combination of matching targets in a queue as a matching query, the request unit 26 extracts unprocessed matching processes that are predicted to be omitted from the matching processes within a predetermined time period from among the matching processes, and then requests the server device 100 to perform the extracted matching processes.

[0168] For example, the request unit 26 can perform the following processing when making a request to the server device 100. For example, as shown in FIG. 13 , the request unit 26 registers a combination (the combination of a trajectory node tNG and a submap sbm) that is the target of matching in an unprocessed matching process in a queue of the server device 100. The request unit 26 also uploads trajectory node data that has not been uploaded to the multi-robot SLAM system 1A to the server device 100. Specifically, the request unit 26 uploads the trajectory node data (sensor data SD corresponding to the trajectory node TG) of the trajectory node TG generated by the SLAM processing unit 23 to the server device 100.

[0169] In this way, in the multiple-robot SLAM system 1B, part of the matching process that would normally be performed by the sensing camera 20 is performed by the server device 100. For this reason, in the multiple-robot SLAM system 1B, the matching unit 120 that was previously excluded from the server device 100 is reinstated.

[0170] In addition, if there is no unprocessed matching process that is predicted to be unable to be calculated within the specified time among the matching processes that match relative positional relationships, the request unit 26 inputs the original combination (combination of trajectory node tNG and submap sbm) that is to be matched in the matching process into the matching unit 25, thereby performing sequential scan matching.

[0171] [3-1-2. Functional Configuration of Information Processing Device] Next, an example of the functional configuration of the server device 100 (information processing device) according to Modification 1 of the proposed method will be described. As described above, in Modification 1, part of the matching process that should be performed by the sensing camera 20 is performed by the server device 100. Therefore, compared to the example in Fig. 8, the server device 100 has a matching unit 120. The server device 100 also has a queue.

[0172] (Matching Unit 120) When the matching unit 120 receives a request for matching processing from one of the sensing cameras 20 (request unit 26), it executes the unprocessed matching processing corresponding to the request. Specifically, the matching unit 120 extracts trajectory node data and submaps sbm to be used for the unprocessed matching processing from the storage unit of its own device, based on the combination of matching targets registered in the queue of its own device by the requesting sensing camera (request unit 26). The matching unit 120 then matches the relative positional relationship between the trajectory nodes tND included in the extracted trajectory node data and the submap nodes sND of the extracted submap sbm, and generates a constraint condition COT that indicates this relative positional relationship. In other words, the matching unit 120 performs the matching processing that the requesting sensing camera 20 was unable to complete.

[0173] [3-2. Operational Procedure] An example of the operational procedure implemented by the multi-robot SLAM system 1B according to Variation 1 described in Fig. 13 will now be described. Specifically, the operational procedure for each scene in the multi-robot SLAM system 1B will be described using Figs. 14 to 16.

[0174] 14, a preprocessing procedure performed by the sensing camera 20 when requesting matching processing from the server device 100 will be described. FIG. 14 is a flowchart showing the preprocessing procedure related to the request for matching processing. Note that the preprocessing procedure shown in FIG. 14 is performed individually by each sensing camera 20.

[0175] First, the request unit 26 determines whether an item for scan matching (a list of combinations to be matched) has been added to the queue of the device itself (step S1401). If an item for scan matching has not been added to the queue (step S1401; No), the request unit 26 waits until an item for scan matching is added to the queue.

[0176] On the other hand, when an item for scan matching is added to the queue (step S1401; Yes), the request unit 26 determines whether any matching processing unit in the matching processing unit of the own device (matching unit 25) has become available for processing (step S1402). If there is no matching processing unit that has become available for processing (step S1402; No), the request unit 26 waits until any matching processing unit becomes available for processing.

[0177] On the other hand, if any of the matching processing units has an available processing time (step S1402; Yes), the request unit 26 assigns the matching task to the matching unit 25 that has an available processing time (step S1403).

[0178] Then, the request unit 26 measures the matching processing speed for each matching processing unit to which the matching task is assigned (step S1404).

[0179] [3-2-2. Request Processing Procedure] Next, a request processing procedure in which the sensing camera 20 requests the server device 100 to perform matching processing will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the request processing procedure for requesting matching processing. Note that the request processing procedure shown in Fig. 15 is performed individually by each sensing camera 20. Furthermore, the request processing procedure may be performed in parallel with the pre-processing procedure in Fig. 14, or may be performed sequentially with respect to the pre-processing procedure in Fig. 14.

[0180] The request unit 26 predicts the processing time required to complete matching for each matching processing unit assigned the matching task based on the queue size of the queue of the own device and the matching processing speed obtained as the measurement result in step S1404 (step S1501). For example, the request unit 26 may perform this processing at regular intervals.

[0181] The request unit 26 determines whether the processing time is within the allowable range (step S1502). If the processing time is within the allowable range (step S1502; Yes), the request unit 26 returns the process to step S1501.

[0182] On the other hand, if the processing time is not within the allowable range (step S1502; No), the request unit 26 extracts a matching process to be requested from the server device 100 (step S1503). For example, the request unit 26 extracts, based on the processing time, a matching process that is predicted to fall outside the allowable range from among the matching processes registered in the queue of the server device 100 and executed by a matching processing unit to which a matching task has been assigned.

[0183] Next, the request unit 26 uploads the extracted trajectory nodes tND necessary for the matching process to the server device 100 (step S1504). The request unit 26 also deletes the matching task corresponding to the extracted matching process from the queue of the server device 100 (step S1505).

[0184] 16 is a flowchart showing the procedure of the matching process executed by the server device 100 in response to a request for the matching process.

[0185] First, the matching unit 120 determines whether an item for scan matching (a list of combinations to be matched) has been added to the queue of its own device (step S1601). For example, the matching unit 120 determines whether a combination to be matched in an unprocessed matching process that has been determined to be difficult to process by the sensing camera 20 because it has not been processed within the allowable range has been added to the queue. In other words, the matching unit 120 determines whether a request for an unprocessed matching process has been received from the sensing camera 20.

[0186] The matching unit 120 executes scan matching (step S1602). Specifically, the matching unit 120 extracts a trajectory node tND and a submap sbm corresponding to a combination of matching targets added to the queue of the matching unit 120's own device, and matches the relative positional relationship between the extracted trajectory node tND and the submap sbm.

[0187] Then, the matching unit 120 generates constraint conditions COT indicating the relative positional relationship between the trajectory node tND and the submap sbm based on the matching result (step S1603), and saves the generated constraint conditions COT (step S1604).

[0188] Although not shown in Figure 16, when the constraint conditions COT are saved by the matching unit 120, the optimization unit 130 performs an optimization process to correct the error between the constraint conditions COT generated by its own device in the matching process in response to a request from the sensing camera 20 and the sub-maps sbm uploaded from each sensing camera 20, based on the COT uploaded from the sensing camera 20.

[0189] The optimization unit 130 also reflects the results of the optimization process in the pose graph PG (step S1203). Specifically, the optimization unit 130 transmits to each sensing camera 20 the corrected overall pose graph PG, which is constructed by combining the pose graphs PG generated by each sensing camera 20 and in which any errors in the overall pose graph PG have been corrected.

[0190] [4. Modification 2] The matching process to be performed by one sensing camera 20 may be allocated to multiple computational resources on the edge side. In this way, the matching process to be performed by one sensing camera 20 may be distributed on the edge side, and if an unprocessed matching process occurs that cannot be handled within the allowable range, this unprocessed matching process may be requested to the server device 100. In other words, in Modification 2 according to the proposed method of the present disclosure, one sensing camera 20 may request another sensing camera 20 to perform part of the matching process that it should perform itself, and if an unprocessed matching process occurs that cannot be handled within the allowable range, this unprocessed matching process is requested to the server device 100.

[0191] 4-1. Detailed System Configuration Example for Modification 2 FIG. 17 is a functional block diagram showing an example of the functional configuration of a multi-robot SLAM system 1 according to Modification 2. Similar to FIG. 13, the multi-robot SLAM system 1 shown in FIG. 17 includes a mobile object 10-1 (robot A) equipped with a sensing camera 20-1, a mobile object 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100. Hereinafter, the multi-robot SLAM system 1 according to Modification 2 of the proposed method will be referred to as a "multi-robot SLAM system 1C."

[0192] In addition, in the example of Figure 17, the sensing camera 20-2 is treated as a sub-machine of the sensing camera 20-1, and is configured to perform part of the matching processing that should be performed by the sensing camera 20-1 in response to a request from the sensing camera 20-1.

[0193] According to the example of FIG. 17, the sensing camera 20-2 has a matching unit 25-2 that performs matching processing on behalf of the sensing camera 20-1 in response to a request from the sensing camera 20-1.

[0194] [4-1-1. Functional Configuration of Terminal Device] First, an example of the functional configuration of the sensing camera 20-1 according to Modification 2 will be described. As shown in FIG. 17 , the sensing camera 20-1, like Modification 1, has a sensor unit 21, a preprocessing unit 22, a SLAM processing unit 23, a detection unit 24, a matching unit 25, and a request unit 26. Although not described further, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1. Also, in FIG. 17 , dotted arrows indicate new information processing flows compared to Modification 1 in FIG. 16 .

[0195] When an unprocessed matching process occurs among the matching processes for matching relative positional relationships that is predicted to be unable to be completed within a predetermined time, the request unit 26 requests the server device 100 to complete the unprocessed matching process. For example, the request unit 26 aims to complete all matching processes within an allowable range by requesting the sensing camera 20-2, which is linked as a sub-device, to perform part of the matching process that should be performed by the request unit 26. However, when an unprocessed matching process occurs that cannot be completed within the allowable range despite the request for part of the matching process being made to the sensing camera 20-2, the request unit 26 requests the server device 100 to complete the unprocessed matching process.

[0196] In this way, in the multi-robot SLAM system 1C, of ​​the matching processes that would normally be performed by multiple edge-side computing resources (sensing cameras 20-1 and 20-2 in the example of FIG. 17 ), the remaining unprocessed matching processes that cannot be handled within the tolerance limits of the multiple computing resources are performed by the server device 100. For this reason, in the multi-robot SLAM system 1C as well, the matching unit 120 that was previously excluded from the server device 100 is reinstated.

[0197] The matching unit 25-2 of the sensing camera 20-2 performs scan matching for each combination (combination of the trajectory node tNG and the submap sbm) that is the target of matching in the matching process requested by the request unit 26 of the sensing camera 20-1. The matching unit 25-2 also uploads the constraint condition COT generated based on the result of the matching process to the server device 100.

[0198] [4-1-2. Functional Configuration of Information Processing Device] Next, an example of the functional configuration of the server device 100 according to Modification 2 of the proposed method will be described. As described above, in Modification 2, part of the matching process that should be performed between multiple edge-side computing resources is performed by the server device 100. Therefore, compared to the example in FIG. 8, the server device 100 includes a matching unit 120. The server device 100 also includes a queue.

[0199] (Matching Unit 120) The matching unit 120 performs unprocessed matching processes that should be performed among multiple edge-side computational resources but that cannot be handled within the tolerance of the multiple computational resources. For example, the matching unit 120 extracts trajectory node data and submaps sbm to be used for the unprocessed matching processes from the storage unit of its own device based on a combination of matching targets registered in the queue of its own device by the requesting sensing camera (request unit 26). The matching unit 120 then matches the relative positional relationship between the trajectory nodes tND included in the extracted trajectory node data and the submap nodes sND of the extracted submap sbm, and generates a constraint condition COT that indicates this relative positional relationship.

[0200] [4-2. Operation Procedure] An example of an operation procedure implemented by the multi-robot SLAM system 1C according to Modification 2 described in FIG. 17 will now be described. Specifically, a request processing procedure in which the sensing camera 20 requests the server device 100 to perform matching processing will be described. FIG. 18 is a flowchart showing the request processing procedure for requesting matching processing. Note that the request processing procedure shown in FIG. 18 is performed individually by each sensing camera 20. The request processing procedure shown in FIG. 18 is a modification of the request processing procedure described in FIG. 15 in accordance with the multi-robot SLAM system 1C according to Modification 2.

[0201] The request unit 26 predicts the processing time required to complete matching for each matching processing unit assigned with the matching task based on the queue size of the queue of the own device and the matching processing speed obtained as the measurement result in step S1404 (step S1801). For example, the request unit 26 may perform this processing at regular intervals.

[0202] The request unit 26 determines whether the processing time is within an allowable range (step S1802). If the processing time is within the allowable range (step S1802; Yes), the request unit 26 determines whether an entity executing the matching process (matching processing unit) exists on the edge side other than the own device (step S1803). For example, if a configuration is adopted in which the matching process that should originally be performed by the own device is distributed among multiple computational resources including the own device, the request unit 26 can determine that an entity executing the matching process exists on the edge side other than the own device.

[0203] If the request unit 26 determines that there are other execution entities of the matching process besides its own device (step S1803; Yes), it excludes the other execution entities from the execution entities that perform the matching process (step S1804). In other words, if there are other execution entities of the matching process besides its own device on the edge side but its own device alone can complete all of the matching process within an allowable range, the request unit 26 does not request the matching process from other execution entities (matching processing units) that are on the edge side.

[0204] If the request unit 26 determines that no entity other than its own device is executing the matching process (step S1803; No), the request unit 26 returns the process to step S1801.

[0205] If the processing time is not within the allowable range (step S1802; No), the request unit 26 determines whether or not there is an entity executing the matching process on the edge side other than the device itself (step S1805).

[0206] If the request unit 26 determines that there are no other execution entities performing the matching process other than its own device (step S1805; No), it excludes other execution entities present on the edge side from the execution entities performing the matching process (step S1806).

[0207] When the request unit 26 determines that there is another entity executing the matching process other than its own device (step S1805; Yes), it causes the matching processes that can be processed within the allowable range to be performed among multiple computational resources including its own device, while extracting matching processes to be requested to the server device 100, i.e., unprocessed matching processes that cannot be processed within the allowable range by the multiple computational resources (step S1807). For example, the request unit 26 extracts matching processes that are registered in the queue of its own device and are predicted to fall outside the allowable range from the matching processes of the matching unit 25 that have been assigned a matching task, based on the processing time.

[0208] Next, the request unit 26 uploads the extracted trajectory nodes tND necessary for the matching process to the server device 100 (step S1808). The request unit 26 also deletes the matching task corresponding to the extracted matching process from the queue of the server device 100 (step S1809).

[0209] [5. Hardware Configuration] An example hardware configuration of a computer corresponding to the device according to each of the above-described embodiments (e.g., the sensing camera 20, the server device 100) will be described using Fig. 19. Fig. 19 is a block diagram showing an example hardware configuration of a computer corresponding to the device according to the embodiments of the present disclosure. Note that Fig. 19 shows an example of the hardware configuration of a computer corresponding to the device according to each embodiment, and is not necessarily limited to the configuration shown in Fig. 19.

[0210] 19 , a computer 1000 includes a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0211] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0212] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0213] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100, data used by such programs, etc. Specifically, HDD 1400 records program data 1450. Program data 1450 is an example of an information processing program for realizing an information processing method according to an embodiment of the present disclosure, and data used by such information processing program.

[0214] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0215] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0216] For example, when the computer 1000 functions as the sensing camera 20, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200, thereby realizing various processing functions executed by the processes shown in Fig. 8 etc. In other words, the CPU 1100, the RAM 1200, etc., work together with software (an information processing program loaded onto the RAM 1200) to realize an information processing method by a terminal device according to an embodiment of the present disclosure.

[0217] Furthermore, when the computer 1000 functions as the server device 100, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200, thereby realizing various processing functions executed by the processes shown in Fig. 8 etc. In other words, the CPU 1100, the RAM 1200, etc., work together with software (an information processing program loaded onto the RAM 1200) to realize an information processing method by an information processing device according to an embodiment of the present disclosure.

[0218] [6. Summary] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of the embodiments and modifications may be combined as appropriate.

[0219] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0220] The present disclosure may also be configured as follows: (1) An information processing method in which an information processing device provides one or more terminal devices with other partial map information generated by a terminal device different from the terminal device, and the terminal device acquires, from the terminal devices, information on a matching result obtained by matching a relative positional relationship between the trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device and the other partial map information, and the partial map information, and performs an optimization process to correct an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices, based on the information on the matching result. (2) The information processing method described in (1), in which the information processing device acquires, as the matching result information, a constraint condition indicating the relative positional relationship generated by matching the relative positional relationship. (3) The information processing method described in (2), in which the information processing device executes a process to correct an error in the overall trajectory position information based on the constraint condition. (4) The information processing method according to (2), wherein the information processing device executes a process of correcting an error in overall map information constructed by combining the partial map information generated by each of the terminal devices based on the constraint condition, and outputting the corrected overall map information to each of the terminal devices. (5) The information processing method according to (3), wherein the information processing device executes a process of synchronizing, for each of the terminal devices, the trajectory position information generated by that terminal device with the corrected overall trajectory position information.(6) An information processing method in which a terminal device executes the following process: acquires other partial map information generated by another terminal device different from the terminal device itself from an information processing device on the cloud side; matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves within an area included in the partial map information generated by the terminal device itself, point cloud information indicating a position of an object within the partial map information, and the other partial map information; and uploads information on the matching result of the relative positional relationship between the trajectory position information and the other partial map information and the partial map information to the information processing device that generates an environmental map corresponding to the partial map. (7) The information processing method according to (6), wherein the terminal device performs a process of: matching a relative positional relationship between the trajectory node and the partial map node based on a trajectory node that is a node including the point cloud information and indicates a representative point of the positions on the trajectory included in the trajectory position information, and a partial map node that indicates a reference point of the other partial map information, thereby generating a constraint condition that indicates the relative positional relationship as information of the matching result, and uploading the constraint condition to the information processing device. (8) The information processing method according to (7), wherein the terminal device performs a process of: acquiring, from the information processing device, overall trajectory position information constructed by combining the trajectory position information generated by its own device and the trajectory position information generated by the other terminal device, the overall trajectory position information being corrected by the information processing device based on the constraint condition generated by its own device and the constraint condition generated by the other terminal device, and estimating its own position in overall map information constructed by combining the partial map information, based on the overall trajectory position information. (9) The information processing method according to (6), wherein the terminal device executes a process in which, when an unprocessed matching process occurs among the matching processes for matching the relative positional relationship and the calculation cannot be completed within a predetermined time, the terminal device requests the information processing device to perform the unprocessed matching process.(10) The information processing method according to (9), wherein the terminal device requests a part of the matching process for matching the relative positional relationships to another terminal device different from the terminal device itself, and when an unprocessed matching process for matching the relative positional relationships occurs among the matching processes for matching the relative positional relationships because calculation cannot be completed within the predetermined time even when part of the matching process has been requested to the other terminal device, requests the unprocessed matching process to the information processing device. (11) An information processing device comprising: a providing unit that provides other partial map information generated by the other terminal device different from the terminal device to one or more terminal devices; an acquiring unit that acquires, from the terminal device, trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device itself, and information on a matching result obtained by matching relative positional relationships between the trajectory position information and the other partial map information, and the partial map information; and an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices, based on the information on the matching result. (12) A terminal device comprising: an acquisition unit that acquires other partial map information generated by another terminal device different from the terminal device itself from an information processing device on the cloud side; a matching unit that matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device itself moves through an area included in the partial map information generated by the terminal device itself, point cloud information indicating a position of an object within the partial map information, and the other partial map information; and a processing unit that uploads information on the matching result of the relative positional relationship between the trajectory position information and the other partial map information and the partial map information to an information processing device that generates an environmental map corresponding to the partial map.(13) A system including an information processing device and a plurality of terminal devices, wherein the information processing device comprises: a providing unit that provides the terminal device with other partial map information generated by a terminal device different from the terminal device; an acquiring unit that acquires, from the terminal device, information on a matching result obtained by matching a relative positional relationship between trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device and the other partial map information, and the partial map information; and an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices, based on the information on the matching result; and wherein the terminal device comprises: an acquiring unit that acquires, from the information processing device on a cloud side, the other partial map information generated by the other terminal device different from the terminal device; A system comprising: a matching unit that matches a relative positional relationship between the trajectory position information and other partial map information based on trajectory position information indicating a position on a trajectory along which the device moves within an area included in partial map information generated by the device, point cloud information indicating a position of an object within the partial map information, and the other partial map information; and a processing unit that uploads information on the matching result of the relative positional relationship between the trajectory position information and the other partial map information, and the partial map information, to the information processing device that generates an environmental map corresponding to the partial map.

[0221] 20 Sensing camera 21 Sensor unit 22 Preprocessing unit 23 SLAM processing unit 24 Detection unit 25 Matching unit 26 Request unit 100 Server device 110 Detection unit 120 Matching unit 130 Optimization unit 140 Visualization unit

Claims

1. An information processing method in which an information processing device provides one or more terminal devices with other partial map information generated by a terminal device different from the terminal device, and the terminal device acquires from the terminal device information and other partial map information, information on a matching result obtained by matching a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device and the other partial map information, and performs an optimization process to correct an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices based on the matching result information.

2. The information processing method according to claim 1, wherein the information processing device executes a process of acquiring, as information of the matching result, a constraint condition that indicates the relative positional relationship and that is generated by matching the relative positional relationship.

3. The information processing method according to claim 2, wherein the information processing device executes a process of correcting an error in the overall trajectory position information based on the constraint condition.

4. The information processing method according to claim 2, wherein the information processing device executes a process of: correcting errors in overall map information constructed by combining the partial map information generated by each of the terminal devices based on the constraint conditions; and outputting the corrected overall map information to each of the terminal devices.

5. The information processing method according to claim 3, wherein the information processing device executes a process of synchronizing, for each of the terminal devices, the trajectory position information generated by that terminal device with the corrected overall trajectory position information.

6. An information processing method in which a terminal device executes the following process: acquires other partial map information generated by another terminal device different from the terminal device itself from an information processing device on the cloud side; matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves within an area included in the partial map information generated by the terminal device itself, point cloud information indicating the position of an object within the partial map information, and the other partial map information; and uploads information on the matching result of the relative positional relationship between the trajectory position information and the other partial map information, and the partial map information to the information processing device which generates an environmental map corresponding to the partial map.

7. The information processing method according to claim 6, wherein the terminal device executes a process of matching the relative positional relationship between a trajectory node, which is a node including the point cloud information and indicates a representative point among the positions on the trajectory included in the trajectory position information, and a partial map node, which indicates a reference point of the other partial map information, to generate a constraint condition indicating the relative positional relationship as information of the matching result, and uploading the constraint condition to the information processing device.

8. The information processing method according to claim 7, wherein the terminal device executes a process of acquiring from the information processing device overall trajectory position information constructed by combining the trajectory position information generated by the terminal device itself and the trajectory position information generated by the other terminal device, the overall trajectory position information being corrected by the information processing device based on the constraint condition generated by the terminal device itself and the constraint condition generated by the other terminal device, and estimating the terminal device's own position in overall map information constructed by combining the partial map information based on the overall trajectory position information.

9. The information processing method according to claim 6, wherein the terminal device executes a process in which, when an unprocessed matching process occurs among the matching processes for matching the relative positional relationship and calculation cannot be completed within a predetermined time, the terminal device requests the unprocessed matching process to be performed by the information processing device.

10. The information processing method according to claim 9, wherein the terminal device requests a part of the matching process for matching the relative positional relationship to another terminal device other than the terminal device itself, and even when a part of the matching process has been requested to the other terminal device, if an unprocessed matching process for matching the relative positional relationship occurs that cannot be calculated within the specified time, the terminal device requests the unprocessed matching process to be performed by the information processing device.

11. An information processing device comprising: a providing unit that provides one or more terminal devices with other partial map information generated by other terminal devices different from the terminal devices; an acquiring unit that acquires from the terminal devices trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device, and information on a matching result in which a relative positional relationship is matched between the trajectory position information and the other partial map information based on the other partial map information; and an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices based on the information on the matching result.

12. A terminal device comprising: an acquisition unit that acquires other partial map information generated by another terminal device different from the terminal device itself from an information processing device on the cloud side; a matching unit that matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves within an area included in the partial map information generated by the terminal device itself, point cloud information indicating the position of an object within the partial map information, and the other partial map information; and a processing unit that uploads information on the matching result of the relative positional relationship between the trajectory position information and the other partial map information and the partial map information to an information processing device that generates an environmental map corresponding to the partial map.

13. A system including an information processing device and a plurality of terminal devices, wherein the information processing device comprises: a providing unit that provides the terminal device with other partial map information generated by a terminal device different from the terminal device; an acquiring unit that acquires from the terminal device information on a matching result obtained by matching a relative positional relationship between trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device and the other partial map information, and the partial map information; and an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices, based on the information on the matching result; and wherein the terminal device comprises: an acquiring unit that acquires the other partial map information generated by the terminal device different from the terminal device from the information processing device on the cloud side; A system comprising: a matching unit that matches a relative positional relationship between the trajectory position information and other partial map information based on trajectory position information indicating a position on a trajectory along which a device moves within an area included in partial map information generated by the device, point cloud information indicating a position of an object within the partial map information, and the other partial map information; and a processing unit that uploads information on a matching result of the relative positional relationship between the trajectory position information and the other partial map information, and the partial map information to the information processing device that generates an environmental map corresponding to the partial map.

Images