Information processing device, control method, and program

Abstract

To be able to detect mis-synchronization between sensors even when the same object is not observed at the same time.SOLUTION: An information processing device includes: sensor parameter acquisition means for acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, and information on relative positions and relative orientations between the sensors; observation information acquisition means for acquiring a group of observation information acquired at a plurality of times by each of the sensors; measurement point information acquiring means for acquiring at least one of a set of velocity and position, and a set of angular velocity and attitude of each of the sensors at the plurality of times, as the measurement point information; and observation time correction amount calculation means for calculating a relative relationship between the sensors at the plurality of times based on the sensor characteristics, the relative positions and orientations between the sensors, the observation information group, and the measurement point information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method, and a program. [Background technology]

[0002] SLAM technology, which uses LiDAR and cameras as sensors, is known as a technology used to control automated guided vehicles (AGVs) in factories and logistics warehouses, and to estimate the position and orientation of HMDs used in VR, etc. AGV is an abbreviation for Automated Guided Vehicle. VR is an abbreviation for Virtual Reality. HMD is an abbreviation for Head Mounted Display. LiDAR is an abbreviation for Laser Imaging Detection and Ranging. SLAM is an abbreviation for Simultaneous Localization and Mapping.

[0003] Also, a method is known in which multiple sensors are used simultaneously to expand the observation range of the surrounding environment and improve the accuracy of position and orientation estimation. In these methods, it is important to synchronize multiple sensors with high accuracy in order to improve the accuracy of position and orientation estimation. For example, Patent Document 1 discloses a technology that detects the same object in an overlapping field of view from images captured almost simultaneously by multiple cameras, detects a synchronization error between the cameras from the detection position, and executes a synchronization process when the detection is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-205498 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method described in Patent Document 1 is based on the premise that the fields of view of multiple cameras overlap, and unless the same object is photographed at the same time, synchronization deviation between the cameras cannot be detected, and therefore synchronization processing cannot be performed.

[0006] An object of the present invention is to make it possible to detect mis-synchronization between sensors even when the same object is not observed at the same time. [Means for solving the problem]

[0007] An information processing device according to one embodiment of the present invention is characterized in that it comprises a sensor parameter acquisition means for acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, as well as relative position and relative attitude information between the sensors, an observation information acquisition means for acquiring a group of observation information acquired by each of the sensors at a plurality of times, a measurement point information acquisition means for acquiring at least one of a pair of velocity and position, and a pair of angular velocity and attitude, of each of the sensors at the multiple times as measurement point information, and an observation time correction calculation means for calculating the relative relationship of each of the sensors at the multiple times based on the sensor characteristics, the relative position and attitude between the sensors, the group of observation information, and the measurement point information. Effect of the Invention

[0008] According to the present invention, it is possible to detect mis-synchronization between sensors even when the same object is not observed at the same time. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating a usage scene of an information processing device according to a first embodiment of the present invention. [Diagram 3] FIG. 4 is a diagram illustrating the influence of synchronization loss in the first embodiment of the present invention. [Figure 4]FIG. 4 is a diagram illustrating a method for correcting a synchronization error in the first embodiment of the present invention. [Diagram 5] 1 is a block diagram showing a functional configuration of an information processing device according to a first embodiment of the present invention. [Figure 6] FIG. 2 is a flowchart showing the flow of an information processing method according to the first embodiment of the present invention. [Figure 7] FIG. 4 is a flowchart showing details of the flow of observation time correction processing in the first embodiment of the present invention. [Figure 8] FIG. 11 is a diagram illustrating the influence of synchronization loss in the second embodiment of the present invention. [Figure 9] FIG. 11 is a diagram illustrating a method for correcting a synchronization error in the second embodiment of the present invention. [Figure 10] FIG. 11 is a diagram showing a configuration of a mobile system according to a third embodiment of the present invention. [Figure 11] FIG. 11 is a block diagram showing a functional configuration of an information processing device according to a third embodiment of the present invention. [Figure 12] FIG. 11 is a flowchart showing the flow of an information processing method according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. Note that the following embodiments do not limit the scope of the present invention, and all of the combinations of features described in the following embodiments are not necessarily essential to constitute the present invention.

[0011] [First embodiment] A first embodiment of the present invention will be described in detail below with reference to the drawings. In the first embodiment, an example will be described in which a shooting synchronization error between cameras is estimated for map data created in advance using a moving body on which multiple cameras are arranged, and the map data is corrected.

[0012] (Configuration of information processing device) FIG. 1 is a block diagram showing the configuration of an information processing device according to a first embodiment of the present invention. The information processing device 101 has the functions of a general embedded PC device. The information processing device 101 has a CPU 111, a ROM 112, a RAM 113, a storage unit 114 such as an HDD or SSD, a communication unit 115, and a system bus 116. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. HDD is an abbreviation for Hard Disk Drive. SSD is an abbreviation for Solid State Drive. In addition, the information processing device 101 can communicate with a moving object 102 having a plurality of sensors (sensor groups) arranged thereon via the communication unit 115.

[0013] The CPU 111 uses the RAM 113 as a work memory, executes an operating system (OS) and various programs stored in the ROM 112 and the storage unit 114, etc., and controls each unit via a system bus 116. For example, the programs executed by the CPU 111 include programs for executing each process described below.

[0014] (Mobile) In this embodiment, the moving body 102 creates map data. The moving body 102 is an unmanned vehicle that can be remotely controlled, and is equipped with three stereo cameras that are attached facing the front, left, and right as a sensor group. The moving body 102 is also equipped with an inertial sensor that can measure the acceleration and angular velocity of the moving body 102. Each sensor is fixed on the moving body 102, and the logical center position of the moving body 102 (for example, a predetermined position such as the center of gravity of the moving body 102) and the relative positions and attitudes between each sensor are known.

[0015] In this embodiment, each camera equipped on the moving body 102 continuously captures images at the same cycle as sensor data. The moving body 102 has a front camera disposed in front of the moving body 102. The moving body 102 has a right camera disposed on the right side of the moving body 102. The moving body 102 has a left camera disposed on the left side of the moving body 102. Here, the front camera and the right camera are precisely synchronized and always capture images at the same time, but the left camera always captures images at a time delayed by a synchronization deviation dt from the front camera and the right camera.

[0016] Here, it is assumed that the synchronization deviation amount dt between the cameras cannot be directly acquired. It is also assumed that the fact that the front camera and the right camera are accurately synchronized is also unknown information. One of the objectives of this embodiment is to identify all of these unknown synchronization deviation amounts of each camera. In the above, the front camera and the right camera are accurately synchronized for the sake of simplicity of explanation, and in reality, the right camera and the left camera have individual synchronization deviation amounts with respect to the front camera. In other words, one of the objectives of this embodiment is to estimate both the synchronization deviation amount between the front camera and the left camera and the synchronization deviation amount between the front camera and the right camera.

[0017] (Usage scenario) FIG. 2 is a diagram illustrating a usage scene of the information processing device according to the first embodiment of the present invention. While moving in an environment 201, a moving body 102 acquires images of the surroundings by a group of sensors arranged on the moving body 102, measures the position and orientation of the moving body 102, and creates map data, which will be described later. The created map data is stored in the storage unit 114 of the information processing device 101 through the communication unit 115. The object 202 is a wall surface or an installation having visual characteristics. The visual characteristics on the object 202 are used as feature points in the environment of the map data. A user 203 is a user who operates the information processing device 101 and the moving body 102.

[0018] (Map data) The map data that is the subject of the synchronization deviation estimation and correction process in this embodiment is map data generated by a known Visual SLAM technique such as ORB-SLAM. This map data holds, as a group of observation information, a number of feature points in the environment and a group of images captured by each camera at a number of measurement points in the environment.

[0019] Feature points in an environment are points that have visual information, such as vertices of stationary objects or patterns. Each feature point has three-dimensional position information (X, Y, Z) that is three-dimensional position information in the environment. Here, the X-axis and Y-axis are axes that are parallel to the horizontal direction of the environment. The X-axis and Y-axis are axes that are perpendicular to each other. The Z-axis is an axis that is perpendicular to both the X-axis and the Y-axis, and has a value that increases in the vertical upward direction. Each feature point in the environment is associated with features on the sensor data of one or more measurement points. In this embodiment, the features on the sensor data are feature points that have two-dimensional coordinates on the image.

[0020] The map data also holds, as measurement point information, the position, orientation, speed, and angular velocity values ​​of the moving body 102 at each measurement point. Here, the position and orientation values ​​are values ​​calculated by Visual SLAM at the same time as the map data is created. Also, the speed and angular velocity values ​​are values ​​calculated from the output values ​​of the inertial sensor.

[0021] The map data also holds, as sensor parameters, the characteristics of each sensor used in the observation, and information on the relative positions and orientations between the sensors. In this embodiment, the characteristics of each sensor include the horizontal and vertical angles of view and the number of pixels of each camera.

[0022] The map data in this embodiment is generated based on a group of image data captured by the camera 104 while the user operates (e.g., remotely controls) the mobile object 102 and moves it through the environment. For example, the well-known ORB-SLAM technology can be used to generate the map data using the camera images as input.

[0023] (Effects of synchronization deviation and compensation principle) The influence of a synchronization error between sensors on map data and position and orientation measurement will be described with reference to the drawings. Fig. 3 is a diagram showing the influence of a synchronization error in the first embodiment of the present invention, and is a diagram explaining the influence of a synchronization error between sensors on map data.

[0024] Reference numeral 301 indicates a state in which the moving object 102 moves in the environment from the measurement point P1 side to the measurement point P3 side by way of the measurement point P2 while making a right turn. The measurement points P1, P2, and P3 are measurement points corresponding to multiple times while the moving object 102 is moving.

[0025] 302 shows the positions of three objects in the environment observed by each camera of the moving body 102. In FIG. 3, the positions of the objects observed by the front camera 303 and the right camera 304 when the moving body 102 moves from the measurement point P1 to the measurement point P3 are shown by solid lines. Each of the front camera 303 and the right camera 304 is a stereo camera as a sensor group arranged on the moving body 102. Also, in FIG. 3, the positions of the objects observed by the left camera 305 when the moving body 102 moves from the measurement point P1 to the measurement point P3 are shown by dashed lines. It is assumed that the solid lines show the correct positions of the objects. The left camera 305 captures images at a time delayed by the synchronization offset amount dt with respect to the front camera 303 and the right camera 304. Therefore, the object observed by the left camera 305 is observed as if it is in a position shifted in the opposite direction to the traveling direction or turning direction of the moving body 102.

[0026] As a result, on the map data generated by Visual SLAM, feature points in the environment originating from each object at the position indicated by 302 are positioned between the solid line and the dashed line. In addition, in the position and orientation measurement, the position and orientation of the moving object 102 at each of the measurement points P1 to P3 are calculated as positions with an error toward the traveling direction side from the correct value.

[0027] 4 is a diagram showing a synchronization deviation correction method in the first embodiment of the present invention, and shows a state in which the shooting position and orientation of the left camera 305 at each measurement point are corrected taking into account the synchronization deviation amount dt. The camera 401 shown by the dashed line shows the left camera 305 at a shooting position and orientation that does not take into account the synchronization deviation at each measurement point. The camera 402 shown by the solid line shows the left camera 305 at a shooting position and orientation corrected taking into account the synchronization deviation at each measurement point.

[0028] Here, the correction amount of the shooting position and orientation of the left camera 305 is calculated from the velocity and angular velocity of the moving body 102 at each measurement point, the relative position and relative orientation of the left camera 305 as viewed from the logical center position of the moving body 102, and the synchronization deviation amount dt. More specifically, first, the position is corrected by translating the original position and orientation of the left camera 305 by an amount obtained by multiplying the velocity and synchronization deviation amount dt at the measurement point. Furthermore, the position and orientation of the left camera are corrected to a position and orientation rotated by an angle obtained by multiplying the angular velocity and synchronization deviation amount dt at the measurement point around the logical center position of the moving body 102.

[0029] By correcting the camera's shooting position and orientation in this way, the positions of feature points in the environment originating from each object at the position indicated by 302 match between the left camera 305, the front camera 303, and the right camera 304, making it possible to obtain correct map data. Also, when measuring position and orientation using map data, it is possible to calculate more correct position and orientation by correcting the camera's shooting position and orientation according to the amount of synchronization deviation.

[0030] (Functional configuration of information processing device) The functional configuration of the information processing device 101 according to this embodiment will be described below. The processing of each unit shown below is implemented as software by loading a program from the ROM 112 or the like onto the RAM 113 and then executing the program by the CPU 111. Fig. 5 is a block diagram showing the functional configuration of the information processing device 101 according to the first embodiment of the present invention. The information processing device 101 has a sensor parameter acquisition unit 501, an observation information acquisition unit 502, a measurement point information acquisition unit 503, and an observation time correction amount calculation unit 504.

[0031] The sensor parameter acquisition unit 501 acquires, from map data previously stored in the memory unit 114, the characteristics of each sensor in a sensor group consisting of multiple sensors arranged on the moving body 102, as well as relative position and relative attitude information between each sensor.

[0032] The observation information acquisition unit 502 acquires a plurality of feature points in the environment and a group of sensor data acquired by each sensor at a plurality of measurement points in the environment (i.e., at a plurality of times) from map data previously stored in the storage unit 114. It also acquires the correspondence between the feature points in the environment and the feature points on the sensor data group (e.g., a group of images) of the measurement points. The sensor data group is an example of an observation information group.

[0033] The measurement point information acquisition unit 503 acquires the position, attitude, velocity, and angular velocity value of the moving body 102 at each measurement point from map data previously stored in the storage unit 114. The measurement point information acquisition unit 503 may acquire at least one of a pair of velocity and position of each sensor at multiple times and a pair of angular velocity and attitude of each sensor.

[0034] The observation time correction amount calculation unit 504 calculates the amount of deviation of the observation time of each sensor based on the sensor characteristics, the relative positions and attitudes between the sensors, the sensor data group, and the speed and angular velocity of each sensor at each measurement point. The observation time correction amount calculation unit 504 may calculate the relative relationship of each sensor at multiple times based on the sensor characteristics, the relative positions and attitudes between the sensors, the observation information group, and the measurement point information. The amount of deviation of the observation time of each sensor is an example of the relative relationship of each sensor at multiple times.

[0035] (Flow of information processing method) Next, an information processing method using the information processing device 101 of this embodiment will be described. Fig. 6 is a flow chart showing the flow of processing of the information processing method according to this embodiment.

[0036] In step S601, the sensor parameter acquisition unit 501 acquires the characteristics of a plurality of sensors arranged on the moving body 102 and information on the relative positions and relative orientations between the sensors.

[0037] In step S602, the observation information acquisition unit 502 acquires position information of multiple feature points in the environment and a group of sensor data acquired by each sensor at multiple measurement points in the environment. It also acquires the correspondence between the feature points in the environment and the features on the sensor data (e.g., a group of images) of the measurement points.

[0038] In step S603, the measurement point information acquisition unit 503 acquires the position, orientation, velocity, and angular velocity value of the moving object 102 at each measurement point.

[0039] In step S604, the observation time correction amount calculation unit 504 calculates the amount of synchronization deviation of each sensor based on the characteristics of the sensor, the relative positions and relative attitudes between the sensors, the sensor data group, the velocity and angular velocity of the moving body 102 at each measurement point, etc. The process of step S604 will be described in detail later.

[0040] (Details of observation time correction process) The process of step S604 will be described in detail below. Fig. 7 is a flow chart showing the details of the flow of the observation time correction process in the first embodiment of the present invention. Fig. 7 shows the details of the flow of the calculation process of the amount of synchronization deviation of each sensor, the position information of the feature points in the environment, and the update value of the position and orientation information of the measurement points.

[0041] In step S701, the observation time correction amount calculation unit 504 sets an evaluation function to be used in optimization based on the various pieces of information acquired in steps S601 to S603.

[0042] The evaluation function first calculates corrected values ​​of the camera's shooting position and orientation based on the synchronization deviation dt for each camera, the relative position and relative orientation of the camera as viewed from the logical center position of the moving body 102, and the velocity and angular velocity values ​​at each measurement point. More specifically, the position is first corrected by translating the original position and orientation of the camera by an amount obtained by multiplying the velocity and synchronization deviation dt at the measurement point. Furthermore, the position and orientation of the camera are corrected to a position and orientation rotated by an angle obtained by multiplying the angular velocity and synchronization deviation dt at the measurement point around the logical center position of the moving body 102.

[0043] Then, based on the corrected shooting position and orientation of each camera, the reprojection error is calculated for each pair of feature points in each environment and their corresponding feature points in the image group. Finally, the sum of squares of the reprojection errors for all pairs is taken as the overall evaluation value. The smaller this evaluation value, the higher the consistency between the synchronization deviation amount dt and each piece of information acquired in steps S601 to S603.

[0044] Here, the reprojection error is the error between the two-dimensional coordinates of a feature point on a group of images and the two-dimensional coordinates of the feature point in the environment projected onto the images based on the position, orientation, and characteristics of the sensor. The three-dimensional coordinates are an example of three-dimensional position information. The calculation of the overall evaluation value is not limited to the sum of squares of the reprojection errors of each group, and a method of linearly evaluating reprojection errors above a certain level may also be used.

[0045] The observation time correction calculation unit 504 evaluates that the consistency of the observation information is high when there is a small error between the position obtained by projecting the three-dimensional position information of the feature point onto the observation information of each sensor in the observation information group and the position of the feature point observed in the observation information group.

[0046] In step S702, the observation time correction amount calculation unit 504 sets the amount of synchronization deviation between the right camera 304 and the left camera 305 based on the front camera 303 as the initial value of the parameter to be varied during optimization. Here, 0 is set as the initial value, but if a predicted value is obtained in advance, this may be set as the amount of synchronization deviation.

[0047] In step S703, the observation time correction amount calculation unit 504 calculates the value of the synchronization deviation amount of each camera using an optimization method using known methods such as the Newton method, the Levenberg-Marquardt method, etc. At this time, the observation time correction amount calculation unit 504 calculates the value of the synchronization deviation amount of each camera that is most consistent with the information acquired in steps S601 to S603.

[0048] That is, the observation time correction amount calculation unit 504 detects one or more common features observed at one or more times by multiple sensors in the sensor group in the observation information group. In addition, the observation time correction amount calculation unit 504 calculates the relative relationship at multiple times of each sensor that increases the consistency of the positions of the observation information of the detected common features.

[0049] (Effects of the First Embodiment) As described above, according to this embodiment, even if the same object is not observed at the same time, the amount of synchronization deviation between sensors can be acquired and corrected, and highly accurate position and orientation estimation can be performed.

[0050] (Modification of the first embodiment) In the first embodiment, values ​​calculated from the output value of the inertial sensor are used as the velocity and angular velocity values ​​of the measurement point information. These values ​​may be calculated by differentiating the position, attitude, and velocity values ​​of the moving body. This makes it possible to obtain the amount of synchronization deviation between sensors even when the inertial sensor cannot be used or the accuracy of the inertial sensor is insufficient.

[0051] In the first embodiment, only the amount of synchronization deviation of each sensor is calculated in step S702. However, in reality, the position information of multiple feature points in the environment acquired in step S602 and the position and orientation values ​​of the moving body at each measurement point acquired in step S603 also contain errors due to synchronization deviation of each sensor. Therefore, a method may be adopted in which the amount of synchronization deviation of each sensor is calculated using an optimization method, and the position information of feature points in the environment and the position and orientation information of measurement points are updated to match the amount of synchronization deviation of each sensor. For example, a method may be considered in which the position information of feature points in the environment and the position and orientation information of measurement points are added to optimization parameters and optimized collectively in step S703. In this case, the values ​​acquired in step S602 are set as the initial values ​​of the parameters in step S702. Alternatively, a method may be adopted in which the position information of feature points in the environment and the position and orientation information of measurement points are updated by a separately provided optimization means after step S702 calculates the amount of synchronization deviation of each sensor. This makes it possible to correct errors in the map data caused by synchronization deviation between sensors, and to achieve more accurate position and orientation measurement.

[0052] In the description of FIG. 4, the shooting position and attitude of the camera are corrected based on the amount of synchronization deviation dt, the velocity and angular velocity of the moving body 102 at each measurement point, and the relative position and attitude of the camera as viewed from the logical center position of the moving body 102. However, the method of correcting the shooting position and attitude of the camera is not limited to this. For example, the shooting position of the camera may be corrected by using only the velocity of the moving body 102 among the velocity and angular velocity of the moving body 102. In this case, the shooting attitude of the camera is not corrected, and it is not necessary to obtain the relative position and attitude of the camera as viewed from the logical center position of the moving body 102. Alternatively, the shooting position and attitude of the camera may be corrected based on the angular velocity and the relative position and attitude of the camera as viewed from the logical center position of the moving body 102 without using the velocity of the moving body 102. Alternatively, the acceleration and angular acceleration of the moving body and the sensor at the measurement point may be additionally obtained, and the shooting position and attitude of the camera may be corrected using a quadratic function for the amount of synchronization deviation dt.

[0053] [Second embodiment] In the first embodiment, a method for acquiring the amount of synchronization deviation between cameras when multiple cameras arranged on a moving object capture images in synchronization has been described. In the second embodiment of the present invention, a method for acquiring the amount of synchronization deviation between sensors will be described, taking as an example a case in which a moving object is arranged with multiple types of sensors with different data acquisition periods.

[0054] (Configuration of information processing device) The configuration of the information processing device in the second embodiment is the same as that described in the first embodiment with reference to FIG. 1, and therefore will not be described here.

[0055] (Mobile) The moving body 102 for creating map data in the second embodiment is an unmanned vehicle that can be remotely controlled, as in the first embodiment. A group of sensors is arranged on the moving body 102. The group of sensors arranged on the moving body 102 includes three stereo cameras and an inertial sensor described in the first embodiment, as well as a 2D-LiDAR capable of acquiring scan data of 270 degrees forward on a horizontal plane. The 2D-LiDAR is a LiDAR that outputs two-dimensional data. The scan data here is data that represents the direction and the distance to an object for each predetermined angle as seen from the 2D-LiDAR. Each sensor including the 2D-LiDAR is fixed to the moving body 102, and the logical center position of the moving body 102 and the relative positions and orientations between each sensor are known.

[0056] In the second embodiment, each camera captures images that are transmitted continuously at the same period. For ease of explanation, each camera is assumed to be precisely synchronized and always captures images at the same time. Each camera captures images at intervals of 30 ms (milliseconds). The 2D-LiDAR periodically acquires scan data at intervals of 40 ms.

[0057] Here, the image capturing time by each camera and the scan time by the 2D-LiDAR are managed by a time stamp assigned by the controller of the moving object 102. The 2D-LiDAR acquires data at a time delayed by a synchronization deviation amount dt compared to the time of the assigned time stamp with respect to the front camera. For example, assume that the controller of the moving object 102 assigns a time stamp indicating that data was acquired by each camera and the 2D-LiDAR at time t. In this case, what was actually acquired are the images captured by each camera at time t and the scan data of the 2D-LiDAR scanned at time t+dt.

[0058] In the above, the cameras are precisely synchronized for ease of explanation, but in reality, the right camera, left camera, and 2D-LiDAR have their own synchronization deviation amount dt with respect to the front camera.

[0059] (Usage scenario) In the second embodiment, in addition to the environment described in the first embodiment with reference to FIG. 2, multiple markers that can be recognized by both the camera and 2D-LiDAR and whose position and orientation as seen from the sensor can be acquired are placed in the environment. The marker is a combination of an uneven shape detectable by the 2D-LiDAR and a geometric pattern that can be recognized by the camera. One or more markers are provided. The markers can be detected by each sensor of the sensor group while being distinguished from one another. When the marker is observed by the camera, the position and orientation as seen from the camera with three dimensions and six degrees of freedom are acquired, and when the marker is observed by the 2D-LiDAR, the position and orientation as seen from the 2D-LiDAR with two dimensions and three degrees of freedom corresponding to the horizontal plane are acquired. In this embodiment, the position and orientation values ​​of the marker with three dimensions and six degrees of freedom acquired by the camera are converted into values ​​with two dimensions and three degrees of freedom projected onto a horizontal plane and used.

[0060] (Map data) The map data that is the subject of correction processing in the second embodiment holds, as a group of observation information, a number of feature points in the environment and feature point observation information obtained by observing the feature points in the environment by each sensor at a number of measurement points in the environment.

[0061] In the second embodiment, the feature points in the environment are markers placed in the environment, and each feature point has a two-dimensional position and orientation value (X, Y, Θ) with three degrees of freedom. Here, the X-axis and Y-axis are mutually orthogonal axes in the horizontal direction of the environment, and Θ is a direction on the horizontal plane.

[0062] In the second embodiment, the measurement points acquire either images captured by each camera or scan data by 2D-LiDAR, or both, as sensor data. The feature point observation information of each measurement point in this embodiment holds the relative position and orientation of the marker recognized by the sensor data from the recognized sensor in two dimensions and three degrees of freedom. Feature points in each environment are associated with feature point observation information of one or more measurement points. Similarly to the first embodiment, the map data holds the characteristics of each sensor used for observation as sensor parameters, and the relative position and relative orientation information between the sensors.

[0063] (Effects of synchronization deviation and compensation principle) Here, the influence of a synchronization error between sensors on map data and position and orientation measurement will be described with reference to Fig. 8. Fig. 8 is a diagram showing the influence of a synchronization error in the second embodiment of the present invention, and is a diagram explaining the influence of a synchronization error of 2D-LiDAR on map data.

[0064] Reference numeral 801 shows a moving object 102 moving in the environment while turning right from the measurement point P0 side to the measurement point P60 side. Here, measurement points P0, P30, and P60 are measurement points photographed by a camera, and measurement points P10 and P50 are measurement points where scan data was acquired by a 2D-LiDAR. The number of each measurement point indicates the time when acquisition of the measurement point data is instructed in milliseconds (ms) based on measurement point P0. In other words, when the camera photographs measurement point P0 at time t=0 ms, data acquisition by the 2D-LiDAR at measurement point P10 is instructed at time t=10 ms, and the time when the scan data is actually acquired is t=10 ms+dt with the synchronization deviation dt added.

[0065] Reference numeral 802 indicates the positions of two markers in the environment observed by each sensor of the moving object 102. Here, the solid line indicates the position of the marker observed by the camera, and the dashed line indicates the position of the marker observed by the 2D-LiDAR. It is assumed that the solid line indicates the correct position of the marker. Since the 2D-LiDAR acquires scan data at a time delayed by a synchronization deviation amount dt from the camera, the marker observed by the 2D-LiDAR is observed as if it is at a position shifted in the opposite direction to the traveling direction and turning direction of the moving object 102.

[0066] FIG. 9 is a diagram showing a method of correcting a synchronization shift in the second embodiment of the present invention, and shows a state in which the scan data acquisition position and orientation of the 2D-LiDAR at each measurement point is corrected taking into account the synchronization shift dt.

[0067] The dashed line 901 indicates the position and orientation of the 2D-LiDAR scan data acquisition that does not take into account the synchronization shift at each measurement point. The solid line 902 indicates the position and orientation of the 2D-LiDAR scan data acquisition that has been corrected taking into account the synchronization shift at each measurement point.

[0068] Here, the correction amount of the scan data acquisition position and orientation is calculated from the velocity and angular velocity of the moving body 102 at each measurement point, the relative position and relative orientation of the 2D-LiDAR as viewed from the logical center position of the moving body 102, and the synchronization deviation amount dt. More specifically, first, the position is corrected by translating the original position and orientation of the 2D-LiDAR by an amount obtained by integrating the velocity and synchronization deviation amount dt at the measurement point. Furthermore, the position and orientation of the 2D-LiDAR are corrected to a position and orientation rotated by an angle obtained by integrating the angular velocity and synchronization deviation amount dt at the measurement point around the logical center position of the moving body 102.

[0069] By correcting the data acquisition position and orientation of the sensors in this way, the positions of the markers 802 observed by each sensor will match, and correct map data can be obtained. Also, when measuring position and orientation using map data, correcting the data acquisition position and orientation of the sensors according to the amount of synchronization deviation will make it possible to calculate more accurate position and orientation.

[0070] Here, only the correction of the scan data acquisition position and orientation considering the synchronization deviation of the 2D-LiDAR has been described. The correction of the camera shooting position and orientation considering the synchronization deviation of the right camera and the left camera is the same as that described with reference to FIG. 4 in the first embodiment, so the description here is omitted.

[0071] (Functional configuration of information processing device) The functional configuration of the information processing device in the second embodiment is the same as that described in the first embodiment with reference to FIG. 5, and therefore will not be described here.

[0072] (Flow of information processing method) Hereinafter, an information processing method using the information processing device of the second embodiment will be described. The basic flow of the process is the same as that described in the first embodiment with reference to Fig. 6. Therefore, only the process steps having unique processes in the second embodiment will be described with reference to Fig. 6.

[0073] The processes in steps S601 and S603 in FIG. 6 are the same as those described in the first embodiment, and therefore a description thereof will be omitted.

[0074] In step S602, the observation information acquisition unit 502 acquires position and orientation information of a plurality of markers in the environment, and feature point observation information acquired by each sensor at a plurality of measurement points in the environment.

[0075] The basic processing content of step S604 is the same as in the first embodiment, and the details of the processing will be described later.

[0076] (Details of observation time correction process) Details of the observation time correction process in step S604 in the second embodiment will be described below. The basic flow of the process is the same as that described in the first embodiment with reference to Fig. 7. Therefore, details of the process steps will be described here with reference to Fig. 7.

[0077] In step S701, the observation time correction amount calculation unit 504 sets an evaluation function to be used in optimization based on the various pieces of information acquired in steps S601 to S603. The output value of this evaluation function is the evaluation value.

[0078] The evaluation function first calculates corrected values ​​of the position and orientation of the sensor when acquiring sensor data. The calculation of the corrected values ​​is performed based on the synchronization deviation dt for each sensor, the relative position and relative orientation of the sensor as viewed from the logical center position of the moving body 102, and the velocity and angular velocity values ​​at each measurement point. More specifically, the position is first corrected by translating the original position and orientation of the sensor by an amount obtained by multiplying the velocity at the measurement point and the synchronization deviation dt. Furthermore, the position and orientation of the sensor are corrected to a position and orientation rotated by an angle obtained by multiplying the angular velocity at the measurement point and the synchronization deviation dt around the logical center position of the moving body 102.

[0079] Then, based on the corrected position and orientation of each sensor when acquiring the sensor data, the relative position and orientation of each marker observed at the measurement point and the sensor are calculated. Then, the error between the calculated relative position and orientation and the relative position and orientation of the marker and the sensor stored as feature point observation information is calculated, and the position error and orientation error are added with a predetermined weighting to obtain a relative position and orientation error value. Finally, the sum of squares of the relative position and orientation error values ​​for all combinations of markers and sensors is obtained as an overall evaluation value. The smaller this evaluation value, the higher the consistency between the synchronization deviation amount dt and each piece of information acquired in steps S601 to S603.

[0080] For example, the observation time correction amount calculation unit 504 sets candidates for the measurement point information and the relative relationship of each sensor at a plurality of times. Next, the observation time correction amount calculation unit 504 calculates a correction value for at least one of the position and the attitude of each sensor when the observation information group was acquired based on the set candidates, and evaluates the consistency of the observation information based on the calculated correction value.

[0081] In the second embodiment, as in the first embodiment, the calculation of the overall evaluation value is not limited to the sum of squares of the reprojection errors of each set, but a method such as linearly evaluating reprojection errors above a certain level may also be used.

[0082] In step S702, the observation time correction amount calculation unit 504 sets the amount of synchronization deviation of the right camera, the left camera, and the 2D-LiDAR based on the front camera as the initial value of the parameter to be varied during optimization. Here, 0 is set as the initial value, but if a predicted value is obtained in advance, it may be set as the amount of synchronization deviation.

[0083] In step S703, the observation time correction amount calculation unit 504 calculates the value of the amount of synchronization deviation for each sensor using an optimization method using known methods such as the Newton method, the Levenberg-Marquardt method, etc. At this time, the observation time correction amount calculation unit 504 calculates the value of the amount of synchronization deviation for each sensor that is most consistent with the information acquired in steps S601 to S603.

[0084] (Effects of the second embodiment) As described above, according to this embodiment, even when multiple types of sensors are used or when the sensor data acquisition periods of the sensors are different, it is possible to acquire and correct the amount of synchronization deviation between sensors and perform highly accurate position and orientation estimation.

[0085] (Modification of the second embodiment) In the second embodiment, only the amount of synchronization deviation of each sensor is calculated in step S702. However, in reality, the position and orientation values ​​of each marker acquired in step S602 and the position and orientation values ​​of the moving body at each measurement point acquired in step S603 also contain errors due to synchronization deviation of each sensor. Therefore, a method may be adopted in which the amount of synchronization deviation of each sensor is calculated using an optimization method, and at the same time, the position and orientation information of each marker and the position and orientation information of the measurement point are updated to be consistent with the amount of synchronization deviation. This makes it possible to correct errors in the map data due to synchronization deviation between sensors, and to achieve more accurate position and orientation measurement.

[0086] In addition, although the position and orientation values ​​of the markers used in the second embodiment can be acquired from each sensor, it is also possible to acquire only the position of the marker. In such a case, only the position of the marker is recorded in the map data as information on the characteristic points in the environment, and the evaluation function set in step S701 evaluates only the error in the relative position between the sensor and the marker. This makes it possible to estimate the amount of synchronization deviation between the sensors even when a marker whose orientation cannot be acquired by the sensor is used.

[0087] [Third embodiment] In the first embodiment, an example is described in which map data created by a mobile body on which a sensor is installed is read into an information processing device independent of the mobile body to estimate a synchronization error and correct the map. In the third embodiment, an example is described in which a mobile body on which an information processing device is installed generates map data while moving through an environment, and performs estimation of a synchronization error and correction of the map in parallel with the generation of a group of observation information and measurement point information.

[0088] (Map data) The format of the map data generated in the third embodiment is similar to that in the first embodiment, and therefore a description thereof will be omitted here.

[0089] (Mobile System Configuration) 10 is a diagram showing the configuration of a mobile system 1001 according to the third embodiment of the present invention. The mobile system 1001 includes a map creation system 1002 including a sensor group 1003 and an information processing device 1004, a communication unit 1005, a control device 1006, and a moving unit 1007.

[0090] The physical configuration of the information processing device 1004 is the same as that of the information processing device 101 described with reference to FIG. 1 in the first embodiment, and therefore a detailed description thereof will be omitted. The information processing device 1004 has a CPU 1111, a ROM 1112, a RAM 1113, a storage unit 1114 such as an HDD or SSD, a general-purpose IF 1115, and a system bus 1116. IF is an abbreviation of interface. The CPU 1111 of the information processing device 1004 corresponds to the CPU 111 of the information processing device 101. The ROM 1112 of the information processing device 1004 corresponds to the ROM 112 of the information processing device 101. The RAM 1113 of the information processing device 1004 corresponds to the RAM 113 of the information processing device 101. The storage unit 1114 of the information processing device 1004 corresponds to the storage unit 114 of the information processing device 101. The general-purpose IF 1115 of the information processing device 1004 corresponds to the communication unit 115 of the information processing device 101. A system bus 1116 of the information processing device 1004 corresponds to the system bus 116 of the information processing device 101. The information processing device 1004 can communicate with the sensor group 1003 via the general-purpose IF 1115. The information processing device 1004 can communicate with the communication unit 1005 and the control device 1006 via the general-purpose IF 1115.

[0091] The mobile body system 1001 has three stereo cameras mounted facing the front, left, and right as the sensor group 1003. The mobile body system 1001 also has inertial sensors capable of measuring acceleration and angular velocity as the sensor group 1003. These sensors are the same as those arranged in the mobile body 102 in the first embodiment.

[0092] The control device 1006 controls the driving of the moving unit 1007. The control device 1006 drives the moving unit 1007 based on an instruction from an external user via the communication unit 1005, and moves and turns the mobile body system 1001. The moving unit 1007 is a plurality of tires, some or all of which are linked to a power source.

[0093] (Functional configuration of mobile systems) Hereinafter, the functional configuration of a mobile system 1001 according to the third embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of an information processing device 1004 according to the third embodiment of the present invention. The information processing device 1004 has a sensor measurement value acquisition unit 1101, a map creation unit 1102, a sensor parameter acquisition unit 1103, an observation information acquisition unit 1104, a measurement point information acquisition unit 1105, an observation time correction amount calculation unit 1106, and a movement control unit 1107.

[0094] The sensor measurement value acquisition unit 1101 continuously acquires sensor data from the sensor group 1003 .

[0095] As the mobile system 1001 moves, the map creation unit 1102 measures the position and orientation of the mobile system 1001 and generates map data in parallel using a known Visual SLAM technique such as ORB-SLAM. Specifically, the map creation unit 1102 receives sensor data obtained from the sensor measurement value acquisition unit 1101, calculates position and orientation values ​​based on the generated map data, and generates additional observation information groups and measurement point information. The map creation unit 1102 also stores the generated map data in the RAM 1113.

[0096] The sensor parameter acquisition unit 1103 acquires the characteristics of each sensor of the sensor group 1003 and information on the relative positions and relative orientations between the sensors from data stored in advance in the storage unit 1114 .

[0097] The observation information acquisition unit 1104 acquires a plurality of feature points in the environment and a group of sensor data acquired by each sensor at a plurality of measurement points in the environment (i.e., at a plurality of times) from map data held by the map creation unit 1102. The observation information acquisition unit 1104 also acquires the correspondence between the feature points in the environment and the feature points on the image group of the measurement points.

[0098] A measurement point information acquisition unit 1105 acquires the position, orientation, velocity, and angular velocity value of the moving object at each measurement point from the map data held by the map creation unit 1102 .

[0099] The observation time correction calculation unit 1106 calculates the amount of deviation in the observation time of each sensor based on the sensor characteristics, the relative positions and relative orientations between the sensors, the image group, and the velocity and angular velocity of the moving object at each measurement point.

[0100] The movement control unit 1107 drives the control device 1006 based on user instructions via the communication unit 1005 or autonomous judgment, and moves the mobile system 1001. Here, the autonomous judgment is an algorithm in which the movement control unit 1107 refers to input values ​​of the sensor measurement values ​​1101 and map data held by the map creation unit 1102, and controls the mobile system 1001 to comprehensively move within the environment.

[0101] (Map data creation process) The following describes a method for creating map data using the mobile system 1001. Fig. 12 is a flow chart showing the flow of an information processing method according to the third embodiment of the present invention.

[0102] In step S1201, the sensor measurement value acquisition unit 1101 acquires sensor data from the sensor group 1003.

[0103] In step S1202, the map creation unit 1102 refers to the generated map data stored in the RAM 1113 and calculates the position and attitude value of the mobile system 1001 at the time of acquiring the sensor data in step S1201. When step S1202 is executed for the first time, valid map data has not yet been generated, so the map creation unit 1102 sets the position and attitude value assuming that the mobile system 1001 is located at the origin in the environment and faces the X-axis direction. When step S1202 is executed for the second or subsequent times, the map creation unit 1102 calculates the position and attitude in the environment based on the generated map data. At that time, if the amount of synchronization deviation of each sensor has been estimated, the position and attitude value of the mobile system 1001 is calculated using the position and attitude of each sensor corrected according to the amount of synchronization deviation. This is expected to improve the accuracy of the calculated position and attitude value.

[0104] In step S1203, the map creation unit 1102 determines whether or not to add new measurement points to the map data. The map creation unit 1102 determines to add new measurement points to the map data when step S1203 is executed for the first time or when the current position or attitude of the mobile system 1001 is away from existing measurement points by a certain amount or more. If new measurement points are to be added to the map data, the process of step 1204 is executed, and if not, the process of step 1210 is executed.

[0105] In step S1204, the map creation unit 1102 adds new measurement point information and observation information to the map data. Here, the position and attitude values ​​of the new measurement points to be added are the values ​​calculated in step S1202. Also, the velocity and angular velocity values ​​at the new measurement points to be added are values ​​calculated from the output values ​​of the inertial sensor among the sensor data acquired in step S1201.

[0106] Furthermore, the map creation unit 1102 extracts feature points on the image from the image data of each camera among the sensor data acquired in step S1201 as a group of observation information observed from the new measurement point to be added. The map creation unit 1102 assigns three-dimensional position information in the environment to the extracted feature points based on the position and orientation value of the new measurement point and the distance calculated by the stereo measurement. Thereafter, the map creation unit 1102 matches the feature points with existing measurement points and merges feature points in the same environment. Here again, if the amount of synchronization deviation of each sensor has been estimated, the map creation unit 1102 calculates three-dimensional position information using the position and orientation of each sensor corrected according to the amount of synchronization deviation. This is expected to improve the accuracy of the map data.

[0107] In step S1205, the sensor parameter acquisition unit 1103 acquires the characteristics of each sensor in the sensor group 1003 and information on the relative positions and orientations between the sensors.

[0108] In step S1206, the measurement point information acquisition unit 1105 first selects measurement points to be corrected. In the third embodiment, since the synchronization deviation estimation process is performed in parallel with the generation of map data, it is necessary to keep the processing load and calculation time below a certain level. Therefore, in the third embodiment, of the measurement points added to the map data, a maximum of 10 measurement points are selected from the most recent ones, and the subsequent processes are performed. Here, the number of measurement points selected and the selection method are merely examples, and other numbers and selection methods may be used. After that, the measurement point information acquisition unit 1105 acquires the position, attitude, velocity, and angular velocity value of the moving body at each selected measurement point.

[0109] In step S1207, the observation information acquisition unit 1104 first extracts feature points in the environment that are observed by two or more of the measurement points selected in step 1206. Then, the observation information acquisition unit 1104 acquires position information of the extracted feature points in the environment and a group of sensor data acquired by each sensor at each measurement point observing the feature points.

[0110] In step S1208, the observation time correction amount calculation unit 1106 judges whether or not to estimate the synchronization deviation. The observation time correction amount calculation unit 1106 investigates whether the velocity and angular velocity values ​​at each measurement point selected in step S1206 are equal to or greater than the threshold, and judges not to estimate the synchronization deviation if all the velocity and angular velocity values ​​are below the threshold. This is to avoid a decrease in the estimation accuracy of the synchronization deviation when the velocity and angular velocity values ​​at each measurement point are small, as the fluctuation in the correction amount of the camera position relative to the fluctuation in the value of the estimated synchronization deviation becomes small. Similarly, when the number of feature points in the environment extracted in step S1207 is below the threshold, the observation time correction amount calculation unit 1106 judges not to estimate the synchronization deviation because there is a high possibility that sufficient estimation accuracy of the synchronization deviation cannot be obtained. If the synchronization deviation is to be estimated, the process of step S1209 is executed, and if the synchronization deviation is not to be estimated, the process of step 1210 is executed.

[0111] In step S1209, the observation time correction amount calculation unit 1106 calculates the amount of synchronization deviation of each sensor based on information on multiple feature points in the environment, the sensor data group of each measurement point, the sensor characteristics, the relative positions and relative orientations between the sensors, and the speed and angular velocity of the moving object at each measurement point. Details of the process in step S1209 will be described later.

[0112] In step S1210, the information processing device 1004 determines whether or not to end the map creation process. If there is an instruction from the user via the communication unit 1005, the series of processes ends. If not, the process of step S1201 is executed.

[0113] (Details of observation time correction process) Details of the process in step S1209 will be described below. The basic flow of the observation time correction process is the same as that described in the first embodiment with reference to Fig. 7. Therefore, details of each process will be described here with reference to Fig. 7.

[0114] In step S701, the observation time correction amount calculation unit 1106 sets an evaluation function to be used in optimization based on each piece of information acquired in steps S1205 to S1207. The specific setting method is the same as in the first embodiment, and therefore a description thereof will be omitted here.

[0115] In step S702, the observation time correction amount calculation unit 1106 sets the amount of synchronization deviation between the right camera and the left camera with respect to the front camera as the initial value of the parameter to be varied during optimization. Here, 0 is set as the initial value at the first execution, but if a predicted value is obtained in advance, this may be set as the amount of synchronization deviation. At the second or subsequent executions, the value of the amount of synchronization deviation estimated previously is set as the initial value.

[0116] In step S703, the observation time correction calculation unit 1106 uses an optimization method using known methods such as the Newton method or the Levenberg-Marquardt method to calculate the synchronization deviation value for each camera that is most consistent with the information obtained in steps S1205 to S1207.

[0117] (Effects of the third embodiment) As described above, according to this embodiment, even when a mobile body on which a group of sensors is arranged is used, it is possible to obtain and correct the amount of synchronization deviation between sensors, and create a highly accurate map.

[0118] (Modification of the third embodiment) In the third embodiment, a moving body in which three stereo cameras and an inertial sensor are arranged as the sensor group 1003 has been described as an example, but the type of sensors to be arranged is not limited to this. For example, the 2D-LiDAR described in the second embodiment may be arranged, or other types of sensors such as a 3D-LiDAR or a depth camera may be arranged. In this case, the map creation unit 1102 may generate map data using a known map creation technology (such as LiDAR SLAM or Depth SLAM) according to the arranged sensors. In any combination of sensors, the present invention can be implemented to estimate the amount of synchronization deviation between the sensors.

[0119] (Another embodiment) In the first, second and third embodiments, the synchronization deviation estimation method is implemented as a program executed on an information processing device. The present invention may take the form of a storage medium having the above-mentioned program written thereon. Various methods can be used to supply the program constituting the present invention, and cases in which the program is distributed via a network such as the Internet, or in which the program is encrypted or divided and distributed are also within the scope of the present invention.

[0120] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0121] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0122] The disclosure of the present embodiment includes the following configurations: a method and a program. (Configuration 1) a sensor parameter acquisition means for acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, and information on relative positions and relative attitudes between the sensors; an observation information acquisition means for acquiring a group of observation information acquired at a plurality of times by each of the sensors; a measurement point information acquiring means for acquiring, as the measurement point information, at least one of a set of a velocity and a position, and a set of an angular velocity and an attitude of each of the sensors at the plurality of times; an observation time correction amount calculation means for calculating a relative relationship between the sensors at the plurality of times based on the sensor characteristics, the relative positions and orientations between the sensors, the observation information group, and the measurement point information; An information processing device comprising: (Configuration 2) The observation time correction amount calculation means detects one or more common features observed at one or more times by a plurality of sensors in the sensor group in the observation information group, and calculates a relative relationship between sensors at the plurality of times that has higher consistency in the positions of the observation information of the detected common features. 2. The information processing device according to configuration 1. (Configuration 3) the common features are feature points in the environment having three-dimensional position information; The information processing device according to configuration 2, wherein the observation time correction amount calculation means evaluates the consistency of the observation information to be high when an error between a position obtained by projecting the three-dimensional position information of the feature point onto the observation information of each of the sensors in the observation information group and a position of the feature point observed in the observation information group is small. (Configuration 4) The set of observations each includes an image, and the common features are feature points on each image that capture the same point on an object in the environment. 4. The information processing device according to configuration 2 or 3. (Configuration 5) The common feature is one or more markers that are placed in the environment and can be detected by each sensor in the group of sensors in a distinguishable manner from one another. 5. The information processing device according to any one of configurations 2 to 4. (Configuration 6) The observation time correction amount calculation means sets candidates for the relative relationship between the measurement point information and each of the sensors at the multiple times, calculates a correction value for at least one of the position and the attitude of each of the sensors when the group of observation information was acquired based on the set candidates, and evaluates the consistency of the observation information based on the calculated correction value. 6. The information processing device according to any one of configurations 2 to 5. (Method 1) a sensor parameter acquisition step of acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, as well as relative position and relative attitude information between the sensors; an observation information acquisition step of acquiring a group of observation information acquired at a plurality of times by each of the sensors; a measurement point information acquiring step of acquiring, as measurement point information, at least one of a set of a velocity and a position of each of the sensors at the plurality of times and a set of an angular velocity and an attitude of each of the sensors; an observation time correction amount calculation step of calculating a relative relationship between the sensors at the multiple times based on the sensor characteristics, the relative positions and orientations between the sensors, the observation information group, and the measurement point information; A control method comprising: (Program 1) Computer, a sensor parameter acquisition means for acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, as well as relative position and relative attitude information between the sensors; an observation information acquiring means for acquiring a group of observation information acquired at a plurality of times by each of the sensors; a measurement point information acquisition means for acquiring, as measurement point information, at least one of a set of velocity and position, and a set of angular velocity and attitude, of each of the sensors at the plurality of times; and an observation time correction amount calculation means for calculating a relative relationship between the sensors at the multiple times based on the sensor characteristics, the relative positions and attitudes between the sensors, the observation information group, and the measurement point information; A program characterized by causing the program to function as a [Explanation of symbols]

[0123] 101: information processing device, 102: moving object, 501: sensor parameter acquisition unit, 502: observation information acquisition unit, 503: measurement point information acquisition unit, 504: observation time correction amount calculation unit

Claims

1. a sensor parameter acquisition means for acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, and information on relative positions and relative attitudes between the sensors; an observation information acquiring means for acquiring a group of observation information acquired at a plurality of times by each of the sensors; a measurement point information acquiring means for acquiring, as the measurement point information, at least one of a set of a velocity and a position, and a set of an angular velocity and an attitude of each of the sensors at the plurality of times; an observation time correction amount calculation means for calculating a relative relationship between the sensors at the plurality of times based on the sensor characteristics, the relative positions and orientations between the sensors, the observation information group, and the measurement point information; An information processing device comprising:

2. The observation time correction amount calculation means detects one or more common features observed at one or more times by a plurality of sensors in the sensor group in the observation information group, and calculates a relative relationship between sensors at the plurality of times that has higher consistency in the positions of the observation information of the detected common features.

2. The information processing apparatus according to claim 1,

3. the common features are feature points in the environment having three-dimensional position information; The information processing device according to claim 2, characterized in that the observation time correction amount calculation means evaluates the consistency of the observation information to be high when there is a small error between the position of the three-dimensional position information of the feature point projected onto the observation information of each sensor in the observation information group and the position of the feature point observed in the observation information group.

4. The set of observations each includes an image, and the common features are feature points on each image that capture the same point on an object in the environment.

3. The information processing apparatus according to claim 2.

5. The common feature is one or more markers that are placed in the environment and can be detected by each sensor in the group of sensors in a distinguishable manner from one another.

3. The information processing apparatus according to claim 2.

6. The observation time correction amount calculation means sets candidates for the relative relationship between the measurement point information and each of the sensors at the multiple times, calculates a correction value for at least one of the position and the attitude of each of the sensors when the group of observation information was acquired based on the set candidates, and evaluates the consistency of the observation information based on the calculated correction value.

3. The information processing apparatus according to claim 2.

7. a sensor parameter acquisition step of acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, as well as relative position and relative attitude information between the sensors; an observation information acquisition step of acquiring a group of observation information acquired at a plurality of times by each of the sensors; a measurement point information acquiring step of acquiring, as measurement point information, at least one of a set of a velocity and a position of each of the sensors at the plurality of times and a set of an angular velocity and an attitude of each of the sensors; an observation time correction amount calculation step of calculating a relative relationship between the sensors at the multiple times based on the sensor characteristics, the relative positions and orientations between the sensors, the observation information group, and the measurement point information; A control method comprising:

8. Computer, a sensor parameter acquisition means for acquiring sensor characteristics of each sensor of a sensor group consisting of a plurality of sensors arranged on a moving body, as well as relative position and relative attitude information between the sensors; an observation information acquiring means for acquiring a group of observation information acquired at a plurality of times by each of the sensors; a measurement point information acquisition means for acquiring, as measurement point information, at least one of a set of velocity and position, and a set of angular velocity and attitude, of each of the sensors at the plurality of times; and an observation time correction amount calculation means for calculating a relative relationship between the sensors at the multiple times based on the sensor characteristics, the relative positions and attitudes between the sensors, the observation information group, and the measurement point information; A program characterized by causing the program to function as a

Images