Data processing apparatus, data processing method, and computer-readable recording medium

Abstract

The camera movement amount acquisition unit (101) acquires, as a camera movement amount (110), an estimated movement amount of the vehicle (10) calculated using captured image data acquired by the camera (20) provided to the vehicle (10). The LiDAR movement amount acquisition unit (102) acquires, as a LiDAR movement amount (120), an estimated movement amount of the vehicle (10) having higher estimation accuracy than the camera movement amount (110) in synchronization with the camera movement amount acquisition unit (101) acquiring the camera movement amount (110). The comparison determination unit (103) compares the camera movement amount (110) and the LiDAR movement amount (120), and determines to use the camera movement amount (110) in the calibration operation in a case where a difference between the camera movement amount (110) and the LiDAR movement amount (120) is smaller than a threshold value.

Description

Technical Field

[0001] This invention relates to automatic calibration. Background Technology

[0002] Multiple sensors are installed on the moving object to identify its surroundings. For example, a vehicle, which is one of the moving objects, is equipped with cameras and other imaging devices, as well as 3D sensors such as LiDAR (Light Detection and Ranging). The detection results of these multiple sensors can be integrated to identify the vehicle's surrounding environment.

[0003] To accurately integrate the sensor detection results, it is necessary to calculate the relative positions and orientations between the sensors beforehand. The automatic calculation of these relative positions and orientations between sensors is called automatic calibration.

[0004] Patent Document 1 discloses a technique for improving the accuracy of automatic calibration. More specifically, in Patent Document 1, the weight (reliability) of 3D point groups that are closer to the camera in a 3D point group obtained by LiDAR is increased, while the weight of 3D point groups that are farther away is decreased. Thus, Patent Document 1 improves the accuracy of automatic calibration.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: International Publication No. WO2019 / 155719 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] Automatic calibration methods include appearance-based methods and motion-based methods.

[0010] In appearance-based methods, edge information from camera-captured images is overlaid with 3D point data from LiDAR. Calibration is then performed by detecting the offset between the edge information and the 3D point data. While appearance-based methods can perform calibration even when the moving object is in motion, they are dependent on the surrounding environment and therefore have low robustness.

[0011] Motion-based methods can improve upon the issue of low robustness. In these methods, the amount of movement calculated from image data captured by a camera is compared with the amount of movement calculated from 3D point data of a LiDAR. Calibration is then performed using this comparison of movement amounts to calculate the relative position and orientation between the camera and the LiDAR. Even if the moving object moves, the relative position and orientation between the camera and the LiDAR remain unchanged; therefore, by comparing the amount of movement, the relative position and orientation between the camera and the LiDAR can be calculated.

[0012] Generally speaking, the scale for calculating the amount of movement based on camera-captured image data is unclear. Furthermore, camera-captured image data tends to have lower detection accuracy for image feature points compared to LiDAR's 3D point set data; therefore, camera-captured image data may contain significant errors.

[0013] Thus, motion-based methods face the following challenges: when the precision of the image data captured by the camera is low, accurate calibration calculations cannot be performed, and accurate calculations of relative position and relative attitude are impossible.

[0014] One of the main objectives of this invention is to solve the aforementioned problem. More specifically, the main objective of this invention is to enable accurate calibration calculations of image data captured using a camera, and to calculate accurate relative position and relative orientation.

[0015] Methods for solving problems

[0016] The data processing apparatus of the present invention includes: a first motion acquisition unit that acquires an estimated motion of the moving body calculated using image data acquired by an imaging device provided on the moving body as a first motion; a second motion acquisition unit that, simultaneously with the first motion acquisition unit acquiring the first motion, acquires an estimated motion of the moving body with an estimation accuracy higher than that of the first motion as a second motion; and a comparison determination unit that compares the first motion and the second motion, and determines to use the first motion in a calibration operation if the difference between the first motion and the second motion is less than a threshold.

[0017] Invention Effects

[0018] According to the present invention, accurate calibration calculations can be performed using image data captured by a camera, thereby enabling the calculation of accurate relative position and relative attitude. Attached Figure Description

[0019] Figure 1 This is a diagram showing a structural example of the calibration system according to Embodiment 1.

[0020] Figure 2 This is a diagram illustrating another structural example of the calibration system according to Embodiment 1.

[0021] Figure 3 This is a diagram illustrating another structural example of the calibration system according to Embodiment 1.

[0022] Figure 4 This is a diagram illustrating an example of the hardware structure of the data processing apparatus according to Embodiment 1.

[0023] Figure 5 This is a diagram illustrating an example of the functional structure of the data processing apparatus according to Embodiment 1.

[0024] Figure 6 This is a flowchart illustrating an example of the operation of the data processing apparatus of Embodiment 1.

[0025] Figure 7 This is a diagram showing a structural example of the calibration system according to Embodiment 2.

[0026] Figure 8 This is a diagram illustrating an example of the functional structure of the data processing apparatus in Embodiment 2.

[0027] Figure 9 This is a flowchart illustrating an example of the operation of the data processing apparatus in Embodiment 2.

[0028] Figure 10 This is a flowchart illustrating an example of the operation of the data processing apparatus in Embodiment 3. Detailed Implementation

[0029] The embodiments will now be described using the accompanying drawings. In the following description of the embodiments and the accompanying drawings, parts labeled with the same reference numerals represent the same or equivalent parts.

[0030] Implementation Method 1

[0031] ***Structure Description***

[0032] Figure 1 , Figure 2 and Figure 3 Examples of the structure of the calibration system 1000 of this embodiment are shown.

[0033] The calibration system 1000 consists of a camera 20, a LiDAR 30, a data processing device 100, a motion calculation device 200, and a calibration device 300.

[0034] exist Figure 1 In the vehicle 10, a camera 20, a LiDAR scanner 30, a data processing device 100, a motion measurement device 200, and a calibration device 300 are respectively installed. Figure 1In the structure, the data processing device 100, the movement calculation device 200, and the calibration device 300 are connected, for example, by wire.

[0035] exist Figure 2 In this context, the data processing device 100 is a portable terminal device such as a smartphone. Therefore, in Figure 2 In this structure, the data processing device 100 is sometimes used inside the vehicle 10 and sometimes used outside the vehicle 10. Figure 2 In the structure, the data processing device 100 is wirelessly connected to the motion calculation device 200 and the calibration device 300.

[0036] exist Figure 3 In the middle, the data processing device 100 and the calibration device 300 are disposed outside the vehicle 10. Figure 3 In the structure, the data processing device 100 is a portable terminal device such as a smartphone. Figure 3 In this system, the data processing device 100 is located outside the vehicle 10; however, the data processing device 100 can also be used inside the vehicle 10. Figure 3 In the structure, the data processing device 100 is also wirelessly connected to the motion calculation device 200 and the calibration device 300.

[0037] Below, with Figure 1 This explanation is based on the premise of the structure, but through comparison with... Figure 2 and Figure 3 For replacements with the same structure, the following instructions also apply. Figure 2 and Figure 3 The structure.

[0038] exist Figure 1 In this system, camera 20 is installed on vehicle 10 and, as vehicle 10 moves, captures images of the surrounding environment of vehicle 10 at regular intervals to obtain image data. For example, camera 20 captures images at 50 millisecond intervals. The captured image data is then output to motion calculation device 200.

[0039] Camera 20 is an example of a shooting device.

[0040] A LiDAR 30 is also installed in the vehicle 10. As the vehicle 10 moves, it captures images of the surrounding environment to obtain 3D point data. The LiDAR 30 captures images of the surrounding environment of the vehicle 10 synchronously with the camera 20 (with an image capture cycle identical to that of the camera 20). For example, the LiDAR 30 captures images at a 50-millisecond cycle. The 3D point data obtained through imaging is output to the motion calculation device 200.

[0041] LiDAR30 is an example of a 3D sensor. Furthermore, 3D point group data acquired by LiDAR30 is an example of 3D sensor data.

[0042] in addition, Figures 1-3 The shown placement of camera 20 and LiDAR 30 is just one example; camera 20 and LiDAR 30 can also be placed in the same location as... Figures 1-3 The positions shown are different.

[0043] The motion calculation device 200 acquires captured image data from the camera 20. Then, the motion calculation device 200 uses the captured image data to calculate the estimated motion of the vehicle 10. Assume that the estimated motion calculated by the motion calculation device 200 includes an estimate of the rotational movement of the vehicle 10 centered on the rotation vector of the camera 20 (estimated rotation).

[0044] The estimated movement of vehicle 10 calculated using the captured image data will be referred to as camera movement 110. Camera movement 110 is equivalent to the first movement.

[0045] The movement calculation device 200 calculates the estimated movement of the vehicle 10, i.e., the camera movement 110, within a shooting cycle (e.g., 50 milliseconds) by comparing the captured image data obtained in the current shooting cycle with the captured image data obtained in the previous shooting cycle.

[0046] Specifically, the motion calculation device 200 calculates the camera motion 110 by using Visual-SLAM (Simultaneous Localization and Mapping) to calculate the correspondence between temporal images.

[0047] Furthermore, the motion calculation device 200 acquires three-dimensional point set data from the LiDAR 30. Then, the motion calculation device 200 uses the three-dimensional point set data to calculate the estimated motion of the vehicle 10. Assume that the estimated motion calculated by the motion calculation device 200 includes an estimate of the rotational movement of the vehicle 10 centered on the rotation vector of the LiDAR 30 (estimated rotation).

[0048] The estimated movement of vehicle 10 calculated using 3D point group data will be referred to as LiDAR movement 120. The estimation accuracy of LiDAR movement 120 is higher than that of camera movement 110. LiDAR movement 120 is equivalent to the second movement.

[0049] The motion calculation device 200 calculates the estimated motion of the vehicle 10, i.e., the LiDAR motion 120, within a camera cycle (e.g., 50 milliseconds) by comparing the three-dimensional point data acquired in the current camera cycle with the three-dimensional point data acquired in the previous camera cycle. Specifically, the motion calculation device 200 calculates the LiDAR motion 120 using LiDAR-SLAM employing the ICP (Iterative Closest Point) algorithm.

[0050] The motion calculation device 200 outputs the calculated camera motion 110 and LiDAR motion 120 to the data processing device 100.

[0051] Furthermore, the motion measurement device 200 also outputs the ID (Identifier) ​​of the captured image data to the data processing device 100. Hereinafter, the ID of the captured image data will be referred to as the captured image data ID 130.

[0052] The data processing device 100 obtains the camera motion 110 and LiDAR motion 120 from the motion calculation device 200. In addition, the data processing device 100 also obtains the captured image data ID 130 from the motion calculation device 200.

[0053] The data processing unit 100 compares the camera movement 110 with the LiDAR movement 120 to determine whether the camera movement 110 is suitable for the calibration operation in the calibration device 300 (described later). If the camera movement 110 is suitable for the calibration operation, it decides to use the camera movement 110 in the calibration operation. Then, the data processing unit 100 outputs the camera movement 110, the LiDAR movement 120, and the captured image data ID 130 to the calibration device 300.

[0054] The calibration device 300 uses camera movement 110, LiDAR movement 120, captured image data determined by captured image data ID 130, and three-dimensional point group data obtained at the same time as the captured image data to perform a motion-based calibration calculation.

[0055] The calibration device 300 can calculate the relative position and relative attitude between the camera 20 and the LiDAR 30 through calibration calculations. Furthermore, the calibration device 300 may be able to obtain, for example, captured image data determined by captured image data ID 130 and three-dimensional point group data obtained at the same time as the captured image data from the motion calculation device 200.

[0056] As a calibration operation, the calibration device 300 mainly performs the following operations.

[0057] The calibration device 300 first aligns the coordinate systems of the camera 20 and the LiDAR 30, performing 2D-3D matching. 2D-3D matching is the process of searching for the correspondence between image feature points detected from the image data captured by the camera 20 and the 3D point sets in the 3D point set data of the LiDAR 30.

[0058] Figure 4 An example of the hardware structure of the data processing apparatus 100 of this embodiment is shown.

[0059] Figure 5 An example of the functional structure of the data processing apparatus 100 of this embodiment is shown.

[0060] First, refer to Figure 4 An example of the hardware structure of the data processing device 100 will be described.

[0061] The data processing device 100 in this embodiment is a computer.

[0062] Furthermore, the operating steps of the data processing device 100 are equivalent to a data processing method. Additionally, the program that implements the operation of the data processing device 100 is equivalent to a data processing program.

[0063] As hardware, the data processing device 100 includes a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication device 904.

[0064] In addition, such as Figure 5 As shown, the data processing device 100 has a camera motion acquisition unit 101, a LiDAR motion acquisition unit 102, and a comparison and determination unit 103 as its functional structure.

[0065] The auxiliary storage device 903 stores a program that implements the functions of the camera motion acquisition unit 101, the LiDAR motion acquisition unit 102, and the comparison and determination unit 103.

[0066] These programs are loaded from the auxiliary storage device 903 into the main storage device 902. Then, the processor 901 executes these programs to perform the operations of the camera motion acquisition unit 101, the LiDAR motion acquisition unit 102, and the comparison and determination unit 103, which will be described later.

[0067] exist Figure 3 The diagram schematically shows the state in which the processor 901 is executing a program that implements the functions of the camera motion acquisition unit 101, the LiDAR motion acquisition unit 102, and the comparison and determination unit 103.

[0068] The communication device 904 communicates with the motion calculation device 200 and the calibration device 300.

[0069] Next, refer to Figure 5 The functional structure of the data processing apparatus 100 in this embodiment will be described.

[0070] The camera movement acquisition unit 101 acquires the camera movement 110 and the captured image data ID 130 from the movement calculation device 200. The camera movement acquisition unit 101 outputs the acquired camera movement 110 and captured image data ID 130 to the comparison and determination unit 103.

[0071] The camera motion acquisition unit 101 is equivalent to the first motion acquisition unit. Furthermore, the processing performed by the camera motion acquisition unit 101 is equivalent to the first motion acquisition processing.

[0072] The LiDAR motion acquisition unit 102 acquires the LiDAR motion 120 from the motion calculation device 200. The LiDAR motion acquisition unit 102 outputs the acquired LiDAR motion 120 to the comparison and determination unit 103.

[0073] The LiDAR motion acquisition unit 102 is equivalent to the second motion acquisition unit. Furthermore, the processing performed by the LiDAR motion acquisition unit 102 is equivalent to the second motion acquisition processing.

[0074] The comparison and determination unit 103 compares the camera movement amount 110 with the LiDAR movement amount 120. If the difference between the camera movement amount 110 and the LiDAR movement amount 120 is less than a threshold, it decides to use the camera movement amount 110 in the calibration operation of the calibration device 300.

[0075] Then, the comparison and decision unit 103 outputs the camera movement amount 110, the LiDAR movement amount 120, and the captured image data ID 130 to the calibration device 300.

[0076] The processing performed by the comparison decision unit 103 is equivalent to comparison decision processing.

[0077] ***Instructions for Action***

[0078] Next, refer to Figure 6 An example of the operation of the data processing apparatus 100 in this embodiment will be described.

[0079] In addition, the data processing device 100 repeatedly performs the following operations whenever a shooting cycle or video recording cycle arrives. Figure 6 The actions shown.

[0080] In step S101, the camera movement acquisition unit 101 acquires the camera movement amount 110 and the captured image data ID 130. The camera movement acquisition unit 101 outputs the acquired camera movement amount 110 and captured image data ID 130 to the comparison and determination unit 103.

[0081] Furthermore, in parallel, in step S102, the LiDAR movement acquisition unit 102 acquires the LiDAR movement amount 120. The LiDAR movement acquisition unit 102 outputs the acquired LiDAR movement amount 120 to the comparison and determination unit 103.

[0082] Next, in step S103, the comparison and determination unit 103 compares the camera movement amount 110 and the LiDAR movement amount 120.

[0083] Specifically, the comparison and decision unit 103 compares the estimated rotation amount contained in the camera movement amount 110 with the estimated rotation amount shown in the LiDAR movement amount 120.

[0084] The rotational movement of camera 20 and LiDAR 30 around the rotation vector is equal. Furthermore, the accuracy of the estimated rotational movement of LiDAR 30 is higher than that of camera 20. Therefore, the comparison determination unit 103 sets the LiDAR movement 120 as the positive value and compares the estimated rotational movement contained in camera movement 110 with the estimated rotational movement contained in LiDAR movement 120.

[0085] If the difference between the camera movement 110 and the LiDAR movement 120 is less than a threshold (step S104: Yes), the process proceeds to step S105. On the other hand, if the difference between the camera movement 110 and the LiDAR movement 120 is greater than or equal to the threshold (step S104: No), the process ends.

[0086] In step S105, the comparison and determination unit 103 determines to use the camera movement amount 110 in the calibration operation.

[0087] That is, if the difference between the camera movement 110 and the LiDAR movement 120 is small, the camera movement 110 is considered to have high accuracy. Therefore, the comparison decision unit 103 decides to use the camera movement 110 in the calibration calculation.

[0088] Then, in step S106, the comparison and determination unit 103 outputs the camera movement amount 110, the LiDAR movement amount 120, and the captured image data ID 130 to the calibration device 300.

[0089] The calibration device 300 acquires camera movement 110, LiDAR movement 120, and captured image data ID 130.

[0090] Then, the calibration device 300 outputs the captured image data ID130 to the motion calculation device 200. The motion calculation device 200 outputs the captured image data determined by the captured image data ID130 obtained from the calibration device 300 and the three-dimensional point group data obtained by taking a picture at the same time as the captured image data to the calibration device 300.

[0091] Next, the calibration device 300 performs the following motion-based calibration calculation.

[0092] (1) The calibration device 300 calculates the relative position and relative attitude between the camera 20 and the LiDAR 30 based on the camera movement 110 and the LiDAR movement 120. However, the relative position and relative attitude contain errors, so the calibration device 300 corrects the errors by the following 2D-3D matching.

[0093] (2) The calibration device 300 uses the relative position and relative attitude containing the calculated error to perform coordinate transformation so that the coordinate systems of the camera 20 and the LiDAR 30 are consistent.

[0094] (3) Next, the calibration device 300 draws a straight line connecting the origin of the coordinate system and the feature points within the captured image data. This captured image data is obtained from the motion calculation device 200.

[0095] (4) Furthermore, the calibration device 300 generates a three-dimensional shape (polygonal surface, etc.) based on the three-dimensional point set measured by the LiDAR 30.

[0096] (5) Next, the calibration device 300 determines the intersection point between the straight line connecting the origin and the feature point and the three-dimensional shape. This intersection point is called the 2D-3D corresponding point.

[0097] (6) Furthermore, the calibration device 300 uses 2D-3D corresponding points to convert the feature points in the captured image data from two-dimensional coordinate information to three-dimensional coordinate information.

[0098] (7) In addition, the calibration device 300 calculates the estimated movement of the camera 20 based on the three-dimensional coordinate information obtained by the transformation.

[0099] (8) Then, the calibration device 300 calculates the relative position and relative attitude between the camera 20 and the LiDAR 30 based on the estimated movement of the camera 20 and the LiDAR movement 120 calculated in (7) above.

[0100] (9) Next, the calibration device 300 evaluates the changes in the relative position and relative attitude obtained in the process of (8) relative to the relative position and relative attitude obtained in the process of (1).

[0101] If the change is above the threshold, it may be possible to calculate the relative position and relative attitude with higher accuracy. Therefore, the calibration device 300 uses the relative position and relative attitude calculated in the process of (8) and performs the processes of (2) to (8) again. On the other hand, if the change is less than the threshold, the relative position and relative attitude obtained in the process of (8) are output.

[0102] (10) After performing the above-mentioned processes (2) to (8) again, the calibration device 300 evaluates the changes in the relative position and relative attitude obtained in the final process (8) relative to the relative position and relative attitude obtained in the previous process (8).

[0103] If the change is above the threshold, it may be possible to calculate the relative position and relative attitude with higher accuracy. Therefore, the calibration device 300 uses the relative position and relative attitude calculated in the process of (8) and performs the processes of (2) to (8) again. On the other hand, if the change is less than the threshold, the relative position and relative attitude obtained in the process of (8) are output.

[0104] In this way, the calibration device 300 repeatedly performs the processes (2) to (8), thereby gradually improving the accuracy of relative position and relative attitude.

[0105] ***Explanation of the effects of the implementation method***

[0106] In this embodiment, the data processing device 100 selects the camera movement 110 with higher accuracy by comparing it with the LiDAR movement 120. Then, the calibration device 300 performs calibration calculations using the camera movement 110 selected by the data processing device 100. Therefore, according to this embodiment, accurate calibration calculations using the camera movement 110 can be performed, and thus, accurate relative position and relative attitude can be calculated.

[0107] When the accuracy of the estimated movement of the camera 20 input to the calibration device 300 is low, the error of the corresponding 2D-3D points is large. Therefore, even if the calibration device 300 repeatedly performs the processing (2) to (8), the evaluation results of relative position and relative attitude will not converge, and the calculation accuracy of relative position and relative attitude will not improve.

[0108] In this embodiment, a high-precision camera movement 110 can be used, so the calibration device 300 repeatedly performs the processing (2) to (8), thereby calculating a high-precision relative position and relative attitude.

[0109] Furthermore, the above explanation, using vehicle 10 as an example of a mobile body, can also be applied to other mobile bodies such as ships, aircraft, helicopters, drones, and people.

[0110] Furthermore, the above explanation, using LiDAR30 as an example of a 3D sensor, demonstrates that other types of sensors, such as millimeter-wave sensors and sonar, can be used instead of LiDAR30 for 3D sensing.

[0111] Furthermore, as described above, the data processing device 100 outputs captured image data ID 130 to the calibration device 300, and the calibration device 300 obtains captured image data and 3D point group data from the motion calculation device 200 based on the captured image data ID 130. If the motion calculation device 200 outputs the captured image data and 3D point group data to the data processing device 100 instead of the captured image data ID 130, and the data processing device 100 decides to use camera motion 110 in the calibration calculation, the data processing device 100 may also output camera motion 110, LiDAR motion 120, captured image data, and 3D point group data to the calibration device 300.

[0112] Implementation Method 2

[0113] In this embodiment, the differences from Embodiment 1 will be explained.

[0114] In addition, the matters not described below are the same as in Implementation 1.

[0115] ***Structure Description***

[0116] Figure 7 An example of the structure of the calibration system 1000 of this embodiment is shown.

[0117] exist Figure 7 In, with Figure 1 In comparison, 40 additional control devices were added.

[0118] Control device 40 controls vehicle 10. While controlling vehicle 10, control device 40 synchronously measures the speed and direction of travel of vehicle 10 (with a measurement cycle identical to the shooting cycle of camera 20) with camera 20. Then, control device 40 outputs the measured values ​​of speed and direction of travel angle to motion calculation device 200.

[0119] In this embodiment, the movement calculation device 200 uses measured values ​​of speed and travel direction angle obtained from the control device 40 to calculate the estimated movement of the vehicle 10 within a measurement period (e.g., 50 milliseconds). Hereinafter, the estimated movement of the vehicle 10 calculated using the measured values ​​of speed and travel direction angle will be referred to as the measured value movement 140. The estimation accuracy of the measured value movement 140 is higher than the estimation accuracy of the camera movement 110. Assume that the estimated rotation amount is also included in the measured value movement 140.

[0120] In this embodiment, the measured value shift 140 corresponds to the second shift.

[0121] In this embodiment, the data processing device 100 obtains the camera movement 110, LiDAR movement 120, captured image data ID 130, and measured value movement 140 from the movement calculation device 200.

[0122] The data processing unit 100 compares the camera movement 110 and the measured value movement 140 to determine whether the camera movement 110 is suitable for the calibration calculation of the calibration unit 300. If the camera movement 110 is suitable for the calibration calculation, it decides to use the camera movement 110 in the calibration calculation. Then, the data processing unit 100 outputs the camera movement 110, the LiDAR movement 120, and the captured image data ID 130 to the calibration unit 300.

[0123] The calibration device 300 is the same as that shown in Embodiment 1.

[0124] Alternatively, it can also be done by Figure 2 and Figure 3 The calibration system 1000 of this embodiment is realized by adding a control device 40 to the structure.

[0125] Below, with Figure 7 This explanation is based on the premise of the structure, but through comparison with... Figure 2 and Figure 3 The following instructions also apply to replacements with the same structure. Figure 2 and Figure 3 The structure of the added control device 40 is as follows.

[0126] Figure 8 An example of the functional structure of the data processing apparatus 100 of this embodiment is shown.

[0127] exist Figure 8 In the data processing device 100 shown, instead of Figure 5 The LiDAR motion acquisition unit 102 shown includes a measurement value motion acquisition unit 104.

[0128] The measured value movement acquisition unit 104 acquires the measured value movement 140 from the movement calculation device 200. The measured value movement acquisition unit 104 outputs the acquired measured value movement 140 to the comparison and determination unit 103.

[0129] In this embodiment, the measurement value movement acquisition unit 104 corresponds to the second movement acquisition unit. Furthermore, the processing performed by the measurement value movement acquisition unit 104 corresponds to the second movement acquisition processing.

[0130] In this embodiment, the camera motion acquisition unit 101 acquires the camera motion 110, the LiDAR motion 120, and the captured image data ID 130. The camera motion acquisition unit 101 outputs the acquired camera motion 110, LiDAR motion 120, and captured image data ID 130 to the comparison and determination unit 103.

[0131] The comparison and decision unit 103 compares the camera movement amount 110 with the measured value movement amount 140. If the difference between the camera movement amount 110 and the measured value movement amount 140 is less than a threshold, it decides to use the camera movement amount 110 in the calibration operation of the calibration device 300.

[0132] Then, the comparison and decision unit 103 outputs the camera movement amount 110, the LiDAR movement amount 120, and the captured image data ID 130 to the calibration device 300.

[0133] ***Instructions for Action***

[0134] Next, refer to Figure 9 An example of the operation of the data processing apparatus 100 in this embodiment will be described.

[0135] In addition, the data processing device 100 repeatedly performs these operations whenever a shooting cycle, video recording cycle, or measurement cycle arrives. Figure 9 The actions shown.

[0136] In step S201, the camera movement acquisition unit 101 acquires the camera movement 110, the LiDAR movement 120, and the captured image data ID 130. The camera movement acquisition unit 101 outputs the acquired camera movement 110, LiDAR movement 120, and captured image data ID 130 to the comparison and determination unit 103.

[0137] Furthermore, in parallel, in step S202, the measurement value movement acquisition unit 104 acquires the measurement value movement 140. The measurement value movement acquisition unit 104 outputs the acquired measurement value movement 140 to the comparison and determination unit 103.

[0138] Next, in step S203, the comparison and determination unit 103 compares the camera movement amount 110 with the measured value movement amount 140. More specifically, the comparison and determination unit 103 compares the estimated rotation amount included in the camera movement amount 110 with the estimated rotation amount included in the measured value movement amount 140.

[0139] If the difference between the camera movement 110 and the measured movement 140 is less than the threshold (step S204: Yes), the process proceeds to step S205. On the other hand, if the difference between the camera movement 110 and the measured movement 140 is greater than or equal to the threshold (step S204: No), the process ends.

[0140] In step S205, the comparison and determination unit 103 determines to use the camera movement amount 110 in the calibration operation.

[0141] That is, if the difference between the camera movement 110 and the measured movement 140 is small, the camera movement 110 is considered to be highly accurate. Therefore, the comparison decision unit 103 decides to use the camera movement 110 in the calibration calculation.

[0142] Then, in step S206, the comparison and determination unit 103 outputs the camera movement amount 110, the LiDAR movement amount 120, and the captured image data ID 130 to the calibration device 300.

[0143] The operation of the calibration device 300 is the same as that of Embodiment 1, therefore, the description of the operation of the calibration device 300 is omitted.

[0144] In addition, the hardware structure of the data processing device 100, for example Figure 4 As shown. In this embodiment, instead of the program that implements the function of the LiDAR motion acquisition unit 102, the processor 901 executes the program that implements the function of the measurement value motion acquisition unit 104.

[0145] ***Explanation of the effects of the implementation method***

[0146] According to this embodiment, even when using the measured value movement amount 140, it is possible to calculate the accurate relative position and relative attitude in the same way as in Embodiment 1.

[0147] Implementation Method 3

[0148] In this embodiment, the differences from Embodiment 1 will be explained.

[0149] In addition, the matters not described below are the same as in Implementation 1.

[0150] In this embodiment, the data processing device 100 selects the camera movement 110 to be used in the calibration operation from a plurality of camera movement 110 whose difference from the LiDAR movement 120 is less than a threshold.

[0151] Furthermore, the hardware structure and functional structure of the data processing apparatus 100 in this embodiment are as follows: Figure 4 and Figure 5 As shown.

[0152] Next, refer to Figure 10 An example of the operation of the data processing apparatus 100 in this embodiment will be described.

[0153] Steps S101 to S105 and Figure 6 The same applies as shown, therefore the explanation is omitted.

[0154] Furthermore, in this embodiment, even if the comparison decision unit 103 decides to use the camera movement amount 110 in the calibration calculation, it does not output the camera movement amount 110 to the calibration device 300. For example, the comparison decision unit 103 stores the camera movement amount 110, the LiDAR movement amount 120, and the captured image data ID 130 in an auxiliary storage device 903 in association with each other.

[0155] In step S301, the comparison and determination unit 103 determines whether the number of camera movements 110 used in the calibration calculation has reached a predetermined target number. The target number is any number of 2 or more.

[0156] If the number of camera movements 110 used in the calibration calculation has reached the target number, the process proceeds to step S302. On the other hand, if the number of camera movements 110 used in the calibration calculation has not reached the target number, the comparison and determination unit 103 waits for the camera movement acquisition unit 101 to acquire the camera movement 110 and the captured image data ID 130, and for the LiDAR movement acquisition unit 102 to acquire the LiDAR movement 120.

[0157] In step S302, the comparison and determination unit 103 determines whether there is a camera movement amount 110 that corresponds to the rotational movement centered on the optical axis of the camera 20 among the camera movement amounts 110 that have been determined for use in the calibration calculation.

[0158] If it has been determined that there is a camera movement amount 110 in the camera movement amount 110 used in the calibration calculation that corresponds to a rotational movement centered on the optical axis of the camera 20, the process proceeds to step S303. On the other hand, if it has been determined that there is no camera movement amount 110 in the camera movement amount 110 used in the calibration calculation that corresponds to a rotational movement centered on the optical axis of the camera 20, the process ends.

[0159] In step S303, the comparison and determination unit 103 selects the camera movement amount 110 corresponding to the rotational movement centered on the optical axis of the camera 20 as the object of the calibration calculation. That is, the data processing device 100 selects the camera movement amount 110 with a small translational movement component as the object of the calibration calculation.

[0160] The reduction effect of the translation vector included in the camera movement amount 110 on calibration error is low. That is, the translation vector included in the camera movement amount 110 does not contribute to improving 2D-3D matching accuracy. On the other hand, the camera movement amount 110 corresponding to rotational movement centered on the optical axis of the camera 20 can improve calibration accuracy.

[0161] Therefore, the comparison and determination unit 103 selects the camera movement amount 110 corresponding to the rotational movement centered on the optical axis of the camera 20 as the object of the calibration calculation.

[0162] Specifically, the comparison and decision unit 103 considers the ratio of the lengths of the translation vectors of the camera movement amount 110, and selects a predetermined number of camera movements 110 in each direction of roll, pitch, and yaw according to the order of translation vectors from shortest to longest.

[0163] Next, in step S304, the comparison and determination unit 103 outputs the camera movement amount 110 selected in step S303, the LiDAR movement amount 120 associated with the camera movement amount 110, and the captured image data ID 130 to the calibration device 300.

[0164] In this embodiment, the data processing device 100 selects a camera movement amount 110 that improves calibration accuracy and outputs the selected camera movement amount 110 to the calibration device 300. Therefore, compared with embodiment 1, it is possible to calculate the relative position and relative attitude with high accuracy.

[0165] In addition, in this embodiment, similar to Embodiment 1, the comparison of camera movement 110 and LiDAR movement 120 by the data processing device 100 is described. However, as in Embodiment 2, the data processing device 100 may also compare camera movement 110 and measured value movement 140.

[0166] Implementation Method 4

[0167] In this embodiment, the differences from Embodiment 1 will be explained.

[0168] In addition, the matters not described below are the same as in Implementation 1.

[0169] In this embodiment, vehicle 10 is an AGV (Automatic Guided Vehicle) that serves as a small mobile unit.

[0170] When an offset in the setting position of the camera 20 and / or LiDAR 30 is detected using an impact sensor or similar means, the AGV detects the space required for automatic calibration (hereinafter referred to as the calibration space). Then, the calibration device 300 mounted on the AGV performs calibration calculations using the camera movement amount 110 and LiDAR movement amount 120 obtained by traveling in the calibration space, thereby enabling rapid correction of the offset in the setting position.

[0171] In automated calibration, the camera movement 110 and LiDAR movement 120 during rotational movement are important. Therefore, it is preferable for the AGV to travel along a circular route with rotational movement within the calibration space. In this respect, AGVs are better suited for small-radius turns during travel compared to larger vehicles.

[0172] In this embodiment, the AGV is designed to complete one loop around the route. That is, the AGV sets any point along the loop as its starting point, completes one loop, and returns to the starting point. Since the AGV returns to the starting point, the accuracy of the camera movement 110 calculated based on the image data captured by camera 20 can be evaluated. In this evaluation, the AGV's movement trajectory generated based on the camera movement 110 is considered, and the accuracy of whether the ending position of the movement trajectory returns to the starting point is evaluated.

[0173] Furthermore, in this embodiment, the movement calculation device 200 determines whether the image data captured at the start of travel and the image data captured at the end of travel are consistent. The image data captured at the start of travel is the image data captured by the camera 20 at the point when the AGV begins traveling from the starting position of the loop route. Similarly, the image data captured at the end of travel is the image data captured by the camera 20 at the point when it is determined that the AGV has completed one loop and returned to the starting position, thus ending its travel. If the image data captured at the start of travel and the image data captured at the end of travel are consistent, the movement calculation device 200 uses any one of the multiple image data captured by the camera 20 during the period when the AGV completes one loop to calculate the estimated movement of the AGV as the camera movement 110.

[0174] Alternatively, in this case, an ID tag can be placed at the start of the journey as a marker. By placing an ID tag or other marker at the start of the journey, it becomes easier to compare image data captured at the start of the journey with image data captured at the end of the journey. Furthermore, the AGV's charging equipment can also be set at the start of the journey.

[0175] In addition, the calibration space can be pre-searched before the AGV starts moving, and the location information of the calibration space obtained as a result of the search can be stored in the AGV.

[0176] The above describes embodiments 1 to 4. However, it is also possible to combine two or more of these embodiments.

[0177] Alternatively, a portion of one of these implementation methods may be implemented.

[0178] Alternatively, a portion of two or more of these implementation methods may be combined.

[0179] Furthermore, the structures and steps described in these embodiments can be modified as needed.

[0180] ***Supplementary Explanation of Hardware Structure***

[0181] Finally, a supplementary description of the hardware structure of the data processing device 100 will be provided.

[0182] Figure 4 The processor 901 shown is an IC (Integrated Circuit) that performs the processing.

[0183] The processor 901 includes CPU (Central Processing Unit), DSP (Digital Signal Processor), etc.

[0184] Figure 4 The main storage device 902 shown is RAM (Random Access Memory).

[0185] Figure 4 The auxiliary storage device 903 shown is ROM (Read Only Memory), flash memory, HDD (Hard Disk Drive), etc.

[0186] Figure 4 The communication device 904 shown is an electronic circuit that performs communication processing of data.

[0187] The communication device 904 is, for example, a communication chip or a NIC (Network Interface Card).

[0188] In addition, the auxiliary storage device 903 also stores the OS (Operating System).

[0189] Moreover, at least a portion of the OS is executed by processor 901.

[0190] The processor 901 executes at least a part of the OS while executing a program that implements the functions of the camera motion acquisition unit 101, the LiDAR motion acquisition unit 102, the comparison and determination unit 103, and the measurement value motion acquisition unit 104.

[0191] The processor 901 executes the OS, thereby performing task management, storage management, file management, communication control, and so on.

[0192] Furthermore, at least one of the information, data, signal value, and variable value representing the processing results of the camera motion acquisition unit 101, LiDAR motion acquisition unit 102, comparison and determination unit 103, and measurement value motion acquisition unit 104 is stored in at least one of the registers and cache memory within the main storage device 902, auxiliary storage device 903, and processor 901.

[0193] Furthermore, the program that implements the functions of the camera motion acquisition unit 101, the LiDAR motion acquisition unit 102, the comparison determination unit 103, and the measurement value motion acquisition unit 104 can also be stored on portable recording media such as disks, floppy disks, optical disks, high-density disks, Blu-ray discs, and DVDs. Moreover, the portable recording medium storing the program that implements the functions of the camera motion acquisition unit 101, the LiDAR motion acquisition unit 102, the comparison determination unit 103, and the measurement value motion acquisition unit 104 can be circulated.

[0194] Alternatively, the word "section" in the camera movement acquisition section 101, LiDAR movement acquisition section 102, comparison determination section 103, and measurement value movement acquisition section 104 can be replaced with "circuit", "process", "step", "processing", or "circuit".

[0195] Furthermore, the data processing device 100 can also be implemented using processing circuits. Examples of processing circuits include integrated circuits (ICs), gate arrays (GAs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0196] In this case, the camera motion acquisition unit 101, the LiDAR motion acquisition unit 102, the comparison and determination unit 103, and the measurement value motion acquisition unit 104 are each implemented as part of the processing circuit.

[0197] In addition, in this specification, the higher-level concept of processor and processing circuit is referred to as "processing circuit".

[0198] That is, the processor and the processing circuit are specific examples of "processing lines".

[0199] Label Explanation

[0200] 10: Vehicle; 20: Camera; 30: LiDAR; 40: Control device; 100: Data processing device; 101: Camera motion acquisition unit; 102: LiDAR motion acquisition unit; 103: Comparison and decision unit; 104: Measured value motion acquisition unit; 110: Camera motion; 120: LiDAR motion; 130: Captured image data ID; 140: Measured value motion; 200: Motion calculation device; 300: Calibration device; 901: Processor; 902: Main storage device; 903: Auxiliary storage device; 904: Communication device; 1000: Calibration system.

Claims

1. A data processing apparatus, comprising: The first movement amount acquisition unit acquires the estimated movement amount of the moving body calculated using the captured image data obtained by the imaging device installed on the moving body as the first movement amount; The second movement amount acquisition unit, synchronously with the first movement amount acquisition unit acquiring the first movement amount, acquires an estimated movement amount of the moving body with a higher estimation accuracy than the first movement amount as the second movement amount; and The comparison and decision unit compares the first movement amount with the second movement amount, and if the difference between the first movement amount and the second movement amount is less than a threshold, it decides to use the first movement amount in the calibration operation.

2. The data processing apparatus according to claim 1, wherein, The second movement amount acquisition unit acquires the estimated movement amount of the moving body calculated using three-dimensional sensor data obtained by a three-dimensional sensor installed on the moving body as the second movement amount.

3. The data processing apparatus according to claim 2, wherein, The first movement amount acquisition unit acquires a first movement amount that includes the estimated rotation amount of the moving body calculated using the captured image data. The second movement acquisition unit acquires a second movement amount that includes the estimated rotation amount of the moving body calculated using the three-dimensional sensor data. The comparison and determination unit compares the estimated rotation amount of the moving body included in the first movement amount with the estimated rotation amount of the moving body included in the second movement amount. If the difference between the estimated rotation amount of the moving body included in the first movement amount and the estimated rotation amount of the moving body included in the second movement amount is less than a threshold, it decides to use the first movement amount in the calibration operation.

4. The data processing apparatus according to claim 1, wherein, The second movement amount acquisition unit acquires the estimated movement amount of the moving body calculated using the measurement value measured by the control device installed on the moving body as the second movement amount.

5. The data processing apparatus according to claim 1, wherein, The first movement amount acquisition unit repeatedly acquires the first movement amount. The second movement amount acquisition unit repeatedly acquires the second movement amount. Whenever the first movement amount acquisition unit acquires the first movement amount and the second movement amount acquisition unit acquires the second movement amount, the comparison and determination unit repeatedly compares the first movement amount with the second movement amount. If, as a result of repeatedly comparing the first movement amount with the second movement amount, there are multiple first movement amounts that are determined to be used in the calibration operation, a first movement amount corresponding to the rotational movement centered on the optical axis of the imaging device is selected from the multiple first movement amounts.

6. The data processing apparatus according to claim 1, wherein, If the difference between the first movement amount and the second movement amount is less than a threshold, the comparison determination unit decides to use the first movement amount in the calibration calculation of the motion-based method for calculating the relative position and relative attitude between the three-dimensional sensor set on the moving body and the imaging device.

7. The data processing apparatus according to claim 1, wherein, The moving body is an automated guided vehicle, or AGV. The first movement amount acquisition unit is connected to a movement amount calculation device that calculates the first movement amount, and acquires the first movement amount calculated by the movement amount calculation device. The movement calculation device determines whether the image data captured at the start of travel and the image data captured at the end of travel are consistent. The image data captured at the start of travel is the image data acquired by the capturing device when the AGV begins traveling from the starting position of the loop route. The image data captured at the end of travel is the image data acquired by the capturing device when it is determined that the AGV has completed one loop around the loop route and returned to the starting position, thus ending its travel. If the image data captured at the start of the journey is consistent with the image data captured at the end of the journey, the estimated movement of the AGV is calculated as the first movement amount using any one of the multiple image data captured by the capturing device during the period when the AGV travels around the loop route.

8. A data processing method, wherein, The computer obtains the estimated movement of the moving body calculated using image data captured by a camera device installed on the moving body, as the first movement amount. Simultaneously with obtaining the first movement amount, the computer obtains an estimated movement amount of the moving body with a higher estimation accuracy than the first movement amount, as a second movement amount. The computer compares the first movement amount with the second movement amount, and if the difference between the first movement amount and the second movement amount is less than a threshold, it decides to use the first movement amount in the calibration operation.

9. A computer-readable recording medium containing a data processing program that causes a computer to perform the following processes: The first movement amount acquisition process acquires the estimated movement amount of the moving body calculated using the captured image data obtained by the imaging device installed on the moving body as the first movement amount; The second movement amount acquisition process, synchronously with the acquisition of the first movement amount in the first movement amount acquisition process, acquires an estimated movement amount of the moving body with a higher estimation accuracy than the first movement amount as the second movement amount; and The comparison decision process compares the first movement amount with the second movement amount, and if the difference between the first movement amount and the second movement amount is less than a threshold, it is decided to use the first movement amount in the calibration operation.

Images