A CALIBRATION METHOD
Patent Information
- Authority / Receiving Office
- TR · TR
- Patent Type
- Applications
- Current Assignee / Owner
- FORD OTOMOTIV SANAYI ANONIM SIRKETI
- Filing Date
- 2024-12-03
- Publication Date
- 2026-06-22
Smart Images
Figure 00000013_0000 
Figure 00000014_0000 
Figure 00000015_0000
Abstract
Description
8343.1167 1 TARIFF A CALIBRATION METHOD Relevant Technical Field 5 The present invention focuses on the calibration of cameras and lidar sensors, particularly in autonomous vehicles. It relates to a method for carrying out the process. Previous Tech 10 Autonomous vehicles can move from one location to another without the need for an active driver. These are vehicles that can move. In such autonomous vehicles, the vehicle moves safely. to be able to do so, it must overcome the obstacles around it (other vehicles, pedestrians and other things on the road). (obstacles) perception and movement in a way that avoids contact with the aforementioned obstacles 15 It is necessary to ensure control. Therefore, obstacles around the vehicle Various sensor structures are used for detection. One of the sensors that enables the vehicle to detect obstacles around it is like a camera. It is an image sensor. The cameras mentioned are used to capture images of the vehicle's surroundings. 20 It is used for this purpose. However, since the images obtained with cameras are 2-dimensional, The distance of an obstacle detected by the camera is only precisely determined by the camera image. It cannot be detected in any way. Therefore, in obstacle detection applications, Sensors called lidar are also used to detect the distance of obstacles. Lidar (light detection and ranging) is used. 25 sensors use light to determine the position of an obstacle with very high precision and accuracy. It can obtain this data. For this reason, especially in autonomous vehicles, camera data from lidar is crucial. By using sensor data in conjunction with other methods, the vehicle can detect obstacles in its surroundings. With location information, it can be identified with great precision. 8343.1167 2 For camera and lidar sensor data to be used together, camera and lidar Determining the spatial positions and orientations of the sensors relative to each other. This is necessary. The process of determining the location and orientation is calibration. This process is called and the accuracy of this process depends on the continuous operation of autonomous vehicles. It is vital for it to be operational. Therefore, the aforementioned 5 a method for performing the calibration process precisely and accurately It is needed. Brief Description of the Invention With the present invention, a sensing system that includes at least one camera and at least one lidar sensor is possible. A calibration method suitable for use in the system is described. The calibration method is based on different positions of at least one calibration target. Data collection from the camera and lidar sensor; from the collected data, the aforementioned The data pair of the camera and lidar sensor belonging to a location of the calibration target is 15. selection; the camera and lidar sensor are aligned with each other according to the selected data pair. determining its position and orientation; the determined position and orientation Calculating an error value based on the information; the calculated error value is a Comparison with the first threshold value; calculated error is lower than the first threshold value. with the determined location and orientation information being defined as the final value 20 Calibration process completed; calculated error lower than the first threshold value. Otherwise, return to the data peer selection step and proceed with the steps to select a new data peer. It includes. In the calibration method developed with the present invention, 25 calibration targets are selected from a calibration target. Using data obtained via camera and lidar sensors, the camera and lidar The position and orientation information of the sensors relative to each other is obtained. Here, location and orientation information obtained using a data peer is also shown. The error value is calculated and it is determined whether this error value is lower than a first threshold value. Since this is not the case, a decision must be made regarding the reliability of the determined location and orientation information. 30 is provided. If the determined location and orientation information is not reliable enough, 8343.1167 3 Calibration process with a different data match in order to obtain more reliable information. This process is repeated, thereby increasing the reliability of the calibration process. Purpose of the Invention The aim of the present invention is to improve the calibration process of the camera and lidar sensor. The goal is to develop a method for its implementation. Another objective of the present invention is to analyze the internal parameters of the camera and lidar sensor. Even if not entirely accurate, the calibration process is reliable. 10 The goal is to develop a method for its implementation. Another objective of the present invention is to develop a reliable method. Figure Description 15 Application examples of the calibration method developed with the present invention are included in the attached figures. These figures are shown as follows: Figure 1 is a flowchart of the developed calibration method. 20 Figure 2 shows the data collection step in the developed calibration method. It is a flowchart of the details. Figure 3 shows a position determination step in the developed calibration method. It is a flowchart of the details. The parts in the figures are individually numbered, and the corresponding numbers correspond to these numbers. given below: Data collection (101) Data peer selection (102) Determining location and orientation (103) 8343.1167 4 Error value calculation (104) Comparison of the error value with a first threshold value (105) Completion of the calibration process (106) Commencement of data collection (200) Positioning of the calibration target (201) Obtaining data from the calibration target (202) Is the calibration target visible in the camera image? checking that it is not there (203) The points are densely distributed at the calibration target. checking that it has not been distributed (204) The scope of the calibration target with a second threshold value comparison (205) Completion of data collection process (206) Starting the position and orientation determination process (300) Determining the corners of the calibration target in the image (301) Vertices of the calibration target in the point set determination (302) Determination of the plane of the calibration target (303) Calculation of calibration result (304) Completion of the position and orientation determination process (305) Description of the Invention For autonomous vehicles to move safely, their surroundings must be protected. They need to be able to detect obstacles along with distance information. Therefore, 5 The vehicles mentioned use sensors such as cameras and lidar. The obstacle here... During the detection process, data from the camera and lidar sensor are combined. For it to be usable, the camera and lidar must be in the correct position and orientation relative to each other. (Orientation) information is needed. Therefore, with the present invention, the aforementioned A calibration method for obtaining position and orientation data 10 It has been developed. 8343.1167 Calibration developed with the present invention and sample flow diagrams are given in Figures 1-3. the method, in a detection system that includes at least one camera and at least one lidar sensor, For example, it is suitable for use in the sensing system of an autonomous vehicle. (mentioned) The calibration method uses 5 different locations of at least one calibration target. Data collection from the camera and lidar sensor (101); from the collected data, Data from a camera and lidar sensor at a location of the mentioned calibration target. Selection of the data partner (102); camera and lidar sensor in line with the selected data partner Determining the position and orientation of each other (103); determined Calculation of an error value based on location and orientation information (104); 10 Comparison of the calculated error value with a first threshold value (105); calculated If the error is lower than the first threshold value, the specified position and orientation information is used. Completion of the calibration process by defining it as the final value (106); If the calculated error is not lower than the first threshold value, proceed to the data peer selection (102) step. It includes the steps of returning and selecting a new data peer (102). 15 The data collection (101) step in a preferred application of the invention is to collect data. initiation (200); at least one calibration target for data collection is a random positioning (201); via camera and lidar sensor obtaining data from the positioned calibration target (202) and image data and 20 Obtaining a set of location points; completing the data collection process (206) It includes the following steps. In this application, at the selected location of the calibration target... Image data detected by the camera and data detected by the lidar sensor The point set is used as a data pair for the relevant location. In this application, the data The collection (101) step is followed by the data acquisition (202) step from the calibration target 25 calibration target in the camera image (image taken via camera) checking whether it is (203); calibration target camera image If it does not remain within, the positioning of the calibration target (201) step the steps of turning around and positioning the calibration target at a different point (201) It includes. The calibration target is located at a position visible to the camera 30°. When not available, image data captured via the camera is useful for calibration. 8343.1167 6 Since there will be no data, when such a situation is encountered, usable data must be obtained. In order to achieve this, the position of the calibration target is changed. Similarly data collection (101) step, after data acquisition from calibration target (202) step (checking whether the calibration target is in the camera image (203)) (This may be before or after step 5) Calibration of points in the point set Checking whether it is heavily distributed in its target (204); point If the points in the cluster are not densely distributed in the calibration target, Calibration by returning to step (201) of positioning the calibration target This includes the steps of repositioning its target at a different point (201). In practice, the points taken from the calibration target are sufficiently dense. If not distributed, the point set is insufficient for the calibration process. Since this will not be the case, the location of the calibration target is being changed. The mentioned data The collection (101) step also includes the acquisition of data from the calibration target (202) step. then (checking if the calibration target is in the camera image) (203) and / or points are densely distributed in the calibration target 15 Checking that it has not been distributed may be before or after step (204) Comparison of the scope of the calibration target with a second threshold value (205); If the scope of the calibration target falls below the second threshold value, calibration is not performed. By returning to the step of positioning the target (201), the calibration target is different. It includes the steps of positioning at the point (201). In this application, 20 Whether the scope of the calibration target remains below the second threshold value, camera and / or in the data obtained via lidar sensors, how much of the calibration target is present It is determined by what the image or point is taken from. For example, camera. less than 40% of the calibration target image in the image If this is the case, it can be determined that the second threshold value has not been exceeded. 25 Similarly, 40% of the calibration target points can be identified via the lidar sensor. If a value below such a level is perceived, then the second threshold value is also considered. It can be determined that it has not been exceeded. Thus, both in the camera image and in In the point set received via the lidar sensor, there is sufficient information regarding the calibration target. The availability of information is guaranteed. The data collection in question is (101) 30 step (sub-steps given above) for at least one other location of the calibration target. 8343.1167 7 This is repeated at least once. Thus, the calibration process is performed for different locations. Data pairs are obtained for use. The step of determining location and orientation (103) in a preferred application of the invention, Starting the position and orientation determination process (300); 5 in the image in the data pair Determining the corners of the calibration target (301); point set in the data pair Determination of the vertices of the calibration target (302); vertex information in the point set and the plane of the calibration target according to the dimensions of the calibration target determination (303); corner information in the image, corner information in the point set and The camera and lidar sensor are calibrated using the plane of the calibration target. calibration result containing information on their relative position and orientation calculation (304); completion of the position and orientation determination process (305) It includes the steps. Location and orientation information determined in a preferred application of the invention 15 In the step of calculating an error value in line with (104), the mentioned location and Reprojection error using orientation information The calculated re-reflection error is the camera's reflection error in three-dimensional space. It is converted into its image. The re-reflection error mentioned here, The center coordinates of the calibration target, in the point set and camera 20 It is the difference in value between the positions in the image (for example, in the form of pixel values). In a sample application, the error value mentioned is different from the re-reflection error. as the error value given on the camera image in three-dimensional space It is the conversion of a value at a specified distance to its actual value, according to the following formula: is being calculated. 25 Error = tan HFOV Square Width ×Re-reflection error ×10 Here, “HFOV” refers to the camera's horizontal field of view; “Square "Width" is defined as the width of the camera image. 30 8343.1167 8 In a sample application of the invention, the calibration method described is used in an autonomous vehicle. Obtaining the position and orientation information of the camera and lidar sensor relative to each other. It is used for calibration. Here, a checkerboard is used for the calibration process in question. A calibration target that can be a board is used. In the data collection step (101), 5 The calibration board is placed at different points in front of the camera and lidar sensor. by positioning cameras and lidar sensors at each location A point set is obtained through this method. Here, the checkerboard is used as the calibration target. Thanks to the use of a whiteboard, corners, edges, and light / dark colored squares are shown in the resulting image. Lidar 10 enables easy and reliable identification of areas such as these. A point set is obtained by collecting point data from these regions via a sensor. This can be done. After data pairs are created in this way for different locations, Calibration using camera images and point sets on a selected dataset. The process is being carried out. The mentioned calibration (position and orientation) In the determination (103)) process, the corner and plane information of the checkerboard is obtained from both camera 15 in both the image and the point set, for example, plane fitting They are determined separately using the camera image and the point set. By comparing this determined corner and plane information, three The relative positions of the camera and lidar sensor in a three-dimensional plane and orientation is determined (103). Afterwards, in line with these determined values, 20 an error value is calculated (104) the calculated error value is a first threshold value is compared (105). The error calculated as a result of the comparison is one first If the result is that it is below the threshold value, the determined position and orientation This information is stored in memory, for example, as final values for the camera and lidar sensor. is recorded and the calibration process is completed (106). The calculated error is 25 If it is not lower than the first threshold value, the data match used in the calibration process For example, the data may not be sufficiently reliable due to internal errors in the camera and lidar sensor. It was determined that there was no matching data pair, and the calibration process was repeated using a new data pair. This is done by adjusting the internal parameters of the camera and lidar sensor. Even if not entirely accurate, the calibration process can be reliably performed. 30 This is ensured. 8343.1167 9 In the calibration method developed with the present invention, from a calibration target Using data obtained via camera and lidar sensors, the camera and lidar The position and orientation information of the sensors relative to each other is obtained. Here, we also show that location and orientation information obtained using a data peer is 5. The error value is calculated and it is determined whether this error value is lower than a first threshold value. Since this is not the case, a decision must be made regarding the reliability of the determined location and orientation information. is provided. If the determined location and orientation information is not reliable enough, Calibration process with a different data match in order to obtain more reliable information. This process is repeated. This increases the reliability of the calibration process. 10
Claims
8343.1167 REQUESTS 1. In a detection system that includes at least one camera and at least one lidar sensor. It is a calibration method suitable for use and its characteristic is; - camera and lidar 5 according to different locations of at least one calibration target. Data collection from the sensor (101); - from the collected data, belonging to a location of the aforementioned calibration target. Selection of the data match of the camera and lidar sensor (102); - the camera and lidar sensor connect to each other based on the selected data pair. Determining the position and orientation of the relative (103); 10 - an error value based on the specified location and orientation information. calculation (104); - Comparison of the calculated error value with a first threshold value (105); - If the calculated error is lower than the first threshold value, the specified location and Calibration 15 by defining orientation information as the final value. completion of the process (106); - Data peer selection if the calculated error is not lower than the first threshold value. (102) Returning to step (102) and selecting a new data peer (102) It includes the steps.
2. A calibration method that complies with Claim 1 and its feature is data collection (101) step, start of data collection (200); at least one calibration for data collection positioning of the target at a random location (201); camera and lidar Obtaining data from the calibration target positioned by means of the sensor (202) Obtaining image data and a set of location points; data collection process 25 its completion includes (206) steps.
3. A calibration method that complies with Claim 2 and its feature is data collection (101) calibration after step of obtaining data from calibration target (202) Checking whether the target is in the camera image (203); 30 If the calibration target does not remain within the camera image, then the calibration target... 8343.1167 11 By returning to step (201) of positioning, the calibration target is different. positioning at the point includes the steps (201).
4. A calibration method that complies with claim 2 or 3, and whose characteristic is data collection. Step (101) is point 5 after step (202) receiving data from the calibration target. the points in the cluster are densely distributed in the calibration target checking that it is not distributed (204); calibration of the points in the point set If the target is not densely distributed, the calibration target By returning to step (201) of positioning, the calibration target is different. It includes the steps of positioning at the point (201). 10 5. A calibration method that conforms to any of claims 2 to 4, and whose characteristic is; data the collection (101) step is the acquisition of data from the calibration target (202) step then comparing the scope of the calibration target with a second threshold value (205); if the scope of the calibration target remains below the second threshold value 15 Calibration by returning to step (201) of positioning the calibration target its objective is to position it at a different point (201) and include the steps.
6. A calibration method that conforms to any of claims 2 to 5, and whose characteristic is; data The collection step (101) must be at least 20 for at least one other location of the calibration target. It is a one-time repetition.
7. Is there a calibration method that meets any of the above requirements? feature; determining location and orientation in the application (103) step, location and Starting the orientation determination process (300); 25 in the image on the data pair Determining the corners of the calibration target (301); point set in the data pair Determination of the vertices of the calibration target (302); vertices in the point set information and the plane of the calibration target according to the dimensions of the calibration target determination (303); corner information in the image, corner information in the point set and The camera and lidar sensor are 30 using the plane of the calibration target. calibration result containing information on their relative position and orientation 8343.1167 12 calculation (304); completion of the position and orientation determination process (305) includes the steps.
8. Is there a calibration method that meets any of the above requirements? Its distinguishing feature is the use of a checkerboard as a calibration target. 5