Sensor external parameter calibration method, device and equipment and storage medium

Abstract

The embodiment of the invention provides a sensor external parameter calibration method and device, equipment and a storage medium. The method comprises the steps of determining a first relative pose according to a pre-acquired first binocular camera picture of a target carrier sent by a calibration detector, and determining a second relative pose according to a pre-acquired second binocular camera picture of a to-be-calibrated vehicle sent by the calibration detector, and determining a third relative pose based on a predetermined three-dimensional coordinate of a target point of the target carrier relative to the target coordinate system and a two-dimensional coordinate of the target point relative to the sensor to be calibrated, and obtaining a calibration result through pose transfer conversion according to the first relative pose, the second relative pose and the third relative pose. According to the method, the calibration precision is improved, the calibration efficiency is enhanced, the adaptability is high, and the automatic driving safety is also improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a calibration method, device, equipment and storage medium for sensor extrinsic parameters. Background Art

[0002] In an autonomous driving system, sensors are key components for sensing the surrounding environment. External parameter calibration can determine the position and orientation of the sensor relative to the vehicle coordinate system, thereby ensuring the accuracy of the sensor data. If the sensor external parameter calibration is inaccurate, the data collected by the sensor will not accurately reflect the environmental information around the vehicle, which will affect the decision-making and driving safety of the autonomous driving system.

[0003] The static calibration methods of common sensors such as cameras and LiDARs currently require the establishment of a calibration room. However, the establishment of a calibration room requires a site, a aligner, and high-precision mapping of the target, which is expensive and inflexible to implement. Some dynamic calibration solutions for cameras and LiDARs can only calibrate the angle information in the external parameters, but not the position information.

[0004] In summary, how to reduce calibration time and errors while ensuring calibration accuracy is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0005] The embodiments of the present application provide a sensor external parameter calibration method, device, equipment and storage medium, which are used to solve the problem of how to reduce calibration time and errors while ensuring calibration accuracy.

[0006] In a first aspect, an embodiment of the present application provides a method for calibrating an external parameter of a sensor, comprising:

[0007] Determine a first relative posture according to a first binocular camera picture of the target carrier sent by the calibration detector and acquired in advance, wherein the first relative posture is the posture of the target carrier relative to the calibration detector;

[0008] Determine a second relative posture according to a second binocular camera picture of the vehicle to be calibrated sent by the calibration detector and acquired in advance, wherein the second relative posture is the posture of the vehicle to be calibrated relative to the calibration detector;

[0009] Based on the predetermined three-dimensional coordinates of the target point of the target carrier relative to the target coordinate system and the two-dimensional coordinates of the target point relative to the sensor to be calibrated, a third relative posture is determined, where the third relative posture is the posture of the target carrier relative to the sensor to be calibrated;

[0010] According to the first relative posture, the second relative posture and the third relative posture, a calibration result is obtained through posture transfer conversion; the calibration result is the posture of the sensor to be calibrated relative to the vehicle to be calibrated.

[0011] In a possible implementation, determining the first relative position and posture according to a first binocular camera image of the target carrier sent by a calibration detector acquired in advance includes:

[0012] Performing feature point detection on the first binocular camera image to obtain at least one pair of feature point pairs;

[0013] For each pair of feature points, a parallax method is used to calculate the depth information of each feature point in the feature point pair;

[0014] Based on the depth information of each feature point in all feature point pairs and the internal parameters of the calibration detector, the three-dimensional coordinates of each feature point are calculated;

[0015] According to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector are calculated;

[0016] The first relative position and posture is determined based on the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector.

[0017] In a possible implementation, determining the second relative posture according to the pre-acquired second binocular camera image of the vehicle to be calibrated and sent by the calibration detector includes:

[0018] For each picture in the second binocular camera pictures, detection is performed through a wheel detection network to obtain contour information of the front wheels and the rear wheels of the vehicle to be calibrated in each picture;

[0019] Based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each picture, generating point cloud information of the front wheels and point cloud information of the rear wheels;

[0020] Performing plane fitting based on the point cloud information of the front wheels and the point cloud information of the rear wheels to obtain a normal vector of the detected surface of the vehicle to be calibrated relative to the calibration detector;

[0021] Calculating the center coordinates of the rear axle of the vehicle to be calibrated according to the point cloud information of the rear wheel, the normal vector and the width of the vehicle to be calibrated;

[0022] Based on the normal vector and the rear axle center coordinates, the second relative posture is determined.

[0023] In a possible implementation, the calculating, based on the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector includes:

[0024] Perform 3D straight line fitting on the 3D coordinates of each feature point to determine the position of each edge;

[0025] Based on the position of each edge, the intersection of the edges is obtained to obtain the vertex position information of the target carrier;

[0026] According to the directions of any three edges, the vertex position information and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector are calculated;

[0027] Two edges are extracted to form a plane, a normal vector is calculated for the plane, and the direction of the target carrier relative to the calibration detector is determined.

[0028] In a possible implementation manner, generating the point cloud information of the front wheels and the point cloud information of the rear wheels based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each picture includes:

[0029] Determine depth information of pixel points in the contour information of the front wheel and pixel points in the contour information of the rear wheel according to the contour information of the front wheel and the contour information of the rear wheel by using a parallax method;

[0030] Based on the contour information of the front wheel, the contour information of the rear wheel and the depth information, point cloud information of the front wheel and point cloud information of the rear wheel are determined.

[0031] In a possible implementation, the method further includes:

[0032] Based on the second binocular camera image and the vehicle width detection network, the vehicle width of the vehicle to be calibrated is determined.

[0033] In a possible implementation, the method further includes:

[0034] Based on the first relative posture and the parameter information of the target carrier, the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated are calculated;

[0035] The coordinates of the vertex relative to the sensor to be calibrated are geometrically transformed to obtain the three-dimensional coordinates of the target point relative to the target coordinate system.

[0036] In a second aspect, an embodiment of the present application provides a sensor extrinsic parameter calibration device, comprising:

[0037] A first determination module is used to determine a first relative posture according to a first binocular camera picture of a target carrier sent by a calibration detector and acquired in advance, wherein the first relative posture is a posture of the target carrier relative to the calibration detector;

[0038] A second determination module is used to determine a second relative posture according to a second binocular camera picture of the vehicle to be calibrated sent by the calibration detector and acquired in advance, wherein the second relative posture is the posture of the vehicle to be calibrated relative to the calibration detector;

[0039] A third determination module is used to determine a third relative posture based on the predetermined three-dimensional coordinates of the target point of the target carrier relative to the target coordinate system and the two-dimensional coordinates of the target point relative to the sensor to be calibrated, wherein the third relative posture is the posture of the target carrier relative to the sensor to be calibrated;

[0040] A conversion module is used to obtain a calibration result through posture transfer conversion according to the first relative posture, the second relative posture and the third relative posture, wherein the calibration result is the posture of the sensor to be calibrated relative to the vehicle to be calibrated.

[0041] In a possible implementation manner, the first determining module is specifically configured to:

[0042] Performing feature point detection on the first binocular camera image to obtain at least one pair of feature point pairs;

[0043] For each pair of feature points, a parallax method is used to calculate the depth information of each feature point in the feature point pair;

[0044] Based on the depth information of each feature point in all feature point pairs and the internal parameters of the calibration detector, the three-dimensional coordinates of each feature point are calculated;

[0045] According to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector are calculated;

[0046] The first relative position and posture is determined based on the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector.

[0047] In a possible implementation manner, the second determining module is specifically configured to:

[0048] For each picture in the second binocular camera pictures, detection is performed through a wheel detection network to obtain contour information of the front wheels and the rear wheels of the vehicle to be calibrated in each picture;

[0049] Based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each picture, generating point cloud information of the front wheels and point cloud information of the rear wheels;

[0050] Performing plane fitting based on the point cloud information of the front wheels and the point cloud information of the rear wheels to obtain a normal vector of the detected surface of the vehicle to be calibrated relative to the calibration detector;

[0051] Calculating the center coordinates of the rear axle of the vehicle to be calibrated according to the point cloud information of the rear wheel, the normal vector and the width of the vehicle to be calibrated;

[0052] Based on the normal vector and the rear axle center coordinates, the second relative posture is determined.

[0053] In a possible implementation, the first determination module calculates the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector according to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, specifically including:

[0054] Perform 3D straight line fitting on the 3D coordinates of each feature point to determine the position of each edge;

[0055] Based on the position of each edge, the intersection of the edges is obtained to obtain the vertex position information of the target carrier;

[0056] According to the directions of any three edges, the vertex position information and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector are calculated;

[0057] Two edges are extracted to form a plane, a normal vector is calculated for the plane, and the direction of the target carrier relative to the calibration detector is determined.

[0058] In a possible implementation manner, the second determination module generates point cloud information of the front wheels and point cloud information of the rear wheels based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each picture, specifically including:

[0059] Determine depth information of pixel points in the contour information of the front wheel and pixel points in the contour information of the rear wheel according to the contour information of the front wheel and the contour information of the rear wheel by using a parallax method;

[0060] Based on the contour information of the front wheel, the contour information of the rear wheel and the depth information, point cloud information of the front wheel and point cloud information of the rear wheel are determined.

[0061] In a possible implementation manner, the second determining module may also be used to:

[0062] Based on the second binocular camera image and the vehicle width detection network, the vehicle width of the vehicle to be calibrated is determined.

[0063] In a possible implementation manner, the device further includes:

[0064] A calculation module, configured to calculate the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated based on the first relative posture and the parameter information of the target carrier;

[0065] The transformation module is used to perform a geometric transformation on the coordinates of the vertex relative to the sensor to be calibrated to obtain the three-dimensional coordinates of the target point relative to the target coordinate system.

[0066] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0067] The memory stores computer-executable instructions;

[0068] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0069] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0070] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0071] The sensor external parameter calibration method, device, equipment and storage medium provided in the embodiment of the present application determine the first relative posture according to the first binocular camera picture of the target carrier sent by the calibration detector in advance, determine the second relative posture according to the second binocular camera picture of the vehicle to be calibrated sent by the calibration detector in advance, determine the third relative posture based on the three-dimensional coordinates of the target point of the predetermined target carrier relative to the target coordinate system, and the two-dimensional coordinates of the target point relative to the sensor to be calibrated, and obtain the calibration result through posture transfer conversion according to the first relative posture, the second relative posture and the third relative posture. The above method improves the calibration accuracy, enhances the calibration efficiency, has strong adaptability, and also improves the safety of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0073] Figure 1 A schematic diagram of a scenario for the calibration method of sensor extrinsic parameters provided in this application;

[0074] Figure 2 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 1 ;

[0075] Figure 3 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 2 ;

[0076] Figure 4 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 3 ;

[0077] Figure 5 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 4 ;

[0078] Figure 6 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 5 ;

[0079] Figure 7 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 6 ;

[0080] Figure 8 Schematic diagram of the target vector;

[0081] Fig. 9 A schematic diagram of the structure of the sensor external parameter calibration device provided in this application;

[0082] Fig.10A schematic diagram of the structure of the electronic device provided in this application.

[0083] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0084] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0085] In the autonomous driving system, sensors are key components for sensing the surrounding environment. External parameter calibration can determine the position and orientation of the sensor relative to the vehicle coordinate system, thereby ensuring the accuracy of the sensor data. If the sensor external parameter calibration is inaccurate, the data collected by the sensor will not accurately reflect the environmental information around the vehicle, which will affect the decision-making and driving safety of the autonomous driving system. Currently, the static calibration methods of common sensors, such as cameras and lidars, mostly require the establishment of a calibration room. The method of establishing a calibration room requires a site, a straightener, and high-precision mapping of the target, which is expensive and inflexible to implement. Some dynamic calibration schemes for cameras and lidars can only calibrate the angle information in the external parameters, but not the position information. Other calibration methods that use lidar detectors as an aid require the use of multi-frame lidar point cloud information to calibrate the car and target and perform three-dimensional reconstruction, and then calculate the position relationship between the sensor and the target, the target and the detector, and the target and the vehicle body through the position relationship between the sensor and the target, the position relationship between the target and the detector, and the position relationship between the target and the vehicle body. However, using a moving LiDAR detector for three-dimensional reconstruction will accumulate large motion errors, causing relative posture errors between the LiDAR detector and the vehicle body. This error will be introduced into the calibration result error, affecting the calibration result and increasing the calibration time. In addition, direct matching of the LiDAR point cloud with the target pattern is prone to inaccurate matching, resulting in errors between the LiDAR detector and the target, which will also be introduced into the calibration result.

[0086] In response to the above problems, the present application provides a calibration method, device, equipment and storage medium for sensor external parameters, which realizes flexible field sensor calibration, and can not only calibrate angle information, but also calibrate position information, avoiding errors between devices and improving calibration accuracy. Specifically, the existing static calibration methods mostly require the establishment of a calibration room, and the method of establishing a calibration room requires a site, a straightener, and high-precision mapping of the target, which is expensive and inflexible to implement. The dynamic calibration scheme can only calibrate the angle information in the external parameters, but cannot calibrate the position information. In addition, some calibration methods assisted by laser radar detectors need to use multi-frame laser radar point cloud information to calibrate the car and target and perform three-dimensional reconstruction, and then calculate the position relationship between the sensor and the target, the position relationship between the target and the detector, and the position relationship between the target and the vehicle body. However, using a moving laser radar detector for three-dimensional reconstruction will accumulate large motion errors, thereby causing a relative posture error between the laser radar detector and the vehicle body. This error will be introduced into the calibration result error, affecting the calibration result and increasing the calibration time. In addition, direct matching of the laser radar point cloud with the target pattern is prone to inaccurate matching, resulting in an error between the laser radar detector and the target, which will also be introduced into the calibration result. In view of these problems, the inventor studied whether it is possible to configure a binocular camera on the calibration detector, set a target carrier, and then use the calibration detector to directly detect the target carrier, thereby obtaining the vertices of the target carrier, and then calculate the coordinates of the target point relative to the calibration detector based on the vertices of the target carrier, thereby eliminating the detection error between the target pattern and the detector. At the same time, it avoids the calibration of fixed scenes, improves the flexibility of calibration, does not require a mobile detector, reduces the calibration time, and improves the accuracy and efficiency of calibration. Based on this, the technical solution of the present application is proposed.

[0087] Figure 1 A schematic diagram of a scenario of a sensor extrinsic parameter calibration method provided in this application, such as Figure 1 As shown, the scene at least includes a target carrier-1, a target pattern-2, a sensor to be calibrated-3, a vehicle to be calibrated-4, a main controller-5, an electronic device-6, and a calibration detector-7. Among them, the target carrier-1 is a regular hexahedron, the target pattern-2 is on the target carrier-1, and each pattern is a square with equal side lengths.

[0088] The main controller-5 is the controller of the vehicle-4 to be calibrated, including a computing unit and a wireless communication module, which can realize data collection and calculation of the sensor-3 to be calibrated, and can transmit the calculation results to the electronic device-6 through the wireless module. The calibration detector-7 is a binocular camera, which is connected to the electronic device-6 by wire; the electronic device-6 can be any electronic device with data receiving and sending and data calculation capabilities, such as a server, a laptop computer, a desktop computer, a cloud server platform, etc. The sensor-3 to be calibrated can be a sensor such as a laser radar, a camera or a millimeter wave radar.

[0089] It should be noted that the above-mentioned calibration vehicle-4 is only an example, and the calibration method of sensor extrinsic parameters provided in this application can be used to calibrate sensors on any device to be calibrated.

[0090] The method provided in the present application can be applied to the above-mentioned scenario. The calibration detector-7 detects the positions of the vehicle to be calibrated-4 and the target carrier-1 respectively. It does not rely on surveying and mapping, does not require a specific site, does not need to move the calibration detector-7, and eliminates the detection error of the target, thereby achieving high-precision angle and position calibration.

[0091] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0092] Figure 2 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0093] S201: Determine a first relative posture according to a first binocular camera image of a target carrier sent by a calibration detector and acquired in advance.

[0094] In this step, in order to improve the efficiency and flexibility of calibration, calibrate accurate angle information and position information, avoid errors between the calibration detector and the vehicle to be calibrated, and reduce the time of calibration, the target carrier can be set in advance, and a binocular camera can be used to detect the target carrier to determine the first relative posture, that is, the posture of the target carrier relative to the calibration detector.

[0095] Specifically, feature point detection is performed on the first binocular camera image to obtain at least one pair of feature point pairs. For each pair of feature point pairs, the parallax method is used to calculate the depth information of each feature point in the feature point pair. Based on the depth information of each feature point in all feature point pairs and the internal parameters of the calibration detector, the three-dimensional coordinates of each feature point are calculated. According to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector are calculated. Based on the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector, the first relative posture is determined.

[0096] S202: Determine a second relative posture according to a second binocular camera image of the vehicle to be calibrated sent by the calibration detector and acquired in advance.

[0097] In order to improve the calibration accuracy, a binocular camera configured on the calibration detector can be used to take a second binocular camera picture of the vehicle to be calibrated, and then the posture of the vehicle to be calibrated relative to the calibration detector, that is, the second relative posture, can be determined through parallax method, plane fitting and clustering.

[0098] Specifically, for each image in the second binocular camera image, detection is performed through the wheel detection network to obtain the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each image, based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each image, generate point cloud information of the front wheels and the point cloud information of the rear wheels, perform plane fitting based on the point cloud information of the front wheels and the point cloud information of the rear wheels, obtain the normal vector of the detected surface of the vehicle to be calibrated relative to the calibration detector, calculate the rear axle center coordinates of the vehicle to be calibrated according to the point cloud information of the rear wheels, the normal vector and the vehicle width of the vehicle to be calibrated, and determine the second relative posture based on the normal vector and the rear axle center coordinates.

[0099] S203: Determine a third relative posture based on the predetermined three-dimensional coordinates of the target point of the target carrier relative to the target coordinate system and the two-dimensional coordinates of the target point relative to the sensor to be calibrated.

[0100] In this step, the first relative posture is determined through the previous steps. Then, based on the three-dimensional coordinates of the target point in the target coordinate system and the two-dimensional coordinates of the target point relative to the sensor to be calibrated, a 3D-2D calibration solution method, such as solvePnP, can be used to obtain the posture of the target carrier relative to the sensor to be calibrated, that is, the third relative posture.

[0101] Specifically, based on the first relative posture and parameter information of the target carrier, the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated can be calculated, and the coordinates of the vertices relative to the sensor to be calibrated can be geometrically transformed to obtain the three-dimensional coordinates of the target point relative to the target coordinate system. Feature point detection is performed on the image of the target carrier captured by the sensor to be calibrated to obtain the 2D position of the target point.

[0102] According to the camera imaging principle, a mapping relationship between a 3D point set and a 2D point set is established. This involves the intrinsic parameters of the camera (such as focal length, optical center, etc.) and distortion parameters. The rotation and translation parameters are solved by minimizing the reprojection error (i.e., the difference between the point projected onto the image plane after the estimated pose transformation of the 3D point and the actual 2D point). For example, an iterative method (such as the Levenberg-Marquardt algorithm) can be used to solve it. Then the third relative pose is obtained.

[0103] For example, in addition to the solvePnP method described above, a direct linear transformation (DLT) can be used to solve a linear equation system to estimate the transformation matrix, thereby realizing the motion transformation of the point pair. It should be noted that at least 4 point pairs are required to calculate the transformation matrix.

[0104] The position and attitude of the camera can also be estimated by the P3P method, given three pairs of 3D space points and corresponding 2D image points. It is necessary to establish a set of equations and solve the camera's pose parameters through an iterative method (such as the Levenberg-Marquardt algorithm).

[0105] Or through EPnP (Efficient PnP), which is an efficient camera pose solution algorithm used to estimate the position and pose of the camera from known 3D points and corresponding 2D image points. Compared with the traditional PnP method, EPnP has higher computational efficiency and accuracy. Using a linear solution method, the pose estimation problem is transformed into a solution to an overdetermined linear equation system, and the accuracy is further improved through iterative optimization strategies.

[0106] In addition to the above methods, some other methods, for example, feature-based methods (such as SIFT, SURF and other feature matching algorithms combined with RANSAC and other outlier removal algorithms), optimization-based methods (such as Bundle Adjustment, etc.), etc., can also be used to solve the pose based on 3D-2D point correspondence.

[0107] S204: According to the first relative posture, the second relative posture and the third relative posture, a calibration result is obtained through posture transfer conversion.

[0108] In this step, after obtaining the first relative posture, the second relative posture and the third relative posture, the posture transfer conversion can be performed using the following formula to obtain the posture of the sensor to be calibrated relative to the vehicle to be calibrated, that is, the calibration result of the sensor to be calibrated.

[0109]

[0110] in, represents the first relative pose, represents the third relative posture, represents the second relative pose, and Indicates the calibration result.

[0111] In the above formula, is the posture transfer equation, which means that the posture from the calibration detector to the sensor to be calibrated can be obtained through a series of relative posture transformations. Therefore, the calibration result can be obtained based on the first relative posture, the second relative posture and the third relative posture through the posture transfer equation.

[0112] The sensor external parameter calibration method provided in the embodiment of the present application determines the first relative posture according to the first binocular camera picture of the target carrier sent by the calibration detector in advance, determines the second relative posture according to the second binocular camera picture of the vehicle to be calibrated sent by the calibration detector in advance, determines the third relative posture based on the three-dimensional coordinates of the target point of the predetermined target carrier relative to the target coordinate system, and the two-dimensional coordinates of the target point relative to the sensor to be calibrated, and obtains the calibration result through posture transfer conversion according to the first relative posture, the second relative posture and the third relative posture. The above method improves the calibration accuracy, enhances the calibration efficiency, has strong adaptability, and also improves the safety of autonomous driving.

[0113] Figure 3 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 2 ,like Figure 3 As shown, based on the above embodiment, step S201 includes:

[0114] S301: Perform feature point detection on a first binocular camera image to obtain at least one pair of feature point pairs.

[0115] In this step, after the left and right pictures are taken by the binocular camera and sent to the data processing device, feature points of the two pictures are extracted, and the extracted feature points are matched to find the corresponding feature points in the two pictures to form feature point pairs.

[0116] Exemplarily, features can be extracted by the ORB feature point detection algorithm. For example, the ORB algorithm is applied to each image to detect feature points. The ORB algorithm combines the FAST (Features from Accelerated Segment Test) feature point detector and the BRIEF (Binary Robust Independent Elementary Features) descriptor, and adds directional information to improve the stability and matching performance of the feature points. For each detected feature point, its descriptor is calculated. The descriptor is a binary string that represents the local image information of the feature point for matching in subsequent steps. A descriptor matching algorithm (such as brute force matching BFMatcher or a more efficient approximate nearest neighbor matching algorithm) is used to find the corresponding feature points in the left and right images. The matching process is based on the similarity measure between the descriptors, usually using the Hamming distance (for the BRIEF descriptor) or the Euclidean distance. Since mismatches may occur during the matching process, the matching results need to be screened. Common screening methods include using a ratio test (Lowe's ratio test) or a RANSAC (Random Sample Consensus) algorithm to remove mismatched points. After screening, a set of high-quality matching point pairs are obtained, which correspond to the same physical points in the left and right pictures. These point pairs constitute at least one pair of feature point pairs, which is the basis for subsequent stereo matching, depth estimation or 3D reconstruction.

[0117] It should be noted that before feature point matching, it is usually necessary to calibrate the binocular camera to obtain the camera's internal parameters (such as focal length, optical center, etc.) and external parameters (such as the relative position and posture between cameras). Before feature point detection, the image can also be preprocessed, such as denoising, grayscale or histogram equalization, to improve the stability and accuracy of feature point detection.

[0118] Different matching strategies and parameter settings can be selected according to specific application scenarios and requirements. For example, if fast matching is required, a brute force matching algorithm can be selected; if high-precision matching is required, a more complex matching algorithm and screening strategy can be used.

[0119] S302: For each pair of feature points, a parallax method is used to calculate the depth information of each feature point in the feature point pair.

[0120] S303: Based on the depth information of each feature point in all feature point pairs and the internal parameters of the calibration detector, the three-dimensional coordinates of each feature point are calculated.

[0121] After obtaining at least one pair of feature points, the binocular vision parallax method is used to calculate the depth information of each feature point in the two images, and then the three-dimensional coordinates of each feature point are calculated.

[0122] Specifically, for each pair of matched feature points, the difference in their horizontal coordinates in the left and right images (i.e., disparity) is calculated. The size of the disparity is inversely proportional to the depth of the feature point (i.e., the distance from the camera): the larger the disparity, the closer the feature point is to the camera; the smaller the disparity, the farther the feature point is from the camera. Based on the internal and external parameters and disparity information of the binocular camera, the three-dimensional coordinates of the feature point can be calculated using the triangulation principle. The triangulation principle is based on the properties of similar triangles, and the three-dimensional coordinates are obtained by calculating the intersection of the projection light of the feature point in the left and right images and the baseline (i.e., the line connecting the optical centers of the two cameras).

[0123] S304: According to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector are calculated.

[0124] S305: Determine a first relative posture based on the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector.

[0125] After obtaining the three-dimensional coordinates of each feature point, the coordinates and direction of the center of the target carrier relative to the calibration detector can be determined based on the three-dimensional coordinates of each feature point and the parameter information of the target carrier, thereby determining the first relative posture.

[0126] Specifically, a three-dimensional straight line fitting is performed on the three-dimensional coordinates of each feature point to determine the position of each edge. Based on the position of each edge, the intersection of the edges is obtained to obtain the vertex position information of the target carrier. According to the directions of any three edges, the vertex position information and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector are calculated, and two edges are extracted to form a plane. The normal vector of the plane is calculated to determine the direction of the target carrier relative to the calibration detector.

[0127] The pose usually includes two parts: position and orientation. The position is represented by three-dimensional coordinates, and the orientation is represented by rotation matrix, Euler angle or quaternion, etc. Then the coordinates of the center of the target carrier relative to the calibration detector can be directly used as the position information of the first relative pose, and the normal vector can be used as the orientation information.

[0128] The sensor external parameter calibration method provided in the embodiment of the present application performs feature point detection on the first binocular camera image to obtain at least one pair of feature point pairs. For each pair of feature point pairs, the parallax method is used to calculate the depth information of each feature point in the feature point pair. Based on the depth information of each feature point in all feature point pairs and the internal parameters of the calibration detector, the three-dimensional coordinates of each feature point are calculated. According to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector are calculated. Based on the coordinates of the center of the target carrier relative to the calibration and the direction of the target carrier relative to the calibration detector, the first relative posture is determined. Through the above method, the calibration accuracy is improved and the detection error between the target pattern and the detector is eliminated.

[0129] Figure 4 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 3 ,like Figure 4 As shown, based on the above embodiment, step S202 includes:

[0130] S401: For each image in the second binocular camera image, a wheel detection network is used to detect and obtain the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each image.

[0131] The second binocular camera image includes two left and right images. The left and right images are preprocessed, including denoising, grayscale, edge detection and other steps to improve the accuracy of wheel detection. Appropriate preprocessing methods and parameters can also be selected according to the characteristics of the wheel and the background environment.

[0132] Use a trained wheel detection network (such as a deep learning model) to perform wheel detection on the preprocessed image. The network will identify and mark the wheel outlines in the image, including the front wheel outline and the rear wheel outline.

[0133] Specific information of the front wheel profile and the rear wheel profile, such as the coordinate points and shape parameters of the profile, is extracted from the output of the wheel detection network.

[0134] S402: Based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each image, point cloud information of the front wheels and point cloud information of the rear wheels are generated.

[0135] In this step, after the contour information of the front wheels and the contour information of the rear wheels are obtained, the point cloud information of the front wheels and the point cloud information of the rear wheels are determined for accurate calibration.

[0136] Specifically, the parallax method is used to determine the depth information of the pixel points in the contour information of the front wheel and the pixel points in the contour information of the rear wheel according to the contour information of the front wheel and the contour information of the rear wheel, and the point cloud information of the front wheel and the point cloud information of the rear wheel are determined based on the contour information of the front wheel, the contour information of the rear wheel and the depth information.

[0137] S403: performing plane fitting based on the point cloud information of the front wheels and the point cloud information of the rear wheels to obtain a normal vector of the detected surface of the vehicle to be calibrated relative to the calibration detector.

[0138] S404: Calculate the center coordinates of the rear axle of the vehicle to be calibrated according to the point cloud information of the rear wheels, the normal vector and the width of the vehicle to be calibrated.

[0139] S405: Determine a second relative posture based on the normal vector and the rear axle center coordinate.

[0140] In order to accurately obtain the position and posture of the vehicle to be calibrated relative to the calibration detector, after obtaining the point cloud information of the front wheel and the rear wheel of the vehicle to be calibrated, the plane is formed by plane fitting of the point clouds of the two wheels, and the normal vector of the calibration detector relative to the detected surface of the vehicle to be calibrated is obtained through the plane. Then, according to the point cloud information of the rear wheels, the normal vector and the width of the vehicle to be calibrated, the rear axle center coordinates of the vehicle to be calibrated are calculated by the following formula.

[0141]

[0142] Among them, PW(x PW ,y PW ,z PW ) represents the point cloud center coordinates of the rear wheels, d is the width of the vehicle to be calibrated, PO(x PO ,y PO ,z PO ) represents the center coordinates of the rear axle of the vehicle to be calibrated.

[0143] It should be noted that the coordinates of the rear axle center of the vehicle to be calibrated are the origin of the vehicle body. In applications such as autonomous driving, vehicle navigation, and parking assistance, knowing the exact position of the vehicle is crucial. The origin of the vehicle body can be used as a reference point for the vehicle's position to help the system accurately determine the vehicle's current position and posture.

[0144] The coordinates of the rear axle center represent a key point of the vehicle. Combined with the normal vector, the attitude of the vehicle to be calibrated in the coordinate system of the calibration detector can be determined (such as pitch, yaw, etc., but the roll angle can usually be assumed to be 0 by the verticality of the wheels unless there is additional information).

[0145] Optionally, the vehicle width of the vehicle to be calibrated may be determined based on the second binocular camera image and the vehicle width detection network.

[0146] Specifically, collect vehicle pictures of different models, angles and lighting conditions, and manually annotate these pictures to mark the actual width of the vehicle. These data will be used for model training. Use deep learning frameworks (such as TensorFlow, PyTorch, etc.) to build an initial vehicle width detection network and train it with the prepared data set. During the training process, the network parameters should be continuously adjusted to optimize the detection performance. Use a test data set independent of the training set to test and verify the trained initial vehicle width detection network. By comparing the network's prediction results with the actual annotation values, the network's detection accuracy and generalization ability are evaluated. Finally, the vehicle width detection network is obtained.

[0147] Use a calibrated binocular camera to take pictures of the vehicle to be calibrated. During the shooting process, ensure that the vehicle is stationary and the distance and angle between the camera and the vehicle are appropriate to obtain a clear vehicle image.

[0148] The captured vehicle image is input into the trained vehicle width detection network, and the network will output the predicted width of the vehicle. This step can be achieved using the inference function of the deep learning framework.

[0149] Verify the prediction results of the vehicle width detection network. This can be done by comparing with vehicles of known width, analyzing the rationality of the prediction results, etc. If there is a large deviation between the prediction results and the actual situation, it may be necessary to recalibrate the camera or adjust the parameters of the vehicle width detection network.

[0150] The calibration method of sensor external parameters provided in the embodiment of the present application is to detect each image in the second binocular camera image through the wheel detection network to obtain the contour information of the front wheel and the contour information of the rear wheel of the vehicle to be calibrated in each image, generate the point cloud information of the front wheel and the point cloud information of the rear wheel based on the contour information of the front wheel and the contour information of the rear wheel of the vehicle to be calibrated in each image, perform plane fitting based on the point cloud information of the front wheel and the point cloud information of the rear wheel, obtain the normal vector of the detected surface of the vehicle to be calibrated relative to the calibration detector, calculate the rear axle center coordinates of the vehicle to be calibrated based on the point cloud information of the rear wheel, the normal vector and the vehicle width of the vehicle to be calibrated, and determine the second relative posture based on the normal vector and the rear axle center coordinates. Through the above method, the calibration accuracy is improved, the error is reduced, the calibration efficiency is enhanced, and the adaptability is improved.

[0151] Figure 5 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 4 ,like Figure 5 As shown, based on the above embodiment, step S304 includes:

[0152] S501: Perform three-dimensional straight line fitting on the three-dimensional coordinates of each feature point to determine the position of each edge.

[0153] S502: Based on the position of each edge, the intersection of the edges is obtained to obtain the vertex position information of the target carrier.

[0154] S503: Calculate the coordinates of the center of the target carrier relative to the calibration detector according to the directions of any three edges, vertex position information and parameter information of the target carrier.

[0155] S504: extract two edges to form a plane, calculate the normal vector of the plane, and determine the direction of the target carrier relative to the calibration detector.

[0156] In order to accurately determine the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector, after determining the feature points, three-dimensional straight line fitting is performed on the feature points to obtain the position of the edge, and then the vertex position information of the target carrier is obtained based on the position of the edge, and the center coordinates of the target carrier are calculated according to the edge direction, vertex position, and the side length of the prior cube in the parameter information of the calibration carrier.

[0157] The plane is formed by extracting two edges, and the normal vector is calculated to determine the direction of the target carrier relative to the calibration detector.

[0158] Specifically, commonly used three-dimensional straight line fitting methods include the least squares method, the RANSAC (random sampling consensus) algorithm, and the like. The least squares method finds the best straight line equation by minimizing the sum of the squares of the distances from the point to the straight line. The RANSAC algorithm fits the straight line by iteratively selecting random points and evaluating the quality of the fit to find the best fit result. Once the straight line is fitted, the position of the edge can be determined. The position of the edge can be represented by the equation or parameters of the straight line. In practical applications, the fitted straight line can also be combined with the geometric properties of the target carrier to more accurately determine the position and direction of the edge. The intersection points between the edge straight lines obtained by fitting can be calculated. These intersection points correspond to the vertex positions of the target carrier.

[0159] Select two adjacent edges on the cube (i.e. two sets of matching feature points) and use these points to fit a plane. Optionally, the plane can be fitted by solving a least squares plane that contains these points. Once the plane is fitted, its normal vector can be calculated. The normal vector is a vector perpendicular to the plane and can be solved by the plane's equation.

[0160] In three-dimensional space, the equation of a plane can be expressed as Ax+By+Cz+D=0, where (A, B, C) is the normal vector of the plane.

[0161] The direction of a cube can be determined by the normal vector of one of its faces. Since a cube has three perpendicular faces, the normal vector of one of the faces can be selected as the direction vector of the cube.

[0162] The sensor external parameter calibration method provided in the embodiment of the present application performs three-dimensional straight line fitting on the three-dimensional coordinates of each feature point to determine the position of each edge. Based on the position of each edge, the intersection of the edges is obtained to obtain the vertex position information of the target carrier. According to the directions of any three edges, the vertex position information and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector are calculated, and two edges are extracted to form a plane. The normal vector of the plane is calculated to determine the direction of the target carrier relative to the calibration detector. This method realizes the precise positioning and orientation of the target carrier relative to the calibration detector through multiple steps such as three-dimensional straight line fitting, edge intersection solution, target carrier center coordinate calculation and plane normal vector determination.

[0163] Figure 6 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 5 ,like Figure 6 As shown, based on the above embodiment, step S402 includes:

[0164] S601: Determine depth information of pixel points in the contour information of the front wheel and pixel points in the contour information of the rear wheel according to the contour information of the front wheel and the contour information of the rear wheel using a parallax method.

[0165] S602: Based on the profile information of the front wheels, the profile information of the rear wheels and the depth information, determine the point cloud information of the front wheels and the point cloud information of the rear wheels.

[0166] In order to more accurately determine the position and posture of the vehicle to be calibrated relative to the calibration detector, the point cloud information of the front wheels and rear wheels of the vehicle to be calibrated can be calculated.

[0167] Specifically, after obtaining the contour information of the front wheel and the rear wheel, for each pixel in the contour information of the front wheel and the rear wheel, find the corresponding matching point in the left and right pictures of the second binocular camera image, and calculate the difference between the horizontal coordinates of the matching point in the left and right pictures, that is, the disparity. According to the principle of triangulation, using the known camera focal length, baseline width (that is, the distance between the two camera lenses) and the calculated disparity, the three-dimensional coordinates of each pixel can be reconstructed to obtain depth information, and each pixel in the contour information of the front wheel and the rear wheel is converted into a point in three-dimensional space in combination with its depth information. By combining and arranging the converted three-dimensional points according to their positions in space, the point cloud information of the front wheel and the rear wheel is constructed. These point cloud information contains key information such as the shape, size and position of the wheel.

[0168] The sensor extrinsic parameter calibration method provided in the embodiment of the present application adopts the parallax method to determine the depth information of the pixel points in the front wheel contour information and the pixel points in the rear wheel contour information according to the contour information of the front wheel and the contour information of the rear wheel, and determines the point cloud information of the front wheel and the point cloud information of the rear wheel based on the contour information of the front wheel, the contour information of the rear wheel and the depth information. The above method realizes the accurate acquisition of depth information, the high-quality generation of point cloud information, and the improvement of the accuracy and efficiency of three-dimensional reconstruction.

[0169] Figure 7 Schematic diagram of the process of calibrating the sensor external parameters provided in this application Figure 6 ,like Figure 7 As shown, based on the above embodiment, the method further includes:

[0170] S701: Based on the first relative posture and parameter information of the target carrier, the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated are calculated.

[0171] S702: Performing a geometric transformation on the coordinates of the vertex relative to the sensor to be calibrated to obtain the three-dimensional coordinates of the target point relative to the target coordinate system.

[0172] After determining the position and posture of the target carrier relative to the calibration detector, that is, the first relative position and posture, in order to more accurately determine the position and posture of the target carrier relative to the sensor to be calibrated, the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated can be calculated based on the position and posture of the target carrier relative to the calibration detector and the parameter information of the target carrier, and then after geometric changes, the three-dimensional coordinates of the target point relative to the target coordinate system are obtained.

[0173] Specifically, Figure 8 is a schematic diagram of the target vector, such as Figure 8 As shown, based on the first relative posture and the parameter information of the target carrier, the coordinates of the four vertices of the target surface can be calculated: W_p1, W_p2, W_p3, W_p4. It is known that the target carrier is a regular hexahedron, the target pattern is a 3x3 square pattern, and the sides of each square pattern are equal. Based on the almost relationship, the three-dimensional coordinates of the target point relative to the target coordinate system are calculated by the following formula:

[0174]

[0175] The sensor extrinsic parameter calibration method provided in the embodiment of the present application calculates the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated based on the first relative position and parameter information of the target carrier, performs a geometric transformation on the coordinates of the vertices relative to the sensor to be calibrated, and obtains the three-dimensional coordinates of the target point relative to the target coordinate system. Through the above method, the positioning accuracy is improved, the flexibility of the geometric transformation is enhanced, the data processing process is optimized, and the system robustness is improved.

[0176] Fig. 9 The schematic diagram of the structure of the calibration device for the sensor external parameters provided in this application is as follows: Fig. 9 As shown, the sensor external parameter calibration device 900 specifically includes:

[0177] The first determination module 901 is used to determine a first relative posture according to a first binocular camera picture of the target carrier sent by the calibration detector that is acquired in advance, and the first relative posture is the posture of the target carrier relative to the calibration detector.

[0178] The second determination module 902 is used to determine a second relative posture according to a second binocular camera image of the vehicle to be calibrated sent by the calibration detector and acquired in advance, where the second relative posture is the posture of the vehicle to be calibrated relative to the calibration detector.

[0179] The third determination module 903 is used to determine a third relative posture based on the predetermined three-dimensional coordinates of the target point of the target carrier relative to the target coordinate system and the two-dimensional coordinates of the target point relative to the sensor to be calibrated. The third relative posture is the posture of the target carrier relative to the sensor to be calibrated.

[0180] The conversion module 904 is used to obtain a calibration result through posture transfer conversion according to the first relative posture, the second relative posture and the third relative posture. The calibration result is the posture of the sensor to be calibrated relative to the vehicle to be calibrated.

[0181] In a possible implementation, the first determining module 901 is specifically configured to:

[0182] Perform feature point detection on the first binocular camera image to obtain at least one pair of feature point pairs;

[0183] For each pair of feature points, the parallax method is used to calculate the depth information of each feature point in the feature point pair;

[0184] Based on the depth information of each feature point in all feature point pairs and the internal parameters of the calibration detector, the three-dimensional coordinates of each feature point are calculated;

[0185] According to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector are calculated;

[0186] A first relative posture is determined based on the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector.

[0187] In a possible implementation, the second determining module 902 is specifically configured to:

[0188] For each image in the second binocular camera image, the wheel detection network is used to detect and obtain the contour information of the front wheel and the contour information of the rear wheel of the vehicle to be calibrated in each image;

[0189] Based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each image, point cloud information of the front wheels and point cloud information of the rear wheels are generated;

[0190] Perform plane fitting based on the point cloud information of the front wheels and the point cloud information of the rear wheels to obtain the normal vector of the detected surface of the vehicle to be calibrated relative to the calibration detector;

[0191] The center coordinates of the rear axle of the vehicle to be calibrated are calculated based on the point cloud information of the rear wheels, the normal vector and the width of the vehicle to be calibrated;

[0192] Based on the normal vector and the rear axle center coordinate, the second relative pose is determined.

[0193] In a possible implementation, the first determination module 901 calculates the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector according to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, specifically including:

[0194] Perform 3D straight line fitting on the 3D coordinates of each feature point to determine the position of each edge;

[0195] Based on the position of each edge, the intersection of the edges is obtained to obtain the vertex position information of the target carrier;

[0196] According to the directions of any three edges, vertex position information and parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector are calculated;

[0197] Two edges are extracted to form a plane, the normal vector of the plane is calculated, and the direction of the target carrier relative to the calibration detector is determined.

[0198] In a possible implementation, the second determination module 902 generates point cloud information of the front wheels and point cloud information of the rear wheels based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each picture, specifically including:

[0199] Using a parallax method, according to the contour information of the front wheel and the contour information of the rear wheel, depth information of the pixel points in the contour information of the front wheel and the pixel points in the contour information of the rear wheel are determined;

[0200] Based on the profile information of the front wheels, the profile information of the rear wheels, and the depth information, point cloud information of the front wheels and point cloud information of the rear wheels are determined.

[0201] In a possible implementation, the second determining module 902 may also be used to:

[0202] Based on the second binocular camera image and the vehicle width detection network, the vehicle width of the vehicle to be calibrated is determined.

[0203] In a possible implementation, the sensor extrinsic parameter calibration device 900 further includes:

[0204] The calculation module 905 is used to calculate the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated based on the first relative posture and parameter information of the target carrier.

[0205] The transformation module 906 is used to perform geometric transformation on the coordinates of the vertex relative to the sensor to be calibrated, so as to obtain the three-dimensional coordinates of the target point relative to the target coordinate system.

[0206] The sensor extrinsic parameter calibration device provided in this embodiment can execute the sensor extrinsic parameter calibration method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0207] Fig.10 This is a schematic diagram of the structure of the electronic device provided in this application. Fig.10 As shown, the electronic device 1000 provided in this embodiment includes: at least one processor 1001 and a memory 1002. Optionally, the device 1000 also includes a communication component 1003. The processor 1001, the memory 1002 and the communication component 1003 are connected via a bus 1004.

[0208] In a specific implementation process, at least one processor 1001 executes the computer-executable instructions stored in the memory 1002, so that at least one processor 1001 executes the above method.

[0209] The specific implementation process of the processor 1001 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0210] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0211] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0212] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0213] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0214] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0215] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0216] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0217] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0218] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0220] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0221] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0222] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for calibrating external parameters of a sensor, characterized in that: include: Determine a first relative posture according to a first binocular camera picture of the target carrier sent by the calibration detector and acquired in advance, wherein the first relative posture is the posture of the target carrier relative to the calibration detector; Determine a second relative posture according to a second binocular camera picture of the vehicle to be calibrated sent by the calibration detector and acquired in advance, wherein the second relative posture is the posture of the vehicle to be calibrated relative to the calibration detector; Based on the predetermined three-dimensional coordinates of the target point of the target carrier relative to the target coordinate system and the two-dimensional coordinates of the target point relative to the sensor to be calibrated, a third relative posture is determined, where the third relative posture is the posture of the target carrier relative to the sensor to be calibrated; According to the first relative posture, the second relative posture and the third relative posture, a calibration result is obtained through posture transfer conversion, and the calibration result is the posture of the sensor to be calibrated relative to the vehicle to be calibrated.

2. The method according to claim 1, characterized in that The determining of the first relative posture according to the first binocular camera picture of the target carrier sent by the calibration detector in advance comprises: Performing feature point detection on the first binocular camera image to obtain at least one pair of feature point pairs; For each pair of feature points, a parallax method is used to calculate the depth information of each feature point in the feature point pair; Based on the depth information of each feature point in all feature point pairs and the internal parameters of the calibration detector, the three-dimensional coordinates of each feature point are calculated; According to the three-dimensional coordinates of each feature point and the parameter information of the target carrier, the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector are calculated; The first relative position and posture is determined based on the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector.

3. The method according to claim 1, characterized in that The determining of the second relative posture according to the second binocular camera picture of the vehicle to be calibrated sent by the calibration detector in advance comprises: For each picture in the second binocular camera pictures, detection is performed through a wheel detection network to obtain contour information of the front wheels and the rear wheels of the vehicle to be calibrated in each picture; Based on the contour information of the front wheels and the contour information of the rear wheels of the vehicle to be calibrated in each picture, generating point cloud information of the front wheels and point cloud information of the rear wheels; Performing plane fitting based on the point cloud information of the front wheels and the point cloud information of the rear wheels to obtain a normal vector of the detected surface of the vehicle to be calibrated relative to the calibration detector; Calculating the center coordinates of the rear axle of the vehicle to be calibrated according to the point cloud information of the rear wheel, the normal vector and the width of the vehicle to be calibrated; Based on the normal vector and the rear axle center coordinates, the second relative posture is determined.

4. The method according to claim 2, characterized in that: The step of calculating the coordinates of the center of the target carrier relative to the calibration detector and the direction of the target carrier relative to the calibration detector according to the three-dimensional coordinates of each feature point and the parameter information of the target carrier comprises: Perform 3D straight line fitting on the 3D coordinates of each feature point to determine the position of each edge; Based on the position of each edge, the intersection of the edges is obtained to obtain the vertex position information of the target carrier; Calculate the coordinates of the center of the target carrier relative to the calibration detector according to the directions of any three edges, the vertex position information and the parameter information of the target carrier; Two edges are extracted to form a plane, a normal vector is calculated for the plane, and the direction of the target carrier relative to the calibration detector is determined.

5. The method according to claim 3, characterized in that: The generating the point cloud information of the front wheel and the point cloud information of the rear wheel based on the contour information of the front wheel and the contour information of the rear wheel of the vehicle to be calibrated in each picture includes: Determine depth information of pixel points in the contour information of the front wheel and pixel points in the contour information of the rear wheel according to the contour information of the front wheel and the contour information of the rear wheel by using a parallax method; Based on the contour information of the front wheel, the contour information of the rear wheel, and the depth information, point cloud information of the front wheel and point cloud information of the rear wheel are determined.

6. The method according to claim 3 or 5, characterized in that: The method further comprises: Based on the second binocular camera image and the vehicle width detection network, the vehicle width of the vehicle to be calibrated is determined.

7. The method according to claim 1, characterized in that The method further comprises: Based on the first relative posture and the parameter information of the target carrier, the coordinates of the vertices of the target surface of the target carrier relative to the sensor to be calibrated are calculated; The coordinates of the vertex relative to the sensor to be calibrated are geometrically transformed to obtain the three-dimensional coordinates of the target point relative to the target coordinate system.

8. A sensor external parameter calibration device, characterized in that: include: A first determination module is used to determine a first relative posture according to a first binocular camera picture of a target carrier sent by a calibration detector and acquired in advance, wherein the first relative posture is a posture of the target carrier relative to the calibration detector; A second determination module is used to determine a second relative posture according to a second binocular camera picture of the vehicle to be calibrated sent by the calibration detector and acquired in advance, wherein the second relative posture is the posture of the vehicle to be calibrated relative to the calibration detector; A third determination module is used to determine a third relative posture based on the predetermined three-dimensional coordinates of the target point of the target carrier relative to the target coordinate system and the two-dimensional coordinates of the target point relative to the sensor to be calibrated, wherein the third relative posture is the posture of the target carrier relative to the sensor to be calibrated; A conversion module is used to obtain a calibration result through posture transfer conversion according to the first relative posture, the second relative posture and the third relative posture, wherein the calibration result is the posture of the sensor to be calibrated relative to the vehicle to be calibrated.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the sensor extrinsic parameter calibration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the sensor external parameter calibration method according to any one of claims 1 to 7.

Images