External parameter calibration method and device, electronic equipment and storage medium

Abstract

The invention provides an external parameter calibration method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring multi-view camera image data and multi-view projection image data; for a target view angle in the multiple view angles, based on a user interaction mode, according to target projection image data corresponding to the target view angle and target camera image data corresponding to the target view angle, a feature point pair is selected, and the selected feature point pair comprises a first feature point selected from the target camera image data and a second feature point selected from the target camera image data; selecting a second feature point matched with the first feature point from the target projection image data to form a feature point pair; constructing an external parameter estimation matrix corresponding to the target view angle based on the plurality of feature point pairs corresponding to the target view angle; global optimization is carried out on the external parameter estimation matrix corresponding to each view angle in the multiple view angles, and the external parameter when the minimum global error is obtained is the calibrated external parameter. The method and the device are suitable for multi-scene real-time external parameter calibration.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an external parameter calibration method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of autonomous driving technology and robot visual perception technology, sensor fusion technology has played a vital role in improving the accuracy and robustness of environmental perception. Especially in autonomous driving and robotic systems, visual sensors (cameras) and laser radar sensors (LiDAR) are two common sensors. Cameras can provide rich texture and color information, while LiDAR can provide high-precision three-dimensional spatial information. Through the combination of the two, the system can obtain more accurate perception results in complex environments. In the sensor fusion process, accurate estimation of the external parameter matrix is ​​a key step in realizing multi-sensor data fusion.

[0003] Existing methods detect target objects in the scene and use the target's 3D information and 2D image information to calculate the external parameters between the camera and the radar. They rely on artificially arranged calibration objects and are highly dependent on a specific calibration environment. They are suitable for relatively stable scenes and cannot meet the external parameter calibration requirements in dynamic or complex environments.

[0004] Therefore, a dynamic external parameter calibration scheme that can be applied to more scenarios is needed. Summary of the invention

[0005] The present application provides an external parameter calibration method to solve the technical problem that the existing external parameter calibration method cannot adapt to multiple scenarios.

[0006] In a first aspect, the present application provides an external reference calibration method, comprising:

[0007] Acquire multi-view camera image data and multi-view projection image data; wherein the projection image data is obtained by projecting the point cloud data onto the camera coordinate system, and the point cloud data is point cloud data of the same view as the camera image data;

[0008] For a target perspective in the multiple perspectives, based on a user interaction method, a feature point pair is selected according to target projection image data corresponding to the target perspective and target camera image data corresponding to the target perspective, wherein the selecting the feature point pair includes selecting a first feature point from the target camera image data, and selecting a second feature point matching the first feature point from the target projection image data, to form a feature point pair;

[0009] Based on multiple feature point pairs corresponding to the target perspective, an external parameter estimation matrix corresponding to the target perspective is constructed;

[0010] The extrinsic parameter estimation matrix corresponding to each view in the multi-view is globally optimized, and the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter.

[0011] Optionally, the above method performs global optimization on the extrinsic parameter estimation matrix corresponding to each view in the multi-view, and the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter, including:

[0012] Applying a preset objective function, globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multi-view, the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter; wherein each extrinsic parameter estimation matrix includes an initial rotation matrix of the n-th view and an initial translation vector of the n-th view; the calibrated extrinsic parameter includes a calibrated rotation matrix and a calibrated translation vector;

[0013] The preset objective function satisfies:

[0014]

[0015] in, represents the Frobenius norm, represents the 2-norm, N represents the number of viewing angles, and R n represents the initial rotation matrix of the nth viewing angle, T n Represents the initial translation vector of the nth viewing angle, R gt represents the rotation matrix after calibration, T gt represents the translation vector after calibration, J(R gt , T gt ) is the error value.

[0016] Optionally, the above method, obtaining multi-view projection image data, includes:

[0017] Obtain an initial internal parameter matrix and an initial external parameter matrix; wherein the initial external parameter matrix includes an initial rotation matrix and an initial translation vector;

[0018] According to the initial intrinsic parameter matrix, the initial rotation matrix, the initial translation vector, and the point cloud data, multi-view projection image data is calculated.

[0019] Optionally, the above method further includes: performing global optimization on the extrinsic parameter estimation matrix corresponding to each view in the multiple views, and obtaining the extrinsic parameter with the minimum global error as the calibrated extrinsic parameter.

[0020] Replace the initial rotation matrix in the initial external parameter matrix with the rotation matrix in the calibrated external parameter matrix;

[0021] The translation vector in the calibrated extrinsic parameter matrix is ​​replaced by the initial translation vector in the initial extrinsic parameter matrix.

[0022] Optionally, the above method, based on the user interaction mode, according to the target projection image data corresponding to the target perspective and the target camera image data corresponding to the target perspective, before selecting the feature point pairs, further includes:

[0023] Determine a projection error value of target projection image data corresponding to the target viewing angle relative to target camera image data;

[0024] It is determined that the projection error value corresponding to the target viewing angle is greater than a preset threshold.

[0025] Optionally, the above method further comprises:

[0026] If the projection error value corresponding to each viewing angle in the multiple viewing angles is less than a preset threshold, the step of acquiring the camera image data of the multiple viewing angles and the projection image data of the multiple viewing angles is performed.

[0027] Optionally, the above method constructs an extrinsic parameter estimation matrix corresponding to the target perspective based on multiple feature point pairs corresponding to the target perspective, including:

[0028] Based on multiple feature point pairs corresponding to the target perspective, a linear equation system is constructed;

[0029] Solve the linear equations and construct the extrinsic parameter estimation matrix corresponding to the target perspective.

[0030] In a second aspect, the present application provides an external reference calibration device, comprising:

[0031] An acquisition module is used to acquire multi-view camera image data and multi-view projection image data; wherein the projection image data is obtained by projecting the point cloud data onto the camera coordinate system, and the point cloud data is point cloud data of the same view as the camera image data;

[0032] A selection module is used to select a feature point pair for a target view in the multi-views based on a user interaction mode and according to target projection image data corresponding to the target view and target camera image data corresponding to the target view, wherein selecting the feature point pair includes selecting a first feature point from the target camera image data and selecting a second feature point matching the first feature point from the target projection image data to form a feature point pair;

[0033] A construction module, used for constructing an external parameter estimation matrix corresponding to a target perspective based on a plurality of feature point pairs corresponding to a target perspective;

[0034] The module is used to globally optimize the extrinsic parameter estimation matrix corresponding to each view in the multi-view, and the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter.

[0035] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0036] Memory stores computer-executable instructions;

[0037] The processor executes the computer-executable instructions stored in the memory to implement the method in the present application.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method of the present application.

[0039] In a fifth aspect, the present application provides a computer program product, comprising: a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementations of the first aspect.

[0040] The extrinsic parameter calibration method, device, electronic device and storage medium provided in the present application obtain camera image data of multiple perspectives and projection image data of multiple perspectives; wherein the projection image data is obtained by projecting the point cloud data onto the camera coordinate system, and the point cloud data is the point cloud data of the same perspective as the camera image data; through multiple perspectives, the extrinsic parameters can be made more accurate. For the target perspective in the multiple perspectives, based on the user interaction method, according to the target projection image data corresponding to the target perspective and the target camera image data corresponding to the target perspective, a feature point pair is selected, wherein the selection of the feature point pair includes selecting a first feature point from the target camera image data, and selecting a second feature point matching the first feature point from the target projection image data to form a feature point pair; by determining the feature point pair through the user interaction method, it can adapt to more scenes, and can accurately determine the feature point pairs for different scenes. Based on multiple feature point pairs corresponding to the target perspective, an extrinsic parameter estimation matrix corresponding to the target perspective is constructed; the extrinsic parameter estimation matrix corresponding to each perspective in the multiple perspectives is globally optimized, and the external parameter at the minimum global error is obtained as the calibrated extrinsic parameter. Through global optimization, the extrinsic parameter estimation matrix of each perspective in the multiple perspectives is comprehensively considered to obtain the calibrated extrinsic parameter at the minimum global error. The increase of the external parameter estimation matrix makes the calibrated extrinsic parameter more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 A schematic diagram of a flow chart of an external reference calibration method provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of another external reference calibration method provided in an embodiment of the present application;

[0044] Figure 3A schematic diagram of the structure of an external reference calibration device provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0049] At present, accurate estimation of the extrinsic parameter matrix is ​​a key step in realizing multi-sensor data fusion. The extrinsic parameter matrix describes the spatial transformation relationship between different sensors, that is, the rotation and translation transformation between the camera coordinate system and the radar coordinate system. An accurate extrinsic parameter matrix can ensure that the pixels in the image correspond precisely to the spatial points in the radar point cloud, thereby realizing effective fusion of multimodal data. The existing extrinsic parameter matrix acquisition method obtains extrinsic parameters based on a fixed scene. When the scene changes, there will be problems with inaccuracy between the pixels in the image and the spatial points in the radar point cloud. In addition, for mobile carriers such as unmanned vehicles and drones, there will be problems with the pixels in the image and the spatial points in the radar point cloud.

[0050] The present application provides an external parameter calibration method, device, electronic device and storage medium, which can determine the target point pair through user interaction, ensure that the pixel points in the image correspond accurately to the spatial points in the radar point cloud, and can be applied to more scenarios. At the same time, combined with the external parameter estimation matrix corresponding to multiple perspectives, global external parameter optimization can be performed, and when the minimum global error is determined according to the objective function, accurate calibrated external parameters can be obtained, and the external parameters can be calibrated in real time.

[0051] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0052] Figure 1 A schematic diagram of a flow chart of an external reference calibration method provided in an embodiment of the present application. Figure 1 As shown, the execution subject of the method can be a server or other servers, and this embodiment is not particularly limited here. Figure 1 As shown, the method includes:

[0053] S101, obtaining multi-perspective camera image data and multi-perspective projection image data; wherein the projection image data is obtained by projecting the point cloud data onto the camera coordinate system, and the point cloud data is point cloud data of the same perspective as the camera image data.

[0054] Among them, multi-view can refer to the images collected from different scenes, or the images from different viewpoints of the same scene. Specifically, when the camera and lidar are placed on a vehicle (or other moving object), the images are continuously captured as the vehicle moves. Each viewpoint contains a camera image data and a point cloud data.

[0055] The image data collected from the camera is camera image data, and the camera image data may be an RGB (Red, Green, Blue) image.

[0056] The point cloud data from the laser radar is collected as point cloud data, and then the point cloud data is projected into the camera coordinate system to obtain the projection image data. The data from the laser radar is stored in the form of point cloud.

[0057] S102. For a target perspective in multiple perspectives, based on a user interaction method, a feature point pair is selected according to target projection image data corresponding to the target perspective and target camera image data corresponding to the target perspective, wherein selecting the feature point pair includes selecting a first feature point from the target camera image data, and selecting a second feature point matching the first feature point from the target projection image data, to form a feature point pair.

[0058] The user interaction method refers to the user marking feature points at corresponding positions by clicking the mouse on the camera image data A1 and the target projection image data B1.

[0059] The target perspective refers to any one of all perspectives.

[0060] Feature point pairs can refer to 2D (Two-Dimensional) feature point pairs, which usually refer to a pair of corresponding feature points used for image matching, recognition or tracking in image processing. These feature points are extracted on the two-dimensional image plane, and key points in the image can be identified through feature detection algorithms (such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and RotatedBRIEF), etc.), or the corresponding feature point pairs can be obtained by user clicks.

[0061] Specifically, the user selects A feature points in the camera image data of any angle by clicking the mouse (in the image coordinate system), and at the same time, the corresponding A feature points are selected by clicking the mouse in the projection image of the same viewing angle Thus according to and The feature point pairs can be obtained. At least 5 pairs of feature point pairs need to be selected, and A is at least 5. When there are 5 pairs of feature point pairs, it is necessary to confirm 5 points in the camera image data as the first feature points and 5 feature points in the projection image as the second feature points to form feature point pairs.

[0062] S103: construct an extrinsic parameter estimation matrix corresponding to the target perspective based on multiple feature point pairs corresponding to the target perspective.

[0063] Among them, according to multiple pairs of feature points of any perspective, the external parameter estimation matrix of the perspective is determined.

[0064] For example, if there are A pairs of feature points from B viewing angles, then it is necessary to determine the extrinsic parameter estimation matrix corresponding to the B viewing angles.

[0065] In the above embodiment of the present application, based on multiple feature point pairs corresponding to the target perspective, an external parameter estimation matrix corresponding to the target perspective is constructed, including:

[0066] Based on multiple feature point pairs corresponding to the target perspective, a linear equation system is constructed;

[0067] Solve the linear equations and construct the extrinsic parameter estimation matrix corresponding to the target perspective.

[0068] Among them, for any pair of feature points, the linear equations are satisfied:

[0069]

[0070] in, is the i-th first feature point, Represents the second feature point of the i-th image. F is the basic matrix, i.e., the external parameter estimation matrix, which describes the mapping relationship between the image coordinate system and the radar projection coordinate system, and contains the relative position information of the camera and the radar (rotation matrix (i.e., the initial rotation matrix R of the n-th view). n ) and the translation vector (i.e. the initial translation vector T of the nth viewing angle n )), any spatial transformation of a rigid body can be expressed by a homogeneous transformation matrix T1 E SE (3), which is in the form of: Here R represents the rotation matrix and T represents the translation vector.

[0071] Specifically, by substituting multiple pairs of feature points of any view into the preset linear equations formula, a linear equations system can be obtained. The linear equations system is solved to obtain F, which is the external parameter estimation matrix of the view.

[0072]

[0073] Among them, the left matrix represents the coordinate set of all the first feature points in the camera image of any view, the number of rows N means there are N pairs of feature points, and the right vector is the homogeneous coordinate (x, y, 1) of the second feature point of the projection image corresponding to the same view. There are also N homogeneous coordinates of the second feature point of the projection image of the same view, which are written here as

[0074] The advantage of this setting is that, through multiple feature point pairs corresponding to the target perspective, the more feature points there are, the more accurate the extrinsic parameter estimation matrix is.

[0075] S104, globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multiple views, and obtaining the extrinsic parameter with the minimum global error as the calibrated extrinsic parameter.

[0076] Among them, global optimization can refer to combining the external parameter matrices of multiple perspectives to obtain the best calibrated matrix so that the global error is minimized.

[0077] Specifically, different weights can be preset for each extrinsic parameter estimation matrix, so as to comprehensively obtain the calibrated extrinsic parameters so as to minimize the global error. The calibrated extrinsic parameters can also be obtained through a preset objective function.

[0078] In the above embodiment of the present application, the extrinsic parameter estimation matrix corresponding to each view in the multi-view is globally optimized, and the extrinsic parameter when the minimum global error is obtained is the calibrated extrinsic parameter, including:

[0079] Applying a preset objective function, globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multi-view, the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter; wherein each extrinsic parameter estimation matrix includes an initial rotation matrix of the n-th view and an initial translation vector of the n-th view; the calibrated extrinsic parameter includes a calibrated rotation matrix and a calibrated translation vector;

[0080] The preset objective function satisfies:

[0081]

[0082] in, represents the Frobenius norm, represents the 2-norm, N represents the number of viewing angles, and R n represents the initial rotation matrix of the nth viewing angle, T n Represents the initial translation vector of the nth viewing angle, R gt represents the rotation matrix after calibration, T gt represents the translation vector after calibration, J(R gt , T gt ) is the error value.

[0083] Among them, the error function is used to calculate the rotation error and the translation error, wherein the rotation error is calculated by the trace of the rotation matrix, and the translation error is calculated by the Euclidean distance. Therefore, it is necessary to determine an objective function (error function) for calculating the rotation error and the translation error. The square of the difference between the initial rotation matrix and the standard rotation matrix of each perspective is the rotation error corresponding to the perspective. Similarly, the square of the difference between the initial translation vector and the standard translation vector of each perspective is the translation error corresponding to the perspective. The rotation errors and translation errors of all perspectives are added to obtain the total error. Therefore, it is necessary to determine a standard rotation matrix (i.e., the rotation matrix in the calibration external parameter) and a standard translation vector (i.e., the translation vector in the calibration external parameter) to minimize the total error.

[0084] Specifically, the extrinsic parameter estimation matrix of each view includes the initial rotation matrix and the initial translation vector of the view. Therefore, the initial rotation matrix and the initial translation vector in the extrinsic parameter estimation matrix of all views are substituted into the preset objective function, and then the objective function is optimized by an optimization algorithm (such as L-BFGS-B (Limited-memory Broyden-Fletcher-Goldfarb-Shannowith Box constraints)) J(R gt , T gt ) to obtain the calibrated translation vector and the calibrated rotation matrix.

[0085] In addition, the external parameter relationship can be learned directly from the labeled camera image data and the labeled point cloud data through a deep neural network, so that the external parameter matrix can be directly optimized through the trained neural network model.

[0086] The advantage of this setting is that by performing global optimization based on the external parameter estimation matrix corresponding to each view in the multi-view, the total error is minimized, so that the calibrated external parameters are more accurate. At the same time, the objective function can also be used to optimize and calibrate the external parameters in real time.

[0087] The embodiment of the present application provides an external parameter calibration method, which can confirm the accurate selection of image data and target camera image data in any perspective based on user interaction, so as to have a wider range of applications. At the same time, by globally optimizing the external parameter estimation matrix corresponding to multiple perspectives, the external parameter with the minimum global error is obtained as the calibrated external parameter, so that the obtained external parameter is more accurate, and when the external parameter changes, it can also be accurately and timely calibrated.

[0088] As a possible implementation of the present application, obtaining multi-view projection image data includes:

[0089] Obtain an initial internal parameter matrix and an initial external parameter matrix; wherein the initial external parameter matrix includes an initial rotation matrix and an initial translation vector;

[0090] According to the initial intrinsic parameter matrix, the initial rotation matrix, the initial translation vector, and the point cloud data, multi-view projection image data is calculated.

[0091] Among them, the initial intrinsic parameter matrix is ​​the intrinsic parameter matrix of the camera, and the initial extrinsic parameter matrix is ​​the extrinsic parameter matrix that can be obtained when the radar position is determined. In other words, as long as the radar is fixed, the extrinsic parameter matrix will not change.

[0092] The initial extrinsic parameter matrix is ​​represented as the current extrinsic parameter matrix. If it is not calibrated, it is represented as the extrinsic parameter matrix at the beginning. If it has undergone a calibration, it is the extrinsic parameter matrix after the calibration. The same applies to other situations.

[0093] If the point cloud data is point cloud data of view A, then the corresponding projection image data is projection image data of view A. If the point cloud data is point cloud data of view A and point cloud data of view B, then the corresponding projection image data is projection image data of view A and projection data of view B.

[0094] By substituting the initial intrinsic parameter matrix, the initial rotation matrix, the initial translation vector, and the multi-view point cloud data into the preset formula, the multi-view projection image data can be obtained.

[0095] The projection image data meets the following requirements:

[0096]

[0097] Among them, K is the internal parameter matrix, X lidar For point cloud data, is the projection image data of any viewing angle, R is the initial selection vector, and T is the initial translation vector.

[0098] The advantage of this setting is that it can obtain projection image data from multiple perspectives, thereby obtaining the external parameter matrix corresponding to the multiple perspectives, making the calibration of the external parameter matrix more accurate, and at the same time being able to accurately determine the situation of the laser radar, specifically, whether the laser radar has changed position.

[0099] As a possible implementation of the present application, after globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multi-view, and obtaining the extrinsic parameter with the minimum global error as the calibrated extrinsic parameter, it also includes:

[0100] Replace the initial rotation matrix in the initial external parameter matrix with the rotation matrix in the calibrated external parameter matrix;

[0101] The translation vector in the calibrated extrinsic parameter matrix is ​​replaced by the initial translation vector in the initial extrinsic parameter matrix.

[0102] The initial external parameter matrix is ​​calibrated by replacing the initial rotation matrix in the initial external parameter matrix with the rotation matrix in the calibrated external parameter and replacing the initial translation vector in the initial external parameter matrix with the translation vector in the calibrated external parameter. When the multi-view projection image data is subsequently calculated, the corresponding external parameter is the calibrated external parameter.

[0103] Until the position of the laser radar changes, specifically, when the laser radar is placed on a moving carrier or vehicle, if the vehicle shakes or bumps, the position of the laser radar will be offset or changed, and the calibrated external parameters need to be recalibrated.

[0104] The advantage of this setting is that the external parameters can be updated and calibrated in real time to ensure their accuracy. It is also more suitable for the fields of unmanned driving and drones.

[0105] In the above embodiment of the present application, based on the user interaction mode, according to the target projection image data corresponding to the target perspective and the target camera image data corresponding to the target perspective, before selecting the feature point pair, the following is further included:

[0106] Determine a projection error value of target projection image data corresponding to the target viewing angle relative to target camera image data;

[0107] It is determined that the projection error value corresponding to the target viewing angle is greater than a preset threshold.

[0108] Among them, the projection error value can be:

[0109]

[0110] Where e is the projection error, P cam To represent the real two-dimensional pixel coordinate point in the camera image, that is, the first feature point in the target camera image data, It is the second feature point of the target projection image data.

[0111] If the external parameter matrix is ​​accurate, P cam and It should be as close as possible (equal in extreme cases), at which point the projection error e will approach zero (equal to 0). If the projection error e is large, it means that the calibrated external parameter matrix is ​​incorrect and needs to be readjusted. In reality, the standard for the projection error e does not have a fixed absolute value because it depends on the specific application scenario, the accuracy of the hardware equipment, the data quality, and the calibration requirements. In this patent, it is believed that if the projection result of the given initial external parameters (referring to the external parameters given at the factory, or the current parameters after we calibrate the factory) is greater than 2cm from the actual distance, recalibration is required. When the vehicle is driving, if a position offset occurs (that is, the fixed position changes due to bumps, etc.), the current initial external parameter projection result and the actual distance are greater than 2cm, then recalibration is required.

[0112] In this embodiment, the first feature point of the target projection image data corresponding to the target viewing angle and the second feature point in the target projection image data are substituted into the preset formula to obtain the projection error value e. At this time, the projection error value is greater than the preset threshold, which indicates that the external parameter matrix is ​​inaccurate and needs to be calibrated.

[0113] The advantage of this setting is that by flexibly setting the projection error, it can adapt to applications in more scenarios involving point cloud data and camera data. At the same time, when in use, by comparing the threshold and the projection error, it is possible to determine in real time whether the lidar has deviated, thereby performing real-time calibration of the external parameters when the external parameters change due to radar offset or position movement.

[0114] As a possible implementation of this application, it also includes:

[0115] If the projection error value corresponding to each viewing angle in the multiple viewing angles is less than a preset threshold, the step of acquiring the camera image data of the multiple viewing angles and the projection image data of the multiple viewing angles is performed.

[0116] Among them, if the projection error value corresponding to each perspective in the perspective is less than the preset threshold, it indicates that the current external parameter is accurate, that is, no calibration is required, but it is necessary to detect whether the position of the laser radar is offset and whether calibration is required. Therefore, it is necessary to re-acquire multi-perspective camera image data and multi-perspective projection image data to determine whether the laser radar is offset.

[0117] The advantage of this setting is that when the projection error value is less than the preset threshold, there is no need to calibrate the external parameters, which saves resources and improves efficiency.

[0118] Figure 2 A flow chart of another external reference calibration method provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0119] S201, acquiring image data from a camera and point cloud data from a laser radar.

[0120] The image data can be RGB images, and the data from the LiDAR is stored in the form of point clouds. The system supports different types of cameras and LiDAR devices, and can process data collected synchronously or asynchronously.

[0121] S202, associating the image data from the camera and the point cloud data from the laser radar.

[0122] Correlating the image data from the camera and the point cloud data from the lidar means:

[0123] Refers to projecting the point cloud data into the camera coordinate system to obtain the radar projection image, specifically:

[0124] The point pairs in the camera image and radar point cloud need to satisfy the projective geometric relationship:

[0125]

[0126] Among them, K is the intrinsic parameter matrix of the camera, which can be determined when the camera is determined, P cam is the projection point on the image data, X lidar It is a point in the radar point cloud data. When the lidar is determined and the position of the lidar is determined, R represents the rotation matrix in the initial external parameters, and T represents the translation vector in the initial external parameters.

[0127] S203: Determine whether to perform external parameter calibration according to the radar projection image and image data.

[0128] Specific:

[0129]

[0130] Where e is the error, Pcam is a point in the image data, is a point in the radar projection data.

[0131] If the external parameter matrix is ​​accurate, P cam and It should be as close as possible (equal in extreme cases), at which point the projection error e will approach zero (equal to 0). If the projection error e is large, it means that the extrinsic parameter matrix is ​​incorrect and needs to be readjusted. In reality, the standard for the projection error e does not have a fixed absolute value because it depends on the specific application scenario, the accuracy of the hardware equipment, the data quality, and the calibration requirements. In this patent, it is believed that if the projection result of the given initial extrinsic parameter (referring to the extrinsic parameter given at the factory, or the current parameter after we calibrate the factory) and the actual distance are greater than 2cm, recalibration is required. When the vehicle is driving, if a position offset occurs (that is, the fixed position changes due to bumps, etc.), when e is greater than 2cm, recalibration is required.

[0132] S204: If external parameter calibration is required at this time, in response to the user's selection operation, 2D points are selected from the radar projection image and 2D points are selected from the image data to obtain multiple pairs of matching points.

[0133] Specifically: Get the coordinates of 2D points in the image data and the 2D point coordinates in the radar projection image by clicking the mouse. The system sets a mouse callback function to select feature points in the image data and the radar projection image respectively. Users can mark the corresponding points of the same object in the image and the radar projection image by clicking on the image and the radar projection image. After each click, the system will mark the 2D point selected by the user with a small red dot on the image to ensure that the user can intuitively see the selected feature point. Check and ensure that the number of feature points in the image and the radar projection image is consistent for subsequent essential matrix calculations.

[0134] The user selects significant feature points on the image data and the radar projection image respectively. These points are selected based on texture features, edge features or other visual saliency. By selecting these feature points at multiple viewing angles, a set of feature point pairs can be formed. There is a clear geometric correspondence between these feature point pairs. It is necessary to ensure that in each viewing angle, each feature point pair in the image data and the radar point cloud corresponds to each other, that is, they all represent the same physical point in space. This geometric relationship provides an important constraint for the estimation of extrinsic parameters. Specifically, suppose we select N feature points in the image data (in the image coordinate system) and corresponding points in the radar projection image. (At least 5 pairs of matching points need to be selected, that is, 5 2D points in the image and 5 corresponding points in the radar point cloud projection image, before the calculation of extrinsic parameters can be performed).

[0135] S205. Obtain an extrinsic parameter estimation matrix based on multiple pairs of matching points.

[0136] Among them, the relationship between multiple pairs of matching points can be expressed as:

[0137]

[0138] Among them, F is the basic matrix, that is, the external parameter estimation matrix, which includes the external parameter estimation matrix, the translation matrix, and the points in the image data (two-dimensional pixel coordinates): Indicates. Among them, The i-th horizontal pixel coordinate in the image data, The i-th vertical pixel coordinate in the image data, 1: The third component in the homogeneous coordinate, used for matrix calculations.

[0139] The radar point cloud is projected onto the image plane (2D pixel coordinates): express.

[0140] The i-th horizontal pixel coordinate after the radar point cloud is projected to the camera coordinates,

[0141] The i-th vertical pixel coordinate after the radar point cloud is projected to the camera coordinates,

[0142] 1: Also the third component in homogeneous coordinates

[0143] The expansion can be:

[0144]

[0145] The matrix on the left represents the coordinate set of all points in the image. The number of rows N indicates that there are N point pairs. The vector on the right is the homogeneous coordinates (x1, y1, 1) of the corresponding radar projection point.

[0146] For the above to facilitate calculation.

[0147]

[0148] The simplification described here describes or expresses the point pair matching relationship rather than directly dealing with geometric matrix transformations.

[0149] In fact, the real

[0150]

[0151] The formula should be:

[0152]

[0153] S206: According to the extrinsic parameter estimation matrix corresponding to each of the multiple perspectives, a final calibrated extrinsic parameter matrix can be obtained.

[0154] Specific:

[0155] Assume that N sets of external parameters are obtained, each set of external parameters is represented by the rotation matrix R i and the translation vector T i composition:

[0156]

[0157] Hope to find a globally optimized rotation matrix R gt and the translation vector T gt . So that they can better represent the actual camera extrinsic parameters. In order to quantify the difference between each set of extrinsic parameters and the global extrinsic parameters, we can define an objective function in the form of:

[0158]

[0159] in, represents the Frobenius norm, represents the 2-norm, N represents the number of viewing angles, and R n represents the initial rotation matrix of the nth viewing angle, T n Represents the initial translation vector of the nth viewing angle, R gt represents the rotation matrix after calibration, T gt represents the translation vector after calibration, J(R gt , T gt ) is the error value.

[0160] By minimizing the objective function JJJ through an optimization algorithm (such as L-BFGS-B), the optimized rotation matrix and translation vector can be obtained. Specifically, L-BFGS-B is a quasi-Newton optimization method suitable for solving large optimization problems.

[0161] The extrinsic parameter calibration method provided in the embodiment of the present application selects multiple pairs of matching points through user selection operation to obtain an extrinsic parameter estimation matrix. Then, by obtaining the extrinsic parameter estimation matrix corresponding to each of the multiple perspectives, the extrinsic parameter estimation matrix is ​​calibrated to obtain the extrinsic parameter matrix. It can reduce the dependence on specific calibration objects and controlled environments, and can be independent of the traditional method based on the calibration plate, and can be based on scene information or any other scene. It includes scenes where the sensor (camera-radar) moves itself (placed on a vehicle, mobile robot, etc., the sensor moves itself) and fixed scenes (fixed in scenes such as buildings), and can be applied to people, cars, objects, etc. that move by passing objects, which significantly broadens the scope of application, from indoor static environment cultivation to outdoor complex dynamic environment cultivation. It is not dependent on scenes with strong feature information, such as edge contours or calibration objects. As long as there is a scene, it can be achieved with the help of point cloud projection and image information, and extrinsic parameter estimation can be completed even in scenes lacking significant features. Scenes with low feature density, such as purely flat ground or areas without geometrically aligned objects. For example, calibration is completed in open areas or sparsely structured scenes. Sensor calibration in low-texture environments (such as workshops and warehouses). The system can adaptively calibrate external parameters to cope with changes in different scenarios. (Meet the real-time needs of low-power devices such as embedded and edge computing). By adaptively adjusting the external parameter matrix through multi-view and multi-frame acquisition and processing of sensor data, the problem of possible changes in external parameters in dynamic environments is solved. Whether it is due to drift in the relative position of the sensor or failure of external parameters due to sensor jitter or environmental changes, the system can maintain the accuracy and stability of data fusion. (Adaptively update external parameters for relative position drift of sensors, sensor jitter or environmental changes. Ensure the fusion accuracy and stability of sensor data in dynamic environments. Maintain the stability of external parameters when the vehicle shakes or vibrates, driving on bumpy roads, etc. Or real-time calibration of sensors in mobile robotic arm operations.)

[0162] Figure 3 This is a structural example diagram of the external reference calibration device provided in the embodiment of the present application. Figure 3 As shown, the external parameter calibration device 30 includes: an acquisition module 301, a selection module 302, a construction module 303 and a obtaining module 304. Among them:

[0163] An acquisition module 301 is used to acquire multi-view camera image data and multi-view projection image data; wherein the projection image data is obtained by projecting the point cloud data onto the camera coordinate system, and the point cloud data is point cloud data of the same view as the camera image data;

[0164] A selection module 302 is used to select a feature point pair for a target view in the multiple view based on a user interaction mode and according to target projection image data corresponding to the target view and target camera image data corresponding to the target view, wherein selecting the feature point pair includes selecting a first feature point from the target camera image data and selecting a second feature point matching the first feature point from the target projection image data to form a feature point pair;

[0165] A construction module 303 is used to construct an external parameter estimation matrix corresponding to the target perspective based on multiple feature point pairs corresponding to the target perspective;

[0166] The module 304 is used to perform global optimization on the extrinsic parameter estimation matrix corresponding to each view in the multiple views, and the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter.

[0167] In a possible implementation, the obtaining module 304 is specifically used for:

[0168] Applying a preset objective function, globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multi-view, the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter; wherein each extrinsic parameter estimation matrix includes an initial rotation matrix of the n-th view and an initial translation vector of the n-th view; the calibrated extrinsic parameter includes a calibrated rotation matrix and a calibrated translation vector;

[0169] The preset objective function satisfies:

[0170]

[0171] in, represents the Frobenius norm, represents the 2-norm, N represents the number of viewing angles, and R n represents the initial rotation matrix of the nth viewing angle, T n Represents the initial translation vector of the nth viewing angle, R gt represents the rotation matrix after calibration, T gt represents the translation vector after calibration, J(R gt , T gt ) is the error value.

[0172] In a possible implementation, the acquisition module 301 is specifically used to:

[0173] Obtain an initial internal parameter matrix and an initial external parameter matrix; wherein the initial external parameter matrix includes an initial rotation matrix and an initial translation vector;

[0174] According to the initial intrinsic parameter matrix, the initial rotation matrix, the initial translation vector, and the point cloud data, multi-view projection image data is calculated.

[0175] In a possible implementation, the obtaining module 304 is further used for:

[0176] Replace the initial rotation matrix in the initial external parameter matrix with the rotation matrix in the calibrated external parameter matrix;

[0177] The translation vector in the calibrated extrinsic parameter matrix is ​​replaced by the initial translation vector in the initial extrinsic parameter matrix.

[0178] In a possible implementation, the selection module 302 is specifically used for:

[0179] Before selecting a feature point pair based on the target projection image data corresponding to the target view angle and the target camera image data corresponding to the target view angle in a user interactive manner, determining a projection error value of the target projection image data corresponding to the target view angle relative to the target camera image data;

[0180] It is determined that the projection error value corresponding to the target viewing angle is greater than a preset threshold.

[0181] In a possible implementation, the selection module 302 is further configured to:

[0182] If the projection error value corresponding to each viewing angle in the multiple viewing angles is less than a preset threshold, the step of acquiring the camera image data of the multiple viewing angles and the projection image data of the multiple viewing angles is performed.

[0183] In a possible implementation, the construction module 303 is specifically used to:

[0184] Based on multiple feature point pairs corresponding to the target perspective, a linear equation system is constructed;

[0185] Solve the linear equations and construct the extrinsic parameter estimation matrix corresponding to the target perspective.

[0186] The device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0187] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0188] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0189] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, Digital Video Discs (DVD)), or semiconductor media (eg, solid state disks (SSD)).

[0190] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 40 includes:

[0191] The electronic device 40 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403 and other components. The processor 401 , the memory 402 and the communication component 403 are connected via a bus 404 .

[0192] In a specific implementation process, at least one processor 401 executes computer-executable instructions stored in the memory 402, so that at least one processor 401 performs an external parameter calibration method as described above.

[0193] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0194] In the above Figure 4In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0195] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0196] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0197] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the steps in any of the above-mentioned external parameter calibration methods when executed by a processor.

[0198] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0199] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0200] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the external parameter calibration methods provided in the embodiment of the present application.

[0201] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0202] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.

[0203] Since the instructions stored in the storage medium can execute the steps in any external parameter calibration method provided in the embodiments of the present application, the beneficial effects that can be achieved by any external parameter calibration method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0204] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0205] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An external parameter calibration method, characterized in that: include: Acquire multi-view camera image data and multi-view projection image data; wherein the projection image data is obtained by projecting point cloud data onto a camera coordinate system, and the point cloud data is point cloud data of the same view as the camera image data; For a target perspective in the multiple perspectives, based on a user interaction method, a feature point pair is selected according to target projection image data corresponding to the target perspective and target camera image data corresponding to the target perspective, wherein the selected feature point pair includes selecting a first feature point from the target camera image data, and selecting a second feature point matching the first feature point from the target projection image data, to form a feature point pair; Based on a plurality of feature point pairs corresponding to the target perspective, construct an extrinsic parameter estimation matrix corresponding to the target perspective; The extrinsic parameter estimation matrix corresponding to each view in the multiple views is globally optimized, and the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter.

2. The method according to claim 1, characterized in that The globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multiple views, and obtaining the extrinsic parameter with the minimum global error as the calibrated extrinsic parameter, comprises: Applying a preset objective function, globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multiple views, and obtaining the extrinsic parameter with the minimum global error as the calibrated extrinsic parameter; wherein each extrinsic parameter estimation matrix includes an initial rotation matrix of the nth view and an initial translation vector of the nth view; and the calibrated extrinsic parameter includes a calibrated rotation matrix and a calibrated translation vector; The preset objective function satisfies: in, represents the Frobenius norm, represents the 2-norm, N represents the number of viewing angles, and R n represents the initial rotation matrix of the nth viewing angle, T n Represents the initial translation vector of the nth viewing angle, R gt represents the rotation matrix after calibration, T gt represents the translation vector after calibration, J(R gt , T gt ) is the error value.

3. The method according to claim 1 or 2, characterized in that: The acquiring of multi-view projection image data comprises: Acquire an initial internal parameter matrix and the initial external parameter matrix; wherein the initial external parameter matrix includes an initial rotation matrix and an initial translation vector; The multi-view projection image data is calculated based on the initial intrinsic parameter matrix, the initial rotation matrix, the initial translation vector, and the point cloud data.

4. The method according to claim 3, characterized in that After globally optimizing the extrinsic parameter estimation matrix corresponding to each view in the multiple views and obtaining the extrinsic parameter with the minimum global error as the calibrated extrinsic parameter, the method further includes: The rotation matrix in the calibrated external parameter replaces the initial rotation matrix in the initial external parameter matrix; The translation vector in the calibrated extrinsic parameter matrix replaces the initial translation vector in the initial extrinsic parameter matrix.

5. The method according to claim 1 or 2, characterized in that: Before selecting the feature point pairs based on the target projection image data corresponding to the target viewing angle and the target camera image data corresponding to the target viewing angle in a user interaction manner, the method further includes: Determine a projection error value of target projection image data corresponding to the target viewing angle relative to the target camera image data; It is determined that the projection error value corresponding to the target viewing angle is greater than a preset threshold.

6. The method according to claim 5, characterized in that Also includes: If the projection error value corresponding to each of the multiple perspectives is less than a preset threshold, the step of acquiring the camera image data of the multiple perspectives and the projection image data of the multiple perspectives is performed.

7. The method according to claim 1 or 2, characterized in that: The step of constructing an external parameter estimation matrix corresponding to the target perspective based on a plurality of feature point pairs corresponding to the target perspective includes: Constructing a linear equation system based on a plurality of feature point pairs corresponding to the target perspective; The linear equations are solved to construct an external parameter estimation matrix corresponding to the target viewing angle.

8. An external parameter calibration device, characterized in that: include: An acquisition module, used to acquire multi-view camera image data and multi-view projection image data; wherein the projection image data is obtained by projecting the point cloud data onto a camera coordinate system, and the point cloud data is point cloud data of the same view as the camera image data; A selection module, for selecting a feature point pair for a target view in the multiple view, based on a user interaction mode, according to target projection image data corresponding to the target view and target camera image data corresponding to the target view, wherein the selected feature point pair includes selecting a first feature point from the target camera image data, and selecting a second feature point matching the first feature point from the target projection image data, to form a feature point pair; A construction module, used to construct an external parameter estimation matrix corresponding to the target perspective based on a plurality of feature point pairs corresponding to the target perspective; The module is used to globally optimize the extrinsic parameter estimation matrix corresponding to each view in the multiple views, and the extrinsic parameter with the minimum global error is the calibrated extrinsic parameter.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed.

Images