Internal and external parameter optimization method and device and storage medium
By obtaining the 3D point cloud and 2D image data in the auto autonomous driving data, the internal and external parameters are optimized by comparing the error between projection information and 2D frame labeling information, the deviation problems caused by low labeling efficiency and equipment jitter are solved, and the deviation problems caused by low labeling efficiency are realized. Automated and efficient internal and external parameter updates.
Patent Information
- Application Number
- CN202411033706.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art in the labeling of automobile autonomous driving data is inaccurate in the labeling of internal and external parameters, and cannot solve the problem of phased deviation caused by equipment parts jitter, which requires additional technical and labor costs.
By obtaining 3D point cloud data and 2D image data in the same scene, the initial internal and external parameters are optimized by comparing the error of the projection information and the 2D box labeling information, the minimum square variance energy equation is constructed and the LM algorithm is used to optimize it to achieve Automated and efficient internal and external parameter updates.
The manpower and material resources required for optimization of internal and external parameters can be reduced, and the phased errors of internal and external parameters during the vehicle driving can be optimized in a timely manner, improving the labeling efficiency and efficiency of internal and external parameters optimization.
Smart Images

Figure CN118918195A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data annotation technology, and more specifically, to an internal and external parameter optimization method, device and storage medium. Background Art
[0002] With the rapid development of technologies such as artificial intelligence and neural network models, big data annotation has become a rapidly growing demand for various technology companies. Due to the popularity of cars and the booming development of autonomous driving technology, big data annotation and training are the key to technological breakthroughs for various intelligent driving companies.
[0003] Most of the existing intelligent driving solutions use the solution of laser radar and multi-camera fusion. Therefore, the big data annotation of intelligent driving scenes usually includes 3D object annotation of laser point cloud data, 2D frame annotation of images, and the relationship between the two.
[0004] In the field of data annotation, external parameters usually refer to the relative position relationship between the lidar and the camera, and internal parameters usually refer to the internal projection parameters of the camera. Due to various reasons such as the installation error of the lidar or camera, the internal and external parameters corresponding to the laser and camera data in the data acquisition process will deviate from the initial given parameters. In addition, the camera or lidar may also shift in position or angle during driving, which may cause the internal and external parameters of the same vehicle to be inconsistent at different time periods.
[0005] Therefore, in the process of jointly annotating the laser data 3D frame and the image 2D frame, due to the inaccuracy of the internal and external parameters, there is a certain deviation when the 3D annotation frame is projected onto the 2D image. At this time, it is necessary to manually observe all objects in the 3D scene and all objects in the 2D image to infer the correspondence between the objects, which greatly reduces the efficiency of annotation. In this regard, the existing technology usually adopts the method of accurately recalibrating the internal and external parameters before data annotation, but this solution requires additional technical and labor costs, and there is a problem that the periodic deviation caused by the jitter of equipment parts during driving cannot be solved in time. Summary of the invention
[0006] The purpose of the embodiments of the present application is to provide an internal and external parameter optimization method, device and storage medium to reduce the manpower and material resources consumed in the optimization of internal and external parameters, and to solve the problem of being unable to optimize the staged errors of internal and external parameters during vehicle driving.
[0007] In a first aspect, an embodiment of the present application provides an internal and external parameter optimization method, including:
[0008] Determine that the initial internal and external parameters of the target vehicle meet the preset update conditions, and obtain at least one frame of 3D point cloud data and at least one corresponding frame of 2D image data under the same scene;
[0009] Acquire 3D frame annotation information of the 3D point cloud data, and acquire 2D frame annotation information of the corresponding 2D image data;
[0010] Determine annotation projection information of the 3D frame annotation information relative to corresponding 2D image data, and filter out comparison projection information corresponding to the 2D frame annotation information from the annotation projection information;
[0011] The initial internal and external parameters are optimized based on a comparison error between the comparison projection information and the 2D frame annotation information to obtain optimized internal and external parameters.
[0012] In an embodiment of the present application, 2D / 3D data of the same scene is obtained, and the error between the information of the projection of the 3D annotation data and the 2D annotation information is obtained, and finally the current internal and external parameters are optimized with the goal of minimizing the error. This allows the internal and external parameters to be automatically optimized and updated according to the set conditions, effectively reducing the manpower and material resources required for the optimization of the internal and external parameters, and can optimize the staged errors of the internal and external parameters during vehicle driving.
[0013] In some possible embodiments, the optimizing the initial internal and external parameters based on the comparison error between the compared projection information and the 2D frame annotation information to obtain the optimized internal and external parameters includes:
[0014] Constructing a minimum square difference energy equation based on the comparison projection information and the 2D frame annotation information;
[0015] The initial internal and external parameters are optimized with the minimum square difference energy equation as the optimization target to obtain optimized internal and external parameters.
[0016] In an embodiment of the present application, by constructing a minimum square difference energy equation for the error between the comparison projection information and the 2D frame annotation information as an optimization target, the efficiency of the internal and external parameter optimization can be further improved.
[0017] In some possible embodiments, the initial internal and external parameters are optimized with the minimum square difference energy equation as the optimization target to obtain the optimized internal and external parameters, including:
[0018] The initial internal and external parameters are optimized by using the LM algorithm with the minimum square difference energy equation as the optimization target to obtain optimized internal and external parameters.
[0019] In the embodiment of the present application, by utilizing the advantage of fast convergence speed of the LM algorithm, the LM algorithm is used to solve the optimization problem of minimizing the error, thereby further improving the efficiency of internal and external parameter optimization.
[0020] In some possible embodiments, the obtaining of 3D frame annotation information of the 3D point cloud data and obtaining of 2D frame annotation information of the corresponding 2D image data includes:
[0021] Acquire 3D frame annotation information of a plurality of target objects in the 3D point cloud data, and generate a 3D frame projection graphic corresponding to the 3D frame annotation information in the associated 2D image data based on an association relationship between the 3D point cloud data and the 2D image data;
[0022] A number of objects to be annotated in the 2D image data are determined based on the 3D frame projection graphic, and 2D frame annotation information of the corresponding 2D image data is acquired based on the number of objects to be annotated.
[0023] In the embodiment of the present application, the 2D image is annotated according to the target object corresponding to the 2D image projected by the 3D annotation information, thereby avoiding unnecessary annotation process and further improving the efficiency of internal and external parameter optimization.
[0024] In some possible embodiments, determining that the initial internal and external parameters of the target vehicle meet a preset update condition includes:
[0025] If it is determined that the application time of the initial internal and external parameters of the target vehicle exceeds a preset update period, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
[0026] In the embodiment of the present application, the optimization process of the internal and external parameters is automatically triggered according to the set cycle, so that the regularity of the optimization of the internal and external parameters can be improved.
[0027] In some possible embodiments, determining that the initial internal and external parameters of the target vehicle meet a preset update condition includes:
[0028] If it is determined that the current jitter monitoring result exceeds a preset jitter allowable range based on the jitter monitoring data of the target vehicle acquired in real time, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
[0029] In an embodiment of the present application, whether to perform internal and external parameter optimization is determined based on the vehicle's vibration monitoring conditions, thereby improving the efficiency of the internal and external parameter optimization.
[0030] In some possible embodiments, determining that the initial internal and external parameters of the target vehicle meet a preset update condition includes:
[0031] Obtain 3D point cloud test data and corresponding 2D image test data under the preset test scene;
[0032] Identify a preset reference point in the 3D point cloud test data and determine the 3D coordinate information of the preset reference point, identify a preset reference point in the 2D image test data and determine the 2D coordinate information of the preset reference point;
[0033] Based on the initial internal and external parameters, 2D projection coordinate information corresponding to the 3D coordinate information is determined. If it is determined that the difference between the 2D projection coordinate information and the 2D coordinate information exceeds a preset threshold, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
[0034] In the embodiment of the present application, by judging whether to perform internal and external parameter optimization based on the error of the reference point with respect to the 2D / 3D data, the timeliness of the internal and external parameter optimization can be improved.
[0035] In a second aspect, an embodiment of the present application provides an internal and external parameter optimization device, including:
[0036] A data acquisition module is used to determine whether the initial internal and external parameters of the target vehicle meet the preset update conditions, and to acquire at least one frame of 3D point cloud data and at least one corresponding frame of 2D image data under the same scene;
[0037] A data annotation module, used to obtain 3D frame annotation information of the 3D point cloud data, and obtain 2D frame annotation information of the corresponding 2D image data;
[0038] An information screening module, used to determine the annotation projection information of the 3D frame annotation information relative to the corresponding 2D image data, and screen out the comparison projection information corresponding to the 2D frame annotation information from the annotation projection information;
[0039] The parameter optimization module is used to optimize the initial internal and external parameters based on the comparison error between the comparison projection information and the 2D frame annotation information to obtain optimized internal and external parameters.
[0040] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the method described in any embodiment of the first aspect when executing the program.
[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented.
[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0044] Figure 1 A schematic diagram of a flow chart of an internal and external parameter optimization method provided in an embodiment of the present application;
[0045] Figure 2 A schematic diagram of the overall process of the internal and external parameter dynamic optimization solution provided in the embodiment of the present application;
[0046] Figure 3 A flowchart of the internal and external parameter optimization algorithm steps provided in the embodiment of the present application;
[0047] Figure 4 A schematic diagram of the structure of an internal and external parameter optimization device provided in an embodiment of the present application;
[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0050] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0051] At present, in the process of jointly annotating the 3D frame of laser data and the 2D frame of the image, the laser point cloud data is usually annotated with 3D frames first, and then the annotated 3D frames are projected into the 2D image through the relative position relationship between the laser device and the camera (external parameters) and the internal projection parameters of the camera (internal parameters). Through the observation of the annotator, the projected 2D frame is associated with the actual corresponding object in the image, and then fine-tuned to finally obtain the associated 2D and 3D frames of a certain object.
[0052] In the process of 2D and 3D frame labeling and association, the inaccuracy of internal and external parameters will lead to a decrease in labeling efficiency. To this end, the existing technology requires accurate recalibration of internal and external parameters before data labeling. However, this solution requires additional technical and labor costs, and there is a problem that the periodic deviation caused by the jitter of equipment parts during driving cannot be solved in time.
[0053] To this end, an embodiment of the present application provides an internal and external parameter optimization method, which can automatically optimize and update the internal and external parameters according to set conditions, effectively reduce the manpower and material resources required for the optimization of internal and external parameters, and can optimize the periodic errors of the internal and external parameters during vehicle driving.
[0054] like Figure 1 As shown, the embodiment of the present application provides an internal and external parameter optimization method, which may include the steps of:
[0055] S1. Determine whether the initial internal and external parameters of the target vehicle meet the preset update conditions, and obtain at least one frame of 3D point cloud data and at least one corresponding frame of 2D image data under the same scene.
[0056] Specifically, the trigger conditions for updating the internal and external parameters can be set according to needs. For example, if you want to make periodic adjustments to the internal and external parameters, you can set the internal and external parameter update cycle according to actual conditions. Whenever the running time reaches an update cycle, it is determined that the preset update conditions are met, and the internal and external parameter optimization process is started.
[0057] Step S1 is to obtain 3D point cloud data and 2D image data of the same scene based on the current initial internal and external parameters when it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions (i.e., it is determined to start the internal and external parameter optimization process). The acquired data can be a data set of multiple frames, each frame of data contains 3D point cloud data and corresponding 2D image data, that is, the 3D point cloud data and the 2D image data are in a one-to-one correspondence.
[0058] S2. Obtain 3D box annotation information of the 3D point cloud data, and obtain 2D box annotation information of the corresponding 2D image data.
[0059] Specifically, for each frame of 3D point cloud data, the target object therein is first annotated with a 3D box, and at the same time, for each frame of 2D image data, the target object therein is also annotated with a 2D box. The number and type of target objects can be set according to demand, and this process can be done manually or automatically. It can be understood that the 3D box annotation information is the information such as the graphics and coordinate points of the target object in the 3D point cloud data, and correspondingly, the 2D box annotation information is the information such as the graphics and coordinate points of the target object in the 2D image data.
[0060] It should be noted that, since the associated 3D box annotation information and 2D box annotation information need to be processed in the subsequent operation process, the number of target objects annotated for 3D point cloud data should be kept consistent with the number of target objects annotated for 2D image data. Usually, some typical target objects such as vehicles can be selected to annotate 3D point cloud data and 2D image data respectively.
[0061] Exemplarily, since each frame of 3D point cloud data and 2D image data are acquired in the same scene, the 3D point cloud data and the corresponding frame of 2D image data have the same target object A. At this time, 3D / 2D annotation is performed on the target object A in each frame of 3D point cloud data and 2D image data, respectively, to obtain a set of 3D box annotation information and 2D box annotation information with associated relationship about the target object A. In the case where the target object A exists in multiple frames of 3D point cloud data and 2D image data, multiple sets of 3D box annotation information and 2D box annotation information with associated relationship about the target object A can be obtained by respectively annotating the multiple frames of data accordingly.
[0062] S3. Determine the annotation projection information of the 3D frame annotation information relative to the corresponding 2D image data, and select the comparison projection information corresponding to the 2D frame annotation information from the annotation projection information.
[0063] It should be noted that since the 3D point cloud data and the 2D image data are in the same scene and correspond one to one, the 3D box annotation information for the target object of the 3D point cloud data can be projected into the 2D image data, and a 2D 3D annotation box will be formed in the 2D image. The original 3D annotation box has 8 spatial coordinates, and based on the current initial internal and external parameter data, the 8 2D coordinate information corresponding to the projection of the 3D annotation box onto the 2D image can be calculated (i.e., the annotation projection information of the 3D box annotation information relative to the corresponding 2D image data).
[0064] Then, the comparison projection information is selected from the 8 projection coordinate information according to the 2D frame annotation information, and is used for error calculation with the 2D frame annotation information.
[0065] S4. Optimize the initial internal and external parameters based on the comparison error between the projection information and the 2D frame annotation information to obtain optimized internal and external parameters.
[0066] It should be noted that, since the 2D frame annotation information includes 4 2D coordinate information pairs, the comparison projection information may be 4 projection coordinate pairs corresponding to the 2D frame annotation information. At this time, the comparison error may be calculated based on the 4 2D coordinate information pairs corresponding to the 4 2D frame annotation information and the 4 projection coordinate pairs. In addition, the comparison projection information may also be a number of projection coordinate values corresponding to certain coordinate values (such as the coordinates of the upper left corner point and the lower right corner point) in the 2D frame annotation information.
[0067] After determining the 2D frame annotation information and its corresponding comparison projection information, the difference can be calculated according to the one-to-one correspondence between the two. Similarly, the difference calculation is performed for the annotation information with associated relationship in each frame of data, and these differences are combined as the comparison error between the comparison projection information and the 2D frame annotation information. Then, the current internal and external parameters are optimized with the goal of minimizing the error, and finally the optimized internal and external parameters are obtained.
[0068] In some possible embodiments, step S4, optimizing the initial internal and external parameters based on the comparison error between the projection information and the 2D frame annotation information to obtain the optimized internal and external parameters, may include:
[0069] S401, constructing a minimum square difference energy equation based on the comparison projection information and the 2D frame annotation information;
[0070] S402, optimizing the initial internal and external parameters with the minimum square difference energy equation as the optimization target to obtain optimized internal and external parameters.
[0071] Further, step S402 may include:
[0072] S4021. Utilize the LM algorithm to optimize the initial internal and external parameters with the minimum square difference energy equation as the optimization target to obtain the optimized internal and external parameters.
[0073] It should be noted that the comparison error between the comparison projection information and the 2D frame annotation information can be reflected based on the minimum square difference. Specifically, the difference operation is performed on the values corresponding to the comparison projection information and the 2D frame annotation information, and then these differences are squared and summed to construct the minimum square difference energy equation. Among them, the value in the comparison projection information is calculated based on the 3D frame annotation information and the internal and external parameters. Therefore, the minimum square difference energy equation is used as the optimization target, and the internal and external parameter data when the algorithm converges can be obtained in the end, that is, the optimized internal and external parameters.
[0074] It should be noted that since the LM algorithm has the advantages of fast convergence speed, insensitivity to initial parameters and nonlinearity, high numerical stability and robustness, the LM algorithm can be used to solve the optimization problem of minimizing errors, thereby further improving the efficiency of internal and external parameter optimization.
[0075] In some possible embodiments, step S2, obtaining 3D box annotation information of 3D point cloud data and obtaining 2D box annotation information of corresponding 2D image data, may include:
[0076] S201, obtaining 3D box annotation information of several target objects in the 3D point cloud data, and generating a 3D box projection graphic corresponding to the 3D box annotation information in the associated 2D image data based on the association relationship between the 3D point cloud data and the 2D image data;
[0077] S202 : determining a number of objects to be annotated in the 2D image data based on the 3D frame projection graphics, and acquiring 2D frame annotation information of the corresponding 2D image data based on the number of objects to be annotated.
[0078] It should be noted that in the annotation process, the 3D point cloud data can first be 3D annotated with the target object (target object), and the corresponding 3D frame projection graphics can be synchronously generated in the associated 2D image data based on these 3D frame annotation information. Based on this, the objects to be annotated in the 2D image data that need to be annotated can be determined based on the generated 3D frame projection graphics, and then 2D frame annotations can be performed on these objects to be annotated in the 2D image data. Among them, after the corresponding 3D frame projection graphics are generated, the target object closest to the position in the 2D image data can be determined as the object to be annotated based on the position information of the generated 3D frame projection graphics. In addition, in the case of manual annotation, the objects to be annotated in the 2D image data can also be directly determined based on the generated 3D frame projection graphics and 2D frame annotation can be performed.
[0079] Based on this, compared with the method of annotating 3D point cloud data and 2D image data separately, by annotating the 2D image according to the graphics projected from the 3D annotation information to the 2D image, the associated objects in the 2D / 3D data can be quickly annotated, avoiding unnecessary annotation process and further improving the efficiency of internal and external parameter optimization.
[0080] In some possible embodiments, in step S1, determining whether the initial internal and external parameters of the target vehicle meet the preset update conditions may include:
[0081] If it is determined that the application time of the initial internal and external parameters of the target vehicle exceeds a preset update period, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
[0082] It should be noted that the parameter update cycle of the internal and external parameters can be set according to the needs, and the application time of the current internal and external parameters can be counted in real time. When it is judged that the application time of the current internal and external parameters exceeds the update cycle, it is considered to meet the preset update conditions. Based on this, the optimization process of the internal and external parameters is automatically triggered according to the set cycle, so that the parameters of the internal and external parameters can be optimized regularly.
[0083] In some possible embodiments, in step S1, determining whether the initial internal and external parameters of the target vehicle meet the preset update conditions may include:
[0084] If it is determined that the current jitter monitoring result exceeds a preset jitter allowable range based on the jitter monitoring data of the target vehicle acquired in real time, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
[0085] Specifically, since the vehicle will be bumpy during driving, which will cause the vibration of various parts on the vehicle, it may be necessary to optimize and adjust the internal and external parameters of the vehicle to offset the parameter error caused by the vibration. In this regard, the vibration monitoring data of the target vehicle can be obtained and counted in real time to determine whether the vibration of the target vehicle exceeds the preset range.
[0086] Exemplarily, the jitter monitoring data can be collected by a vibration sensor deployed on the vehicle. Based on the collected jitter monitoring data, the number of jitters occurring within a certain period and the amplitude of each jitter can be recorded. Accordingly, the jitter allowable range can be the allowed number of jitters or the allowed amplitude of jitters. For example, when the number of jitters greater than a preset amplitude exceeds the preset allowed number of jitters, it is determined that the preset update condition is met; for another example, when the total amplitude of multiple jitters exceeds the allowed amplitude of jitters, it is determined that the preset update condition is met. The specific determination logic can be set according to actual needs.
[0087] In some possible embodiments, in step S1, determining whether the initial internal and external parameters of the target vehicle meet the preset update conditions may include:
[0088] Obtain 3D point cloud test data and corresponding 2D image test data under the preset test scene;
[0089] Identify the preset reference points in the 3D point cloud test data and determine the 3D coordinate information of the preset reference points, identify the preset reference points in the 2D image test data and determine the 2D coordinate information of the preset reference points;
[0090] Based on the initial internal and external parameters, 2D projection coordinate information corresponding to the 3D coordinate information is determined. If it is determined that the difference between the 2D projection coordinate information and the 2D coordinate information exceeds a preset threshold, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
[0091] It is understandable that in addition to determining whether to optimize the internal and external parameters based on a preset cycle or according to vehicle jitter factors, it is also possible to directly judge the accuracy of the current internal and external parameters to determine whether they need to be optimized and updated.
[0092] Specifically, 3D point cloud test data and corresponding 2D image test data are obtained under a preset test scenario, and for a preset reference point, the 3D coordinate information of the reference point in the 3D point cloud test data and the 2D coordinate information of the reference point in the 2D image test data are obtained respectively. Based on the current initial internal and external parameters, the 2D projection coordinate information projected into the 2D image test data can be calculated based on the 3D coordinate information, and then whether the preset update condition is met is determined based on whether the difference between the 2D projection coordinate information and the 2D coordinate information exceeds a preset threshold.
[0093] It is understandable that when the difference between the 2D projection coordinate information and the 2D coordinate information is too large, it means that the initial internal and external parameters at this time have a large error, so the internal and external parameters need to be optimized and updated.
[0094] In addition, in some possible embodiments, the annotator may also initiate a request to update the current initial internal and external parameters. When the system receives the internal and external parameter update request initiated by the annotator, it is determined that the preset update condition is met.
[0095] Based on this, by directly judging whether the error of the current internal and external parameters is too large based on the error of the reference point with respect to the 2D / 3D data, the timeliness of the optimization of the internal and external parameters can be further improved.
[0096] See also Figures 2 to 3 , the following specific examples are given to illustrate the internal and external parameter optimization method provided by the present application. In this example, the system for executing the internal and external parameter optimization method may include a marking client and a server, and the method may include the following steps:
[0097] Step 1: First, obtain the 2D image corresponding to the 3D point cloud data;
[0098] Step 2: Annotate the target object of the 3D point cloud data with a 3D box. Based on the given initial internal and external parameters, project the annotated 3D box to the 2D image, and annotate the corresponding target object in the 2D image with a 2D box. Repeat this step to obtain the 2D annotated boxes and corresponding 3D annotated boxes of multiple objects.
[0099] Step 3: Repeat the previous step for multiple frames of images to obtain annotation results for multiple frames, which include multiple groups of 2D and 3D box association information of multiple target objects.
[0100] Step 4: Calculate the error between the coordinate information of the 3D frame projected onto the 2D image and the 2D frame, and use the LM algorithm that minimizes the error to optimize the initial internal and external parameters. After iterative calculation until the algorithm converges, the optimized internal and external parameters are obtained.
[0101] Step 5: Use the optimized internal and external parameters to replace the initial internal and external parameters, and use the optimized internal and external parameters to label the data in the subsequent labeling process.
[0102] Specifically, see Figure 3 The algorithm flow, the internal and external parameter optimization process of step 4 is as follows:
[0103] Step 401, traverse the annotation results corresponding to each frame of data, and the 3D box and 2D box of the same object ID in each frame of annotation results are regarded as a matching pair (pt3d, img2d), and the annotation results of multiple frames obtain a total of N 2D and 3D annotation box matching pairs. The annotation information of each pt3d object includes 8 corner point coordinates (x, y, z), and the annotation information of each img2d object includes 4 corner point coordinates (u, v).
[0104] Step 402: For each matching pair, the projection coordinate information corresponding to the eight corner points of the 3D frame is solved (annotated projection information), that is, the coordinates of each projection point are calculated using the current initial internal and external parameters: (u1, v1), (u2, v2), (u3, v3), (u4, v4), (u5, v5), (u6, v6), (u7, v7), (u8, v8); the specific calculation method of the projection coordinates is as follows:
[0105] [x1,y1,z1] T =R*[x,y,z] T +T
[0106] x n =x1 / z1,y n =y1 / z1,r 2 =x n 2 +y n 2
[0107] s=x n (1+k1*r 2 +k2*r 4 +k5*r 6 )+2*k3*x n *y n +k4*(r 2 +2*x n 2 )
[0108] t=y n (1+k1*r2 +k2*r 4 +k5*r 6 )+k3*(r 2 +2*y n 2 )+2*k4*x n *y n
[0109] u=f x *s+c x
[0110] v=f y *t+c y
[0111] Among them, [x, y, z] are the point coordinates in the laser coordinate system, [x1, y1, z1] are the point coordinates in the camera coordinate system, R and T are the external parameters of the camera, where R is the rotation matrix (including 3 rotation angles) and T is the translation matrix (including 3 direction translations). x 、f y 、c x and c y is the internal parameter of the camera, where f x and f y is the focal length of the camera in the x and y directions, c x and c y is the center point coordinate of the image, that is, the principal point coordinate (c x , c y ) values; k1, k2, k3, k4, k5 are camera distortion coefficients.
[0112] From the above-mentioned annotated projection information, the comparison projection information corresponding to the 2D box annotation information is selected. In this example, the maximum and minimum values of u and v among the 8 projection points are obtained: max 、u min 、v max and v min .
[0113] Among them, since the coordinates of the four points of the 2D box can be expressed as (u min-2D , v min-2D )(u min-2D , v max-2D )(u max-2D , v min-2D )(u max-2D , v max-2D ). Therefore, u min u with the coordinates of the upper left corner (or lower left corner) of the 2D box min-2D Correspondingly, v max v with the coordinates of the upper left corner (or upper right corner) of the 2D box max-2DCorrespondingly, u max u with the coordinates of the lower right corner (or upper right corner) of the 2D box max-2D Correspondingly, v min v with the coordinates of the lower right corner (or lower left corner) of the 2D box min-2D So far, 4 sets of corresponding coordinate value comparison information are formed: (u max -u max-2D )、(u min -u min-2D )、(v max -v max-2D )、(v min -v min-2D ).
[0114] Step 403: The internal and external parameters to be optimized include 15 parameters in total: the camera internal parameters include f x 、f y 、c x 、c y (focal length f x and f y , principal point coordinates c x and c y ), camera distortion coefficients k1, k2, k3, k4, k5, and external parameters include the rotation matrix R (3 rotation angles) and the translation matrix T (3 direction translations).
[0115] Step 404, construct an energy equation for the minimum square difference between the coordinates of each spatial point of the 3D frame projected onto the image and the corresponding value u or v of the actual 2D annotation (i.e., subtract the comparison information of each pair of coordinate values mentioned above, then square them respectively and sum them up), wherein the projection formula of each spatial point of the 3D frame to the image plane contains the expressions of each internal and external parameter to be optimized, and this energy equation is the final optimization target.
[0116] Step 405: Figure 3 As shown, the LM algorithm is used to solve the above optimization objectives. First, the partial derivatives of each parameter of the above projection formula are solved. And the current projection error: Fd = F u (x)-u, or Fd=F v (x)-v, specifically calculated according to the comparison relationship determined in step 402; calculating the iterative update amount h: h = (J T ·J+λ·I) -1 ·J T Fd, where I is the identity matrix, and again calculate the equation value corresponding to the update amount: If the loss decreases, update it. If it does not decrease, increase the λ value in the LM algorithm (λ is a non-negative parameter that controls the strength of regularization. Regularization is a necessary condition to ensure the convergence of the algorithm, also known as the damping coefficient). Repeat this optimization process until the maximum number of iterations or the update change is almost 0 (for example, less than e -5 ) to obtain the optimized internal and external parameters.
[0117] Step 406: Replace the initial internal and external parameters with the optimized internal and external parameters, and the optimization process ends.
[0118] Please refer to Figure 4 , Figure 4 FIG. 1 shows a block diagram of the internal and external parameter optimization device provided in some embodiments of the present application. It should be understood that the internal and external parameter optimization device is similar to the above Figure 1 Corresponding to the method embodiment, it is able to execute each step involved in the above method embodiment. The specific functions of the internal and external parameter optimization device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.
[0119] Figure 4 The internal and external parameter optimization device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the internal and external parameter optimization device, and the internal and external parameter optimization device includes:
[0120] The data acquisition module 410 is used to determine whether the initial internal and external parameters of the target vehicle meet the preset update conditions, and acquire at least one frame of 3D point cloud data and at least one corresponding frame of 2D image data under the same scene;
[0121] The data annotation module 420 is used to obtain 3D frame annotation information of the 3D point cloud data and obtain 2D frame annotation information of the corresponding 2D image data;
[0122] An information filtering module 430 is used to determine the annotation projection information of the 3D frame annotation information relative to the corresponding 2D image data, and to filter out the comparison projection information corresponding to the 2D frame annotation information from the annotation projection information;
[0123] The parameter optimization module 440 is used to optimize the initial internal and external parameters based on the comparison error between the projection information and the 2D frame annotation information to obtain optimized internal and external parameters.
[0124] It can be understood that the above-mentioned device item embodiment corresponds to the method item embodiment of the present invention. An internal and external parameter optimization device provided by the embodiment of the present invention can implement the internal and external parameter optimization method provided by any method item embodiment of the present invention.
[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.
[0126] like Figure 5 As shown, some embodiments of the present application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 through a bus 530 and executes the program to implement a method of any embodiment included in the above-mentioned internal and external parameter optimization method.
[0127] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.
[0128] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the disclosed embodiment may be used to execute instructions in the memory 510 to implement the method shown above. The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.
[0129] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor to execute the method described in the method embodiment.
[0130] Some embodiments of the present application further provide a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.
[0131] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0132] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0133] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0134] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0135] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0136] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0137] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. A method for optimizing internal and external parameters, characterized in that: include: Determine that the initial internal and external parameters of the target vehicle meet the preset update conditions, and obtain at least one frame of 3D point cloud data and at least one corresponding frame of 2D image data under the same scene; Acquire 3D frame annotation information of the 3D point cloud data, and acquire 2D frame annotation information of the corresponding 2D image data; Determine annotation projection information of the 3D frame annotation information relative to corresponding 2D image data, and filter out comparison projection information corresponding to the 2D frame annotation information from the annotation projection information; The initial internal and external parameters are optimized based on a comparison error between the comparison projection information and the 2D frame annotation information to obtain optimized internal and external parameters.
2. The internal and external parameter optimization method according to claim 1, characterized in that: The optimizing the initial internal and external parameters based on the comparison error between the comparison projection information and the 2D frame annotation information to obtain the optimized internal and external parameters includes: Constructing a minimum square difference energy equation based on the comparison projection information and the 2D frame annotation information; The initial internal and external parameters are optimized with the minimum square difference energy equation as the optimization target to obtain optimized internal and external parameters.
3. The internal and external parameter optimization method according to claim 2, characterized in that: The initial internal and external parameters are optimized by taking the minimum square difference energy equation as the optimization target to obtain the optimized internal and external parameters, including: The initial internal and external parameters are optimized by using the LM algorithm with the minimum square difference energy equation as the optimization target to obtain optimized internal and external parameters.
4. The internal and external parameter optimization method according to claim 1, characterized in that: The obtaining of 3D frame annotation information of the 3D point cloud data and obtaining of 2D frame annotation information of the corresponding 2D image data includes: Acquire 3D frame annotation information of a plurality of target objects in the 3D point cloud data, and generate a 3D frame projection graphic corresponding to the 3D frame annotation information in the associated 2D image data based on an association relationship between the 3D point cloud data and the 2D image data; A number of objects to be annotated in the 2D image data are determined based on the 3D frame projection graphic, and 2D frame annotation information of the corresponding 2D image data is acquired based on the number of objects to be annotated.
5. The internal and external parameter optimization method according to claim 1, characterized in that: The step of determining that the initial internal and external parameters of the target vehicle meet a preset update condition includes: If it is determined that the application time of the initial internal and external parameters of the target vehicle exceeds a preset update period, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
6. The internal and external parameter optimization method according to claim 1, characterized in that: The step of determining that the initial internal and external parameters of the target vehicle meet a preset update condition includes: If it is determined that the current jitter monitoring result exceeds a preset jitter allowable range based on the jitter monitoring data of the target vehicle acquired in real time, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
7. The internal and external parameter optimization method according to claim 1, characterized in that: The step of determining that the initial internal and external parameters of the target vehicle meet a preset update condition includes: Obtain 3D point cloud test data and corresponding 2D image test data under the preset test scene; Identify a preset reference point in the 3D point cloud test data and determine the 3D coordinate information of the preset reference point, identify a preset reference point in the 2D image test data and determine the 2D coordinate information of the preset reference point; Based on the initial internal and external parameters, 2D projection coordinate information corresponding to the 3D coordinate information is determined. If it is determined that the difference between the 2D projection coordinate information and the 2D coordinate information exceeds a preset threshold, it is determined that the initial internal and external parameters of the target vehicle meet the preset update conditions.
8. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the internal and external parameter optimization method described in any one of claims 1 to 7 when executing the program.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the internal and external parameter optimization method according to any one of claims 1 to 7 is executed.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the internal and external parameter optimization method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Radar and camera external parameter calibration method, electronic equipment and storage medium
CN116721162A
Three-dimensional laser radar and camera joint calibration external parameter optimization method based on LM algorithm
CN117111040A
