Camera external parameter optimization method and system, electronic equipment and medium

Abstract

The invention provides a camera extrinsic parameter optimization method and system, electronic equipment and a medium, and belongs to the technical field of vehicle-mounted cameras, and the method comprises the steps: obtaining input data for optimizing camera extrinsic parameters; constructing a target function based on the input data; performing iterative optimization according to a Bayesian optimization search algorithm, the input data and the target function to obtain a camera external parameter optimization result; and screening according to the camera external parameter optimization result of each iteration to obtain a target camera external parameter optimization result. According to the method, the external parameters of the vehicle-mounted camera are finely adjusted based on Bayesian hyper-parameter optimization, when the external parameters of the camera fail, the external parameters of the camera can be efficiently optimized without being limited to a specific calibration workshop and scene, the precision and accuracy of image capturing of the camera are improved, and therefore the performance and accuracy of a camera system are improved.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted camera technology, and in particular to a method, system, electronic device, and medium for optimizing camera extrinsic parameters. Background Technology

[0002] Camera extrinsic parameters are the mathematical foundation of autonomous driving perception systems. They define the rigid transformation relationship (rotation and translation) between the camera coordinate system and the world coordinate system, providing geometric constraints for spatial alignment in multi-sensor data fusion, 3D scene reconstruction, and vehicle localization. In autonomous driving tasks, the accuracy of camera extrinsic parameters directly affects the reliability of mapping image pixel coordinates to the vehicle coordinate system, and is a core prerequisite for functions such as lane line 3D localization, object detection, and distance estimation.

[0003] Traditional extrinsic parameter calibration often relies on manual targets and fixed scenes, requiring pre-set calibration scenarios and struggling to adapt to dynamic environments, easily leading to problems such as extrinsic parameter drift in real-world road scenarios. Furthermore, the optimization process after camera extrinsic parameter failure is costly in terms of both hardware and time. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a camera extrinsic parameter optimization method, system, electronic device and medium.

[0005] In a first aspect, embodiments of the present invention provide a method for optimizing camera extrinsic parameters, including:

[0006] Obtain input data for optimizing camera extrinsic parameters;

[0007] Construct an objective function based on the input data;

[0008] The camera extrinsic parameter optimization results are obtained by iterative optimization based on the Bayesian optimization search algorithm, the input data, and the objective function.

[0009] The optimization results of the camera extrinsic parameters are filtered based on the results of each iteration to obtain the optimization results of the target camera extrinsic parameters.

[0010] In some embodiments, obtaining the input data for optimizing camera extrinsic parameters includes:

[0011] Obtain initial data for the camera to be optimized; wherein, the initial data includes camera images, point cloud data, and initial camera intrinsic and extrinsic parameter data;

[0012] The vehicle grounding point is obtained by labeling and extracting the initial data.

[0013] Based on the vehicle grounding point, find the corresponding projection point on the image plane to establish an image point set and a point cloud point set;

[0014] The image point set and point cloud point set are used as input data for optimizing camera extrinsic parameters.

[0015] In some embodiments, the step of labeling and extracting the initial data to obtain the vehicle grounding point includes:

[0016] The initial data is imported into the annotation platform to annotate target boxes in multiple sets of point cloud-based and image-based radar coordinate systems, resulting in annotated target boxes.

[0017] The selection script automatically extracts the vehicle grounding point from the labeled target box.

[0018] In some embodiments, constructing the objective function based on the input data includes:

[0019] Multiple sets of image point sets and point cloud point sets are obtained based on the input data;

[0020] An objective function is constructed based on the matching error between the image point set and the point cloud point set.

[0021] In some embodiments, the iterative optimization based on the Bayesian optimization search algorithm, the input data, and the objective function to obtain the camera extrinsic parameter optimization result includes:

[0022] Input the input data and objective function into the Bayesian hyperparameter optimization tool;

[0023] Define the six degrees of freedom variables of the camera extrinsic parameters, and set the number of iterations and the search range of the six degrees of freedom variables;

[0024] The camera extrinsic parameters are iteratively updated based on a Bayesian optimization search algorithm to obtain the updated extrinsic parameters.

[0025] Calculate the average Euclidean distance error based on the updated extrinsic parameters;

[0026] The updated extrinsic parameters and the average Euclidean distance error are used as the results of camera extrinsic parameter optimization.

[0027] In some embodiments, obtaining the average Euclidean distance error based on the updated extrinsic parameters includes:

[0028] Based on the initial camera intrinsic and extrinsic parameter data, the image point set is projected onto the image plane to obtain the projected point set;

[0029] Use the set of projection points as the reference point set;

[0030] The updated projection point set is obtained based on the updated extrinsic parameters;

[0031] The average Euclidean distance error is calculated based on the reference point set and the updated projection point set.

[0032] In some embodiments, the step of filtering based on the camera extrinsic optimization results of each iteration to obtain the target camera extrinsic optimization results includes:

[0033] Obtain the camera extrinsic optimization results for each iteration;

[0034] Based on the camera extrinsic parameter optimization results, the target bounding box corresponding to the point cloud under the camera extrinsic parameters is obtained;

[0035] The target bounding box is visualized to obtain the visualization result;

[0036] The target camera extrinsic parameter optimization results are selected based on the visualization results and the average Euclidean distance error.

[0037] Secondly, embodiments of the present invention provide a camera extrinsic parameter optimization system, comprising:

[0038] The data processing module is used to acquire input data for optimizing camera extrinsic parameters;

[0039] The objective function construction module is used to construct an objective function based on the input data;

[0040] The iterative optimization module is used to perform iterative optimization based on the Bayesian optimization search algorithm, the input data, and the objective function to obtain the camera extrinsic parameter optimization results.

[0041] The result filtering module is used to filter the camera extrinsic optimization results based on the results of each iteration to obtain the target camera extrinsic optimization results.

[0042] Thirdly, embodiments of the present invention provide an electronic device, including:

[0043] One or more processors;

[0044] Memory, used to store one or more programs;

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described above.

[0046] Fourthly, embodiments of the present invention provide a computer-readable medium on which a computer program is stored, the computer program being executed by a processor to implement the steps of any of the methods described above.

[0047] The camera extrinsic parameter optimization method provided by this invention includes: acquiring input data for optimizing camera extrinsic parameters; constructing an objective function based on the input data; performing iterative optimization according to a Bayesian optimization search algorithm, the input data, and the objective function to obtain the camera extrinsic parameter optimization result; and filtering the camera extrinsic parameter optimization result based on each iteration to obtain the target camera extrinsic parameter optimization result. This invention fine-tunes the extrinsic parameters of an onboard camera by using Bayesian hyperparameter optimization. When camera extrinsic parameters fail, it can efficiently optimize the camera extrinsic parameters without being limited to specific calibration workshops or scenarios, improving the accuracy and precision of image capture by the camera, thereby improving the performance and accuracy of the camera system. Attached Figure Description

[0048] Figure 1 A flowchart illustrating a camera extrinsic parameter optimization method provided in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the implementation process involved in the embodiments of the present invention;

[0050] Figure 3 This is a schematic diagram comparing the camera extrinsic parameter optimization effects involved in the embodiments of the present invention;

[0051] Figure 4 This is a structural block diagram of a camera extrinsic parameter optimization system provided in an embodiment of the present invention;

[0052] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0054] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0055] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0056] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0057] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0058] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0059] The key terms involved in this invention are defined as follows:

[0060] box: Represents the three-dimensional bounding box of the target object in two-dimensional / three-dimensional space;

[0061] MEDE (Mean Euclidean Distance Error): Mean Euclidean distance error.

[0062] In related technologies, traditional extrinsic parameter calibration often relies on manual targets such as checkerboard calibration boards, using feature point matching and nonlinear optimization to calculate parameters. However, these methods require pre-defined calibration scenarios and are difficult to adapt to dynamic environments, leading to extrinsic parameter drift problems in real-world road scenarios (e.g., camera pose changes caused by vehicle vibration and temperature deformation). In recent years, extrinsic parameter estimation methods based on self-supervised learning have gradually emerged, achieving real-time extrinsic parameter calibration in dynamic environments by jointly optimizing image feature matching and geometric projection consistency. For example, using prior knowledge such as lane vanishing points and road plane assumptions to construct loss functions drives the network to infer extrinsic parameter changes from consecutive frame images, significantly improving calibration robustness. Nevertheless, extrinsic parameter calibration still faces multiple challenges. One problem is that irreversible shifts in camera mounting pose during long-term use can cause extrinsic parameter models to fail, and frequent manual recalibration will severely restrict system availability and calibration costs.

[0063] On one hand, an optimization method for the extrinsic parameters of an onboard camera is proposed. Based on calibration data obtained in a calibration workshop, feature points are extracted from the 2D images obtained during calibration. The extracted feature points are then transformed into 3D coordinates using the original camera's extrinsic and extrinsic parameters. Then, RANSAC estimation is used to find that the projection error between the new 3D coordinate feature points projected onto the image and the feature points in the original image is less than a threshold h, generating 3D matching points corresponding to the feature points in the original image. The same method is then used to reproject the 3D matching points onto the image to obtain 2D matching points, establishing the final matching relationship between the feature points in 2D and 3D. To accelerate the processing, the calibration data is divided into data blocks, and the camera extrinsic parameters of the target camera are optimized using Bundle Adjustment (BA). In this scheme, the calculation of the final matching relationship through the 2D-to-3D transformation of feature points and the 3D-to-2D projection is relatively time-consuming. Furthermore, the obtained calibration data consists of points on a 2D image, which often differs from the actual driving conditions of the vehicle, and the robustness of the optimized result also requires verification.

[0064] On the other hand, a method for online optimization of camera extrinsic parameters based on lane line features, vehicle body attitude, and images from surround-view cameras is proposed. The method involves determining the vehicle's body attitude and optimizing the extrinsic parameters of the surround-view camera when the vehicle is traveling in a straight line on a flat road. Lane line point sets are obtained by detecting lane lines in the surround-view images. Based on the original camera extrinsic and extrinsic parameters, these point sets are projected onto the vehicle coordinate system. Adjustment errors in the vehicle coordinate system data, such as camera pitch, yaw, roll, and position, are analyzed based on the detected lane line width, slope, and actual lane line width. When the adjustment error does not meet preset error conditions, the camera attitude and position are adjusted by a fixed optimization angle or length to make the adjustment error meet the preset error conditions. This allows for online optimization of camera extrinsic parameters, regardless of the public road scene, number of cameras, or type of rule-based calibration objects. However, this approach optimizes camera extrinsic parameters by detecting lane line features in the surround-view images and analyzing the adjustment errors of the lane line point sets in the vehicle coordinate system. The optimization results heavily depend on the lane line model detection performance, and the verification process after optimization cannot be visually displayed. Furthermore, the online optimization of camera extrinsic parameters is based on the vehicle's movement, and the optimization process introduces uncertainties caused by factors such as camera vibration, thus affecting the robustness of the camera extrinsic parameters in different scenarios. Moreover, the algorithm for adjusting the camera's attitude and position is based on iterative calculations using fixed lengths or angles, resulting in relatively high computational complexity.

[0065] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a method for optimizing camera extrinsic parameters. Figure 1 This is a flowchart illustrating a camera extrinsic parameter optimization method provided in an embodiment of the present invention.

[0066] As one embodiment of the present invention, such as Figure 1 As shown, the camera extrinsic parameter optimization method includes:

[0067] Step S1: Obtain input data for optimizing camera extrinsic parameters;

[0068] Step S2: Construct the objective function based on the input data;

[0069] Step S3: Perform iterative optimization based on the Bayesian optimization search algorithm, the input data, and the objective function to obtain the camera extrinsic parameter optimization results;

[0070] Step S4: Filter the results of the camera extrinsic parameter optimization in each iteration to obtain the target camera extrinsic parameter optimization results.

[0071] It should be noted that the execution subject in this embodiment can be an electronic device, which can be a computer device with data processing function, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, the execution subject is a computer device as an example for explanation.

[0072] Specifically, this embodiment proposes a camera extrinsic parameter optimization method, which is a method for fine-tuning the extrinsic parameters of an onboard camera based on Bayesian hyperparameter optimization. In this embodiment, as shown... Figure 2 As shown, the data preparation steps yield an accurate, one-to-one correspondence of image point sets and point cloud point sets, which are used to solve for camera extrinsic parameters. Bayesian hyperparameter optimization is employed, constructing a hyperparameter optimization function with 6 degrees of freedom for the input variables. The Bayesian optimization algorithm searches for the values ​​of the 6 degrees of freedom parameters that minimize the objective value. Camera extrinsic parameter optimization is not limited to expensive calibration workshops; the entire optimization process allows for efficient and intuitive quantitative and qualitative analysis. Even when camera extrinsic parameters fail, it can efficiently optimize them without being limited to specific calibration workshops or scenarios, and can quantitatively and qualitatively analyze the optimized effects. The specific steps are explained below.

[0073] In some embodiments, obtaining input data for optimizing camera extrinsic parameters includes: obtaining initial data of the camera to be optimized; wherein the initial data includes camera images, point cloud data, and initial camera extrinsic and extrinsic parameter data; labeling and extracting the initial data to obtain vehicle grounding points; finding the corresponding projection points on the image plane based on the vehicle grounding points to establish image point sets and point cloud point sets; and using the image point sets and point cloud point sets as input data for optimizing camera extrinsic parameters.

[0074] In some embodiments, the process of labeling and extracting vehicle grounding points based on the initial data includes: importing the initial data into a labeling platform to label target boxes in multiple sets of point cloud-based and image-based radar coordinate systems to obtain labeled target boxes; and automatically extracting vehicle grounding points from the labeled target boxes based on a selection script.

[0075] Specifically, in the data preparation stage, high-quality manual data labeled by an annotation platform can be used, and the data sources can include data collected from vehicles when they are stationary.

[0076] For example, such as Figure 2 As shown, data preparation involves importing the images, point clouds, and initial camera intrinsic and extrinsic parameter data corresponding to the cameras requiring optimization into the annotation platform. For vehicle target boxes with complete point cloud data, multiple sets of target boxes in radar coordinate systems (point cloud accuracy and image accuracy) can be manually annotated. A selection script is written to extract the vehicle grounding points from the eight vertices of the target box, which will then be used as input data for optimizing camera extrinsic parameters.

[0077] In one example, the image data captured by the camera that needs optimization, the corresponding point cloud data (3D data captured by radar or lidar), and the camera's initial intrinsic and extrinsic parameters (intrinsic parameters include focal length, principal point coordinates, etc., and extrinsic parameters include camera position and attitude) are imported into an annotation platform to assist manual data annotation and preprocessing. In the point cloud data, vehicle targets are usually represented by bounding boxes (e.g., 3D bounding boxes), which should be complete, i.e., include all key parts of the vehicle.

[0078] For example, a human operator annotates vehicles in point cloud data on an annotation platform, ensuring that these bounding boxes conform to both the actual positions in the point cloud data (point cloud accurate) and the positions in the image data (image accurate) in the radar coordinate system. A script is then written to automatically extract the vehicle's ground contact point from the annotated bounding boxes; the ground contact point can refer to the lowest point where the vehicle contacts the ground. This script can analyze the eight vertices of the bounding box to determine the point among these vertices that contacts the ground, or infer the vehicle's ground contact point through calculation.

[0079] In this embodiment, by employing dual constraints of manually labeled image accuracy and 3D point cloud accuracy, the optimization time cost is reduced from the data optimization perspective. Furthermore, in this embodiment, feature points can be selected using the grounding points of road vehicles to obtain the data required for calibration, making the calibration scene closer to the actual driving state of the vehicle. This improves the precision and accuracy of the images captured by the camera, thereby enhancing the performance and accuracy of the camera system.

[0080] It is understood that, in this embodiment, the vehicle grounding point is obtained by annotation and extraction. In addition to using the vehicle grounding point as the feature point, other target objects with stable features can also be used as feature points. This embodiment does not limit this.

[0081] In some embodiments, constructing an objective function based on the input data includes: obtaining multiple sets of image point sets and point cloud point sets based on the input data; and constructing an objective function based on the matching error between the image point sets and the point cloud point sets.

[0082] In some embodiments, six variables (six degrees of freedom variables) of camera extrinsic parameters are defined, including rotation variables about the x-axis, rotation variables about the y-axis, rotation variables about the z-axis, translation variables about the x-axis, translation variables about the y-axis, and translation variables about the z-axis.

[0083] Specifically, such as Figure 2 As shown, the objective function is constructed as follows: Based on multiple sets of point cloud-based quasi-points and image-based quasi-points, the image-based quasi-point set is projected onto the image using the original camera intrinsic and extrinsic parameters as a reference. The changes in six camera extrinsic parameters (rotation variable) are determined based on the input. yaw (around the y-axis) pitch (around the x-axis) roll (around the z-axis) and translate the variable x, y, z) Update the camera extrinsic parameters. Project the point set from the point cloud extrinsic parameters onto the image under the new camera extrinsic parameters. Finally, calculate the mean Euclidean distance error (MEDE) between the two sets of points, one from the image extrinsic parameters and the other from the point cloud extrinsic parameters, using the following formula:

[0084]

[0085] Where N represents the number of point set pairs; Represents the i-th "image reference point"; c i This represents the point on the image projected by the i-th "point cloud accuracy point" under the new extrinsic parameters.

[0086] As is understandable, the objective function is a function used in machine learning to guide model optimization, guiding model training by evaluating the difference between the model's predicted values ​​and the true values. In this embodiment, the objective function is used to evaluate and optimize camera extrinsic parameters, making the point cloud data more consistent with the image data.

[0087] For example, multiple sets of point cloud-accurate points and image-accurate points are input. Using the camera's initial intrinsic and extrinsic parameters, the "image-accurate points" are projected onto the image plane; this projected point set serves as the reference point set. The objective function considers six variables of the camera's extrinsic parameters, including three rotation variables and three translation variables. These variables represent minor adjustments to the original extrinsic parameters, used to find the optimal extrinsic parameter values. Based on the feedback from the objective function, the changes in these six variables are determined, thereby updating the camera's extrinsic parameters. Based on the updated camera extrinsic parameters, the "point cloud-accurate points" are projected onto the image plane, resulting in a new projected point set. The mean Euclidean distance error (MEDE) is calculated using the formula. The optimization objective is to minimize this MEDE. When the MEDE is minimized, it means the camera's extrinsic parameters are optimal, and the point cloud data and image data are most consistent. By iteratively optimizing these six variables until the extrinsic parameter values ​​that minimize the MEDE are found, the optimization of the camera's extrinsic parameters is completed.

[0088] In some embodiments, iterative optimization is performed based on a Bayesian optimization search algorithm, the input data, and the objective function to obtain camera extrinsic parameter optimization results. This includes: inputting the input data and the objective function into a Bayesian hyperparameter optimization tool; defining six degrees of freedom variables for the camera extrinsic parameters, setting the number of iterations and the search range of the six degrees of freedom variables; iteratively updating the camera extrinsic parameters based on the Bayesian optimization search algorithm to obtain updated extrinsic parameters; calculating the mean Euclidean distance error based on the updated extrinsic parameters; and using the updated extrinsic parameters and the mean Euclidean distance error as the camera extrinsic parameter optimization results.

[0089] In some embodiments, obtaining the average Euclidean distance error based on the updated extrinsic parameters includes: projecting an image point set onto an image plane based on initial camera extrinsic and extrinsic parameter data to obtain a projected point set; using the projected point set as a reference point set; obtaining an updated projected point set based on the updated extrinsic parameters; and calculating the average Euclidean distance error based on the reference point set and the updated projected point set.

[0090] Specifically, based on the Bayesian hyperparameter optimization tool: according to the constructed objective function input, the search range of the input variables is set, for example: the rotation variables (yaw, pitch, roll) are [-30°, 30°], the translation variables (x, y, z) are [-0.3m, 0.3m], the number of iterations is 240, and the MEDE value calculated in each iteration is output.

[0091] For example, for rotation variables (yaw, pitch, roll), the search range can be set to [-30°, 30°], and for translation variables (x, y, z), the search range can be set to [-0.3m, 0.3m]. A Bayesian hyperparameter optimization tool is used, with the constructed objective function as input. The number of iterations can be set to 240, meaning the optimization process will run for 240 iterations. In each iteration, the mean Euclidean distance error (MEDE) is calculated using the objective function based on the current changes in the extrinsic parameters. Based on the MEDE value, the Bayesian optimization tool will guide how to adjust the changes in the extrinsic parameters to reduce the error. During each iteration, the calculated MEDE value is recorded and output; these values ​​will be used to evaluate the effectiveness of the optimization process and provide the optimal changes in the extrinsic parameters after the iteration is complete.

[0092] In this embodiment, the Bayesian optimization tool effectively guides the search direction, accelerates the optimization process, and improves the accuracy of the solution. Through continuous iteration and adjustment, the camera's extrinsic parameters are gradually optimized until the solution with the minimum error is found.

[0093] It should be noted that this embodiment uses the Bayesian optimization algorithm as an example. Besides using the Bayesian optimization algorithm to search for the optimal solution that minimizes the objective value, other algorithms such as grid search and random search can also be used to find the optimal solution; this embodiment does not impose any limitations on this.

[0094] In some embodiments, the step of filtering based on the camera extrinsic optimization results of each iteration to obtain the target camera extrinsic optimization results includes: obtaining the camera extrinsic optimization results of each iteration; obtaining the target bounding box corresponding to the point cloud point under the camera extrinsic parameters based on the camera extrinsic optimization results; visualizing the target bounding box to obtain the visualization result; and filtering the target camera extrinsic optimization results based on the visualization result and the average Euclidean distance error.

[0095] Specifically, the results of the iteration process are organized and analyzed: the target bounding boxes corresponding to the point cloud points under the camera extrinsic parameters obtained in each iteration are visualized, and the iteration result with the best camera extrinsic parameter optimization effect is selected by comprehensively considering the visualization results and the mean Euclidean distance error (MEDE) value. Figure 3 The results show the quantitative and qualitative results after camera extrinsic parameter optimization. (Reference) Figure 3 Before adjustment, the mean Euclidean distance error (MEDE) was 40.563. After optimization using the camera extrinsic optimization method provided in this embodiment, the mean Euclidean distance error (MEDE) was 4.420. Obviously, the MEDE value was significantly reduced after adjustment, and the point cloud data and image data were more consistent. The camera extrinsic optimization method provided in this embodiment can improve the accuracy and precision of the camera in capturing images, thereby improving the performance and accuracy of the camera system.

[0096] This embodiment proposes a method for fine-tuning the extrinsic parameters of an onboard camera based on Bayesian hyperparameter optimization. By utilizing the dual constraints of manually labeled image accuracy and 3D point cloud accuracy, it reduces optimization time costs from the data optimization perspective. Furthermore, feature points can be selected using the grounding points of road vehicles to obtain the data needed for calibration, making the calibration scenario closer to the actual driving state of the vehicle. This embodiment proposes using high-quality manually labeled data from a labeling platform, with the data source being data collected from the vehicle in a stationary state, allowing control over the accuracy of the input data before optimization. Regarding computational complexity, this embodiment proposes using a Bayesian hyperparameter optimization tool to construct an objective function for the average geometric error of the projection of the point cloud data (accurate point cloud data and image accuracy) onto the image, thereby efficiently optimizing the camera extrinsic parameters.

[0097] The camera extrinsic parameter optimization method provided in this embodiment includes: acquiring input data for optimizing camera extrinsic parameters; constructing an objective function based on the input data; performing iterative optimization according to a Bayesian optimization search algorithm, the input data, and the objective function to obtain the camera extrinsic parameter optimization result; and filtering the camera extrinsic parameter optimization result based on the result of each iteration to obtain the target camera extrinsic parameter optimization result. This embodiment uses Bayesian hyperparameter optimization to fine-tune the extrinsic parameters of an onboard camera. When camera extrinsic parameters fail, it can efficiently optimize the camera extrinsic parameters, not limited to specific calibration workshops and scenarios, and can quantitatively and qualitatively analyze the optimization effect.

[0098] Reference Figure 4 , Figure 4 This is a structural block diagram of an embodiment of the camera extrinsic parameter optimization system of the present invention. Figure 4 As shown, the camera extrinsic parameter optimization system includes:

[0099] Data processing module 10 is used to acquire input data for optimizing camera extrinsic parameters;

[0100] Objective function construction module 20 is used to construct an objective function based on the input data;

[0101] The iterative optimization module 30 is used to perform iterative optimization based on the Bayesian optimization search algorithm, the input data, and the objective function to obtain the camera extrinsic parameter optimization results.

[0102] The result filtering module 40 is used to filter the camera extrinsic optimization results based on the results of each iteration to obtain the target camera extrinsic optimization results.

[0103] Specifically, data preparation involves importing the corresponding images, point clouds, and initial camera intrinsic and extrinsic parameter data of the cameras requiring optimization into the annotation platform. For vehicle target boxes with complete point cloud data, multiple sets of target boxes in radar coordinate systems (point cloud-accurate and image-accurate) are manually annotated. A selection script is written to extract the vehicle grounding points from the eight vertices of the target box, which are then used as input data for optimizing camera extrinsic parameters.

[0104] Specifically, the objective function is constructed as follows: based on multiple sets of input point cloud-based quasi-points and image-based quasi-points, the image-based quasi-point set is projected onto the image using the original camera intrinsic and extrinsic parameters as a reference. The changes in six camera extrinsic parameters (rotation variable) are determined based on the input. yaw (around the y-axis) pitch (around the x-axis) roll (around the z-axis) and translate the variable x, y, z) Update the camera extrinsic parameters. The point set of the point cloud is projected onto the image under the new camera extrinsic parameters. Finally, according to the formula... Calculate the mean Euclidean distance error (MEDE) between the two sets of points: the image reference and the point cloud reference.

[0105] Specifically, based on the Bayesian hyperparameter optimization tool: according to the constructed objective function input, the search range of the input variables is set: the rotation variables (yaw, pitch, roll) are [-30°, 30°], the translation variables (x, y, z) are [-0.3m, 0.3m], the number of iterations is 240, and the average Euclidean distance error (MEDE) value calculated in each iteration is output.

[0106] Specifically, the results of the iteration process are organized and analyzed: the target bounding boxes corresponding to the point cloud points under the camera extrinsic parameters obtained in each iteration are visualized, and the iteration result with the best camera extrinsic parameter optimization effect is selected by comprehensively considering the visualization results and the mean Euclidean distance error (MEDE) value. Figure 3 The results shown are the quantitative and qualitative results after the optimization of the camera extrinsic parameters.

[0107] The camera extrinsic parameter optimization system provided in this embodiment reduces optimization time costs from the data optimization perspective by utilizing the dual constraints of manually labeled image accuracy and 3D point cloud accuracy. Furthermore, feature points can be selected using the grounding points of road vehicles to obtain the data required for calibration, making the calibration scenario closer to the actual driving state of the vehicle. This embodiment proposes using high-quality manually labeled data from a labeling platform; the data source is data collected from vehicles in a stationary state, allowing control over the accuracy of the input data before optimization. Regarding computational complexity, this embodiment proposes using a Bayesian hyperparameter optimization tool to construct an objective function for the average geometric error of the projection of point cloud data (both point cloud accuracy and image accuracy) onto the image. This efficiently optimizes camera extrinsic parameters, improving the accuracy and precision of the captured images, thereby enhancing the performance and accuracy of the camera system.

[0108] In addition, for technical details not described in detail in this embodiment of the camera extrinsic optimization system, please refer to the camera extrinsic optimization method provided in any embodiment of the present invention, which will not be repeated here.

[0109] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the camera extrinsic parameter optimization methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0110] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0111] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0112] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0113] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps in any of the camera extrinsic parameter optimization methods described in the above embodiments. The computer-readable storage medium can be volatile or non-volatile.

[0114] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described camera extrinsic optimization method.

[0115] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0116] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0117] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0118] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0119] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0120] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0121] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0122] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0124] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for camera extrinsic parameter optimization, characterized in that, The method comprises the following steps: obtaining input data for optimizing camera extrinsic parameters; constructing a target function based on the input data; iterative optimization according to a Bayesian optimization search algorithm and the input data and the target function to obtain camera extrinsic parameter optimization results; screening the camera extrinsic parameter optimization results of each iteration to obtain target camera extrinsic parameter optimization results.

2. The method of claim 1, wherein, The input data for optimizing camera extrinsic parameters comprises: obtaining initial data of a camera to be optimized; wherein the initial data comprises camera pictures, point cloud data and initial camera intrinsic and extrinsic parameter data; annotating and extracting according to the initial data to obtain a vehicle grounding point; finding a corresponding projection point on an image plane according to the vehicle grounding point to establish an image point set and a point cloud point set; taking the image point set and the point cloud point set as input data for optimizing camera extrinsic parameters.

3. The method of claim 2, wherein, The annotating and extracting according to the initial data to obtain a vehicle grounding point comprises: importing the initial data into an annotation platform to annotate a target box in a radar coordinate system of multiple groups of point cloud and image to obtain an annotated target box; automatically extracting a vehicle grounding point from the annotated target box based on a selection script.

4. The method of claim 1, wherein, The constructing a target function based on the input data comprises: obtaining multiple image point sets and point cloud point sets from the input data; constructing a target function based on matching errors between the image point sets and the point cloud point sets.

5. The method of claim 1, wherein, The iterative optimization according to a Bayesian optimization search algorithm and the input data and the target function to obtain camera extrinsic parameter optimization results comprises: inputting the input data and the target function into a Bayesian hyperparameter optimization tool; defining six degrees of freedom variables of camera extrinsic parameters, setting the number of iterations and the search range of the six degrees of freedom variables; iteratively updating camera extrinsic parameters based on a Bayesian optimization search algorithm to obtain updated extrinsic parameters; calculating an average Euclidean distance error according to the updated extrinsic parameters; taking the updated extrinsic parameters and the average Euclidean distance error as camera extrinsic parameter optimization results.

6. The method of claim 5, wherein, The calculating an average Euclidean distance error according to the updated extrinsic parameters comprises: projecting an image point set onto an image plane according to initial camera intrinsic and extrinsic parameter data to obtain a projection point set; taking the projection point set as a reference point set; obtaining an updated projection point set according to the updated extrinsic parameters; calculating an average Euclidean distance error according to the reference point set and the updated projection point set.

7. The method according to any one of claims 1 to 6, characterized in that, The screening the camera extrinsic parameter optimization results of each iteration to obtain target camera extrinsic parameter optimization results comprises: obtaining camera extrinsic parameter optimization results of each iteration; obtaining a target box corresponding to a point cloud point according to camera extrinsic parameters of the camera extrinsic parameter optimization results; visualizing the target box to obtain a visualization result; screening target camera extrinsic parameter optimization results according to the visualization result and an average Euclidean distance error.

8. A camera extrinsic parameter optimization system, comprising: The method comprises the following steps: a data processing module for obtaining input data for optimizing camera extrinsic parameters; a target function construction module for constructing a target function based on the input data; an iterative optimization module for iterative optimization according to a Bayesian optimization search algorithm and the input data and the target function to obtain camera extrinsic parameter optimization results; A result screening module is configured to screen the camera extrinsic parameter optimization results according to each iteration of the camera extrinsic parameter optimization results to obtain a target camera extrinsic parameter optimization result.

9. An electronic device, comprising: Comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 7.

10. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 7.

Images