Camera parameter calibration method and device and electronic equipment

Abstract

The invention provides a camera parameter calibration method and device and electronic equipment, which are applied to the technical field of camera calibration, and the method comprises the steps: obtaining camera data, point cloud data and a first camera parameter group set of a target object; determining a first objective function value based on the first camera parameter group and a mapping relation between the camera data and the point cloud data, and determining a target camera external parameter and a target camera internal parameter based on a minimum value of a plurality of first objective function values; determining a second camera parameter group set according to the target camera external parameter, the target camera internal parameter and the plurality of reference offset times; and determining second target function values based on the second camera parameter group and the mapping relationship between the camera data and the point cloud data, and determining target migration time based on the minimum value of the plurality of second target function values. Thus, by accurately solving the millisecond-level time offset, an external parameter calibration error introduced by sensor movement and time desynchrony is greatly eliminated, and the calibration precision in a dynamic scene and the performance of subsequent sensor fusion are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of camera calibration, and more specifically, to a method, apparatus, and electronic device for calibrating camera parameters. Background Technology

[0002] To achieve accurate fusion of camera data and radar point cloud data, it is necessary to accurately calibrate the extrinsic and intrinsic parameters of the camera relative to the radar coordinate system. At the same time, due to factors such as hardware triggering, data transmission delay, exposure time variation, and system clock asynchrony, there is often a millisecond-level time offset between the timestamp of the camera data and the actual timestamp of the LiDAR scan.

[0003] In related technologies, calibrating the camera's extrinsic and intrinsic parameters relative to the radar coordinate system requires setting up a calibration object and a specific calibration scene, and treating the calibration time offset as an independent step. This separate processing method disrupts the inherent coupling relationship between the parameters and affects the accuracy of camera parameter calibration. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for calibrating camera parameters. First, a preset time offset is fixed, and different combinations of camera intrinsic and extrinsic parameters are iteratively optimized to determine the optimal camera intrinsic and extrinsic parameters. Then, the calibrated intrinsic and extrinsic parameters are fixed, and different time offsets are iteratively optimized to determine the optimal time offset. In this way, by co-optimizing camera intrinsic and extrinsic parameters and time offset, error propagation in traditional separate calibration methods is effectively reduced, achieving accurate estimation of millisecond-level time offsets. This significantly suppresses extrinsic parameter calibration errors caused by the asynchronous movement of sensors and time. It not only greatly improves calibration accuracy in dynamic scenes and the performance of subsequent sensor fusion, but also eliminates the dependence on specific calibration objects and scenes, allowing direct online calibration using regular vehicle driving data, greatly enhancing the applicability and engineering practicality of the method.

[0005] Firstly, a method for calibrating camera parameters is provided. This method includes: acquiring camera data, point cloud data, and a first set of camera parameters for a target object, wherein the first set of camera parameters includes multiple first camera parameter sets, each including camera extrinsic parameters, camera intrinsic parameters, and a preset offset time; determining a first objective function value based on the mapping relationship between the first camera parameter sets, camera data, and point cloud data, and determining target camera extrinsic parameters and target camera intrinsic parameters based on the minimum value of the multiple first objective function values; determining a second set of camera parameters based on the target camera extrinsic parameters, target camera intrinsic parameters, and multiple reference offset times, wherein the second set of camera parameters includes multiple second camera parameter sets, each including target camera extrinsic parameters, target camera intrinsic parameters, and a reference offset time; determining a second objective function value based on the mapping relationship between the second camera parameter sets, camera data, and point cloud data, and determining a target offset time based on the minimum value of the multiple second objective function values. In this way, by co-optimizing the camera's intrinsic and extrinsic parameters and time offset, the error propagation in the traditional separation calibration method is effectively reduced, and the accurate estimation of the time offset at the millisecond level is achieved. This significantly suppresses the extrinsic parameter calibration error caused by the asynchronous movement of the sensor and time. It not only greatly improves the calibration accuracy and subsequent sensor fusion performance in dynamic scenes, but also eliminates the dependence on specific calibration objects and scenes. It can directly use the vehicle's regular driving data for online calibration, which greatly enhances the applicability and engineering practicality of the method.

[0006] According to one embodiment of this application, determining a first objective function value based on the mapping relationship between a first camera parameter set, camera data, and point cloud data includes: projecting point cloud data onto a two-dimensional plane based on the first camera parameter set to determine first projection data; calculating multiple first pixel distances between camera data and corresponding first projection data, and performing a weighted summation of the multiple first pixel distances to determine the first objective function value.

[0007] According to one embodiment of this application, determining a second objective function value based on the mapping relationship between a second camera parameter set, camera data, and point cloud data includes: projecting point cloud data onto a two-dimensional plane based on the second camera parameter set to determine second projection data; calculating multiple second pixel distances between camera data and corresponding second projection data, and performing a weighted summation of the multiple second pixel distances to determine the second objective function value.

[0008] According to one embodiment of this application, the method further includes: determining synthetic geometry data, synthetic depth data, and synthetic rendering mask based on point cloud data and an initial camera parameter set; constructing a corresponding point set between camera data and point cloud data based on the synthetic geometry data, synthetic depth data, synthetic rendering mask, and camera data, wherein the corresponding point set includes the mapping relationship between camera data and point cloud data.

[0009] According to one embodiment of this application, constructing a corresponding point set between camera data and point cloud data based on synthetic geometric data, synthetic depth data, synthetic rendering mask, and camera data includes: performing feature extraction on camera data to determine a first set of feature points; performing feature extraction on synthetic geometric data to determine a second set of feature points; performing feature matching on the first set of feature points and the second set of feature points to determine multiple feature pairs; determining target second feature points in the multiple feature pairs based on the synthetic rendering mask; determining the depth value of each target second feature point based on the synthetic depth data, and back-projecting the coordinates and depth values ​​of each target second feature point based on an initial camera parameter set to determine the corresponding virtual point cloud; searching for the target point cloud in the point cloud data that is closest to the multiple virtual point clouds; determining target camera data corresponding to the multiple target point clouds based on the multiple feature pairs; and constructing a corresponding point set between camera data and point cloud data based on the multiple point clouds and the multiple target camera data.

[0010] According to one embodiment of this application, after constructing the corresponding point set of camera data and point cloud data, the above method further includes: performing spatial uniform sampling on the corresponding point set.

[0011] According to one embodiment of this application, the above method further includes: determining the minimum value of a plurality of first objective function values ​​based on a first preset optimization algorithm.

[0012] According to one embodiment of this application, the above method further includes: determining the minimum value of a plurality of second objective function values ​​based on a second preset optimization algorithm.

[0013] Secondly, a camera parameter calibration device is provided. The device includes: a first acquisition module, used to acquire camera data, point cloud data, and a first camera parameter set of a target object, wherein the first camera parameter set includes multiple first camera parameter sets, each including camera extrinsic parameters, camera intrinsic parameters, and a preset offset time; a first determination module, used to determine a first objective function value based on the mapping relationship between the first camera parameter sets, camera data, and point cloud data, and to determine target camera extrinsic parameters and target camera intrinsic parameters based on the minimum value of multiple first objective function values; a second determination module, used to determine a second camera parameter set based on the target camera extrinsic parameters, target camera intrinsic parameters, and multiple reference offset times, wherein the second camera parameter set includes multiple second camera parameter sets, each including target camera extrinsic parameters, target camera intrinsic parameters, and a reference offset time; and a third determination module, used to determine a second objective function value based on the mapping relationship between the second camera parameter set, camera data, and point cloud data, and to determine a target offset time based on the minimum value of multiple second objective function values.

[0014] Thirdly, an electronic device is provided, including a memory, a processor, and a camera parameter calibration program stored in the memory and executable on the processor. When the processor executes the camera parameter calibration program, it implements the aforementioned camera parameter calibration method.

[0015] The camera parameter calibration method, apparatus, and electronic device of this application acquire camera data, point cloud data, and a first camera parameter set for a target object. The first camera parameter set includes multiple first camera parameter sets, each including camera extrinsic parameters, camera intrinsic parameters, and a preset offset time. Based on the mapping relationship between the first camera parameter sets, camera data, and point cloud data, a first objective function value is determined. The target camera extrinsic parameters and target camera intrinsic parameters are determined based on the minimum value of the multiple first objective function values. A second camera parameter set is determined based on the target camera extrinsic parameters, target camera intrinsic parameters, and multiple reference offset times. The second camera parameter set includes multiple second camera parameter sets, each including target camera extrinsic parameters, target camera intrinsic parameters, and a reference offset time. A second objective function value is determined based on the mapping relationship between the second camera parameter sets, camera data, and point cloud data. The target offset time is determined based on the minimum value of the multiple second objective function values. In this way, by co-optimizing the camera's intrinsic and extrinsic parameters and time offset, the error propagation in the traditional separation calibration method is effectively reduced, and the accurate estimation of the time offset at the millisecond level is achieved. This significantly suppresses the extrinsic parameter calibration error caused by the asynchronous movement of the sensor and time. It not only greatly improves the calibration accuracy and subsequent sensor fusion performance in dynamic scenes, but also eliminates the dependence on specific calibration objects and scenes. It can directly use the vehicle's regular driving data for online calibration, which greatly enhances the applicability and engineering practicality of the method. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for calibrating camera parameters according to some embodiments of this application; Figure 2 This is a schematic diagram of synthetic visual data according to some embodiments of this application; Figure 3 This is a flowchart illustrating the construction of a corresponding point set of camera data and point cloud data according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a camera parameter calibration system according to some embodiments of this application; Figure 5 This is a block diagram of a camera parameter calibration device according to some embodiments of this application; Figure 6 This is a block diagram of an electronic device according to some embodiments of this application. Detailed Implementation

[0017] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0018] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0019] The camera parameter calibration method, apparatus, and electronic device of this application embodiment will now be described in detail with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a camera parameter calibration method according to some embodiments of this application. (Refer to...) Figure 1 The camera parameter calibration method in this application embodiment may include the following steps: S110: Obtain camera data, point cloud data, and a first camera parameter set for the target object. The first camera parameter set includes multiple first camera parameter sets, each including camera extrinsic parameters, camera intrinsic parameters, and a preset offset time. The target object can be the road in front of the vehicle; no specific restrictions are imposed here.

[0021] Specifically, camera data of the target object can be acquired through vehicle-mounted cameras, while point cloud data of the target object can be acquired through vehicle-mounted LiDAR. After acquiring the point cloud data, it needs to be preprocessed, such as data loading and filtering (removing noise below the ground or excessively high noise), voxel downsampling (reducing data redundancy and improving data processing efficiency), and normal vector and intensity calculation (calculating the normal vector information of each point and normalizing the intensity information of the point cloud).

[0022] When calibrating the camera parameters relative to the radar coordinate system, it is necessary to set a first set of camera parameters. Each first set of camera parameters includes camera extrinsic parameters, camera intrinsic parameters, and a preset offset time. The preset offset time can be calibrated according to the actual situation. For example, the preset offset time can be 0. There is no specific restriction here.

[0023] In other words, the preset offset time is fixed first, and multiple different combinations of camera extrinsic and camera intrinsic parameters are set. The target camera extrinsic and target camera intrinsic parameters are then found from these combinations. The camera extrinsic parameters include the 6-DOF transformation T_ego_cam of the camera relative to the carrier coordinate system (e.g., vehicle), which can be represented by its se(3) Lie algebra ξ_ego_cam. The camera intrinsic parameters include the camera focal length, principal point coordinates, and distortion coefficients.

[0024] S120, determine the first objective function value based on the mapping relationship between the first camera parameter group, camera data and point cloud data, and determine the target camera extrinsic parameters and target camera intrinsic parameters based on the minimum value of multiple first objective function values.

[0025] Specifically, camera data includes multiple two-dimensional pixels, and point cloud data includes multiple three-dimensional spatial points. After acquiring the camera data and point cloud data, feature matching is performed on the two-dimensional pixels in the camera data and the three-dimensional spatial points in the point cloud data to determine the three-dimensional spatial point corresponding to each two-dimensional pixel, thereby determining the mapping relationship between the camera data and the point cloud data.

[0026] After determining the mapping relationship between camera data and point cloud data, the corresponding camera data for the point cloud data can be determined based on this mapping relationship. Projecting the point cloud data onto a two-dimensional plane based on the first camera parameter set yields the corresponding first projection data. In other words, there is a one-to-one correspondence between camera data and point cloud data, and also a one-to-one correspondence between the first projection data and the point cloud data. Furthermore, the two-dimensional pixel coordinates of the camera data and the two-dimensional pixel coordinates of the first projection data are input into a preset formula to calculate the first objective function value. Similarly, the first objective function value corresponding to each first camera parameter set in the first camera parameter set is calculated using the above method. The first camera parameter set corresponding to the minimum value among multiple first objective function values ​​is determined as the target first camera parameter set. The camera extrinsic and intrinsic parameters in the target camera parameter set are the target camera intrinsic and target camera extrinsic parameters.

[0027] S130, determine a set of second camera parameters based on the target camera extrinsic parameters, the target camera intrinsic parameters, and multiple reference offset times. The set of second camera parameters includes multiple second camera parameter sets, and each second camera parameter set includes the target camera extrinsic parameters, the target camera intrinsic parameters, and the reference offset times.

[0028] Specifically, after determining the target camera's extrinsic and intrinsic parameters, multiple reference time offsets are set. The target camera's extrinsic and intrinsic parameters are then combined with different reference time offsets to obtain a second set of camera parameters. In other words, the target camera's extrinsic and intrinsic parameters are fixed, multiple different offset times are set, and the target offset time is found from these multiple different offset times.

[0029] S140, determine the second objective function value based on the mapping relationship between the second camera parameter group, camera data and point cloud data, and determine the target offset time based on the minimum value of multiple second objective function values.

[0030] Specifically, based on the mapping relationship between camera data and point cloud data, the corresponding camera data for the point cloud data can be determined. Projecting the point cloud data onto a two-dimensional plane based on the second camera parameter set yields the corresponding second projection data. In other words, there is a one-to-one correspondence between camera data and point cloud data, and also a one-to-one correspondence between the second projection data and point cloud data. Furthermore, the two-dimensional pixel coordinates of the camera data and the two-dimensional pixel coordinates of the second projection data are input into a preset formula to calculate the second objective function value. This process is repeated to calculate the second objective function value for each second camera parameter set in the second camera parameter set. The second camera parameter set corresponding to the minimum value among multiple second objective function values ​​is determined as the target second camera parameter set, and the time offset in the target camera parameter set is the target time offset.

[0031] The calibration method of this application first fixes a preset time offset and iterates through and optimizes different combinations of camera intrinsic and extrinsic parameters to determine the optimal camera intrinsic and extrinsic parameters. Then, fixing the calibrated intrinsic and extrinsic parameters, it iterates through and optimizes different time offsets to determine the optimal time offset. Thus, by co-optimizing camera intrinsic and extrinsic parameters and time offsets, it effectively reduces error propagation in traditional separate calibration methods, achieves accurate estimation of millisecond-level time offsets, and significantly suppresses extrinsic parameter calibration errors caused by sensor motion and time asynchrony. This not only greatly improves calibration accuracy and subsequent sensor fusion performance in dynamic scenes but also eliminates dependence on specific calibration objects and scenes, allowing direct online calibration using regular vehicle driving data, greatly enhancing the applicability and engineering practicality of the method. In some embodiments, determining the first objective function value based on the mapping relationship between the first camera parameter set, camera data, and point cloud data includes: projecting point cloud data onto a two-dimensional plane based on the first camera parameter set to determine first projected data; calculating multiple first pixel distances between camera data and corresponding first projected data, and performing a weighted summation of the multiple first pixel distances to determine the first objective function value.

[0032] For example, suppose the first camera parameter set includes first camera parameter set 1, first camera parameter set 2 and first camera parameter set 3, suppose the two-dimensional pixels in the camera data include D1, D2 and D3, and the three-dimensional spatial points in the point cloud data include P1, P2 and P3. According to the mapping relationship between the camera data and the point cloud data, it can be seen that D1 corresponds to P1, D2 corresponds to P2 and D2 corresponds to P2.

[0033] Then, based on the first camera parameter set 1, P1, P2, and P3 are projected onto a two-dimensional plane to obtain first projection 1, first projection 2, and first projection 3, respectively. Specifically, the first camera parameter set 1 includes camera intrinsic parameter 1 and camera extrinsic parameter 1. Using camera extrinsic parameter 1, the three-dimensional spatial point P1 is transformed from the world coordinate system to the camera coordinate system, and then using camera intrinsic parameter 1, the camera coordinate system is transformed into the image coordinate system to determine first projection 1; using camera extrinsic parameter 1, the three-dimensional spatial point P2 is transformed from the world coordinate system to the camera coordinate system, and then using camera intrinsic parameter 1, the camera coordinate system is transformed into the image coordinate system to determine first projection 2; using camera extrinsic parameter 1, the three-dimensional spatial point P3 is transformed from the world coordinate system to the camera coordinate system, and then using camera intrinsic parameter 1, the camera coordinate system is transformed into the image coordinate system to determine first projection 3. That is to say, P1 corresponds to first projection 1 and D1, P2 corresponds to first projection 2 and D2, and P3 corresponds to first projection 3 and D3. Therefore, first projection 1 corresponds to D1, first projection 2 corresponds to D2, and first projection 3 corresponds to D3.

[0034] Then, the first pixel distance between the two-dimensional pixel and the first projection is calculated based on the coordinates of the two-dimensional pixel and the corresponding coordinates of the first projection. In the following explanation, the calculation of the first pixel distance 1 between the first projection 1 and D1 will be used as an example. Specifically, assuming the coordinates of the first projection 1 are (u1, v1) and the coordinates of D1 are (u1', v1'), the formula for calculating the first pixel distance 1 is as follows:

[0035] in, This indicates that the distance from the first pixel is 1.

[0036] Similarly, the first pixel distance 2 between the first projection 2 and D2, and the first pixel distance 3 between the first projection 3 and D3 are calculated using the above method.

[0037] Then, a weighted sum of the first pixel distance 1, the first pixel distance 2, and the first pixel distance 3 is performed using a preset robust loss function to determine the first objective function value 1 corresponding to the first camera parameter group 1. For example, the first pixel distance 1, the first pixel distance 2, and the first pixel distance 3 can be input into the preset robust loss function formula to output the first objective function value 1.

[0038] Similarly, based on the first camera parameter set 2, P1, P2, and P3 are projected onto a two-dimensional plane to obtain the first projection 11, the first projection 22, and the first projection 33, respectively. Specifically, the first camera parameter set 2 includes camera intrinsic parameters 2 and camera extrinsic parameters 2. Using the camera extrinsic parameters 2, the three-dimensional spatial point P1 is transformed from the world coordinate system to the camera coordinate system, and then the camera intrinsic parameters 2 are used to transform the camera coordinate system into the image coordinate system, thus determining the first projection 11; using the camera extrinsic parameters 2, the three-dimensional spatial point P2 is transformed from the world coordinate system to the camera coordinate system, and then the camera intrinsic parameters 2 are used to transform the camera coordinate system into the image coordinate system, thus determining the first projection 22; using the camera extrinsic parameters 2, the three-dimensional spatial point P3 is transformed from the world coordinate system to the camera coordinate system, and then the camera intrinsic parameters 2 are used to transform the camera coordinate system into the image coordinate system, thus determining the first projection 33. That is to say, P1 corresponds to the first projection 11 and D1 respectively, P2 corresponds to the first projection 22 and D2 respectively, and P3 corresponds to the first projection 33 and D3 respectively. So the first projection 11 corresponds to D1, the first projection 22 corresponds to D2, and the first projection 33 corresponds to D3.

[0039] Then, the first pixel distance between the two-dimensional pixel and the first projection is calculated based on the coordinates of the two-dimensional pixel and the corresponding coordinates of the first projection. In the following explanation, the calculation of the first pixel distance 11 between the first projection 11 and D1 will be used as an example. Specifically, assuming the coordinates of the first projection 11 are (u11, v11) and the coordinates of D1 are (u1', v1'), the formula for calculating the first pixel distance 11 is as follows:

[0040] in, This indicates that the distance to the first pixel is 11.

[0041] Similarly, the first pixel distance 22 between the first projection 22 and D2, and the first pixel distance 33 between the first projection 33 and D3 are calculated using the above method.

[0042] Then, a weighted sum of the first pixel distances 11, 22, and 33 is performed using a preset robust loss function to determine the first objective function value 2 corresponding to the first camera parameter group 2. For example, the first pixel distances 11, 22, and 33 can be input into the preset robust loss function formula to output the first objective function value 2.

[0043] Similarly, based on the first camera parameter set 3, P1, P2, and P3 are projected onto a two-dimensional plane to obtain the first projection 111, the first projection 222, and the first projection 333, respectively. Specifically, the first camera parameter set 3 includes camera intrinsic parameters 3 and camera extrinsic parameters 3. Using the camera extrinsic parameters 3, the three-dimensional spatial point P1 is transformed from the world coordinate system to the camera coordinate system, and then the camera intrinsic parameters 3 are used to transform the camera coordinate system into the image coordinate system to determine the first projection 111; using the camera extrinsic parameters 3, the three-dimensional spatial point P2 is transformed from the world coordinate system to the camera coordinate system, and then the camera intrinsic parameters 3 are used to transform the camera coordinate system into the image coordinate system to determine the first projection 222; using the camera extrinsic parameters 3, the three-dimensional spatial point P3 is transformed from the world coordinate system to the camera coordinate system, and then the camera intrinsic parameters 3 are used to transform the camera coordinate system into the image coordinate system to determine the first projection 333. That is to say, P1 corresponds to the first projection 111 and D1 respectively, P2 corresponds to the first projection 222 and D2 respectively, and P3 corresponds to the first projection 333 and D3 respectively. Therefore, the first projection 111 corresponds to D1, the first projection 222 corresponds to D2, and the first projection 333 corresponds to D3.

[0044] Then, the first pixel distance between the two-dimensional pixel and the first projection is calculated based on the coordinates of the two-dimensional pixel and the corresponding coordinates of the first projection. In the following explanation, the calculation of the first pixel distance 111 between the first projection 111 and D1 is used as an example. Specifically, assuming the coordinates of the first projection 111 are (u111, v111) and the coordinates of D1 are (u1', v1'), the formula for calculating the first pixel distance 111 is as follows:

[0045] in, This indicates that the distance to the first pixel is 111.

[0046] Similarly, the first pixel distance 222 between the first projection 222 and D2, and the first pixel distance 333 between the first projection 333 and D3 are calculated using the above method.

[0047] Then, a weighted sum of the first pixel distances 111, 222, and 333 is performed using a preset robust loss function to determine the first objective function value 3 corresponding to the first camera parameter group 3. For example, the first pixel distances 111, 222, and 333 can be input into the preset robust loss function formula to output the first objective function value 3.

[0048] Finally, the first objective function value 1, the first objective function value 2, and the first objective function value 3 are compared to determine the minimum value. Based on the first camera parameter set corresponding to the minimum value, the target camera extrinsic parameters and target camera intrinsic parameters are determined. For example, if the first objective function value 1 is the minimum value, camera extrinsic parameter 1 and camera intrinsic parameter 1 in the first camera parameter set 1 can be determined as the target camera extrinsic parameters and target camera intrinsic parameters; similarly, if the first objective function value 2 is the minimum value, camera extrinsic parameter 2 and camera intrinsic parameter 2 in the first camera parameter set 2 can be determined as the target camera extrinsic parameters and target camera intrinsic parameters; and again, if the first objective function value 3 is the minimum value, camera extrinsic parameter 3 and camera intrinsic parameter 3 in the first camera parameter set 3 can be determined as the target camera extrinsic parameters and target camera intrinsic parameters.

[0049] This application transforms the parameter calibration problem into an objective function optimization problem by establishing a projection error model from the point cloud to the image plane. Under a fixed time offset, it iterates through the combinations of camera intrinsic and extrinsic parameters and calculates their corresponding projection errors. A robust loss function is then used to perform a weighted summation of pixel-level distances, effectively suppressing the interference of mismatched points on the optimization process and ensuring good convergence characteristics of the objective function in the parameter space. This significantly reduces the error accumulation caused by parameter decoupling, achieving millimeter-level accuracy in camera-radar extrinsic parameter calibration, while enhancing the robustness and practicality of the algorithm in dynamic scenes.

[0050] In some embodiments, determining the second objective function value based on the mapping relationship between the second camera parameter set, camera data, and point cloud data includes: projecting the point cloud data onto a two-dimensional plane based on the second camera parameter set to determine the second projection data; calculating multiple second pixel distances between the camera data and the corresponding second projection data, and performing a weighted summation of the multiple second pixel distances to determine the second objective function value.

[0051] For example, suppose the second camera parameter set includes second camera parameter set 1, second camera parameter set 2 and second camera parameter set 3. Suppose the two-dimensional pixels in the camera data include D1, D2 and D3, and the three-dimensional spatial points in the point cloud data include P1, P2 and P3. According to the mapping relationship between the camera data and the point cloud data, D1 corresponds to P1, D2 corresponds to P2 and D2 corresponds to P2.

[0052] Then, based on the second camera parameter set 1, P1, P2, and P3 are projected onto a two-dimensional plane to obtain second projection 1, second projection 2, and second projection 3, respectively. Specifically, the second camera parameter set 1 includes target camera intrinsic parameters, target camera extrinsic parameters, and time offset 1. First, motion compensation is performed on the three-dimensional spatial point P1 using time offset 1 to eliminate motion distortion of the three-dimensional spatial point cloud caused by sensor time asynchrony. Then, the motion-compensated three-dimensional spatial point P1 is transformed from the world coordinate system to the camera coordinate system using the target camera extrinsic parameters. Finally, the camera coordinate system is transformed into the image coordinate system using the target camera intrinsic parameters, thus determining second projection 1. Similarly, motion compensation is performed on the three-dimensional spatial point P2 using time offset 1 to eliminate motion distortion of the three-dimensional spatial point cloud caused by sensor time asynchrony. To address motion distortion, the motion-compensated 3D spatial point P2 is transformed from the world coordinate system to the camera coordinate system using the target camera's extrinsic parameters. Then, the camera coordinate system is transformed into the image coordinate system using the target camera's intrinsic parameters, thus determining the second projection 2. First, motion compensation is applied to the 3D spatial point P3 using a time offset of 1 to eliminate motion distortion of the 3D spatial point cloud caused by sensor time asynchrony. The motion-compensated 3D spatial point P3 is then transformed from the world coordinate system to the camera coordinate system using the target camera's extrinsic parameters. Finally, the camera coordinate system is transformed into the image coordinate system using the target camera's intrinsic parameters, thus determining the second projection 3. In other words, P1 corresponds to both second projection 1 and D1, P2 corresponds to both second projection 2 and D2, and P3 corresponds to both second projection 3 and D3. Therefore, second projection 1 corresponds to D1, second projection 2 corresponds to D2, and second projection 3 corresponds to D3.

[0053] Then, the second pixel distance between the two-dimensional pixel and the second projection is calculated based on the coordinates of the two-dimensional pixel and the corresponding coordinates of the second projection. In the following explanation, the calculation of the second pixel distance 1 between the second projection 1 and D1 will be used as an example. Specifically, assuming the coordinates of the second projection 1 are (u2, v2) and the coordinates of D1 are (u1', v1'), the formula for calculating the second pixel distance 1 is as follows:

[0054] in, This indicates that the distance to the second pixel is 1.

[0055] Similarly, the second pixel distance 2 between the second projection 2 and D2, and the second pixel distance 3 between the second projection 3 and D3 are calculated using the above method.

[0056] Then, a weighted sum of the second pixel distance 1, the second pixel distance 2, and the second pixel distance 3 is performed using a preset robust loss function to determine the second objective function value 1 corresponding to the second camera parameter group 1. For example, the second pixel distance 1, the second pixel distance 2, and the second pixel distance 3 can be input into the preset robust loss function formula to output the second objective function value 1.

[0057] Similarly, based on the second camera parameter set 2, P1, P2, and P3 are projected onto a two-dimensional plane to obtain the second projection 11, the second projection 22, and the second projection 33, respectively. Specifically, the second camera parameter set 2 includes the target camera intrinsic parameters, the target camera extrinsic parameters, and the time offset 2. First, the time offset 2 is used to perform motion compensation on the three-dimensional spatial point P1 to eliminate the motion distortion of the three-dimensional spatial point cloud caused by sensor time asynchrony. Then, the target camera extrinsic parameters are used to transform the motion-compensated three-dimensional spatial point P1 from the world coordinate system to the camera coordinate system. Finally, the target camera intrinsic parameters are used to transform the camera coordinate system to the image coordinate system, thus determining the second projection 11. Similarly, the time offset 2 is used to perform motion compensation on the three-dimensional spatial point P2 to eliminate the motion distortion of the three-dimensional spatial point cloud caused by sensor time asynchrony. To address motion distortion, the motion-compensated 3D spatial point P2 is transformed from the world coordinate system to the camera coordinate system using the target camera's extrinsic parameters. Then, the camera coordinate system is transformed into the image coordinate system using the target camera's intrinsic parameters, thus determining the second projection 22. First, motion compensation is applied to the 3D spatial point P3 using a time offset of 2 to eliminate motion distortion of the 3D spatial point cloud caused by sensor time asynchrony. The motion-compensated 3D spatial point P3 is then transformed from the world coordinate system to the camera coordinate system using the target camera's extrinsic parameters. Finally, the camera coordinate system is transformed into the image coordinate system using the target camera's intrinsic parameters, thus determining the second projection 33. In other words, P1 corresponds to second projections 11 and D1, P2 corresponds to second projections 22 and D2, and P3 corresponds to second projections 33 and D3. Therefore, second projection 11 corresponds to D1, second projection 22 corresponds to D2, and second projection 33 corresponds to D3.

[0058] Then, the second pixel distance between the two-dimensional pixel and the second projection is calculated based on the coordinates of the two-dimensional pixel and the corresponding coordinates of the second projection. In the following explanation, the calculation of the second pixel distance 11 between the second projection 11 and D1 will be used as an example. Specifically, assuming the coordinates of the second projection 11 are (u22, v22) and the coordinates of D1 are (u1', v1'), the formula for calculating the second pixel distance 11 is as follows:

[0059] in, This indicates that the distance to the second pixel is 11.

[0060] Similarly, the second pixel distance 22 between the second projection 22 and D2, and the second pixel distance 33 between the second projection 33 and D3 are calculated using the above method.

[0061] Then, a weighted sum of the second pixel distances 11, 22, and 33 is performed using a preset robust loss function to determine the second objective function value 2 corresponding to the second camera parameter group 2. For example, the second pixel distances 11, 22, and 33 can be input into the preset robust loss function formula to output the second objective function value 2.

[0062] Similarly, based on the second camera parameter set 3, P1, P2, and P3 are projected onto a two-dimensional plane to obtain the second projection 111, the second projection 222, and the second projection 333, respectively. Specifically, the second camera parameter set 3 includes the target camera intrinsic parameters, the target camera extrinsic parameters, and the time offset 3. First, the time offset 3 is used to perform motion compensation on the three-dimensional spatial point P1 to eliminate the motion distortion of the three-dimensional spatial point cloud caused by sensor time asynchrony. Then, the target camera extrinsic parameters are used to transform the motion-compensated three-dimensional spatial point P1 from the world coordinate system to the camera coordinate system. Finally, the target camera intrinsic parameters are used to transform the camera coordinate system to the image coordinate system, thus determining the second projection 111. Similarly, the time offset 3 is used to perform motion compensation on the three-dimensional spatial point P2 to eliminate the motion distortion of the three-dimensional spatial point cloud caused by sensor time asynchrony. To address motion distortion, the motion-compensated 3D spatial point P2 is transformed from the world coordinate system to the camera coordinate system using the target camera's extrinsic parameters. Then, the camera coordinate system is transformed into the image coordinate system using the target camera's intrinsic parameters, thus determining the second projection 222. First, motion compensation is applied to the 3D spatial point P3 using a time offset of 3 to eliminate motion distortion of the 3D spatial point cloud caused by sensor time asynchrony. The motion-compensated 3D spatial point P3 is then transformed from the world coordinate system to the camera coordinate system using the target camera's extrinsic parameters. Finally, the camera coordinate system is transformed into the image coordinate system using the target camera's intrinsic parameters, thus determining the second projection 333. In other words, P1 corresponds to the second projections 111 and D1, P2 corresponds to the second projections 222 and D2, and P3 corresponds to the second projections 333 and D3. Therefore, the second projections 111 and D1 correspond, the second projections 222 and D2 correspond, and the second projections 333 and D3 correspond.

[0063] Then, the second pixel distance between the two-dimensional pixel and the second projection is calculated based on the coordinates of the two-dimensional pixel and the corresponding coordinates of the second projection. In the following explanation, the calculation of the second pixel distance 111 between the second projection 111 and D1 will be used as an example. Specifically, assuming the coordinates of the second projection 111 are (u222, v222) and the coordinates of D1 are (u1', v1'), the formula for calculating the second pixel distance 11 is as follows:

[0064] in, This indicates that the distance to the second pixel is 111.

[0065] Similarly, the second pixel distance 222 between the second projection 222 and D2, and the second pixel distance 333 between the second projection 333 and D3 are calculated using the above method.

[0066] Then, a weighted sum of the second pixel distances 111, 222, and 333 is performed using a preset robust loss function to determine the second objective function value 3 corresponding to the second camera parameter group 3. For example, the second pixel distances 111, 222, and 333 can be input into the preset robust loss function formula to output the second objective function value 3. Finally, the second objective function value 1, 2, and 3 are compared to determine the minimum value, and the target time offset is determined based on the second camera parameter group corresponding to the minimum value. For example, if the second objective function value 1 is the minimum value, the time offset 1 in the second camera parameter group 1 can be determined as the target time offset; if the second objective function value 2 is the minimum value, the time offset 2 in the second camera parameter group 2 can be determined as the target time offset; and if the second objective function value 3 is the minimum value, the time offset 3 in the second camera parameter group 3 can be determined as the target time offset.

[0067] This application establishes a functional relationship between projection error and time offset by fixing the calibrated camera intrinsic and extrinsic parameters. By traversing different time offset values ​​and calculating the corresponding reprojection error, a robust loss function is used to weight and aggregate pixel distances, effectively eliminating motion distortion caused by sensor timing asynchrony in dynamic scenes. By systematically minimizing timing errors, accurate estimation of millisecond-level time offsets is achieved. This not only significantly improves the time alignment accuracy of multi-sensor data but also provides a reliable time reference for subsequent fusion sensing, enhancing the stability and accuracy of the system in practical applications.

[0068] In some embodiments, the method further includes: determining synthetic geometry data, synthetic depth data, and synthetic rendering mask based on point cloud data and an initial camera parameter set; constructing a corresponding point set between camera data and point cloud data based on the synthetic geometry data, synthetic depth data, synthetic rendering mask, and camera data, wherein the corresponding point set includes the mapping relationship between camera data and point cloud data.

[0069] Specifically, the initial camera parameter set includes initial camera intrinsic parameters, initial camera extrinsic parameters, and an initial time offset. The initial camera intrinsic parameters can be a set of nominal camera intrinsic parameters provided by the camera manufacturer in the product specifications; the initial camera extrinsic parameters can be directly read from the CAD design model; and the initial time offset can be 0. Vehicle pose information is acquired through an IMU (Inertial Measurement Unit). Based on the vehicle pose information and the initial camera parameter set, a coordinate transformation is performed to determine the camera's initial pose in the global coordinate system. For example, the initial pose of the camera in the global coordinate system can be determined based on the product of the vehicle pose information matrix and the initial camera extrinsic parameter matrix. Then, the point cloud data is rendered into a series of two-dimensional synthetic data based on the initial pose, such as... Figure 2 As shown, this specifically includes synthetic geometric data, synthetic depth data, and synthetic rendering mask. Among them, referencing... Figure 2 (a) Synthetic geometric data simulates the scene appearance as seen by the camera in this pose. The color of a point can be determined by its normal vector or checkerboard pattern, used for subsequent feature matching; refer to Figure 2 (b) In the synthetic depth data, the value of each pixel represents the depth value of its corresponding 3D point in the camera coordinate system; Figure 2 (c) Synthesize a rendering mask for a binary image, marking which pixels are generated by point cloud projection.

[0070] After determining the synthetic geometry data, synthetic depth data, and synthetic rendering mask, a corresponding point set between the camera data and the point cloud data is constructed based on the synthetic geometry data, synthetic depth data, synthetic rendering mask, and camera data. This includes: extracting features from the camera data to determine a first set of feature points; extracting features from the synthetic geometry data to determine a second set of feature points; performing feature matching between the first and second set of feature points to determine multiple feature pairs; determining target second feature points in the multiple feature pairs based on the synthetic rendering mask; determining the depth value of each target second feature point based on the synthetic depth data, and back-projecting the coordinates and depth values ​​of each target second feature point based on the initial camera parameter set to determine the corresponding virtual point cloud; searching for the target point cloud in the point cloud data that is closest to the multiple virtual point clouds; determining the target camera data corresponding to the multiple target point clouds based on the multiple feature pairs; and constructing a corresponding point set between the camera data and the point cloud data based on the multiple target point clouds and the multiple target camera data.

[0071] For example, refer to Figure 3 Constructing the corresponding point set between camera data and point cloud data based on synthetic geometric data, synthetic depth data, synthetic rendering mask, and camera data includes the following steps: Feature extraction is performed on camera data to determine a first set of feature points, and feature extraction is performed on synthetic geometric data to determine a second set of feature points. The extracted feature points can be either SuperPoint or SURF (Speeded-UpRobustFeatures).

[0072] Although both camera data and synthetic geometric data are two-dimensional pixels, camera data consists of real two-dimensional pixels, while synthetic geometric data is determined based on point cloud data. The feature points of the two do not correspond completely. Therefore, it is necessary to perform feature matching (e.g., through descriptor distance or pixel coordinate distance) on the first and second feature point sets to determine multiple feature pairs and remove non-corresponding feature points to improve accuracy.

[0073] The second feature point in a feature pair may be a virtual point generated when acquiring synthetic geometric data. In other words, some second feature points do not correspond to the real point cloud. Therefore, in order to further improve accuracy, it is necessary to determine whether the second feature point corresponds to the real point cloud based on the synthetic rendering mask. For example, when the synthetic rendering mask is 1, it is determined that the second feature point corresponds to the real point cloud. When the synthetic rendering mask is 0, it is determined that the second feature point does not correspond to the real point cloud. Furthermore, the point cloud that corresponds to the real point cloud is determined as the target second feature point.

[0074] Then, based on the synthetic depth data, the depth value of the target's second feature point is determined. The coordinates and depth value of each target's second feature point are back-projected through the initial camera parameter set to obtain a virtual point cloud in the camera coordinate system. The point cloud in the camera coordinate system is transformed to the world coordinate system, and the nearest real point cloud to each virtual point cloud is searched in the point cloud data to determine the target point cloud. It should be noted that, to improve the search speed, after preprocessing the point cloud data, an efficient spatial index structure such as a K-Dimensional Tree can be constructed on the preprocessed point cloud, and the K-Dimensional Tree can be used to search for the nearest real point cloud to each virtual point cloud.

[0075] The target point cloud corresponding to the second feature point is determined through the above process. In the above embodiment, multiple feature pairs are also established through feature matching. Each feature pair includes a first feature point and a corresponding second feature point. That is, the second feature point corresponds one-to-one with the target point cloud, and the first feature point corresponds one-to-one with the second feature point. Therefore, the target point cloud can be one-to-one with the first feature point. Since the first feature point represents camera data, the target camera data (target two-dimensional pixels) corresponding to the target point cloud can be found. Based on multiple target point clouds and multiple target camera data, a corresponding point set of camera data and point cloud data can be constructed. This corresponding point set includes the mapping relationship between camera data and point cloud data.

[0076] This application generates synthetic geometric data, synthetic depth data, and synthetic rendering mask by combining point cloud data and initial camera parameter sets, thereby accurately constructing the mapping relationship between camera data and point cloud data. This not only improves the accuracy of feature matching, but also further filters out effective feature points through depth information and rendering mask, ensuring a one-to-one correspondence between feature points and real point clouds.

[0077] In some embodiments, after constructing the corresponding point set of camera data and point cloud data, the above method further includes: performing spatial uniform sampling on the corresponding point set.

[0078] Specifically, to mitigate the impact of feature point clustering on the robustness of calibration results, spatial uniformity sampling is performed on the corresponding point sets of the camera data and point cloud data. For example, the camera data is divided into a grid (e.g., 20x20), and a limited number of two-dimensional pixels are randomly selected from each grid or according to specific rules (e.g., selecting the point closest to the grid center). Based on the point cloud corresponding to these two-dimensional pixels, a more uniformly distributed subset is formed for subsequent optimization. This approach can, to some extent, avoid calibration bias caused by local clustering of feature points, enhancing the overall robustness of the algorithm.

[0079] In some embodiments, the method further includes: determining the minimum value of a plurality of first objective function values ​​based on a first preset optimization algorithm, wherein the first preset optimization algorithm includes one of differential evolution algorithm, particle swarm optimization algorithm, simulated annealing algorithm and genetic algorithm.

[0080] In some embodiments, the method further includes: determining the minimum value of a plurality of second objective function values ​​based on a second preset optimization algorithm, wherein the second preset optimization algorithm includes one of nonlinear least squares method, Gauss-Newton method and gradient descent method.

[0081] Specifically, refer to Figure 4 In the first stage, a first preset optimization algorithm is used to search for the target camera's extrinsic and intrinsic parameters within a large parameter range. This first preset optimization algorithm effectively avoids global optimization algorithms that get trapped in local optima, such as differential evolution, particle swarm optimization, simulated annealing, and genetic algorithms. This provides a good initial value for subsequent fine-tuning. In this stage, the time offset can be fixed at 0.

[0082] Then, in the second stage, the target camera extrinsic and intrinsic parameters obtained in the first stage are used as initial values. One of the following methods—nonlinear least squares, Gauss-Newton method, and gradient descent method—is used to jointly optimize all parameters until convergence. The target camera intrinsic, extrinsic, and time offset are then output to complete the camera parameter calibration.

[0083] In this way, the first preset optimization algorithm is used to perform a global and large-scale parameter search, effectively avoiding local optima. Then, the excellent results of the first preset optimization algorithm are used as initial values, and the second preset optimization algorithm is used to jointly and finely optimize all spatiotemporal parameters, which can ensure the global optimality of camera parameters and the high accuracy of the final result.

[0084] In summary, firstly, a preset time offset is fixed, and a first preset optimization algorithm is used to iterate and optimize different combinations of camera intrinsic and extrinsic parameters to determine the optimal camera intrinsic and extrinsic parameters. Then, the calibrated intrinsic and extrinsic parameters are fixed, and a second preset optimization algorithm is used to iterate and optimize different time offsets to determine the optimal time offset. In this way, by co-optimizing camera intrinsic and extrinsic parameters and time offsets, the error propagation in traditional separate calibration methods is effectively reduced, achieving accurate estimation of millisecond-level time offsets. This significantly suppresses extrinsic parameter calibration errors caused by the asynchronous movement of sensors and time. It not only greatly improves calibration accuracy in dynamic scenes and the performance of subsequent sensor fusion, but also eliminates the dependence on specific calibration objects and scenes, allowing direct online calibration using regular vehicle driving data, greatly enhancing the applicability and engineering practicality of the method.

[0085] Corresponding to the above embodiments, this application also proposes a camera parameter calibration device.

[0086] Reference Figure 5 The camera parameter calibration device 700 includes: a first acquisition module 710, a first determination module 720, a second determination module 730, and a third determination module 740.

[0087] The first acquisition module 710 acquires camera data, point cloud data, and a first camera parameter set for the target object. The first camera parameter set includes multiple first camera parameter sets, each containing camera extrinsic parameters, camera intrinsic parameters, and a preset offset time. The first determination module 720 determines a first objective function value based on the mapping relationship between the first camera parameter sets, camera data, and point cloud data, and determines the target camera extrinsic parameters and target camera intrinsic parameters based on the minimum value of the multiple first objective function values. The second determination module 730 determines a second camera parameter set based on the target camera extrinsic parameters, target camera intrinsic parameters, and multiple reference offset times. The second camera parameter set includes multiple second camera parameter sets, each containing target camera extrinsic parameters, target camera intrinsic parameters, and a reference offset time. The third determination module 740 determines a second objective function value based on the mapping relationship between the second camera parameter sets, camera data, and point cloud data, and determines the target offset time based on the minimum value of the multiple second objective function values.

[0088] According to one embodiment of this application, the first determining module 720 is specifically used to: project point cloud data onto a two-dimensional plane based on a first camera parameter set to determine first projection data; calculate multiple first pixel distances between the camera data and the corresponding first projection data, and perform a weighted summation of the multiple first pixel distances to determine a first objective function value.

[0089] According to one embodiment of this application, the third determining module 740 is specifically used to: project point cloud data onto a two-dimensional plane based on a second camera parameter set to determine second projection data; calculate multiple second pixel distances between the camera data and the corresponding second projection data, and perform a weighted summation of the multiple second pixel distances to determine a second objective function value.

[0090] According to one embodiment of this application, synthetic geometry data, synthetic depth data, and synthetic rendering mask are determined based on point cloud data and an initial camera parameter set; a corresponding point set of camera data and point cloud data is constructed based on the synthetic geometry data, synthetic depth data, synthetic rendering mask, and camera data, and the corresponding point set includes the mapping relationship between camera data and point cloud data.

[0091] According to one embodiment of this application, feature extraction is performed on camera data to determine a first set of feature points, and feature extraction is performed on synthetic geometric data to determine a second set of feature points; feature matching is performed on the first set of feature points and the second set of feature points to determine multiple feature pairs; target second feature points in the multiple feature pairs are determined based on a synthetic rendering mask; the depth value of each target second feature point is determined based on synthetic depth data, and the coordinates and depth values ​​of each target second feature point are back-projected based on an initial camera parameter set to determine the corresponding virtual point cloud; a target point cloud that is closest to the multiple virtual point clouds is searched in the point cloud data; target camera data corresponding to the multiple target point clouds are determined based on the multiple feature pairs; and a corresponding point set of camera data and point cloud data is constructed based on the multiple point clouds and the multiple target camera data.

[0092] According to one embodiment of this application, after constructing the corresponding point set of camera data and point cloud data, spatial uniform sampling is performed on the corresponding point set.

[0093] According to one embodiment of this application, the minimum value of a plurality of first objective function values ​​is determined based on a first preset optimization algorithm.

[0094] According to one embodiment of this application, the minimum value of a plurality of second objective function values ​​is determined based on a second preset optimization algorithm.

[0095] It should be noted that the above explanation of the embodiments and beneficial effects of the camera parameter calibration method also applies to the camera parameter calibration device of the embodiments of this application. To avoid redundancy, it will not be elaborated in detail here.

[0096] Corresponding to the above embodiments, this application also proposes a computer-readable storage medium.

[0097] The computer-readable storage medium of this application stores a camera parameter calibration program thereon, which, when executed by a processor, implements the aforementioned camera parameter calibration method.

[0098] It should be noted that the above-described embodiments and explanations of the beneficial effects of the camera parameter calibration method are also applicable to the computer-readable storage medium of the embodiments of this application. To avoid redundancy, they will not be elaborated in detail here.

[0099] Corresponding to the above embodiments, this application also proposes an electronic device.

[0100] See Figure 6 As shown, the electronic device 300 of this application includes a memory 310, a processor 320, and a camera parameter calibration program stored in the memory 310 and executable on the processor 320. When the processor executes the camera parameter calibration program, it implements the aforementioned camera parameter calibration method.

[0101] It should be noted that the above-described embodiments and explanations of the beneficial effects of the camera parameter calibration method are also applicable to the electronic devices in the embodiments of this application. To avoid redundancy, they will not be elaborated in detail here.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0103] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for calibrating camera parameters, characterized in that, The method includes: Acquire camera data, point cloud data and a first camera parameter set for the target object. The first camera parameter set includes multiple first camera parameter sets, and each first camera parameter set includes camera extrinsic parameters, camera intrinsic parameters and a preset offset time. The first objective function value is determined based on the mapping relationship between the first camera parameter group, the camera data and the point cloud data, and the target camera extrinsic parameters and target camera intrinsic parameters are determined based on the minimum value of multiple first objective function values. A second camera parameter set is determined based on the target camera extrinsic parameters, the target camera intrinsic parameters, and multiple reference offset times. The second camera parameter set includes multiple second camera parameter sets, and each second camera parameter set includes the target camera extrinsic parameters, the target camera intrinsic parameters, and reference offset times. The second objective function value is determined based on the mapping relationship between the second camera parameter group, the camera data, and the point cloud data, and the target offset time is determined based on the minimum value of multiple second objective function values.

2. The camera parameter calibration method according to claim 1, characterized in that, The step of determining the first objective function value based on the mapping relationship between the first camera parameter group, the camera data, and the point cloud data includes: The point cloud data is projected onto a two-dimensional plane based on the first camera parameter set to determine the first projection data. Calculate multiple first pixel distances between the camera data and the corresponding first projection data, and perform a weighted summation of the multiple first pixel distances to determine the first objective function value.

3. The camera parameter calibration method according to claim 1, characterized in that, The step of determining the second objective function value based on the mapping relationship between the second camera parameter set, the camera data, and the point cloud data includes: The point cloud data is projected onto a two-dimensional plane based on the second camera parameter set to determine the second projection data; Calculate multiple second pixel distances between the camera data and the corresponding second projection data, and perform a weighted summation of the multiple second pixel distances to determine the value of the second objective function.

4. The camera parameter calibration method according to claim 1, characterized in that, The method further includes: Based on the point cloud data and the initial camera parameter set, determine the synthetic geometry data, synthetic depth data, and synthetic rendering mask; Based on the synthetic geometric data, the synthetic depth data, the synthetic rendering mask, and the camera data, a corresponding point set is constructed between the camera data and the point cloud data. The corresponding point set includes the mapping relationship between the camera data and the point cloud data.

5. The camera parameter calibration method according to claim 4, characterized in that, The step of constructing a corresponding point set between the camera data and the point cloud data based on the synthetic geometry data, the synthetic depth data, the synthetic rendering mask, and the camera data includes: Feature extraction is performed on the camera data to determine a first set of feature points, and feature extraction is performed on the synthetic geometric data to determine a second set of feature points; The first set of feature points and the second set of feature points are used for feature matching to determine multiple feature pairs; Based on the synthetic rendering mask, target second feature points are determined among the multiple feature pairs; The depth value of each target second feature point is determined based on the synthetic depth data, and the coordinates and depth value of each target second feature point are back-projected based on the initial camera parameter set to determine the corresponding virtual point cloud; Search the point cloud data for the target point cloud that is closest to the plurality of virtual point clouds; Based on the multiple features, target camera data corresponding to the multiple target point clouds are determined; Construct a corresponding point set between the camera data and the point cloud data based on multiple point clouds and multiple target camera data.

6. The camera parameter calibration method according to claim 4, characterized in that, After constructing the corresponding point set of the camera data and the point cloud data, the method further includes: Spatial homogenization sampling is performed on the corresponding point set.

7. The camera parameter calibration method according to claim 1, characterized in that, The method further includes: The minimum value of multiple first objective function values ​​is determined based on the first preset optimization algorithm.

8. The camera parameter calibration method according to claim 1, characterized in that, The method further includes: The minimum value of multiple second objective function values ​​is determined based on the second preset optimization algorithm.

9. A camera parameter calibration device, characterized in that, The device includes: The first acquisition module is used to acquire camera data, point cloud data and a first camera parameter set of the target object. The first camera parameter set includes multiple first camera parameter sets, and each first camera parameter set includes camera extrinsic parameters, camera intrinsic parameters and preset offset time. The first determining module is used to determine a first objective function value based on the first camera parameter group, the mapping relationship between the camera data and the point cloud data, and to determine the target camera extrinsic parameters and the target camera intrinsic parameters based on the minimum value of multiple first objective function values. The second determining module is used to determine a second camera parameter set based on the target camera extrinsic parameters, the target camera intrinsic parameters, and multiple reference offset times, wherein the second camera parameter set includes multiple second camera parameter sets, and the second camera parameter sets include the target camera extrinsic parameters, the target camera intrinsic parameters, and reference offset times; The third determining module is used to determine the second objective function value based on the mapping relationship between the second camera parameter group, the camera data and the point cloud data, and to determine the target offset time based on the minimum value of multiple second objective function values.

10. An electronic device, characterized in that, The system includes a memory, a processor, and a camera parameter calibration program stored in the memory and executable on the processor. When the processor executes the camera parameter calibration program, it implements the camera parameter calibration method according to any one of claims 1-8.

Images