4D millimeter wave radar and camera calibration method and system based on projection error

Abstract

The invention provides a 4D millimeter-wave radar and camera calibration method based on projection errors. The method comprises the following steps: acquiring radar point cloud and an original image of a calibration board in a static scene through a 4D millimeter-wave radar and a camera; performing dynamic object filtering and noise suppression on the synchronous data to obtain a static enhanced point cloud and a corrected image; performing coordinate extraction and checkerboard corner detection on the filtered data to obtain point cloud 3D coordinates and pixel 2D coordinates; performing projection conversion on the point cloud 3D coordinate through the external parameter matrix to obtain a conversion coordinate, and calculating a projection error between the conversion coordinate and the pixel 2D coordinate; and performing iterative optimization on the extrinsic parameter matrix through a nonlinear optimization algorithm based on the projection error to obtain an optimized matrix. According to the method, through dynamic interference suppression, noise collaborative filtering and a nonlinear optimization mechanism, the calibration precision and the environmental adaptability are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of sensor calibration, in particular to a 4D millimeter wave radar and camera calibration method and system based on projection error. BACKGROUND

[0002] With the rapid development of intelligent sensing systems, multi-modal sensor fusion has become a core technology for environmental perception. 4D millimeter wave radar, with its all-weather working ability, speed detection advantage and point cloud density improvement, is complementary to high-resolution cameras, but the high-precision fusion of the data of the two depends on strict space-time calibration. Traditional calibration methods are mostly based on checkerboard calibration boards in static scenes, and the external parameter matrix is solved through feature point matching, the accuracy of which is limited by the structure of the calibration board, environmental interference and sensor noise.

[0003] At present, there are mainly three methods for data fusion calibration, one is a calibration method based on point cloud and image feature matching, which relies on manual extraction of corner points or edge features and is easily affected by light changes and dynamic objects; the second is an online calibration method based on motion trajectory, which needs to rely on the motion consistency between consecutive frames and is easy to fail in static scenes; the third is a calibration method based on reflection intensity, which uses radar strong reflection points and image features for association, but the 4D radar point cloud is sparse and noisy, and the camera is easily affected by mirror reflection and atmospheric scattering, resulting in insufficient feature association reliability. Therefore, it is very necessary to design a 4D millimeter wave radar and camera calibration method and system based on projection error. SUMMARY

[0004] The purpose of the present application is to provide a 4D millimeter wave radar and camera calibration method and system based on projection error, which improves the calibration accuracy and environmental adaptability through dynamic interference suppression, noise collaborative filtering and nonlinear optimization mechanism.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A 4D millimeter wave radar and camera calibration method based on projection error, comprising the following steps:

[0007] Synchronously collecting data of a calibration board in a static scene by a 4D millimeter wave radar and a camera; the synchronous data includes radar point cloud and original image;

[0008] Filtering dynamic objects and suppressing noise on the synchronous data to obtain filtered data; the filtered data includes static enhanced point cloud and corrected image;

[0009] Extracting coordinates and detecting checkerboard corner points on the filtered data to obtain data coordinates; the data coordinates include point cloud 3D coordinates and pixel 2D coordinates;

[0010] The 3D coordinates of the point cloud are projected and converted by an external parameter matrix to obtain converted coordinates, and the projection error between the converted coordinates and the 2D pixel coordinates is calculated.

[0011] Based on the projection error, the external parameter matrix is iteratively optimized by a nonlinear optimization algorithm to obtain an optimized matrix.

[0012] Optionally, the synchronous data of the calibration board in the static scene is collected by the 4D millimeter wave radar and the camera, including:

[0013] A calibration board with a third-order fractal structure is designed. The base of the calibration board is an aluminum honeycomb plate, the surface of the calibration board is covered with an LED array with a pitch of 10 mm, the LED array is attached to a radar reflective film, and the radar reflective film is engraved with a golden angle spiral pattern.

[0014] The calibration board is periodically scanned by the chirp signal emitted by the 4D millimeter wave radar to obtain initial point cloud; the LED array flashes at the same frequency as the periodic scanning according to the chaotic sequence.

[0015] The pose of the calibration board is constructed in real time by the camera, and the initial point cloud is dynamically adjusted by the pose of the calibration board to obtain optimized point cloud.

[0016] Pseudo-random phase coding is added to the optimized point cloud to obtain radar point cloud, and the camera is subjected to optical flow constraint by LED pulse and exposure time to obtain original image.

[0017] Optionally, the synchronous data is filtered and noise suppressed to obtain filtered data, including:

[0018] The optical flow field of the original image is decomposed by the chaotic sequence to obtain the optical flow vector field;

[0019] The speckle autocorrelation coefficient of each pixel in the original image is calculated by the laser speckle field of the LED array;

[0020] Based on the optical flow vector field and the speckle autocorrelation coefficient, the pixel offset compensation of the original image is performed by the correlation coefficient gradient descent method to obtain a compensated image;

[0021] The atmospheric transmittance map is obtained by spectral radiation inversion of the LED array, the mirror reflection compensation mask is generated by the optical properties of the pyramid microstructure of the radar reflective film, and the compensated image is subjected to reflection compensation enhancement according to the atmospheric transmittance map and the mirror reflection compensation mask to obtain a corrected image.

[0022] Optionally, the synchronous data is filtered and noise suppressed to obtain filtered data, further including:

[0023] A dynamic probability heat map is constructed according to the phase correlation relationship between the Doppler velocity distribution of the radar point cloud and the optical flow vector field.

[0024] The radar point cloud is decomposed into an azimuth-elevation-Doppler three-dimensional tensor, the entanglement of the three-dimensional tensor and the reflection spectrum characteristics of the golden angle spiral pattern is obtained through a quantum derivation algorithm, and the point cloud clusters with an entanglement higher than 0.8 in the radar point cloud are coherently synthesized and suppressed to obtain path point clouds;

[0025] The path point clouds are multi-scale decomposed in the time domain, the spatial domain and the frequency domain through a complex wavelet packet transform to obtain filtered point clouds;

[0026] The filtered point clouds are sparsely coded and reconstructed through the manifold trajectory of the pose of the calibration board to separate the rigid motion component in the filtered point clouds and obtain static enhanced point clouds;

[0027] The confidence of the corrected image and the static enhanced point clouds is calculated through a pre-constructed double-flow Bayesian network, and the corrected image and the static enhanced point clouds are screened through the confidence.

[0028] Optionally, the filtered data is subjected to coordinate extraction and checkerboard corner detection to obtain data coordinates, including:

[0029] An iterative function is constructed based on the self-similarity ratio of the calibration board, and the point cloud Hausdorff dimension obtained through the iterative function is used to remove non-structure points in the static enhanced point clouds to obtain skeleton point clouds;

[0030] The reflection spectrum characteristics of the golden angle spiral pattern are encoded into quantum states, and the spiral control points in the skeleton point clouds are located through the Hamiltonian of the quantum states to obtain point cloud 3D coordinates;

[0031] The Harris corner points of the corrected image are detected to obtain a corner point response map, and then non-maximum suppression is performed to obtain a preliminary corner point coordinate set;

[0032] The preliminary corner point coordinate set is subjected to triangular subdivision to obtain a corner point topology relationship map, and then distance variance screening is performed to obtain an effective corner point set;

[0033] The effective corner point set is subjected to main direction projection sorting to obtain pixel 2D coordinates.

[0034] Optionally, the point cloud 3D coordinates are projected and converted through an external parameter matrix to obtain converted coordinates, and the projection error of the converted coordinates and the pixel 2D coordinates is calculated, including:

[0035] The point cloud 3D coordinates are subjected to non-local correlation operation through the quantum entanglement field of the photon orbital angular momentum to obtain entanglement enhanced coordinates;

[0036] The external parameter matrix is subjected to dynamic polarization modulation operation through a liquid crystal topological phase transition metasurface to obtain a space-time coding matrix;

[0037] The entanglement-enhanced coordinates are projected in fractional dimension by a space-time encoding matrix to obtain fractal projection coordinates;

[0038] The fractal projection coordinates are subjected to superconducting quantum interference operation to obtain Josephson phase coordinates;

[0039] The Josephson phase coordinates are subjected to near-field optical compression operation to obtain plasmonic fingerprint coordinates;

[0040] The plasmonic fingerprint coordinates are subjected to pulse timing coding operation to obtain pulse neural coordinates;

[0041] The pulse neural coordinates are subjected to observation locking operation to obtain conversion coordinates;

[0042] Based on the conversion coordinates and the pixel 2D coordinates, projection errors are obtained by Chen-Simon integral operation.

[0043] Optionally, based on the projection errors, an external parameter matrix is iteratively optimized by a nonlinear optimization algorithm to obtain an optimized matrix, including:

[0044] The projection errors are subjected to polarization conversion by sunflower light tracking principle to obtain error energy field;

[0045] The error energy field is subjected to gradient marking by bee pheromone diffusion to obtain pheromone concentration map;

[0046] The aerial root of the pheromone concentration map grows towards the geotropism to explore the path to obtain an optimized branch network;

[0047] The optimized branch network is subjected to energy dissipation by damped oscillation to obtain metastable state trajectory;

[0048] The metastable state trajectory is subjected to phase locking by firefly algorithm to obtain coherent optimization points;

[0049] The coherent optimization points are subjected to matrix reconstruction to obtain external parameter update;

[0050] The external parameter update is subjected to verification propagation by chaotic synchronization to obtain an optimized matrix.

[0051] A 4D millimeter wave radar and camera calibration system based on projection errors includes:

[0052] A data acquisition module is configured to acquire synchronous data of a calibration board in a static scene by a 4D millimeter wave radar and a camera; the synchronous data includes radar point cloud and original image;

[0053] A data processing module is configured to filter dynamic objects and suppress noise of the synchronous data to obtain filtered data; the filtered data includes static enhanced point cloud and corrected image;

[0054] The coordinate extraction module is configured to perform coordinate extraction and checkerboard corner point detection on the filtered data to obtain data coordinates, wherein the data coordinates include point cloud 3D coordinates and pixel 2D coordinates.

[0055] The projection conversion module is configured to perform projection conversion on the point cloud 3D coordinates by using an external parameter matrix to obtain conversion coordinates and calculate projection errors between the conversion coordinates and the pixel 2D coordinates.

[0056] The iterative optimization module is configured to perform iterative optimization on the external parameter matrix by using a nonlinear optimization algorithm based on the projection errors to obtain an optimized matrix.

[0057] According to the embodiments of the present application, the following technical effects are achieved: the 4D millimeter wave radar and camera calibration method based on projection errors provided by the present application comprises the following steps: collecting synchronous data of a calibration board in a static scene by using a 4D millimeter wave radar and a camera; the synchronous data comprises radar point clouds and original images; performing dynamic object filtering and noise suppression on the synchronous data to obtain filtered data; the filtered data comprises static enhanced point clouds and corrected images; performing coordinate extraction and checkerboard corner point detection on the filtered data to obtain data coordinates; the data coordinates comprise point cloud 3D coordinates and pixel 2D coordinates; performing projection conversion on the point cloud 3D coordinates by using an external parameter matrix to obtain conversion coordinates and calculating projection errors between the conversion coordinates and the pixel 2D coordinates; performing iterative optimization on the external parameter matrix by using a nonlinear optimization algorithm based on the projection errors to obtain an optimized matrix. The method significantly improves the calibration accuracy and environmental adaptability by using dynamic interference suppression, noise collaborative filtering and a nonlinear optimization mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0059] Figure 1 The 4D millimeter wave radar and camera calibration method of the present application is shown in the flowchart.

[0060] Figure 2 The coordinate extraction and checkerboard corner point detection method of the present application is shown in the flowchart. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1 As shown, the present invention provides a 4D millimeter wave radar and camera calibration method based on projection error, comprising the following steps:

[0064] Step 100: Collecting synchronized data of a calibration plate in a static scene using a 4D millimeter-wave radar and a camera; the synchronized data includes: radar point cloud and original image;

[0065] Specifically, a calibration plate with a third-order fractal structure is first designed; the base of the calibration plate is made of aviation-grade aluminum honeycomb panel with a thickness of 20 mm and a density of 3.5 kg / m 3 The surface is precisely laid with a micro LED array with a spacing of 10mm. The wavelength of the LED array is 850nm and the single point brightness is 200cd / m 2 A composite radar reflective film with a thickness of 0.2 mm and a dielectric constant of 4.2 is applied beneath the array. Laser micro-engraving is used to create a golden spiral pattern with a line width of 0.5 mm, a spiral angle of 137.5°, and a fractal order of 3. The film is also covered with an anti-reflection layer consisting of pyramidal microstructures with a cone angle of 70° and a height of 50 μm to enhance millimeter-wave backscattering and suppress specular reflections. The calibration plate measures 1.2 m × 0.8 m and features a three-order fractal structure: a primary structure consisting of an overall rectangular frame, a secondary structure consisting of eight golden spiral patterns, and a tertiary structure consisting of miniature Koch snowflake patterns nested within the spiral arms.

[0066] Then the 4D millimeter-wave radar emits a linear frequency chirp signal with a period of 50 ms and a slope of 200 MHz / μs, and the MIMO antenna array performs periodic scanning of the calibration board in the azimuth angle ±60° and the pitch angle ±15°. At the same time, the LED array receives the synchronization signal based on chaotic logic, and through the iterative equation dx / dt = 10(y-x), dy / dt = x(28-z)-y, dz / dt = xy-8z / 3, initial value x0 = 0.1, y0 = 0, z0 = 0, and then through the FPGA conversion to the duty cycle adjustable PWM pulse, the LED flicker is strictly synchronized with the radar scanning period. The radar receiving end processes the echo signal according to the 3D-FFT algorithm and generates the initial point cloud.

[0067] Then the camera continuously captures the calibration board image at a frame rate of 60 fps, extracts the candidate light spots of the image through adaptive threshold segmentation, and filters out the noise with a circularity of 0.85< circularity <1.2. The light spot center coordinates are matched with the pre-stored LED topological map. The matching process first calculates the initial pose through the 2D-3D correspondence relationship, and then removes the mismatched points according to the re-projection error. Finally, a plurality of stable control points are obtained. The control points are represented as the weighted sum of the four virtual control points, and the weight coefficients are decomposed through the SVD algorithm. Then the rotation matrix and translation vector are calculated, and the Dog-leg gradient descent method is used for iterative optimization to obtain the 6DOF pose and its covariance matrix. The initial point cloud is converted from the radar coordinate system to the calibration board coordinate system, and the linear velocity and angular velocity of the calibration board are calculated through the central difference method, so as to perform point-level motion compensation on the initial point cloud, and the expression is:

[0068]

[0069] where [ω] is the skew-symmetric matrix of angular velocity, v is linear velocity, ω is angular velocity, P i is the radar point coordinate, ΔP i is the compensation coordinate offset, τ is the time normalization coefficient, T s is the scanning period, t s is the initial time, t i is the current time. If the distance between the nearest neighbor points after compensation is >3 cm, then the real-time position is corrected through the error function to obtain the optimized point cloud, where e is the total projection error, P r is the radar feature point coordinate, is the transformation matrix, P b is the calibration board feature point coordinate, and ||·|| is the Euclidean distance norm.

[0070] Finally, the 31-bit Gold sequence is used as the phase encoding source, and the binary sequence is generated by a linear feedback shift register. After BPSK modulation and conversion to a phase sequence, it is added to each point of the optimized point cloud. The point cloud is divided into an 8x8 grid, and each grid is assigned a unique sequence starting index. The sequence position of the grid is calculated according to the timestamp, and the phase encoding is written into the attribute field of the point cloud to generate the radar point cloud. At the same time, the camera exposure starting time is aligned with the pulse rising edge of the LED, and the collected image is filtered in the time domain and enhanced by difference according to the laser characteristics of the LED array, thereby obtaining the original image.

[0071] Step 200: Dynamic object filtering and noise suppression of the synchronization data to obtain filtered data; the filtered data includes: static enhanced point cloud and corrected image;

[0072] Specifically, the step of obtaining the corrected image is: constructing a spatiotemporal gradient according to the original image sequence I(x, y, t), and using the chaotic sequence s(t) as the regularization weight, thereby constructing an energy function where u x is the x-direction optical flow, v y is the y-direction optical flow, and α is the balance coefficient, which is 0.3 in this embodiment. The optical flow vector field is obtained by iteratively solving the Euler-Lagrange equation 20 times by the optical flow algorithm, and the equation expression is:

[0073]

[0074] where is the neighborhood mean of the x-direction optical flow u k of the kth iteration, is the neighborhood mean of the y-direction optical flow v k of the kth iteration. The speckle autocorrelation coefficient R(x, y, δx, δy) of each pixel (x, y) is calculated according to the coherence length and speckle size of the LED array laser, and the calculation formula is:

[0075]

[0076] wherein δx, δy∈[-3, 3] is the offset of the pixel, Ω is the neighborhood of the pixel. Based on the optical flow vector field and the speckle autocorrelation coefficient, the target optimization function is constructed, and the target optimization function is solved by the gradient descent method of the correlation coefficient, and the original image is bilinearly interpolated and reconstructed to compensate for the pixel offset, and the compensated image is obtained. Finally, based on the spectral radiation of the LED array, the atmospheric transmittance map is obtained by the atmospheric scattering model, and the specular reflection compensation mask is generated based on the optical properties of the pyramid microstructure of the radar reflecting film, and the adaptive gamma correction is performed on the compensated image based on the atmospheric transmittance map and the specular reflection compensation mask, so as to perform reflection compensation enhancement, and obtain the corrected image.

[0077] Specifically, the step of obtaining the static enhanced point cloud is: according to the Doppler velocity distribution v d and the phase correlation relationship of the optical flow vector field F(u, v), a dynamic probability heat map P d is constructed, and the expression is:

[0078]

[0079] wherein is the phase gradient of the Doppler velocity, and ∠F is the direction angle of the optical flow vector. The radar point cloud is decomposed into an azimuth-elevation-Doppler three-dimensional tensor, and the entanglement degree E j of the three-dimensional tensor and the reflection spectral characteristics of the golden angle spiral pattern is obtained by a quantum derivation algorithm, and the calculation formula is:

[0080]

[0081] wherein Ψ(·) is a quantum state mapping function, T is a three-dimensional tensor, Sr is a golden angle spiral pattern reflection spectrum feature, and <·|·> is a Hilbert space inner product. If the entanglement degree of a point cloud cluster is higher than 0.8, it is determined as a strong correlation static point and coherent synthesis is suppressed to eliminate dynamic point cloud, and a path point cloud is obtained. The path point cloud is decomposed in time domain, space domain and frequency domain by complex wavelet packet transform, the time series transient component is extracted in time domain with a scanning period as a window, the outlier in the point cloud is detected in space domain according to the Hausdorff distance, and the high-frequency noise is identified and the low-frequency coherent component is reserved in frequency domain, and the filtered point cloud is obtained by reconstruction according to the decomposition results in three dimensions.

[0082] Step 300: coordinate extraction and checkerboard corner detection are performed on the filtered data to obtain data coordinates; the data coordinates include: point cloud 3D coordinates and pixel 2D coordinates; the specific steps are as shown in Figure 2

[0083] Step 301: based on the self-similarity ratio of the calibration board, an iterative function is constructed, and the Hausdorff dimension of the point cloud obtained by the iterative function is used to remove non-structural points in the static enhanced point cloud to obtain a skeleton point cloud.

[0084] Specifically, according to the self-similarity ratio an iterative function f i (x)=ρR(θ i )x+t i is generated, wherein φ is a golden angle, which is 137.5° in this embodiment, R(θ i ) is a rotation matrix, θ i is a rotation angle, θ i ∈{0°,72°,144°,216°,288°} in this embodiment, t i is the i-th level spiral translation vector. Then the box counting method is used to calculate the required minimum number of spheres with a coverage radius r in the static enhanced point cloud, and the points with a minimum sphere number less than 2.73 are reserved to form a skeleton point cloud. ​

[0085] Step 302: encode the reflection spectrum feature of the golden angle spiral pattern into a quantum state, and locate the spiral control points in the skeleton point cloud through the Hamiltonian of the quantum state, to obtain the point cloud 3D coordinates;

[0086] Specifically, the reflection pulse sequence of the reflection spectrum feature is subjected to short-time Fourier transform to obtain a time-frequency spectrum, and the main frequency component thereof is extracted to construct a quantum state. The eigenstate of the quantum state is solved through a preset Hamiltonian, and a point cloud cluster with eigenenergy in a preset range is screened out. The nearest neighbor of the position expectation value in the point cloud cluster to the skeleton point cloud is calculated as a spiral control point, which is collected and output as the point cloud 3D coordinates.

[0087] Step 303: Harris corner point detection is performed on the corrected image to obtain a corner point response map, and then non-maximum suppression is performed to obtain a preliminary corner point coordinate set;

[0088] Specifically, the corrected image is subjected to grayscale processing, and a grayscale gradient matrix and a corner point response are calculated. Points with a corner point response greater than 0.01 and being a local maximum value in a 5x5 window are retained to generate a preliminary corner point coordinate set.

[0089] Step 304: Delaunay triangulation is performed on the preliminary corner point coordinate set to obtain a corner point topology graph, and then distance variance screening is performed to obtain an effective corner point set;

[0090] Specifically, Delaunay triangulation is performed on the preliminary corner point coordinate set to generate a triangular mesh, and the length variance of the adjacent edges thereof is calculated. Points with a length variance greater than 0.5 are determined as abnormal twisted points and removed to obtain an effective corner point set.

[0091] Step 305: the effective corner point set is subjected to main direction projection sorting to obtain pixel 2D coordinates.

[0092] Specifically, the covariance matrix of the effective corner point set is calculated, the eigenvector corresponding to the maximum eigenvalue is taken as the main direction, the effective corner point set is projected along the main direction and sorted in ascending order to obtain ordered pixel 2D coordinates.

[0093] Step 400: project the point cloud 3D coordinates through an external parameter matrix to obtain converted coordinates, and calculate the projection error of the converted coordinates and the pixel 2D coordinates;

[0094] Specifically, a two-photon entangled state is constructed according to the photon orbital angular momentum (OAM), the point cloud 3D coordinates are mapped into the OAM Hilbert space, then the entanglement fidelity is calculated, and the Hamiltonian is converted through unitary transformation to enhance the coordinate correlation, and the entanglement enhanced coordinates are output. A transmission function is constructed according to the liquid crystal topological phase transition super surface where β is the phase modulation depth, f uFor and f v For the spatial frequency of the coordinates, and using the transfer function to encode the extrinsic matrix into the Jones matrix, the Stokes parameters are extracted to generate the space-time encoding matrix. The entangled coordinates are projected into the fractal space by the projection operator generated by the Mandelbrot set iteration, and the space-time encoding matrix is used to perform coordinate compression on the projected coordinates to obtain fractal projection coordinates. The phase difference of the modulation of the fractal projection coordinates is input into the pre-constructed superconducting loop inductance, the critical current is calculated using the nonlinear Langevin equation, and the Josephson phase coordinates are obtained by solving the Josephson phase dynamics formula. The finite difference time domain method is used to solve the Maxwell equation of the coordinates to generate the plasmonic fingerprint coordinates. Then the coordinates are converted into pulse neural coordinates represented by pulse sequences, and the phase coherence thereof is extracted through a phase-locked amplifier, and the converted coordinates are output when the coherence is greater than 0.9. Finally, the norm field is constructed according to the converted coordinates and the 2D pixel coordinates, and the Chen-Simon integral is performed to obtain the projection error.

[0095] Step 500: based on the projection error, the extrinsic matrix is iteratively optimized by a nonlinear optimization algorithm to obtain an optimized matrix.

[0096] Specifically, first, an error vector field is constructed according to the sunflower light tracking principle, a polarization energy density field is generated by Maxwell-Boltzmann distribution, and a three-dimensional distribution of error energy field is formed. Then, taking the error energy field as the initial condition, the anisotropic diffusion equation of the energy field is solved by adopting the bee colony pheromone diffusion mechanism to obtain a pheromone concentration map. Subsequently, based on the fluid dynamics equation, the steady-state distribution of the concentration map is solved by the finite element method, and the connected domain with a steady-state distribution greater than 0.7K is extracted to generate an optimized branch network. An RLC equivalent circuit model is introduced, and the metastable trajectory of the current decay trajectory of the circuit model is solved by the fourth-order Runge-Kutta method. Each metastable point is regarded as a firefly, and the iteration is performed by a step size of 0.05 until the convergence phase difference variance is less than 0.01. The optimized point set is a point set, and the weighted centroid thereof is calculated. The centroid is taken as the starting point, and the point set is decomposed into a rotation vector and a translation vector. The point set matrix is updated by using Lie algebra to obtain the extrinsic update. Finally, the Lyapunov index is calculated according to the extrinsic update, and the synchronization is determined to be successful when the Lyapunov index is less than 0, so as to output the final optimized matrix.

[0097] The application also provides a 4D millimeter wave radar and camera calibration system based on projection error, comprising:

[0098] A data acquisition module is configured to acquire synchronous data of a calibration board in a static scene by a 4D millimeter wave radar and a camera. The synchronous data includes radar point clouds and original images.

[0099] The data processing module is used for dynamic object filtering and noise suppression on the synchronous data to obtain filtered data; the filtered data comprises static enhanced point clouds and a corrected image;

[0100] The coordinate extraction module is used for coordinate extraction and checkerboard corner detection on the filtered data to obtain data coordinates; the data coordinates comprise point cloud 3D coordinates and pixel 2D coordinates;

[0101] The projection conversion module is used for projection conversion on the point cloud 3D coordinates through an external parameter matrix to obtain converted coordinates and calculate projection errors of the converted coordinates and the pixel 2D coordinates;

[0102] The iterative optimization module is used for iterative optimization on the external parameter matrix through a nonlinear optimization algorithm based on the projection errors to obtain an optimized matrix.

[0103] The beneficial effects of the present application are as follows:

[0104] 1) The influence of environmental interference on sensor data is effectively eliminated through dynamic object filtering and noise suppression, ensuring the accuracy of feature extraction, and the iterative optimization of the external parameter matrix is performed through the firefly algorithm and the nonlinear optimization mechanism of chaotic synchronization verification, overcoming the defect that the traditional linear method is easy to fall into local optimization, and the calibration accuracy is significantly improved;

[0105] 2) Through the calibration plate structure design of the third-order fractal structure, the LED array and the golden angle spiral reflective film, combined with the active modulation of the pseudo-random phase coding, the stability of obtaining high recognition features under complex light, dynamic interference and atmospheric scattering conditions is improved, and the environmental adaptability of calibration is limited;

[0106] 3) Through the atmospheric transmittance inversion and the mirror reflection compensation mask, the problems of camera mirror reflection and radar point cloud sparsity are solved, and the robustness of calibration in harsh environments is improved;

[0107] 4) Through quantum state coding and Hamiltonian positioning, the problem of insufficient millimeter wave point cloud structure information is solved, the matching accuracy of point cloud coordinates and image corner points is improved, the physical relevance of coordinate conversion is enhanced combined with fractal projection and space-time coding, and the reliability of calibration is enhanced;

[0108] 5) The calibration board pose is used to compensate the point cloud motion in real time, and the LED pulse is used to constrain the camera exposure, avoiding the occurrence of calibration failure caused by motion blur, and combining with lightweight algorithms such as wavelet packet decomposition, the calculation complexity is reduced while the accuracy is ensured, and the real-time performance and stability of the whole method are improved.

[0109] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other.

[0110] The principles and implementations of the present application are described in the specific examples. The above examples are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A 4D millimeter-wave radar and camera calibration method based on projection error, characterized in that: The steps include: The 4D millimeter-wave radar and camera are used to collect synchronous data of the calibration plate in the static scene; the synchronous data includes: radar point cloud and original image; Performing dynamic object filtering and noise suppression on the synchronized data to obtain filtered data; the filtered data includes: a static enhanced point cloud and a corrected image; Performing coordinate extraction and checkerboard corner point detection on the filtered data to obtain data coordinates; the data coordinates include: point cloud 3D coordinates and pixel 2D coordinates; Performing a projection transformation on the point cloud 3D coordinates using an extrinsic parameter matrix to obtain transformed coordinates, and calculating a projection error between the transformed coordinates and the pixel 2D coordinates; Based on the projection error, the extrinsic parameter matrix is ​​iteratively optimized by a nonlinear optimization algorithm to obtain an optimized matrix.

2. The 4D millimeter-wave radar and camera calibration method based on projection error according to claim 1, characterized in that: The 4D millimeter-wave radar and camera collect synchronized data of the calibration plate in the static scene, including: A calibration plate with a third-order fractal structure was designed; the base of the calibration plate was an aluminum honeycomb panel, the surface of the calibration plate was covered with an LED array with a spacing of 10 mm, the LED array was bonded with a radar reflective film, and the radar reflective film was engraved with a golden angle spiral pattern; The calibration plate is periodically scanned by the chirp signal emitted by the 4D millimeter-wave radar to obtain an initial point cloud; the LED array flashes at the same frequency as the periodic scanning according to a chaotic sequence; Constructing a calibration plate pose in real time through the camera, and dynamically adjusting the initial point cloud through the calibration plate pose to obtain an optimized point cloud; Pseudo-random phase coding is added to the optimized point cloud to obtain the radar point cloud, and optical flow constraints are performed on the camera using LED pulses and exposure time to obtain the original image.

3. The 4D millimeter-wave radar and camera calibration method based on projection error according to claim 2, characterized in that: Performing dynamic object filtering and noise suppression on the synchronized data to obtain filtered data includes: Performing optical flow field decomposition on the original image through the chaotic sequence to obtain an optical flow vector field; Calculating the speckle autocorrelation coefficient of each pixel in the original image using the laser speckle field of the LED array; Based on the optical flow vector field and the speckle autocorrelation coefficient, performing pixel offset compensation on the original image by a correlation coefficient gradient descent method to obtain a compensated image; An atmospheric transmittance map is obtained by inverting the spectral radiation of the LED array, a specular reflection compensation mask is generated by the optical properties of the pyramid microstructure of the radar reflective film, and reflection compensation and enhancement are performed on the compensated image based on the atmospheric transmittance map and the specular reflection compensation mask to obtain the corrected image.

4. The 4D millimeter-wave radar and camera calibration method based on projection error according to claim 3 is characterized in that: Performing dynamic object filtering and noise suppression on the synchronized data to obtain filtered data further includes: Constructing a dynamic probability heat map based on the Doppler velocity distribution of the radar point cloud and the phase correlation relationship of the optical flow vector field; Decomposing the radar point cloud into a three-dimensional azimuth-elevation-Doppler tensor, obtaining the degree of entanglement between the three-dimensional tensor and the reflection spectrum characteristics of the golden angle spiral pattern through a quantum derivative algorithm, and performing coherent synthesis suppression on point cloud clusters in the radar point cloud having an entanglement degree greater than 0.8 to obtain a path point cloud; Performing multi-scale decomposition on the path point cloud in the time domain, spatial domain and frequency domain respectively by complex wavelet packet transform to obtain a filtered point cloud; Performing sparse coding reconstruction on the filtered point cloud through the manifold trajectory of the calibration plate posture, separating the rigid motion component in the filtered point cloud, and obtaining the static enhanced point cloud; The confidences of the corrected image and the static enhanced point cloud are respectively calculated by a pre-built two-stream Bayesian network, and the corrected image and the static enhanced point cloud are screened according to the confidences.

5. The 4D millimeter-wave radar and camera calibration method based on projection error according to claim 1, characterized in that: Performing coordinate extraction and checkerboard corner point detection on the filtered data to obtain data coordinates includes: constructing an iterative function based on the self-similarity ratio of the calibration plate, and eliminating non-structural points in the static enhanced point cloud using the Hausdorff dimension of the point cloud obtained by the iterative function to obtain a skeleton point cloud; Encoding the reflection spectrum characteristics of the golden angle spiral pattern into a quantum state, and locating the spiral control points in the skeleton point cloud using the Hamiltonian of the quantum state to obtain the 3D coordinates of the point cloud; Performing Harris corner detection on the corrected image to obtain a corner response map and then performing non-maximum suppression to obtain a preliminary corner coordinate set; Triangulating the preliminary corner point coordinate set to obtain a corner point topology relationship diagram and then performing distance variance screening to obtain a valid corner point set; The valid corner point set is projected and sorted in the main direction to obtain the pixel 2D coordinates.

6. The 4D millimeter-wave radar and camera calibration method based on projection error according to claim 1, characterized in that: Performing a projection transformation on the point cloud 3D coordinates using an extrinsic parameter matrix to obtain transformed coordinates, and calculating a projection error between the transformed coordinates and the pixel 2D coordinates, including: Through the quantum entangled field of photon orbital angular momentum, non-local correlation operations are performed on the 3D coordinates of the point cloud to obtain entanglement-enhanced coordinates; Performing a dynamic polarization modulation operation on the extrinsic parameter matrix through a liquid crystal topological phase change metasurface to obtain a spatiotemporal coding matrix; Performing a fractal dimension projection operation on the entanglement enhancement coordinates through the space-time coding matrix to obtain fractal projection coordinates; performing a superconducting quantum interference operation on the fractal projection coordinates to obtain Josephson phase coordinates; performing a near-field optical compression operation on the Josephson phase coordinates to obtain plasmon fingerprint coordinates; Performing a pulse timing encoding operation on the plasmon fingerprint coordinates to obtain pulse neural coordinates; Performing an observation lock operation on the pulse neural coordinates to obtain the transformed coordinates; The projection error is obtained based on the transformed coordinates and the pixel 2D coordinates through a Chern-Simons integration operation.

7. The 4D millimeter-wave radar and camera calibration method based on projection error according to claim 1, characterized in that: Based on the projection error, the extrinsic parameter matrix is ​​iteratively optimized by a nonlinear optimization algorithm to obtain an optimized matrix, including: Polarization conversion is performed on the projection error using the sunflower ray tracing principle to obtain an error energy field; Gradient marking of the error energy field is performed by bee colony pheromone diffusion to obtain a pheromone concentration map; Simulating the geotropic growth of aerial roots according to the pheromone concentration map to perform path exploration and obtain an optimized branch network; dissipating energy of the optimized branch network through damped oscillation to obtain a metastable trajectory; Phase locking is performed on the metastable trajectory by using a firefly algorithm to obtain a coherent optimization point; Performing matrix reconstruction on the coherent optimization points to obtain an updated external parameter; The updated amount of the external parameters is verified and propagated through chaotic synchronization to obtain an optimized matrix.

8. A 4D millimeter-wave radar and camera calibration system based on projection error, characterized in that: include: The data acquisition module is used to collect synchronous data of the calibration plate in the static scene through the 4D millimeter wave radar and the camera; The synchronized data includes: radar point cloud and original image; A data processing module is used to perform dynamic object filtering and noise suppression on the synchronized data to obtain filtered data; the filtered data includes: a static enhanced point cloud and a corrected image; A coordinate extraction module is used to perform coordinate extraction and checkerboard corner point detection on the filtered data to obtain data coordinates; the data coordinates include: point cloud 3D coordinates and pixel 2D coordinates; A projection conversion module, configured to perform a projection conversion on the 3D coordinates of the point cloud using an extrinsic parameter matrix to obtain a conversion coordinate, and calculate a projection error between the conversion coordinate and the 2D coordinate of the pixel; The iterative optimization module is used to iteratively optimize the extrinsic parameter matrix based on the projection error by a nonlinear optimization algorithm to obtain an optimized matrix.

Images