Target fusion method and system based on radar track projection

Through sparse optimization, Liqun manifold optimization, Gaussian process regression model and adaptive nuclear method, the problems of accuracy loss and data quality reduction in radar track projection are solved, and high-precision and real-time radar tracks and photoelectric tracks are achieved. Fusion reduces computational complexity and maintenance costs.

CN118736369BActive Publication Date: 2024-12-03NANJING HAOLIN TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411225487.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-12-03
Estimated Expiration
2044-09-03

AI Technical Summary

Technical Problem

The prior art has problems of accuracy loss and data quality reduction when projecting radar tracks into camera images, especially under the influence of environmental factors, and high computational complexity leads to processing delays and maintenance difficulties in real-time applications.

Method used

Adaptive camera calibration based on sparse optimization and Liqun manifold optimization method are used for coordinate transformation, combined with Gaussian process regression model and sequence importance sampling algorithm for track projection, and introduced adaptive kernel method and sparse dictionary learning technology for real-time correction and model update , multimodal data fusion is performed using constrained non-negative tensor decomposition model.

Benefits of technology

It improves the accuracy, reliability and real-timeness of radar tracks and photoelectric tracks, reduces calculation complexity and maintenance costs, and enhances the accuracy and robustness of target tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118736369B_ABST
    Figure CN118736369B_ABST
Patent Text Reader

Abstract

The present invention discloses a target fusion method and system based on radar track projection, the method comprises obtaining original parameters of a camera and a radar, performing adaptive camera calibration, obtaining optimized camera parameters, performing coordinate conversion, obtaining coordinate conversion parameters; calling a preconfigured Gaussian process regression model for track projection, generating projection coordinates; generating a probabilistic track projection result based on the projection coordinates; constructing a correction model, using an online kernel ridge regression algorithm to update the parameters of the correction model in real time, using a trained correction model, obtaining corrected track point data; obtaining target tracking data, constructing a multimodal tensor; constructing a constrained non-negative tensor decomposition model, extracting potential factors, realizing high-precision fusion of radar tracks and optoelectronic tracks, and obtaining the final fused track result. The present invention improves the accuracy, reliability and real-time performance of the fusion of radar tracks and optoelectronic tracks, and avoids the influence of environmental factors on data quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of data fusion, and in particular to a target fusion method and system based on radar track projection. Background Art

[0002] The technical background of projecting radar tracks into camera images involves multiple fields, including photogrammetry, computer vision, radar signal processing, and geographic information systems. In practical applications, these technologies are used in many fields such as drone navigation, traffic monitoring, and disaster response. By projecting radar tracks into camera images, operators can understand radar data more intuitively and use visual information to improve the accuracy of target recognition.

[0003] Although the technology of projecting radar tracks into camera images has broad application prospects, it also has some disadvantages and challenges. Since radar and camera are two different sensors, their measurement accuracy and error characteristics are different, which may lead to accuracy loss. In addition, environmental factors such as weather and atmospheric conditions and lighting conditions may also lead to data quality degradation. When data processing is complex, a large amount of computing resources are required, which may lead to processing delays in real-time applications, and maintenance difficulties also need to be considered. Summary of the invention

[0004] The purpose of the invention is to propose a target fusion method and system based on radar track projection to solve the above problems existing in the prior art.

[0005] The technical solution is a target fusion method based on radar track projection, comprising the following steps:

[0006] S1. Obtain the original parameters of the camera and the radar, use the pre-collected calibration plate image data to perform adaptive camera calibration based on sparse optimization, and obtain the optimized camera parameters; based on the optimized camera parameters, use the Lie group manifold optimization method to perform coordinate transformation to obtain the coordinate transformation parameters from the radar to the camera; wherein the original parameters of the camera and the radar include the original intrinsic parameter data of the camera, the extrinsic parameter data of the camera, and the installation parameters of the radar;

[0007] S2, obtain the original track data, and based on the original track data, the optimized camera parameters and the coordinate conversion parameters, call the pre-configured Gaussian process regression model to perform track projection, generate the projection coordinates and their uncertainty estimates; obtain the particle set at the previous moment, and dynamically update it using the sequence importance sampling algorithm to obtain the updated particle set; based on the projection coordinates and their uncertainty estimates and the updated particle set, generate the probabilistic track projection result;

[0008] S3. Based on the probabilistic track projection results, a correction model based on the adaptive kernel method is constructed, the online kernel ridge regression algorithm is used to update the parameters of the correction model in real time, and the sparse dictionary learning technology is introduced to improve the updating efficiency, so as to obtain a trained correction model. The trained correction model is used to obtain the corrected track point data;

[0009] S4. Obtain target tracking data, and construct a multimodal tensor based on the target tracking data and the corrected track point data; based on the multimodal tensor, construct a constrained non-negative tensor decomposition model to extract shared and modality-specific latent factors; based on the latent factors, achieve high-precision fusion of radar tracks and optoelectronic tracks to obtain the final fused track result.

[0010] The target fusion system based on radar track projection includes:

[0011] at least one processor; and,

[0012] a memory communicatively connected to at least one of the processors; wherein,

[0013] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the target fusion method based on radar track projection.

[0014] Beneficial effects: The present invention improves the accuracy, reliability and real-time performance of the fusion of radar track and optoelectronic track, avoids the influence of environmental factors on data quality, improves the accuracy and reliability of the projection of radar track to the optoelectronic image plane, reduces the calculation complexity, and also reduces the maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention.

[0016] Figure 2 This is a flow chart of step S1 of the present invention.

[0017] Figure 3 This is a flow chart of step S2 of the present invention.

[0018] Figure 4 This is a flow chart of step S3 of the present invention.

[0019] Figure 5 This is a flow chart of step S4 of the present invention.

[0020] Figure 6 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0021] like Figure 1 As shown, the present application proposes a target fusion method based on radar track projection, comprising the following steps:

[0022] S1. Obtain the original parameters of the camera and the radar, use the pre-collected calibration plate image data to perform adaptive camera calibration based on sparse optimization, and obtain the optimized camera parameters; based on the optimized camera parameters, use the Lie group manifold optimization method to perform coordinate transformation to obtain the coordinate transformation parameters from the radar to the camera; wherein the original parameters of the camera and the radar include the original intrinsic parameter data of the camera, the extrinsic parameter data of the camera, and the installation parameters of the radar;

[0023] S2, obtain the original track data, and based on the original track data, the optimized camera parameters and the coordinate conversion parameters, call the pre-configured Gaussian process regression model to perform track projection, generate the projection coordinates and their uncertainty estimates; obtain the particle set at the previous moment, and dynamically update it using the sequence importance sampling algorithm to obtain the updated particle set; based on the projection coordinates and their uncertainty estimates and the updated particle set, generate the probabilistic track projection result;

[0024] S3. Based on the probabilistic track projection results, a correction model based on the adaptive kernel method is constructed, the online kernel ridge regression algorithm is used to update the parameters of the correction model in real time, and the sparse dictionary learning technology is introduced to improve the updating efficiency, so as to obtain a trained correction model. The trained correction model is used to obtain the corrected track point data;

[0025] S4. Obtain target tracking data, and construct a multimodal tensor based on the target tracking data and the corrected track point data; based on the multimodal tensor, construct a constrained non-negative tensor decomposition model to extract shared and modality-specific latent factors; based on the latent factors, achieve high-precision fusion of radar tracks and optoelectronic tracks to obtain the final fused track result.

[0026] This embodiment can improve the accuracy of camera parameters by obtaining the original parameters of the camera and radar and using sparse optimization technology to perform adaptive camera calibration. The optimized camera parameters provide a reliable basis for subsequent coordinate transformation. The Lie group manifold optimization method is used for coordinate transformation, and the radar coordinate system can be accurately converted to the camera coordinate system. It has high robustness and accuracy when processing nonlinear transformations, ensuring the accurate alignment of different sensor data in the same coordinate system. The Gaussian process regression model is used for track projection, which can generate high-precision projection coordinates and their uncertainty estimates, can effectively deal with noise and uncertainty in radar data, and improve the reliability of track projection. The particle set is dynamically updated by the sequence importance sampling algorithm, and the motion state of the target can be tracked in real time. The updated particle set is combined with the projection coordinates and their uncertainty estimates to generate a more accurate and stable probability track projection result. Based on the probability track projection results, a correction model of the adaptive kernel method is constructed, and the online kernel ridge regression algorithm is used to update the model parameters in real time, which can quickly adapt to environmental changes and improve the real-time and accuracy of the correction model. The sparse dictionary learning technology is introduced to improve the efficiency of model update and reduce the consumption of computing resources. The trained correction model can provide high-precision corrected track point data, providing a reliable data basis for subsequent target fusion. Obtaining target tracking data and corrected track point data and constructing a multi-modal tensor can make full use of the data characteristics of different sensors. By constraining the non-negative tensor decomposition model, shared and modality-specific latent factors are extracted to achieve high-precision fusion of radar tracks and optoelectronic tracks, which can effectively fuse multi-source data and improve the accuracy and robustness of target tracking. The final fused track result has higher accuracy and reliability, and can better support target tracking and recognition tasks in practical applications.

[0027] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0028] S11, retrieving the original intrinsic parameter data of the camera from the pre-stored parameter database, including focal length, principal point coordinates and distortion coefficient; obtaining the external parameter data of the camera, including installation position and orientation information; and extracting the installation parameters of the radar, including position and direction data relative to the camera; organizing the original intrinsic parameter data of the camera, the external parameter data of the camera and the installation parameters of the radar into structured data, and storing them in a temporary cache as the original parameters of the camera and the radar;

[0029] S12, based on the original parameters and the pre-collected calibration plate image data, extract the coordinates of the image feature points; based on the coordinates of the image feature points, use the Zhang Zhengyou method to calculate the initial camera parameter estimation; based on the initial camera parameter estimation, construct an L1 norm optimization objective function including reprojection error and parameter complexity penalty, use the alternating direction multiplier method to solve the L1 norm optimization objective function, and output the optimized camera parameters;

[0030] S13. Based on the radar installation parameters, a Lie group representation is constructed; radar measurement points and camera image points with known corresponding relationships are extracted from the registration database; based on the optimized camera parameters, Lie group representation, radar measurement points and camera image points, a manifold optimization objective function is constructed; the Levenberg-Marquardt algorithm is used to update the parameters of the manifold optimization objective function, and the Huber loss function is introduced to handle outliers, and the optimized coordinate transformation parameters are output.

[0031] According to one aspect of the present application, step S1 further includes:

[0032] S14, obtaining pre-stored test data, and performing projection transformation based on the optimized camera parameters and coordinate transformation parameters to obtain coordinates after projection transformation; wherein the pre-stored test data is a data set containing actual image coordinates;

[0033] S15, calculating the average reprojection error between the coordinates after projection transformation and the actual image coordinates;

[0034] S16. If the average reprojection error exceeds the preset threshold, return to step S12; otherwise, save the optimized camera parameters and coordinate transformation parameters into the system parameter database.

[0035] In this embodiment, the coordinates of the image feature points are extracted using a pre-collected calibration plate image dataset; the initial camera parameter estimation is calculated based on the Zhang Zhengyou method. An L1 norm optimization objective function including reprojection error and parameter complexity penalty is constructed. The alternating direction multiplier method (ADMM) is used to solve the optimization problem. During the optimization process, an adaptive weight adjustment mechanism is introduced to dynamically adjust the weight parameters according to the field of view and magnification of the current camera, and finally the optimized camera intrinsic and extrinsic parameters are output and stored in the camera parameter database.

[0036] Based on the acquired radar installation parameters, the initial SO(3) rotation representation and SE(3) pose transformation are constructed, and the radar measurement points and camera image points with known corresponding relationships are extracted from the registration database. The manifold optimization objective function that integrates these corresponding point information and camera parameters is constructed. The Levenberg-Marquardt algorithm is used to iteratively optimize on the Lie group manifold, and the Lie group parameters are updated through exponential mapping in each iteration. The Huber loss function is introduced to handle outliers and improve the robustness of the algorithm. Finally, the optimized coordinate transformation parameters are output and saved in the coordinate transformation parameter database.

[0037] Load the reserved test data from the validation dataset and apply the camera parameters and coordinate transformation parameters for projection transformation. Calculate the average reprojection error and maximum error between the transformed coordinates and the actual image coordinates. If the error exceeds the preset threshold, return to the corresponding optimization step, adjust the optimization weight or initial value and re-execute. When the error meets the requirements, output the final camera parameters and coordinate transformation parameters to the system parameter database as input data for subsequent track processing steps.

[0038] In one embodiment of the present application, the original parameters of the camera and radar are obtained. The camera internal parameters include focal length: f = [fx, fy], in pixels; principal point coordinates: c = [cx, cy], in pixels; distortion coefficients: k = [k1, k2, p1, p2, k3], where k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients; the camera external parameters include rotation matrix: R∈R 3×3 , translation vector: t∈R 3 ; Radar installation parameters include the position vector relative to the camera: trl∈R 3 And the rotation matrix relative to the camera: Rrl∈R 3×3 ; Retrieve these parameters from a pre-stored parameter database and store them in an in-memory structure.

[0039] Construct a calibration plate image dataset, specifically: prepare a calibration plate of known size (such as a chessboard); take N (N ≥ 20) calibration plate images at different positions and angles; extract the corner coordinates of each image and store them as: X = {x i,j | i = 1, ..., N; j = 1, ..., M}, where x i,j ∈R 2 Represents the pixel coordinates of the jth corner point in the i-th image. Use Zhang Zhengyou calibration method to initialize the camera intrinsic and extrinsic parameters: Calculate the homography matrix H i , i = 1, ..., N; solve the camera internal parameter matrix K and external parameter [R i |ti ]; Initialize the distortion coefficient k = [0, 0, 0, 0, 0]. Construct a sparse optimization problem and define the objective function: J(θ) = Σ i Σ j ρ(||x i,j -P(X j , K, R i , t i , k)||2) + λ||k||1, where: θ= {K, {R i , t i}, k} is the parameter to be optimized, P(·) is the projection function, ρ(·) is the Huber loss function, λ is the L1 regularization coefficient, and the recommended initial value is λ = 0.1, R i represents the rotation matrix of the i-th image, t i Represents the translation vector of the ith image.

[0040] Solve using the alternating direction method of multipliers (ADMM): Initialize: u = 0, z = θ; repeat until convergence (maximum number of iterations 100, error threshold 1e-6): θ = argmin θ (J(θ) + (ρ / 2)||θ - z + u||2 2 ); z = S λ / ρ(θ +u); u = u + θ – z; where S λ / ρ(·) is a soft threshold operator, u and z represent different auxiliary variables, and argmin represents finding the variable value that minimizes the objective function. Perform adaptive weight adjustment and adjust λ according to the field of view FOV and magnification Z of the current camera: λ= λ0 * (1 + α * FOV) * (1 + β * log(Z)), where λ0 = 0.1, α = 0.01, β = 0.05. Store the optimized camera parameters in the camera parameter database, including the intrinsic parameter matrix K, the distortion coefficient k, and the extrinsic parameters of each image [R i |t i ].

[0041] Construct the Lie group representation and use the SO(3) Lie group as the rotation representation: R∈SO(3) = {R∈R 3×3 | R T R = I, det(R) = 1}, where det( ) represents the determinant of the matrix R and I represents the identity matrix; the SE(3) Lie group is used as the pose transformation representation: T∈SE(3) = {[R t; 0 1] | R∈SO(3), t∈R 3}. Get M pairs of corresponding points from the pre-stored database: {(pi , q i ) | i = 1, ..., M}, where p i ∈R 3 is the radar measurement point, q i ∈R 3 is the 3D reconstruction result of the camera image point. Construct the optimization objective function: J(T) = Σ i ρ(||q i -T*p i ||2), where ρ(·) is the Huber loss function: if |x|≤Δ, ρ(x) = x 2 / 2; if |x| > Δ, ρ(x) = Δ(|x| - Δ / 2), set Δ = 1.345 (standard choice). Optimize on the Lie group manifold using the Levenberg-Marquardt algorithm: Initialize: T0 = [Rrltrl; 0 1]; Repeat until convergence (maximum number of iterations 50, error threshold 1e-8): Compute the Jacobian matrix J i and residual r i , solve the increment: Δξ= -(J T J +λI) -1 J T r; Update T k+1 = T k * exp(Δξ^), where exp(·) is an exponential map, ^ means converting a vector into a Lie algebra, T k represents the pose transformation matrix at the kth iteration. In each iteration, the weight of each point pair is dynamically adjusted according to the Huber loss function: if |r i |≤Δ, wi =1; if |r i | > Δ, wi = Δ / |r i |. The optimized transformation matrix T is stored in the coordinate transformation parameter database.

[0042] Select K point pairs from the reserved test data set: {(p j , q j ) | j = 1, ..., K}, generate verification data. Calculate the error indicator: average reprojection error e avg = (1 / K) * Σ j ||q j -T*p j ||2; Maximum error e max =max j ||q j -T*p j ||2. If e avg> 1 pixel or e max > 5 pixels, adjust the initial value or the damping factor of the Levenberg-Marquardt algorithm, and re-execute the Lie group manifold optimization. Save the verified camera parameters and coordinate transformation parameters into the system parameter database.

[0043] This embodiment improves the accuracy and reliability of the radar track and optoelectronic track fusion system by introducing an adaptive camera calibration method based on sparse optimization and a robust coordinate transformation algorithm based on Lie group manifold optimization. In the camera calibration link, the sparse optimization objective function of the L1 norm is adopted to effectively suppress the influence of noise and outliers, making the calibration result more stable. In particular, in complex lighting and dynamic scenes, traditional methods are easily disturbed, while this embodiment dynamically adjusts the optimization weights to adapt to different imaging conditions and ensures the consistency of calibration. Experiments show that compared with traditional methods, this embodiment reduces the reprojection error by an average of 15%, and the performance improvement under extreme lighting conditions is as high as 30%. In terms of coordinate transformation, the optimization method based on Lie group manifold cleverly avoids the singularity problem in the Euclidean space representation, especially when dealing with large-angle rotations, and exhibits superior numerical stability. By optimizing on the Lie group manifold, the algorithm can more accurately capture the geometric characteristics of rigid body transformation and reduce the cumulative error. In actual tests, compared with the traditional ICP algorithm, the cumulative error of this embodiment after long-term operation is reduced by about 40%. In addition, the introduction of Huber loss function as M-estimator improves the robustness of the algorithm to outliers, and can maintain stable performance in the presence of local occlusion or sensor failure. This feature is particularly important in complex battlefield environments, ensuring the reliable operation of the system under harsh conditions. The entire calibration and coordinate transformation process adopts an iterative optimization strategy, and adaptive parameter adjustment is achieved through verification and fine-tuning steps. This closed-loop feedback mechanism enables the system to continuously optimize its performance and adapt to dynamic changes in the environment. In practical applications, this adaptive capability improves the long-term stability of the system, reduces the need for manual intervention, and reduces maintenance costs. This embodiment makes the spatial correspondence between radar tracks and optoelectronic images more accurate, lays a solid foundation for subsequent track projection and fusion, and improves the reliability and accuracy of the entire system.

[0044] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0045] S21, obtaining original track data, including the three-dimensional spatial coordinates, speed and timestamp of the target; constructing and training a Gaussian process regression model based on the original track data, the optimized camera parameters and the coordinate conversion parameters to obtain a trained Gaussian process regression model; optimizing the trained Gaussian process regression model using a completely independent conditional approximation technique to obtain an optimized Gaussian process regression model;

[0046] S22, inputting the original track data into the optimized Gaussian process regression model to obtain the projection coordinates of each track point on the image plane and its uncertainty estimation;

[0047] S23, obtaining the particle set of the previous moment and the current observation, generating new particles through the importance function; calculating the weight of each particle, and normalizing it to obtain normalized particles; resampling based on the normalized particles to obtain an updated particle set;

[0048] S24, integrating the projection coordinates and their uncertainty estimates with the updated particle set to generate a probabilistic track projection result.

[0049] In this embodiment, the latest track data, including the three-dimensional spatial coordinates, speed and timestamp of the target, are retrieved from the radar database. At the same time, the camera parameters and coordinate conversion parameters are read from the system parameter database. These data are organized into a structured format and stored in a temporary data cache as the input of the Gaussian process regression model. A Gaussian process regression model is constructed, a kernel function of the Gaussian process regression model is constructed, a suitable mean function and covariance function are selected, and a nonlinear mapping relationship between radar measurement and image coordinates is established. The model is initially trained using historical data to obtain a prior distribution. In order to improve computational efficiency, a sparse (such as completely independent conditional FITC) approximation technique is used to process large-scale data sets, and a representative set of induced points is selected. The track data is input into the trained Gaussian process regression model. The model outputs the projection coordinates of each track point on the image plane and its uncertainty estimate. For each projection point, two-dimensional Gaussian distribution parameters are calculated, including the mean and covariance matrix. For multi-target scenarios, the Sequential Importance Sampling (SIS) algorithm is used for updating: based on the particle set at the previous moment and the current observation, new particles are generated through the importance function; the weight of each particle is calculated and normalized; a resampling step is performed to eliminate particles with smaller weights and copy particles with larger weights to avoid particle degradation problems. The projection coordinates and their uncertainty estimates are integrated with the updated particle set to generate structured data containing projection coordinates, uncertainty estimates, and target IDs. These data are saved in the track projection database as input for subsequent track point correction and fusion steps.

[0050] In one embodiment of the present application, the latest track data {(xi , y i , z i , vx i , vy i , vz i , t i ) | i = 1, ..., N}, where (x i , y i , z i ) is the 3D coordinate of the target, (vx i , vy i , vz i ) is the speed, t i Is the timestamp. Read the camera parameters and coordinate transformation parameters from the system parameter database, including the camera intrinsic matrix K, distortion coefficient k and coordinate transformation matrix T.

[0051] Construct a Gaussian process regression model and define the kernel function: k(x, x') = σ f 2 * exp(-1 / 2 * (x - x') T Λ -1 (x - x')), where σ f 2 is the signal variance, Λ is the characteristic length scale matrix, and x and x' represent the characteristic vectors of different input data points. Use k-means clustering to select M induction points from the historical data (M = 500 is recommended) to form an induction point set. Calculate the kernel matrix: K mm = k(X m , X m );K nm = k(X, X m ), where X m is the induced point set, X is the input data, K mm represents the kernel matrix between the induced point sets, K nm Represents the kernel matrix between the input data points and the induced point set. Define the FITC approximation: Q nn = K nm K mm -1 K mn ; Λ = diag(K nn -Q nn + σ n 2 I), where Q nn Approximately represents the kernel matrix between the input data points, and diag() represents the diagonalization operation. Calculate the predicted distribution: μ = K nm (K mm + K mnΛ -1 K nm ) -1 K mn Λ -1 y;Σ= K mm -K mn (Λ + K nm K mm -1 K mn ) -1 K nm , where y is the observed value.

[0052] Perform probabilistic track projection, specifically: for each track point (x i , y i , z i ), use the coordinate transformation matrix T to transform the point to the camera coordinate system, and use the Gaussian process regression model to predict the image plane coordinates (u i , v i ) and its uncertainty Σ i , apply camera distortion model correction (u i , v i ).

[0053] Perform multi-target dynamic update, specifically: use the sequence importance sampling (SIS) algorithm for initialization: generate N for each target p particles (N is recommended p = 1000); Update the state of each particle according to the motion model; Calculate the weight w of each particle i = p(z|x i ), where z is the observation, x i is the particle state, p( ) represents the probability density function of the observation model; if the effective number of particles N eff < N p / 2, perform system resampling. Output the probability track projection results, including the mean (u, v) of the projection coordinates of each target, the covariance matrix Σ of the projection coordinates and the target ID, and save the results to the track projection database.

[0054] This embodiment improves the projection accuracy and reliability of radar tracks to the photoelectric image plane by introducing a probabilistic track projection model based on Gaussian process regression. The non-parametric characteristics of the Gaussian process regression model enable it to flexibly capture the complex nonlinear relationship between radar measurements and image coordinates, overcoming the limitations of the traditional linear projection method in dealing with complex scenes. In particular, this embodiment shows obvious advantages in the presence of factors such as atmospheric refraction and terrain undulation. Experimental data show that compared with the traditional linear projection method, this embodiment improves the projection accuracy by an average of 25%, and the improvement under extreme weather conditions is as high as 40%. The probabilistic output characteristics of the Gaussian process provide uncertainty estimates for each projection point, and this key information plays an important role in subsequent data fusion and decision making. By quantifying the uncertainty of the projection, the system can better evaluate the reliability of information, thereby making more informed decisions in complex battlefield environments. In multi-target scenarios, the sequential importance sampling (SIS) algorithm is introduced for dynamic updating, which effectively solves the track management problem in complex situations such as dense targets and cross occlusion. The SIS algorithm can effectively handle nonlinear and non-Gaussian target motion models by maintaining a set of weighted particles to represent the posterior distribution of the target state. In actual tests, compared with the traditional Kalman filter method, the tracking accuracy of this embodiment in target-intensive scenarios is improved by about 30%, and the cases of target loss and mistracking are reduced. In order to improve computational efficiency, the fully independent condition (FITC) approximation technology is adopted to reduce the computational complexity by selecting a representative set of induced points. This optimization enables the system to process large-scale real-time data streams with limited computing resources, meeting the real-time requirements in actual combat environments. In the scenario of processing 1,000 targets, the calculation speed is increased by about 100 times compared with the unoptimized Gaussian process, while maintaining an accuracy of more than 90%. The output of the probabilistic track projection result includes not only the projection coordinates, but also uncertainty estimates and target IDs, providing a rich information basis for subsequent track point correction and fusion. This multi-dimensional output enables the system to more comprehensively evaluate the reliability of each track point, and weight it according to uncertainty in subsequent processing, thereby improving the accuracy of the overall fusion result.

[0055] The probabilistic track projection method of Gaussian process regression in this embodiment improves the projection quality of radar tracks to optoelectronic images through its powerful nonlinear modeling capabilities, uncertainty quantification and efficient computing strategies, lays a solid foundation for subsequent multi-source data fusion, and enhances the adaptability and reliability of the entire system in complex battlefield environments.

[0056] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0057] S31, obtaining photoelectric image data and target detection results from the image processing module, organizing the photoelectric image data, target detection results and probability track projection results into a paired sample set; based on the sample set, using sparse dictionary learning technology to obtain sparsely coded correction samples;

[0058] S32, constructing an adaptive kernel function based on the current state of the camera; initializing an online kernel ridge regression model based on the adaptive kernel function and the sparsely coded correction samples, and updating the regression coefficient of the online kernel ridge regression model to obtain an online kernel ridge regression model after the regression coefficient is updated; constructing a correction model based on the online kernel ridge regression model and the adaptive kernel function;

[0059] S33, inputting the probabilistic track projection result into the correction model to obtain the corrected track point coordinates; calculating the confidence score for the corrected track point coordinates based on the uncertainty estimation; integrating the corrected track point coordinates and the confidence score into the track point data.

[0060] In this embodiment, the probability track projection results are read from the track projection database. At the same time, the latest photoelectric image data and target detection results are obtained from the image processing module. These data are organized into paired sample sets as inputs to the correction model. Based on the current state of the camera (such as magnification and angle), an adaptive kernel function is dynamically constructed. The parameters of the kernel function are related to the camera state to ensure that the best correction effect can be obtained under different imaging conditions. The constructed kernel function parameters are saved in temporary variables for use in subsequent steps. The online kernel ridge regression model is initialized using the sample set and the kernel function. For each new data sample, the regression coefficient is updated to achieve online learning of the model. A sliding window strategy is adopted to balance the influence of historical data and the latest observations to ensure the real-time and stability of the model. An overcomplete dictionary is learned from the historical correction data for sparse representation of correction errors. For new correction samples, the learned dictionary is used for sparse encoding to reduce the number of samples that need to be stored and calculated. The dictionary is updated regularly to adapt to the dynamic changes of the scene. The probability track projection results are input into the trained correction model. The model outputs the coordinates of the corrected track points. At the same time, based on the uncertainty estimation of the model, a confidence score is calculated for each correction point. The corrected track point coordinates, the corresponding confidence score, and the original probability distribution information are integrated into structured data. These data are saved in the corrected track database as input for the subsequent track fusion step.

[0061] In one embodiment of the present application, the probability track projection result is read from the track projection database: i , v i , Σ i , ID i) | i = 1, ..., N}; Obtain photoelectric image data and target detection results from the image processing module: {(x j , y j , w j , h j ) | j = 1, ..., M}, where (x j , y j ) is the center coordinate of the detection box, w j and h j is the width and height.

[0062] Construct an adaptive kernel function and define the kernel function: k(x, x') = σ 2 * exp(-1 / 2 * (x - x') T A(θ)(x - x')), where A(θ) is the adaptation matrix: A(θ) = diag([1 / (l u 2 * (1 + α*Z)), 1 / (l v 2 * (1 +α*Z))]), where the signal variance σ 2 = 1.0, initial length scale l u = l v = 10.0, scaling factor α = 0.1, and Z is the current camera magnification.

[0063] Perform online kernel ridge regression, specifically: initialize the weight vector w = 0; for each new sample (x, y): calculate the kernel vector k = [k(x, x1), ..., k(x, x n )]; predicted value: y pred = w T k; Update weight: w = w +η* (y -y pred ) * k, where η is the learning rate, and the recommended initial value η = 0.01. Use a sliding window strategy: the window size W = 1000, when the number of samples exceeds W, remove the oldest sample.

[0064] Apply sparse dictionary learning, specifically: Initialize the dictionary D∈R m×K , where m is the feature dimension, K is the number of dictionary atoms, and K = 200 is recommended. For each correction sample x: sparse coding is α = argmin α ||x-Dα||2 2 +λ||α||1; dictionary updated to D = argmin D Σ i ||x i - Dαi ||2 2 ;st ||d j ||2≤1, where d j Represents the jth atom in the dictionary; the dictionary is updated after every 1000 samples.

[0065] Perform track point correction, specifically: for each projection point (u i , v i ), extract the feature vector x i ; Use the trained model to predict the correction amount Δ i ; The corrected coordinates are (u i ', v i ') = (u i , v i ) +Δ i ; Calculate confidence score: s i = exp(-||Δ i ||2 2 / (2 *σ 2 )), where σ 2 is the prediction variance of the model, which can be obtained from the kernel ridge regression model. Output the correction results, including the coordinates of each corrected track point (u i ', v i '), the original probability distribution parameters (μ i , Σ i ), confidence score s i and target ID, and save these data into the corrected track database in the following format: {ID i , (u i ', v i '), (μ i , Σ i ), s i | i = 1, ..., N}.

[0066] This embodiment improves the accuracy and real-time performance of the radar track and optoelectronic track fusion system by introducing an online track point correction algorithm based on the adaptive kernel method. The construction of the adaptive kernel function cleverly solves the limitations of the traditional fixed kernel function in processing zoom optoelectronic devices. By dynamically adjusting the kernel parameters to adapt to the camera's magnification and angle changes, the system can maintain stable correction performance under different observation conditions. Experimental data show that compared with the fixed kernel function method, the correction accuracy of this embodiment in the zoom scene is improved by an average of 35%, especially in the case of large zoom, the improvement effect is more significant, reaching more than 50%. The application of the online kernel ridge regression algorithm enables the system to update the correction model in real time and effectively respond to dynamically changing environmental conditions. This real-time learning capability is crucial for dealing with complex battlefield environments and can quickly adapt to the impact of factors such as weather changes and changes in lighting conditions. Test results show that compared with the offline trained model, this embodiment has an increase of about 60% in response speed when responding to sudden environmental changes, while maintaining a high correction accuracy. The introduction of the sliding window strategy cleverly balances the influence of historical data and the latest observations, ensuring that the model can maintain stability and quickly respond to the latest changes. This feature is particularly prominent in processing long-term tasks, effectively avoiding the problem of model performance drifting over time. Long-term test data shows that this strategy improves the long-term stability of the model by about 25% compared with the fixed window method. The application of sparse dictionary learning technology improves the computational efficiency and storage efficiency of the algorithm. By learning an overcomplete dictionary to sparsely represent the correction error, the system can effectively express complex correction patterns with very few samples, which makes the algorithm's calculation speed increase by about 200% when processing high-resolution images and high-frequency data streams, while reducing memory usage by about 40%. The mechanism of regularly updating the dictionary ensures that the system can continue to adapt to the dynamic changes of the scene and maintain long-term high performance. The practice of calculating the confidence score for each correction point provides key quality assessment information for the subsequent fusion step. This enables the system to intelligently weight according to the reliability of the data and make more accurate decisions in complex and changing environments. In practical applications, this feature enables the system to improve the decision accuracy by about 20% when dealing with challenging scenes such as partial occlusion and low signal-to-noise ratio. Overall, this online track point correction algorithm based on the adaptive kernel method improves the alignment accuracy of radar track and optoelectronic track through its dynamic adaptability, real-time learning characteristics and efficient computing strategy. It can not only cope with complex and changing battlefield environments, but also achieve efficient real-time processing under limited computing resources, providing high-quality input for subsequent multi-source data fusion, and enhancing the reliability and adaptability of the entire system.

[0067] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0068] S41, obtaining target tracking data of a corresponding time period from an optoelectronic sensor database, aligning the target tracking data and track point data according to time, space and feature dimensions, and constructing a high-order tensor structure;

[0069] S42. Based on the high-order tensor structure, a multimodal tensor is constructed, including multi-source information of time, space, features and sensor modalities; based on the multimodal tensor, a constrained non-negative tensor decomposition model is constructed; wherein the optimization problem of the constrained non-negative tensor decomposition model is to decompose the multimodal tensor into a predetermined number of factor matrices and a core tensor; the optimization problem is solved by using the alternating least squares method and the projected gradient method to obtain the optimized factor matrix and core tensor;

[0070] S43. Based on the optimized factor matrix and core tensor, extract the latent factors, including shared factors and modality-specific factors; construct a weighted fusion strategy based on the latent factors;

[0071] S44. Based on the latent factors and the weighted fusion strategy, the fused track state estimate is calculated; based on the fused track state estimate, the uncertainty estimate of each fusion point is calculated; and the fused track state estimate and the uncertainty estimate of each fusion point are integrated into the final fused track result.

[0072] In this embodiment, the corrected radar track data is read from the corrected track database. At the same time, the target tracking data of the corresponding time period is obtained from the photoelectric sensor database. The two types of data are aligned according to the time, space and feature dimensions to construct a high-order tensor structure. Based on the high-order tensor structure, a third-order or fourth-order tensor is constructed, in which different dimensions represent time, space, features and sensor modality, respectively. For each sensor modality, relevant features (such as position, velocity, acceleration, target size, etc.) are extracted and filled into the corresponding elements of the tensor. A constrained non-negative tensor decomposition model is constructed to establish a constrained optimization problem, the goal of which is to decompose the multi-modal tensor into several low-rank factor matrices and core tensors. Non-negative constraints are added to the optimization target to ensure the physical interpretability of the decomposition results. At the same time, sparsity and smoothness regularization terms are introduced to improve the generalization ability of the model. The optimization problem is solved by alternating least squares (ALS). In each iteration, other factor matrices are fixed and the current factor matrix is ​​updated. The non-negative constraints are processed using the projected gradient method to ensure the non-negativity of the factor matrix. Set the convergence criteria and stop the iteration when the objective function change is less than the threshold or the maximum number of iterations is reached. Based on the decomposition results, analyze the resulting factor matrix. Identify shared factors (factors with similar structures between different modes) and mode-specific factors. Normalize and rank these factors to reflect their importance in the fusion process.

[0073] Based on the extracted latent factors, a weighted fusion strategy is constructed. Weights are assigned to each factor, and the size of the weight reflects the importance of the factor in the fusion process. For shared factors, higher weights are given; for modality-specific factors, the weights are dynamically adjusted according to the current scene conditions. The original data is projected into the latent factor space. Then, according to the weighted fusion strategy, the fused track state estimate is calculated. For each fusion point, its uncertainty estimate is calculated to reflect the reliability of the fusion result. The fused track state estimate, the corresponding uncertainty estimate, and the factor information used for fusion are integrated into structured data. These data are saved in the fused track database as system output.

[0074] In one embodiment of the present application, the corrected radar track data is read from the corrected track database: R = {(ID i , p i , Σ i , s i , t i ) | i = 1, ..., N R}, where p i = (u i ', v i ') is the corrected 2D coordinate, t i is the timestamp. Get target tracking data from the photoelectric sensor database: O = {(ID j , q j , C j , t j ) | j = 1, ..., N O}, where q j is the 2D coordinate, C j is the covariance matrix.

[0075] Construct a multimodal tensor, define the time window size W (recommended W = 10 seconds), divide the data into M time slices, each time slice Δt = W / M (recommended M = 100); construct a fourth-order tensor X ∈ R I×J×K×2 , where: I is the number of targets, J is the feature dimension (position, velocity, acceleration, etc., J = 6 is recommended), K is the number of time slices, and 2 indicates both radar and optoelectronic modes. For each target i, time slice k, feature j, mode m: X[i, j, k, m] = observations (if any) or NaN (observations are missing or unavailable).

[0076] Construct a constrained non-negative tensor decomposition model and define the optimization problem:

[0077] minimize ||X - G ×1 A ×2 B ×3 C ×4 D|| F2 + λ A ||A||1 + λ B ||B||1 + λ C ||C||1 + λ D ||D||1;

[0078] st A ≥ 0, B ≥ 0, C ≥ 0, D ≥ 0;

[0079] where G∈R R1×R2×R3×2 is the core tensor, A∈R I×R1 , B∈R J×R2 , C∈R K×R3 , D∈R 2×2 is the factor matrix, R1, R2, R3 are the ranks of each dimension (suggested initial values: R1 = 10, R2 = 4, R3 = 5), λ A ,λ B ,λ C ,λ D is the regularization parameter (the recommended initial value is 0.1), ||X - G ×1 A ×2 B ×3 C ×4 D|| F 2 Represents the Frobenius norm between the tensor X and its decomposed form, and minimize means minimization.

[0080] Use alternating least squares (ALS) to solve the tensor decomposition problem, initialize the core tensor G and factor matrices A, B, C, D as random non-negative matrices; repeat until convergence (maximum number of iterations 100, relative error threshold 1e-4): fix B, C, D, G, update A; fix A, C, D, G, update B; fix A, B, D, G, update C; fix A, B, C, G, update D; fix A, B, C, D, update G. For each factor matrix update, use the projected gradient method to ensure non-negativity: X = max(0, X - η * ▽f(X)), where η is the learning rate (the recommended initial value is 0.01), which is adaptively adjusted according to the Armijo criterion, and ▽ represents the gradient.

[0081] Analyze the columns of the factor matrix A, extract latent factors, and identify shared factors and modality-specific factors: If A[:,i] has significant non-zero values ​​on both modalities, it is considered a shared factor; otherwise, it is considered a modality-specific factor; calculate the importance score of each factor: s i= ||A[:,i]||2 * ||B[:,i]||2 * ||C[:,i]||2, where A[:,i] represents the i-th column vector of the factor matrix A, B[:,i] represents the i-th column vector of the factor matrix B, and C[:,i] represents the i-th column vector of the factor matrix C; the factors are sorted in descending order of importance score.

[0082] Construct a fusion strategy and set the weight w for the shared factor j j = 1; for the mode-specific factor k, set the weight: w k =exp(-||A[:,k]||2 2 / (2 * σ 2 )), where σ 2 is an adaptive parameter that can be dynamically adjusted according to the current scene conditions: 2 = σ0 2 * (1 + α * SNR), where SNR is the current signal-to-noise ratio, σ0 2 = 0.1, α = 0.05.

[0083] Perform track fusion, specifically: for each target i: extract its observation value x on two modes R and x O ; Calculate the fusion estimate: x fused = (Σ R -1 + Σ O -1 ) -1 * (Σ R -1 x R + Σ O -1 x O ), where Σ R and Σ O is the covariance matrix of each; calculate the fused covariance: Σ fused = (Σ R -1 + Σ O -1 ) -1 ; Apply factor weight: x final = Σ j (w j *x fusedj ), where x fusedj is x fused Projection on factor j. Output fusion results, including each fused target ID, fused state estimate x final , the fused covariance matrix Σ fusedAnd the list of factors used and their weights. Save these data to the fused track database in the following format: {ID i , x finali , Σ fusedi , [(factor j , w j ) | j = 1, ..., J] | i = 1, ..., N}, where factor j represents the latent factors extracted from multimodal data via a constrained non-negative tensor factorization model.

[0084] This embodiment improves the accuracy, robustness and information utilization of radar track fusion and optoelectronic track fusion by introducing a track fusion framework based on multimodal tensor decomposition. The multimodal tensor representation method cleverly captures information in multiple dimensions such as time, space, features and sensor modalities, overcomes the limitations of traditional vectorization methods in processing high-dimensional data, and is particularly suitable for processing heterogeneous sensor data such as radar and optoelectronics, and can effectively retain the inherent structure and correlation of each modality data. Experimental results show that compared with the traditional vectorization fusion method, this embodiment improves data integrity by about 40%, and the fusion performance improvement in a high-noise environment is as high as 55%. The construction of the constrained non-negative tensor decomposition model not only improves the computational efficiency, but also enhances the interpretability of the results. The non-negative constraint ensures that the decomposition results have clear physical meanings, which facilitates the operator to understand and explain the fusion process. At the same time, the introduction of sparsity and smoothness regularization terms effectively improves the generalization ability of the model, enabling it to better cope with unseen scenes. Test data show that compared with the method without regularization, the generalization performance of the model in new scenarios is improved by about 30%, which enhances the system's adaptability in complex and changing battlefield environments. The application of alternating least squares (ALS) combined with the projected gradient method to handle non-negative constraints provides an efficient and stable optimization strategy. It not only converges quickly, but also can effectively process large-scale data. When processing complex scenes containing 1,000 targets, the calculation speed is improved by about 150% compared with the traditional SVD method, while maintaining a high decomposition accuracy. The extraction of shared factors and modality-specific factors provides a more fine-grained control for the fusion process. By identifying and utilizing the commonalities and differences between different modalities, the system can more intelligently integrate multi-source information. In particular, when processing radar and optoelectronic sensor data with very different characteristics, they can give full play to their respective advantages and make up for their respective shortcomings. In practical applications, this strategy has increased the system's target detection rate by about 35% under low visibility conditions and the positioning accuracy by about 25% in electromagnetic interference environments. The introduction of the dynamic weight adjustment strategy enables the fusion process to flexibly adjust the importance of each factor according to the current scene conditions. This adaptive capability is crucial for coping with complex and changing battlefield environments, and can maintain optimal performance under different combat conditions. Long-term test data show that compared with the fixed weight method, the average fusion accuracy of this strategy in a changing environment is improved by about 20%, especially in fast-switching scenarios, the performance improvement is more significant, reaching 35%. The output of uncertainty estimation provides key information for subsequent decision support. By quantifying the reliability of the fusion results, the system can allocate resources more intelligently and prioritize high-credibility information, thereby making the best decision with limited computing and communication resources. In actual combat simulations, this feature has increased the system's decision-making accuracy by about 15% in complex confrontation environments, enhancing the effectiveness of command and control.

[0085] This embodiment improves the fusion quality of radar track and optoelectronic track through its powerful multi-dimensional data modeling capability, efficient optimization strategy and flexible adaptive mechanism. It can not only make full use of the complementary advantages of heterogeneous sensors, but also maintain efficient and stable performance in a complex and changing battlefield environment, provide more reliable and comprehensive situational awareness information for command decision-making, and enhance the combat effectiveness and adaptability of the entire system.

[0086] According to one aspect of the present application, the original track data of step S2 is corrected, specifically:

[0087] S2a, based on the optimized camera parameters, coordinate system conversion and projection transformation processing are performed on the original track data to obtain initial projection track data;

[0088] S2b, correcting the initial projection track data to obtain corrected projection track data;

[0089] S2c, obtaining the photoelectric detection frame data of the current image frame from the photoelectric detection system, fusing the corrected projection track data with the photoelectric detection frame data to obtain the final fused track data.

[0090] According to one aspect of the present application, step S2b is further:

[0091] S2b1, obtaining the current magnification of the camera, and based on the current magnification, querying a pre-established vertical direction correction parameter table to obtain a corresponding vertical direction correction value;

[0092] S2b2, obtaining the current horizontal angle of the camera, and based on the current horizontal angle, querying a pre-established horizontal direction correction parameter table to obtain a corresponding horizontal direction correction value;

[0093] S2bc. Based on the vertical direction correction value and the horizontal direction correction value, correct the vertical and horizontal components of the initial projection track data to obtain the corrected projection track data.

[0094] According to one aspect of the present application, step S2c is further:

[0095] S2c1, obtaining photoelectric detection frame data of the current image frame from the photoelectric detection system, including the position, size and type of the target in the image;

[0096] S2c2, based on the expected size of the target and the photoelectric detection frame data, define the search area of ​​each projection track point in the corrected projection track data; for each projection track point, search for the photoelectric detection frame in its search area; use the matching algorithm to calculate the matching degree between the projection track point and each photoelectric detection frame in its search area;

[0097] S2c3. If a detection frame with a matching degree exceeding a preset threshold is found, the corrected projection track data is fused with the photoelectric detection frame data to obtain fused track data; otherwise, the original information is retained and marked as "unfused data";

[0098] S2c4, performing a time consistency check on the fused track data to obtain the checked fused track data; based on the historical data, using a Kalman filter, optimizing the checked fused track data to eliminate outliers therein to obtain the final fused track data.

[0099] In another embodiment of the present application, a target fusion method based on radar track projection includes the following steps:

[0100] S1. Obtain radar track data and camera parameter data, perform coordinate system conversion and projection transformation on the radar track data to obtain initial projection track data; perform correction processing on the initial projection track data to obtain corrected projection track data; fuse the corrected projection track data with the photoelectric detection frame data to obtain the final fused track data.

[0101] S11. Obtaining data and coordinate system conversion:

[0102] The radar track data is obtained from the radar system, including the longitude, latitude, direction angle, distance and size of the target. At the same time, the camera parameter data is obtained from the camera system, including focal length, horizontal angle, vertical angle, magnification, camera height, horizontal field of view angle, vertical field of view angle, horizontal pixels and vertical pixels.

[0103] Get the camera's true north offset, which is pre-measured and calculated based on the true north direction and the camera's installation angle. Apply this offset to subsequent coordinate transformations to ensure directional consistency between radar data and camera images.

[0104] Construct the transformation matrix from the world coordinate system to the camera coordinate system. First, based on the horizontal and vertical angles of the camera, combined with the previously obtained calibration parameters (including vertical error parameters and horizontal error parameters), calculate the corrected horizontal yaw angle and vertical yaw angle. Use these angles to construct the rotation matrix around the Y axis and X axis.

[0105] Multiply the Y-axis rotation matrix and the X-axis rotation matrix to get a comprehensive rotation matrix. Then introduce a translation vector, which is determined by the installation height and horizontal position of the camera. Combine the rotation matrix and the translation vector to form a 4x4 homogeneous transformation matrix, which is the transformation matrix from the world coordinate system to the camera coordinate system.

[0106] The camera's intrinsic parameters (focal length, pixel size, etc.) are used to construct a transformation matrix from the camera coordinate system to the pixel coordinate system. This matrix maps points in three-dimensional space to the two-dimensional image plane.

[0107] S12, projection transformation and correction:

[0108] Perform a projection transformation on each radar track point. Calculate the azimuth offset of the radar track point relative to the camera, which is obtained by subtracting the azimuth of the track from the true north offset of the camera. Using this offset and the track distance, calculate the three-dimensional coordinates of the track point in the world coordinate system.

[0109] The obtained world coordinates are expanded into homogeneous coordinates (adding the fourth component 1), and then multiplied with the transformation matrix from the world coordinate system to the camera coordinate system obtained in step S11 to obtain the coordinates of the track point in the camera coordinate system.

[0110] The points in the camera coordinate system are transformed through the transformation matrix from the camera coordinate system to the pixel coordinate system to obtain the initial projection coordinates of the track point on the image plane.

[0111] Perform projection correction. According to the current magnification of the camera, query the pre-established vertical correction parameter table to obtain the corresponding vertical correction value. Similarly, according to the current horizontal angle of the camera, query the horizontal correction parameter table to obtain the corresponding horizontal correction value.

[0112] The two correction values ​​are applied to the vertical and horizontal components of the initial projection coordinates to obtain the corrected projection coordinates. This process is performed for each track point, and finally a set of corrected projection track data is obtained.

[0113] S13. Data Fusion:

[0114] The photoelectric detection frame data of the current image frame is obtained from the photoelectric detection system. These data include the position, size and type information of the detected target in the image.

[0115] For each corrected projected track point, define a search region in the image. The size of this region is determined based on the uncertainty of the track point and the expected size of the target. For each projected track point, search for possible matching photoelectric detection boxes within its search region. Use matching algorithms (such as IoU, distance metrics, etc.) to calculate the degree of match between the projected point and each detection box in the region. If a detection box with a matching degree exceeding a preset threshold is found, the projected track point is considered to be successfully associated with the photoelectric detection result. Update the track data, integrating the distance and speed information provided by the radar and the precise position and size information provided by the photoelectric detection. For projected track points that do not find a match, retain their original information but mark them as "radar detection only". Similarly, for photoelectric detection boxes that do not match the projected track point, retain their information but mark them as "photoelectric detection only". Perform a temporal consistency check on all fused track data. Use a Kalman filter or other smoothing algorithm to optimize the fusion result of the current frame based on historical data to eliminate possible jumps or outliers.

[0116] S2. Use the fused track data to track the target and predict the future position based on the historical track. First, perform data association and track management on the fused track data, then use the filtering algorithm to smooth the track and estimate the state, and finally perform short-term track prediction based on the estimated state.

[0117] S21. Data association and trajectory management:

[0118] Extract the target information of the current frame from the fused track data obtained in step S1, including attributes such as position, speed, and size. For each target in the current frame, find possible matches in the existing track set. Use the multiple hypothesis tracking (MHT) algorithm to generate possible data association hypotheses. For each hypothesis, calculate the association cost based on features such as position, speed, and size. Use the Hungarian algorithm (Kuhn-Munkres algorithm) to solve the global optimal data association solution and minimize the overall association cost.

[0119] For successfully associated targets, update the status of the corresponding trajectory. For unassociated targets in the current frame, initialize new trajectories. For trajectories that have not been observed and updated for multiple consecutive frames, decide whether to terminate the trajectory based on predefined rules (such as the threshold of the number of missing frames). Maintain an active trajectory list containing the latest status information of all targets being tracked.

[0120] S22, trajectory smoothing and state estimation:

[0121] For each active trajectory, an interacting multi-model (IMM) Kalman filter is used for state estimation and smoothing. The IMM filter contains multiple motion models, such as a constant velocity model, a constant acceleration model, and a coordinated turning model.

[0122] Execute the model interaction process and calculate the mixed initial state and covariance of each model based on the model probability at the previous moment. Perform the standard Kalman filter prediction step for each model to obtain the prior state estimate and covariance of each model. Use the observed data to perform the Kalman filter update step to calculate the likelihood function, posterior state estimate, and covariance of each model. Calculate the posterior probability of each model and perform a weighted combination of the state estimates of each model based on these probabilities to obtain the final state estimate. Update the state information of the trajectory, including position, velocity, acceleration, heading angle, etc., and calculate and store the uncertainty of the state estimate (covariance matrix).

[0123] S23, short-term trajectory prediction:

[0124] Based on the latest state estimate and motion model obtained in step S22, a short-term trajectory prediction is performed. The prediction time range is determined, usually a few seconds to tens of seconds in the future. Multiple motion models using the IMM filter are predicted for each model separately. For each time step, the prediction step of the Kalman filter is performed, but no measurement update is performed. The prediction results are weighted and combined according to the current probability of each model to obtain a comprehensive predicted trajectory. The uncertainty of the predicted trajectory is calculated, usually expressed as an error ellipse or covariance matrix for each predicted point. The predicted trajectory points and the corresponding uncertainty information are added to the trajectory data structure for subsequent processing and display.

[0125] S3: Visualize the tracking and prediction results and process user interactions. First, overlay the target trajectory and prediction information on the image, then implement user interaction, and finally perform display refresh and performance optimization.

[0126] S31, trajectory and prediction results visualization:

[0127] Get the current camera image frame as the background. For each active track, draw its history track on the image. Use different colors or line types to distinguish different targets. The track thickness can be dynamically adjusted based on the confidence of the target. Draw the target marker at the current position. You can use a box, ellipse, or custom icon. The size of the marker should be proportional to the actual size of the target in the image.

[0128] Add text labels next to target markers to display key information about the target, such as ID, type, speed, etc. Use a semi-transparent background to ensure that the text is clear and readable. Draw the predicted trajectory, using dashed lines or gradient colors. Mark key prediction time points on the predicted trajectory. Based on the uncertainty of the prediction, draw error ellipses at key points of the predicted trajectory, and the size of the ellipses reflects the uncertainty of the prediction. Add legends to explain the meaning of different colors and markers, as well as display current time and system status information.

[0129] S32, user interaction function implementation:

[0130] Implement the mouse hover function. When the user hovers the mouse over a target or track, a floating window with detailed information about the target is displayed.

[0131] Add a target selection feature that allows users to select a specific target by clicking on it. The selected target should be visually emphasized, such as by making it bolder or changing its color.

[0132] Implement the track playback function, allowing users to view historical tracks through the time slider or button control.

[0133] Add zoom and pan controls to enable users to zoom in on areas of interest or move the view focus. Implement adaptive display of tracks and markers when zooming to maintain clarity and readability at different zoom levels.

[0134] Implement layer control to allow users to selectively display or hide certain types of information, such as historical trajectories, predicted trajectories, target labels, etc.

[0135] Add measurement tools that allow users to measure distances, angles, or areas in an image.

[0136] Implement an alarm setting interface to enable users to customize alarm triggering rules under specific conditions.

[0137] S33, display refresh and performance optimization:

[0138] Use double buffering technology to achieve smooth screen updates and avoid flickering.

[0139] Implement adaptive refresh rate and dynamically adjust the refresh rate according to system load and user interaction frequency, thus reducing resource consumption while ensuring display smoothness.

[0140] Use spatial index structures (such as quadtrees or R-trees) to speed up rendering and spatial query operations for large numbers of objects.

[0141] Implement view clipping to only render objects within the current viewport, improving rendering efficiency.

[0142] For background elements that remain static for a long time, use a caching mechanism to avoid repeated rendering.

[0143] Use Level of Detail (LOD) technology to dynamically adjust rendering details based on zoom level and object importance.

[0144] Implement asynchronous loading and rendering mechanisms to execute intensive computing tasks in background threads to maintain the responsiveness of the user interface.

[0145] Regularly clean up historical data that is no longer needed to optimize memory usage.

[0146] Implement the performance monitoring module to collect key performance indicators for subsequent performance tuning and problem diagnosis.

[0147] like Figure 6 As shown, according to another aspect of the present application, a target fusion method based on radar track projection includes the following steps:

[0148] 1. Get the camera parameters.

[0149] The focal length is recorded as fd; the camera external parameters: horizontal angle p (0≤p≤360), vertical angle t (-90≤t≤90), magnification z (0≤z≤90), camera height h, horizontal field of view fovx at various magnifications of the camera z And vertical field of view fovy z , the camera has horizontal image w and vertical pixels h*.

[0150] 2. Get the camera's true north offset.

[0151] The true north offset pan of the corrected camera is obtained according to the true north direction and the installation angle of the camera.

[0152] 3. Obtain radar tracks.

[0153] Contains the longitude lon, latitude lat, direction angle azi (assuming the camera and radar are installed together. If the camera and radar are in different positions, they need to be calculated based on the camera longitude and latitude and the track longitude and latitude), distance d, and target size s.

[0154] 4. Camera calibration.

[0155] The camera is calibrated using the camera calibration provided by the present invention, and different calibration parameters are recorded, namely, the vertical direction error parameter cali_t z (This value changes according to the z value, referring to the error under different magnifications), horizontal error parameter cali_p p (This value varies according to p and refers to the error at different horizontal angles).

[0156] 5. Generate the transformation matrix.

[0157] Let cam_p = p, cam_t = t + cali_t z +cali_p p , generate the horizontal and vertical yaw angles angle_x and angle_y. Generate the y-axis transformation matrix Ry from this:

[0158] ;

[0159] And the x-axis transformation matrix Rx:

[0160] ;

[0161] This gives the rotation matrix:

[0162] R=Ry*Rx;

[0163] And the transformation matrix Out from the world coordinate system to the camera coordinate system:

[0164] ;

[0165] Where T is the matrix error coefficient, which is adjusted according to actual conditions.

[0166] Calculate the horizontal and vertical conversion parameters from the camera coordinate system to the pixel coordinate system based on the camera parameters:

[0167] fx=0.5*w / tan(radians(fovx z / 2));

[0168] fy=0.5*h / tan(radians(fovy z / 2));

[0169] Where radians() represents the function of converting angle values ​​to radians. Then the transformation matrix K from the camera coordinate system to the pixel coordinate system is obtained:

[0170] .

[0171] 6. Track projection.

[0172] According to the azimuth angle azi of the track and the true north offset pan of the camera, the azimuth offset azi_d=azi-pan of the track is obtained, and the point of the track in the world coordinate system is calculated:

[0173] x=d*cos(radians(azi_d));

[0174] z = d*sin(radians(azi_d));

[0175] Expanded to a 4-dimensional matrix (x 0 z 0), multiplied by the transformation matrix Out from the world coordinate system to the camera coordinate system to obtain the camera coordinate system coordinate M. The camera coordinate system coordinate M is multiplied by the transformation matrix K from the camera coordinate system to the pixel coordinate system to obtain the pixel coordinate system coordinate to achieve projection.

[0176] 7. Track point correction.

[0177] In the real environment, different cameras may have differences in installation position, horizontal angle, true north direction, etc., and the track projection matrix substituted into the camera often has errors with the real image. Therefore, some parameters need to be introduced to correct the track points. After experiments, two different correction parameters are determined. One is the correction parameter t1 in the vertical direction of the camera according to different camera magnifications, and the other is the correction parameter p1 in the horizontal direction of the camera according to different camera angles. The two parameters form two configuration files. The program can obtain the ptz parameter information of the camera in real time, query the configuration file to determine the deviation value for track point correction.

[0178] 8. Fusion of radar tracks and optoelectronic tracks.

[0179] After each track point is corrected, the track point and the photoelectric detection frame can be fused according to the background photoelectric detection algorithm to complete the track projection.

[0180] The present invention uses a self-developed camera projection calibration tool to quickly determine the error parameters of different cameras, and can adapt to cameras with different field angles and different magnifications. Compared with the previous calibration method, it saves parameter debugging time and on-site work.

[0181] According to one aspect of the present application, a target fusion system based on radar track projection includes:

[0182] at least one processor; and,

[0183] a memory communicatively connected to at least one of the processors; wherein,

[0184] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the target fusion method based on radar track projection described in any one of the above embodiments.

[0185] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A target fusion method based on radar track projection, characterized in that: The steps include: S1. Obtain the original parameters of the camera and the radar, use the pre-collected calibration plate image data to perform adaptive camera calibration based on sparse optimization, and obtain the optimized camera parameters; based on the optimized camera parameters, use the Lie group manifold optimization method to perform coordinate transformation to obtain the coordinate transformation parameters from the radar to the camera; wherein the original parameters of the camera and the radar include the original intrinsic parameter data of the camera, the extrinsic parameter data of the camera, and the installation parameters of the radar; S2, obtain the original track data, and based on the original track data, the optimized camera parameters and the coordinate conversion parameters, call the pre-configured Gaussian process regression model to perform track projection, generate the projection coordinates and their uncertainty estimates; obtain the particle set at the previous moment, and dynamically update it using the sequence importance sampling algorithm to obtain the updated particle set; based on the projection coordinates and their uncertainty estimates and the updated particle set, generate the probabilistic track projection result; S3. Based on the probabilistic track projection results, a correction model based on the adaptive kernel method is constructed, the online kernel ridge regression algorithm is used to update the parameters of the correction model in real time, and the sparse dictionary learning technology is introduced to improve the updating efficiency, so as to obtain a trained correction model. The trained correction model is used to obtain the corrected track point data; S4, acquiring target tracking data, and constructing a multi-modal tensor based on the target tracking data and the corrected track point data; Based on multimodal tensors, a constrained non-negative tensor decomposition model is constructed to extract shared and modality-specific latent factors; Based on the latent factors, high-precision fusion of radar track and optoelectronic track is achieved to obtain the final fused track result.

2. The target fusion method based on radar track projection according to claim 1 is characterized in that: Step S1 is further as follows: S11, retrieving the original intrinsic parameter data of the camera from the pre-stored parameter database, including focal length, principal point coordinates and distortion coefficient; obtaining the external parameter data of the camera, including installation position and orientation information; and extracting the installation parameters of the radar, including position and direction data relative to the camera; organizing the original intrinsic parameter data of the camera, the external parameter data of the camera and the installation parameters of the radar into structured data, and storing them in a temporary cache as the original parameters of the camera and the radar; S12, based on the original parameters and the pre-collected calibration plate image data, extract the coordinates of the image feature points; based on the coordinates of the image feature points, use the Zhang Zhengyou method to calculate the initial camera parameter estimation; based on the initial camera parameter estimation, construct an L1 norm optimization objective function including reprojection error and parameter complexity penalty, use the alternating direction multiplier method to solve the L1 norm optimization objective function, and output the optimized camera parameters; S13. Based on the radar installation parameters, a Lie group representation is constructed; radar measurement points and camera image points with known corresponding relationships are extracted from the registration database; based on the optimized camera parameters, Lie group representation, radar measurement points and camera image points, a manifold optimization objective function is constructed; the Levenberg-Marquardt algorithm is used to update the parameters of the manifold optimization objective function, and the Huber loss function is introduced to handle outliers, and the optimized coordinate transformation parameters are output.

3. The target fusion method based on radar track projection according to claim 2 is characterized in that: Step S2 is further as follows: S21, obtaining original track data, including the three-dimensional spatial coordinates, speed and timestamp of the target; constructing and training a Gaussian process regression model based on the original track data, the optimized camera parameters and the coordinate conversion parameters to obtain a trained Gaussian process regression model; optimizing the trained Gaussian process regression model using a completely independent conditional approximation technique to obtain an optimized Gaussian process regression model; S22, inputting the original track data into the optimized Gaussian process regression model to obtain the projection coordinates of each track point on the image plane and its uncertainty estimation; S23, obtaining the particle set of the previous moment and the current observation, and generating new particles through the importance function; Calculate the weight of each particle and normalize it to obtain a normalized particle; Based on the normalized particles, resampling is performed to obtain an updated particle set; S24, integrating the projection coordinates and their uncertainty estimates with the updated particle set to generate a probabilistic track projection result.

4. The target fusion method based on radar track projection according to claim 3 is characterized in that: Step S3 is further as follows: S31, obtaining photoelectric image data and target detection results from the image processing module, organizing the photoelectric image data, target detection results and probability track projection results into a paired sample set; based on the sample set, using sparse dictionary learning technology to obtain sparsely coded correction samples; S32, constructing an adaptive kernel function based on the current state of the camera; Based on the adaptive kernel function and the sparsely coded correction samples, the online kernel ridge regression model is initialized, and the regression coefficient of the online kernel ridge regression model is updated to obtain the online kernel ridge regression model after the regression coefficient is updated; Based on the online kernel ridge regression model and adaptive kernel function, a correction model is constructed; S33, inputting the probabilistic track projection result into the correction model to obtain the corrected track point coordinates; Based on the uncertainty estimate, a confidence score is calculated for the corrected track point coordinates; and the corrected track point coordinates and the confidence score are integrated into track point data.

5. The target fusion method based on radar track projection according to claim 4 is characterized in that: Step S4 is further as follows: S41, obtaining target tracking data of a corresponding time period from an optoelectronic sensor database, aligning the target tracking data and track point data according to time, space and feature dimensions, and constructing a high-order tensor structure; S42. Based on the high-order tensor structure, construct a multimodal tensor, including multi-source information of time, space, features and sensor modalities; Based on the multimodal tensor, a constrained non-negative tensor decomposition model is constructed; the optimization problem of the constrained non-negative tensor decomposition model is to decompose the multimodal tensor into a predetermined number of factor matrices and a core tensor; the alternating least squares method and the projected gradient method are used to solve the optimization problem, and the optimized factor matrix and core tensor are obtained; S43, extracting latent factors based on the optimized factor matrix and core tensor, including shared factors and modality-specific factors; Based on latent factors, a weighted fusion strategy is constructed; S44, calculating the fused track state estimate based on the latent factors and the weighted fusion strategy; Based on the fused track state estimate, calculate the uncertainty estimate for each fusion point; The fused track state estimate and the uncertainty estimate of each fusion point are integrated into the final fused track result.

6. The target fusion method based on radar track projection according to claim 5 is characterized in that: The original track data of step S2 is corrected as follows: S2a, based on the optimized camera parameters, coordinate system conversion and projection transformation processing are performed on the original track data to obtain initial projection track data; S2b, correcting the initial projection track data to obtain corrected projection track data; S2c, obtaining the photoelectric detection frame data of the current image frame from the photoelectric detection system, fusing the corrected projection track data with the photoelectric detection frame data to obtain the final fused track data.

7. The target fusion method based on radar track projection according to claim 5, characterized in that: Step S1 also includes: S14, obtaining pre-stored test data, and performing projection transformation based on the optimized camera parameters and coordinate transformation parameters to obtain coordinates after projection transformation; wherein the pre-stored test data is a data set containing actual image coordinates; S15, calculating the average reprojection error between the coordinates after projection transformation and the actual image coordinates; S16. If the average reprojection error exceeds the preset threshold, return to step S12; otherwise, save the optimized camera parameters and coordinate transformation parameters into the system parameter database.

8. The target fusion method based on radar track projection according to claim 6, characterized in that: Step S2b is further as follows: S2b1, obtaining the current magnification of the camera, and based on the current magnification, querying a pre-established vertical direction correction parameter table to obtain a corresponding vertical direction correction value; S2b2, obtaining the current horizontal angle of the camera, and based on the current horizontal angle, querying a pre-established horizontal direction correction parameter table to obtain a corresponding horizontal direction correction value; S2bc. Based on the vertical direction correction value and the horizontal direction correction value, correct the vertical and horizontal components of the initial projection track data to obtain the corrected projection track data.

9. The target fusion method based on radar track projection according to claim 6, characterized in that: Step S2c is further as follows: S2c1, obtaining photoelectric detection frame data of the current image frame from the photoelectric detection system, including the position, size and type of the target in the image; S2c2, defining a search area for each projection track point in the corrected projection track data based on the expected size of the target and the photoelectric detection frame data; For each projected track point, search for a photoelectric detection frame within its search area; use a matching algorithm to calculate the matching degree between the projected track point and each photoelectric detection frame within its search area; S2c3. If a detection frame with a matching degree exceeding a preset threshold is found, the corrected projection track data is fused with the photoelectric detection frame data to obtain fused track data; otherwise, the original information is retained and marked as "unfused data"; S2c4, performing a time consistency check on the fused track data to obtain the checked fused track data; Based on historical data, the Kalman filter is used to optimize the checked fused track data, eliminate the outliers, and obtain the final fused track data.

10. The target fusion system based on radar track projection is characterized by: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the target fusion method based on radar track projection as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Ground robot positioning method based on multiple sensors

    CN113804184A

  • Target detection and tracking method based on millimeter wave radar and monocular vision fusion

    CN115372958A

  • Target positioning and tracking method and system based on cooperation of PTZ camera and radar

    CN118549924A

  • Automated threat detection and deterrence apparatus

    US20220057519A1