Target detection method based on radar and camera data fusion

Abstract

The present invention relates to target detection, and in particular to a target detection method based on radar and camera data fusion, wherein the radar and the camera collect data separately; an adaptive multi-angle joint calibration strategy based on multi-feature constraints is adopted to achieve spatial sparse alignment of heterogeneous data collected by the radar and the camera, so that the heterogeneous data are spatially consistent and comparable; a YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures is utilized to perform target detection on the calibrated radar data and camera data respectively; a cross-modal feature fusion algorithm is adopted to fuse target information detected by the radar with the camera image plane to obtain fused features; a method combining deep learning and reinforcement learning is adopted, and a reward function is used to optimize the parameters of the reinforcement learning model to perform target classification; the technical solution provided by the present invention can effectively overcome the defect of the prior art that it is difficult to accurately detect dynamic targets in complex scenes.

Description

Technical Field

[0001] The present invention relates to target detection, and in particular to a target detection method based on radar and camera data fusion. Background Art

[0002] As the demand for precise target detection continues to grow across various industries, especially in critical scenarios like security monitoring, intelligent transportation, and industrial automation, accurate and efficient target detection technology has become a core element in ensuring stable system operation, improving safety, and enhancing production efficiency. Accurate target detection can quickly locate and identify different targets in various scenarios, providing a reliable data foundation for subsequent decision-making and application, and is of great significance in promoting the intelligent development of related industries.

[0003] Traditional target detection methods often rely on a single sensor for data collection and analysis. While relying solely on cameras can capture rich visual image information, it is susceptible to lighting conditions. At night, in direct sunlight, or in dimly lit environments, image quality degrades significantly, blurring target features and significantly reducing detection accuracy. While relying solely on radar can effectively detect information such as target distance and speed, it struggles to perform detailed target classification and accurately identify their appearance. Furthermore, traditional methods are highly susceptible to interference when processing targets in complex backgrounds, unable to accurately distinguish between targets and background noise, making it difficult to meet the stringent requirements for high-precision and high-reliability target detection in complex scenarios.

[0004] While some attempts have been made to employ data fusion technology, most involve simple data splicing, failing to fully exploit the complementary advantages of radar and camera data and resulting in suboptimal fusion results. Regarding the application of deep learning algorithms, conventional deep learning models lack targeted optimization and adaptation for heterogeneous data collected by radar and cameras, and are unable to effectively integrate the characteristics of both types of data, resulting in suboptimal target detection accuracy and recall. Furthermore, existing technologies struggle to track the motion of dynamic targets in complex scenarios in real time, making detection efficiency and real-time performance insufficient for practical applications.

[0005] The above-mentioned defects of the existing technology not only hinder the rapid response of the security monitoring system to emergencies and affect the safe driving of autonomous vehicles in the intelligent transportation system, but also restrict the efficient product quality inspection and equipment operation monitoring on the industrial automation production line, thereby limiting the intelligent upgrading and development of related industries. Summary of the Invention

[0006] (1) Technical problems solved

[0007] In response to the above-mentioned shortcomings of the existing technology, the present invention provides a target detection method based on radar and camera data fusion, which can effectively overcome the defect of the existing technology that it is difficult to accurately detect dynamic targets in complex scenes.

[0008] (2) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] The target detection method based on radar and camera data fusion includes the following steps:

[0011] S1, radar and camera collect data separately;

[0012] S2. Adopting an adaptive multi-angle joint calibration strategy based on multi-feature constraints to achieve spatial sparse alignment of heterogeneous data collected by radar and camera, making the heterogeneous data spatially consistent and comparable;

[0013] S3. Use the YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures to perform target detection on the calibrated radar data and camera data respectively;

[0014] S4, using a cross-modal feature fusion algorithm to fuse the target information detected by the radar with the camera image plane to obtain a fusion feature;

[0015] S5. Using a method that combines deep learning and reinforcement learning, construct a deep learning model and a reinforcement learning model and set a reward function. The reward function is used to optimize the parameters of the reinforcement learning model to perform target classification.

[0016] S6. Output the final target detection and target classification results.

[0017] Preferably, the adaptive multi-angle joint calibration strategy based on multi-feature constraints in S2 includes:

[0018] S21. Extract the geometric features of the target:

[0019] Extract the target's three-dimensional radar coordinates P in the radar coordinate system from the radar data R ;

[0020] Get the 2D pixel coordinates of the target from the camera image , using the camera intrinsic parameter matrix K, through Estimate the 3D camera coordinates P of the target in the camera coordinate system C ;

[0021] in, is the depth factor;

[0022] S22. Extract the target's motion features:

[0023] In the radar data, extract the target's velocity v in the radar coordinate system R , and calculate the acceleration a of the target in the radar coordinate system according to the change of radar echo data in a continuous time period R ;

[0024] In the camera image, obtain the target's motion trajectory and velocity v I , and converted to the target's velocity v in the camera coordinate system C , and further estimate the acceleration of the target in the camera coordinate system based on the velocity change rate ;

[0025] S23. Since the working angles of radar and camera change over time, a dynamic angle adaptive matrix is ​​introduced to dynamically adapt radar data and camera data based on the real-time collected angle data and time series information:

[0026] ;

[0027] in, is the dynamic angle adaptive matrix, t is the time, 、 are the pitch angle and yaw angle respectively;

[0028] S24. To achieve spatial sparse alignment of heterogeneous data, construct multi-feature constraints:

[0029] ;

[0030] Among them, T RC (t) is the transformation matrix from the radar coordinate system to the camera coordinate system that changes with time t, is the correction vector used to compensate for the error of the ensemble model, is the correction vector used to compensate for velocity model errors and environmental disturbances, is the correction vector used to compensate for acceleration model errors and environmental disturbances;

[0031] S25. Solve the transformation matrix T by minimizing the following objective function RC (t) and dynamic angle adaptation matrix :

[0032] ;

[0033] in, 、 、 are the three-dimensional camera coordinates, velocity, and acceleration of the data point i in the camera coordinate system, 、 、 are the three-dimensional radar coordinates, velocity, and acceleration of data point i in the radar coordinate system, N is the number of collected data points, 、 are the weight factors of velocity constraint and acceleration constraint respectively.

[0034] Preferably, the YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures in S3 includes:

[0035] I. Adjust the YOLOv8 model based on the radar data structure:

[0036] 1) Adjust the input layer: In order to make the YOLOv8 model better adapt to the input radar data, a nonlinear transformation layer is embedded in the input layer. The transformed radar data is:

[0037] ;

[0038] Among them, R t is the transformed radar data, R' is the input calibrated radar data, W R-in is the transformation matrix, b R-in is the bias term, is the activation function;

[0039] 2) Adjust the feature extraction layer: Considering the sparsity of radar data and the target motion characteristics, a hierarchical temporal convolutional network is constructed. Assume that the radar data has T continuous observations in the time dimension and form them into a three-dimensional tensor R seq , design a set of layered temporal convolution kernels , For the Convolution kernel in the layer time dimension, L is the number of layers, , is the field of real numbers, For the The convolution kernel size in the layer time dimension, 7 represents the dimension of the radar data;

[0040] First, perform the first convolution operation on the three-dimensional tensor composed of radar data:

[0041] ;

[0042] in, is the radar data feature in the first time dimension, R seq is a three-dimensional tensor composed of radar data, , 7 represents the dimension of radar data, is the convolution kernel in the time dimension of the first layer;

[0043] Secondly, subsequent layers perform feature extraction through residual connections:

[0044] ;

[0045] in, For the Radar data characteristics in the layer time dimension, For the Radar data characteristics in the layer time dimension;

[0046] Finally, the radar data features on each layer of time dimension are fused and combined with the spatial dimension features to obtain the fused radar data features F R ;

[0047] II. Adjust the YOLOv8 model based on the camera data structure:

[0048] 1) Multi-scale feature fusion optimization: Based on the original multi-scale feature fusion of the YOLOv8 model, a dynamic scale selection mechanism is introduced to dynamically generate the selection probability vector through a reinforcement learning-based intelligent agent. , , the agent adjusts the probability according to the environmental feedback, and the feature map of the fused camera data is:

[0049] ;

[0050] Among them, F C is the fused camera data feature map, F C,si is the camera data feature map at the i-th scale, p Ci To select the camera data feature map F at the i-th scale C,si The probability of , N is the number of scales of the feature map;

[0051] 2) Semantic-guided feature enhancement: The semantic segmentation information of the camera image is used to enhance the target detection performance. Let the semantic segmentation result of the camera image be S C , and combine it with the fused camera data feature map F C Through a gated fusion mechanism, a learnable gated vector is designed. , generated by convolution operation, H and W are the fused camera data feature maps F C The height and width of the fused camera data enhancement feature map are:

[0052] ;

[0053] Among them, F C,enhanced is the enhanced feature map of the fused camera data, and ⊙ represents element-wise multiplication.

[0054] Preferably, in S4, a cross-modal feature fusion algorithm is used to fuse the target information detected by the radar with the camera image plane to obtain a fusion feature, including:

[0055] S41. Calculate the fused radar data feature F R Enhance the feature map F with the fused camera data C,enhanced The attention weight matrix between:

[0056] ;

[0057] Among them, A cross is the attention weight matrix, Q R =W Q F R , K C =W K F C,enhanced , W Q and W K are all learnable weight matrices, is the Softmax activation function;

[0058] S42. Calculate the fusion feature by weighted summation:

[0059] ;

[0060] Among them, F fusion For fusion features.

[0061] Preferably, S5 adopts a method combining deep learning and reinforcement learning to construct a deep learning model and a reinforcement learning model and set a reward function, and uses the reward function to optimize the parameters of the reinforcement learning model to perform target classification, including:

[0062] S51, combining Swin Transformer and ConvNeXtV2 deep learning networks, adopting a multimodal feature fusion method based on bilinear pooling to extract deep features, and determining the preliminary classification probability based on the deep features;

[0063] S52. Input the deep features and preliminary classification probabilities into the improved A2C reinforcement learning network to perform target classification and obtain target classification results.

[0064] Preferably, S51 combines the Swin Transformer and ConvNeXtV2 deep learning networks, adopts a multimodal feature fusion method based on bilinear pooling to extract deep features, and determines the preliminary classification probability based on the deep features, including:

[0065] S511, in order to maximize the balance between radar data features and camera data features, from the fusion feature F fusionExtract radar feature vector from input data , m is the number of radar feature vectors, and the SwinTransformer deep learning network is used to extract the remaining features F fusion - Extract camera feature vectors in R , n is the number of camera feature vectors, and the radar feature vector R and the camera feature vector C are linearly transformed:

[0066] ;

[0067] Among them, R trans 、C trans are the transformed radar feature vector and camera feature vector respectively, W R and W C are all learnable weight matrices, b R and b C is the bias term;

[0068] S512, the transformed radar feature vector R trans , camera feature vector C trans Perform bilinear pooling to capture high-order interaction information between multimodal features and obtain a bilinear feature vector:

[0069] ;

[0070] Among them, F bilinear is the bilinear eigenvector, represents the outer product operation;

[0071] S513, for the bilinear eigenvector F bilinear Perform dimensionality reduction to obtain a low-dimensional feature vector:

[0072] ;

[0073] Among them, F low is a low-dimensional feature vector, Represents vectorized operation, W red is the dimension reduction weight matrix, b red is the bias term;

[0074] S514, use ConvNeXtV2 deep learning network to calculate the low-dimensional feature vector F low Perform deep feature extraction to obtain deep feature F deep ;

[0075] S515, the depth feature F deep Input into a fully connected layer to get the preliminary classification probability:

[0076] ;

[0077] Among them, P pre is the initial classification probability, W pre is the weight matrix, b pre is the bias term, is the Softmax activation function.

[0078] Preferably, the ConvNeXtV2 deep learning network is improved on the basis of the original ConvNeXt deep learning network, introducing more efficient modules and optimization strategies, and obtaining the deep feature F after passing through multiple ConvNeXtV2 blocks. deep .

[0079] Preferably, in S52, the deep features and preliminary classification probabilities are input into the improved A2C reinforcement learning network to perform target classification, and the target classification results are obtained, including:

[0080] S521, state definition: the depth feature F deep And the preliminary classification probability P pre splicing, as the state s=[P pre ;F deep ];

[0081] S522, Action Space: The action a of the agent is to calculate the initial classification probability P pre The adjustment strategy is achieved through the learnable transformation matrix W adj For the initial classification probability P pre Perform a linear transformation and add a bias term b adj , get the adjusted classification probability P adj :

[0082] ;

[0083] S523. Reward function: Considering the classification accuracy, classification uncertainty and action complexity, the reward function is designed:

[0084] ;

[0085] Among them, R(s,a) is the reward function, which represents the immediate reward obtained by taking action a in state s, Accuracy(P adj ) is the adjusted classification probability P adj The corresponding classification accuracy, Entropy (P adj ) is the adjusted classification probability P adj The corresponding classification uncertainty, Complexity(W adj ) is the transformation matrix W adj The corresponding action complexity, 、 and They are all weight coefficients, and the weight coefficients should be set reasonably 、 and , can balance the three factors of classification accuracy, classification uncertainty and action complexity, and guide the intelligent agent to learn the optimal adjustment strategy;

[0086] S524. Improved A2C reinforcement learning algorithm used by the improved A2C reinforcement learning network:

[0087] 1) Develop a strategic network , taking state s as input and outputting the probability of action a , is the parameter of the policy network. Under a given state s, according to the probability The sampled action is actually executed, and the above adjusted classification probability P adj That is, the result corresponding to the action actually performed;

[0088] 2) The value function network is recorded as , takes state s as input and outputs the value estimate of state s, that is, the expectation of the long-term reward obtained by executing the optimal adjustment strategy starting from the current state, are the parameters of the value function network;

[0089] 3) Policy network update: policy network parameters Updated by policy gradient, the calculation formula of policy gradient is:

[0090] ;

[0091] in, is the policy gradient, which represents the policy network parameters The update direction and size of is the objective function of the policy network, i.e., maximizing long-term rewards, is the expectation operator, s~p(s) means that the state s is sampled according to the state distribution p(s), a~ Indicates that action a is distributed according to probability Sampling obtained, Strategic Network The logarithmic probability of the policy network parameters The gradient of , which represents the policy network The output action probability is the policy network parameter sensitivity, R(s,a) is the reward function, is the value function, is the advantage function, which indicates the advantage of taking action a relative to the average action in state s and is used to guide the strategy network The update of the policy network Prefer to choose actions with positive advantages, that is, actions that can obtain higher than average rewards;

[0092] 4) Value function network update: value function network parameters Update is performed by minimizing the mean square error loss, and the loss function is:

[0093] ;

[0094] in, is the network parameter with respect to the value function The loss function is is the maximum value function, which means the maximum value function obtained by taking all possible actions a' in state s'. State s' is the next state transferred to after executing action a. is a discount factor used to balance immediate rewards with future rewards. is the value function network's estimate of the value of state s, is the target value, is the predicted value;

[0095] S525. The agent continuously interacts with the environment and adjusts the parameters of the policy network and the value function network according to the update rules of the policy network and the value function network based on the sampled state s, the executed action a, and the reward R(s,a).

[0096] As training progresses, the policy network gradually learns to output the optimal action according to different states s, that is, it can obtain the adjusted classification probability P that maximizes the reward function R(s,a). adj , and finally according to the adjusted classification probability P adj Determine the target classification results.

[0097] (3) Beneficial effects

[0098] Compared with the existing technology, the target detection method based on radar and camera data fusion provided by the present invention has the following beneficial effects:

[0099] 1) In the data calibration phase, an adaptive multi-angle joint calibration strategy based on multi-feature constraints is adopted to accurately align heterogeneous data collected by radar and camera, effectively improving the quality of data fusion;

[0100] 2) In the target detection phase, the YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures is used, combined with a new cross-modal feature fusion algorithm to effectively improve the accuracy and recall of target detection;

[0101] 3) In the target classification phase, we use a combination of deep learning and reinforcement learning, and utilize a reward function to optimize the parameters of the reinforcement learning model to enhance the model's adaptability and robustness in complex scenarios.

[0102] Through practice, the present invention can provide efficient solutions for various fields, effectively promote the intelligent upgrading and development of related industries, and reduce the risks and losses of detection errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0104] Figure 1 It is a schematic diagram of the process of the present invention;

[0105] Figure 2 Schematic diagram of the process of obtaining fusion features by using the YOLOv8 dynamic structure reconfiguration mechanism and the cross-modal feature fusion algorithm in the present invention;

[0106] Figure 3 This is a schematic diagram of the structure of the improved YOLOv8 model in the present invention;

[0107] Figure 4 This is a flow chart of the target classification process using a method combining deep learning and reinforcement learning in the present invention;

[0108] Figure 5 This is a graph showing the results of target detection and target classification performed on the training set using the technical solution of the present invention;

[0109] Figure 6 This is a graph showing the results of target detection and target classification on a test set using the technical solution of the present invention. DETAILED DESCRIPTION

[0110] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0111] Target detection methods based on radar and camera data fusion, such as Figure 1 As shown, the following steps are included:

[0112] S1, radar and camera collect data separately. The radar transmits electromagnetic waves and receives echo signals to obtain information such as the distance and speed of the target. The camera obtains visual image information of the target through optical imaging.

[0113] S2. An adaptive multi-angle joint calibration strategy based on multi-feature constraints is adopted to achieve spatial sparse alignment of heterogeneous data collected by radar and camera, making the heterogeneous data spatially consistent and comparable.

[0114] Specifically, the adaptive multi-angle joint calibration strategy based on multi-feature constraints includes:

[0115] S21. Extract the geometric features of the target:

[0116] Extract the target's three-dimensional radar coordinates P in the radar coordinate system from the radar data R ;

[0117] Get the 2D pixel coordinates of the target from the camera image , using the camera intrinsic parameter matrix K, through Estimate the 3D camera coordinates P of the target in the camera coordinate system C ;

[0118] in, is the depth factor;

[0119] S22. Extract the target's motion features:

[0120] In the radar data, extract the target's velocity v in the radar coordinate system R , and calculate the acceleration a of the target in the radar coordinate system according to the change of radar echo data in a continuous time period R ;

[0121] In the camera image, obtain the target's motion trajectory and velocity v I , and converted to the target's velocity v in the camera coordinate system C , and further estimate the acceleration of the target in the camera coordinate system based on the velocity change rate ;

[0122] S23. Since the working angles of radar and camera change over time, a dynamic angle adaptive matrix is ​​introduced to dynamically adapt radar data and camera data based on the real-time collected angle data and time series information:

[0123] ;

[0124] in, is the dynamic angle adaptive matrix, t is the time, 、 are the pitch angle and yaw angle respectively;

[0125] S24. To achieve spatial sparse alignment of heterogeneous data, construct multi-feature constraints:

[0126] ;

[0127] Among them, T RC (t) is the transformation matrix from the radar coordinate system to the camera coordinate system that changes with time t, is the correction vector used to compensate for the error of the ensemble model, is the correction vector used to compensate for velocity model errors and environmental disturbances, is the correction vector used to compensate for acceleration model errors and environmental disturbances;

[0128] S25. Solve the transformation matrix T by minimizing the following objective function RC (t) and dynamic angle adaptation matrix :

[0129] ;

[0130] in, 、 、 are the three-dimensional camera coordinates, velocity, and acceleration of the data point i in the camera coordinate system, 、 、 are the three-dimensional radar coordinates, velocity, and acceleration of data point i in the radar coordinate system, N is the number of collected data points, 、 are the weight factors of velocity constraint and acceleration constraint respectively.

[0131] S3. Use the YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures to perform target detection on the calibrated radar data and camera data respectively.

[0132] Specifically, the YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures, such as Figure 2 and Figure 3 Shown, including:

[0133] I. Adjust the YOLOv8 model based on the radar data structure:

[0134] 1) Adjust the input layer: In order to make the YOLOv8 model better adapt to the input radar data, a nonlinear transformation layer is embedded in the input layer. The transformed radar data is:

[0135] ;

[0136] Among them, R t is the transformed radar data, R' is the input calibrated radar data, W R-in is the transformation matrix, b R-in is the bias term, is the activation function;

[0137] 2) Adjust the feature extraction layer: Considering the sparsity of radar data and the target motion characteristics, a hierarchical temporal convolutional network is constructed. Assume that the radar data has T continuous observations in the time dimension and form them into a three-dimensional tensor R seq , design a set of layered temporal convolution kernels , For the Convolution kernel in the layer time dimension, L is the number of layers, , is the field of real numbers, For the The convolution kernel size in the layer time dimension, 7 represents the dimension of the radar data;

[0138] First, perform the first convolution operation on the three-dimensional tensor composed of radar data:

[0139] ;

[0140] in, is the radar data feature in the first time dimension, R seq is a three-dimensional tensor composed of radar data, , 7 represents the dimension of radar data, is the convolution kernel in the time dimension of the first layer;

[0141] Secondly, subsequent layers perform feature extraction through residual connections:

[0142] ;

[0143] in, For the Radar data characteristics in the layer time dimension, For the Radar data characteristics in the layer time dimension;

[0144] Finally, the radar data features on each layer of time dimension are fused and combined with the spatial dimension features to obtain the fused radar data features F R ;

[0145] II. Adjust the YOLOv8 model based on the camera data structure:

[0146] 1) Multi-scale feature fusion optimization: Based on the original multi-scale feature fusion of the YOLOv8 model, a dynamic scale selection mechanism is introduced to dynamically generate the selection probability vector through a reinforcement learning-based intelligent agent. , , the agent adjusts the probability according to the environmental feedback, and the feature map of the fused camera data is:

[0147] ;

[0148] Among them, F C is the fused camera data feature map, F C,si is the camera data feature map at the i-th scale, p Ci To select the camera data feature map F at the i-th scale C,si The probability of , N is the number of scales of the feature map;

[0149] 2) Semantic-guided feature enhancement: The semantic segmentation information of the camera image is used to enhance the target detection performance. Let the semantic segmentation result of the camera image be S C , and combine it with the fused camera data feature map F C Through a gated fusion mechanism, a learnable gated vector is designed. , generated by convolution operation, H and W are the fused camera data feature maps F C The height and width of the fused camera data enhancement feature map are:

[0150] ;

[0151] Among them, F C,enhanced is the enhanced feature map of the fused camera data, and ⊙ represents element-wise multiplication.

[0152] S4, using the cross-modal feature fusion algorithm to fuse the target information detected by the radar with the camera image plane to obtain the fusion feature, such as Figure 2 As shown, specifically including:

[0153] S41. Calculate the fused radar data feature F R Enhance the feature map F with the fused camera data C,enhanced The attention weight matrix between:

[0154] ;

[0155] Among them, A cross is the attention weight matrix, Q R =W Q F R , K C =W K F C,enhanced, W Q and W K are all learnable weight matrices, is the Softmax activation function;

[0156] S42. Calculate the fusion feature by weighted summation:

[0157] ;

[0158] Among them, F fusion For fusion features.

[0159] S5. Use a method that combines deep learning and reinforcement learning to build a deep learning model and a reinforcement learning model and set a reward function. Use the reward function to optimize the parameters of the reinforcement learning model to perform target classification, including:

[0160] S51, combining Swin Transformer and ConvNeXtV2 deep learning networks, adopting a multimodal feature fusion method based on bilinear pooling to extract deep features, and determining the preliminary classification probability based on the deep features;

[0161] S52. Input the deep features and preliminary classification probabilities into the improved A2C reinforcement learning network to perform target classification and obtain target classification results.

[0162] Specifically, S51 combines the Swin Transformer and ConvNeXtV2 deep learning networks, adopts a multimodal feature fusion method based on bilinear pooling to extract deep features, and determines the preliminary classification probability based on the deep features, such as Figure 4 Shown, including:

[0163] S511, in order to maximize the balance between radar data features and camera data features, from the fusion feature F fusion Extract radar feature vector from input data , m is the number of radar feature vectors, and the SwinTransformer deep learning network is used to extract the remaining features F fusion - Extract camera feature vectors in R , n is the number of camera feature vectors, and the radar feature vector R and the camera feature vector C are linearly transformed:

[0164] ;

[0165] Among them, R trans 、C trans are the transformed radar feature vector and camera feature vector respectively, W R and W C are all learnable weight matrices, b Rand b C is the bias term;

[0166] S512, the transformed radar feature vector R trans , camera feature vector C trans Perform bilinear pooling to capture high-order interaction information between multimodal features and obtain a bilinear feature vector:

[0167] ;

[0168] Among them, F bilinear is the bilinear eigenvector, represents the outer product operation;

[0169] S513, for the bilinear eigenvector F bilinear Perform dimensionality reduction to obtain a low-dimensional feature vector:

[0170] ;

[0171] Among them, F low is a low-dimensional feature vector, Represents vectorized operation, W red is the dimension reduction weight matrix, b red is the bias term;

[0172] S514, use ConvNeXtV2 deep learning network to calculate the low-dimensional feature vector F low Perform deep feature extraction to obtain deep feature F deep ;

[0173] S515, the depth feature F deep Input into a fully connected layer to get the preliminary classification probability:

[0174] ;

[0175] Among them, P pre is the initial classification probability, W pre is the weight matrix, b pre is the bias term, is the Softmax activation function.

[0176] In the technical solution of this application, the ConvNeXtV2 deep learning network is improved on the basis of the original ConvNeXt deep learning network, introducing more efficient modules and optimization strategies. After passing through multiple ConvNeXtV2 blocks, the deep feature F is obtained. deep .

[0177] Specifically, in S52, the deep features and preliminary classification probabilities are input into the improved A2C reinforcement learning network for target classification, and the target classification results are obtained, such as Figure 4 Shown, including:

[0178] S521, state definition: the depth feature F deep And the preliminary classification probability P pre splicing, as the state s=[P pre ;F deep ];

[0179] S522, Action Space: The action a of the agent is to calculate the initial classification probability P pre The adjustment strategy is achieved through the learnable transformation matrix W adj For the initial classification probability P pre Perform a linear transformation and add a bias term b adj , get the adjusted classification probability P adj :

[0180] ;

[0181] S523. Reward function: Considering the classification accuracy, classification uncertainty and action complexity, the reward function is designed:

[0182] ;

[0183] Among them, R(s,a) is the reward function, which represents the immediate reward obtained by taking action a in state s, Accuracy(P adj ) is the adjusted classification probability P adj The corresponding classification accuracy, Entropy (P adj ) is the adjusted classification probability P adj The corresponding classification uncertainty, Complexity(W adj ) is the transformation matrix W adj The corresponding action complexity, 、 and They are all weight coefficients, and the weight coefficients should be set reasonably 、 and , can balance the three factors of classification accuracy, classification uncertainty and action complexity, and guide the intelligent agent to learn the optimal adjustment strategy;

[0184] S524. Improved A2C reinforcement learning algorithm used by the improved A2C reinforcement learning network:

[0185] 1) Develop a strategic network , taking state s as input and outputting the probability of action a , is the parameter of the policy network. Under a given state s, according to the probability The sampled action is actually executed, and the above adjusted classification probability P adj That is, the result corresponding to the action actually performed;

[0186] 2) The value function network is recorded as , takes state s as input and outputs the value estimate of state s, that is, the expectation of the long-term reward obtained by executing the optimal adjustment strategy starting from the current state, are the parameters of the value function network;

[0187] 3) Policy network update: policy network parameters Updated by policy gradient, the calculation formula of policy gradient is:

[0188] ;

[0189] in, is the policy gradient, which represents the policy network parameters The update direction and size of is the objective function of the policy network, i.e., maximizing long-term rewards, is the expectation operator, s~p(s) means that the state s is sampled according to the state distribution p(s), a~ Indicates that action a is distributed according to probability Sampling obtained, Strategic Network The logarithmic probability of the policy network parameters The gradient of , which represents the policy network The output action probability is the policy network parameter sensitivity, R(s,a) is the reward function, is the value function, is the advantage function, which indicates the advantage of taking action a relative to the average action in state s and is used to guide the strategy network The update of the policy network Prefer to choose actions with positive advantages, that is, actions that can obtain higher than average rewards;

[0190] 4) Value function network update: value function network parameters Update is performed by minimizing the mean square error loss, and the loss function is:

[0191] ;

[0192] in, is the network parameter with respect to the value function The loss function is is the maximum value function, which means the maximum value function obtained by taking all possible actions a' in state s'. State s' is the next state transferred to after executing action a. is a discount factor used to balance immediate rewards with future rewards. is the value function network's estimate of the value of state s, is the target value, is the predicted value;

[0193] S525. The agent continuously interacts with the environment and adjusts the parameters of the policy network and the value function network according to the update rules of the policy network and the value function network based on the sampled state s, the executed action a, and the reward R(s,a).

[0194] As training progresses, the policy network gradually learns to output the optimal action according to different states s, that is, it can obtain the adjusted classification probability P that maximizes the reward function R(s,a). adj , and finally according to the adjusted classification probability P adj Determine the target classification results.

[0195] S6. Output the final target detection and target classification results to provide a reliable data basis for subsequent decision-making and application, such as target recognition and processing in security monitoring, intelligent transportation, industrial automation and other scenarios.

[0196] In order to test the effectiveness of the technical solution of the present application, a data set was constructed by collecting specific image data to train and verify the technical solution of the present application. The data set used is shown in Table 1 below:

[0197] Table 1 Dataset information table

[0198]

[0199] This dataset has a balanced distribution of categories, with 250 samples per category. This provides comprehensive and balanced learning material for the model, helping it fully capture the characteristic differences between objects of different categories. The total number of samples, reaching 1,500, provides sufficient data support for model training and validation, enabling the model to learn more universal and accurate patterns with the support of this large amount of data.

[0200] In terms of dataset utilization strategy, a classic and effective partitioning method was adopted: 80% of the dataset was divided into a training set for model parameter learning and optimization, while the remaining 20% ​​was used as a test set to evaluate the model's performance on unknown data. This partitioning method ensures that the model fully learns while accurately testing its generalization ability.

[0201] like Figure 5 and Figure 6 As shown, it can be seen intuitively that the model accurately locates the detection boxes of various targets, and the classification labels are highly consistent with the actual targets. The classification accuracy of each category is clearly presented in the following Table 2 through detailed calculation and statistics:

[0202] Table 2 Target detection and classification results

[0203]

[0204] Table 2 shows that the accuracy of each category in the training set varies somewhat. For example, the bus category has the highest accuracy, reaching 98.5%. This indicates that the model has fully learned the characteristics of bus objects during training and can accurately identify them. However, the accuracy of the electric vehicle and motorcycle categories is relatively low, at 93%. This may be due to the complex variations in appearance, color, and model of these objects, which poses certain challenges to model learning.

[0205] On the test set, the accuracy of each category also varied. The accuracy for bicycles reached 98%, demonstrating the model's strong generalization ability for unknown bicycle data. The accuracy for trucks was 90%, slightly lower than other categories. This may be due to the unique scenarios or angles of the truck samples in the test set, which affected the model's judgment.

[0206] Taking the average values ​​of the comprehensive training set and the test set as an example, the average accuracy of the training set is 95.08%, and the average accuracy of the test set is 93.33%. This not only shows that the model can learn effectively during the training process, but also has good generalization ability when facing new data, further verifying the effectiveness and reliability of the technical solution of this application.

[0207] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A target detection method based on radar and camera data fusion, characterized by: The following steps are involved: S1, radar and camera collect data separately; S2. Adopting an adaptive multi-angle joint calibration strategy based on multi-feature constraints to achieve spatial sparse alignment of heterogeneous data collected by radar and camera, making the heterogeneous data spatially consistent and comparable; S3. Use the YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures to perform target detection on the calibrated radar data and camera data respectively; S4, using a cross-modal feature fusion algorithm to fuse the target information detected by the radar with the camera image plane to obtain a fusion feature; S5. Using a method that combines deep learning and reinforcement learning, construct a deep learning model and a reinforcement learning model and set a reward function. The reward function is used to optimize the parameters of the reinforcement learning model to perform target classification. S6, output the final target detection and target classification results; S5 uses a method that combines deep learning and reinforcement learning to build deep learning models and reinforcement learning models and set a reward function. The reward function is used to optimize the parameters of the reinforcement learning model to perform target classification, including: S51, combining Swin Transformer and ConvNeXtV2 deep learning networks, adopting a multimodal feature fusion method based on bilinear pooling to extract deep features, and determining the preliminary classification probability based on the deep features; S52. Input the deep features and preliminary classification probabilities into the improved A2C reinforcement learning network to perform target classification and obtain target classification results.

2. The target detection method based on radar and camera data fusion according to claim 1, characterized in that: The adaptive multi-angle joint calibration strategy based on multi-feature constraints in S2 includes: S21. Extract the geometric features of the target: Extract the target's three-dimensional radar coordinates P in the radar coordinate system from the radar data R ; Get the 2D pixel coordinates p of the target from the camera image I , using the camera intrinsic parameter matrix K, through Estimate the 3D camera coordinates P of the target in the camera coordinate system C ; in, is the depth factor; S22. Extract the target's motion features: In the radar data, extract the target's velocity v in the radar coordinate system R , and calculate the acceleration a of the target in the radar coordinate system according to the change of radar echo data in a continuous time period R ; In the camera image, obtain the target's motion trajectory and velocity v I , and converted to the target's velocity v in the camera coordinate system C , and further estimate the acceleration of the target in the camera coordinate system based on the velocity change rate ; S23. Since the working angles of radar and camera change over time, a dynamic angle adaptive matrix is ​​introduced to dynamically adapt radar data and camera data based on the real-time collected angle data and time series information: ； in, is the dynamic angle adaptive matrix, t is the time, 、 are the pitch angle and yaw angle respectively; S24. To achieve spatial sparse alignment of heterogeneous data, construct multi-feature constraints: ； Among them, T RC (t) is the transformation matrix from the radar coordinate system to the camera coordinate system that changes with time t, is the correction vector used to compensate for the error of the ensemble model, is the correction vector used to compensate for velocity model errors and environmental disturbances, is the correction vector used to compensate for acceleration model errors and environmental disturbances; S25. Solve the transformation matrix T by minimizing the following objective function RC (t) and dynamic angle adaptation matrix : ； in, 、 、 are the three-dimensional camera coordinates, velocity, and acceleration of the data point i in the camera coordinate system, 、 、 are the three-dimensional radar coordinates, velocity, and acceleration of data point i in the radar coordinate system, N is the number of collected data points, 、 are the weight factors of velocity constraint and acceleration constraint respectively.

3. The target detection method based on radar and camera data fusion according to claim 1, characterized in that: YOLOv8 dynamic structure reconfiguration mechanism for radar and camera data structures in S3, including: I. Adjust the YOLOv8 model based on the radar data structure: 1) Adjust the input layer: In order to make the YOLOv8 model better adapt to the input radar data, a nonlinear transformation layer is embedded in the input layer. The transformed radar data is: ； Among them, R t is the transformed radar data, R' is the input calibrated radar data, W R-in is the transformation matrix, b R-in is the bias term, is the activation function; 2) Adjust the feature extraction layer: Considering the sparsity of radar data and the target motion characteristics, a hierarchical temporal convolutional network is constructed. Assume that the radar data has T continuous observations in the time dimension and form them into a three-dimensional tensor R seq , design a set of layered temporal convolution kernels , For the Convolution kernel in the layer time dimension, L is the number of layers, , is the field of real numbers, For the The convolution kernel size in the layer time dimension, 7 represents the dimension of the radar data; First, perform the first convolution operation on the three-dimensional tensor composed of radar data: ； in, is the radar data feature in the first time dimension, R seq is a three-dimensional tensor composed of radar data, , 7 represents the dimension of radar data, is the convolution kernel in the time dimension of the first layer; Secondly, subsequent layers perform feature extraction through residual connections: ； in, For the Radar data characteristics in the layer time dimension, For the -1 layer of radar data features in the time dimension; Finally, the radar data features on each layer of time dimension are fused and combined with the spatial dimension features to obtain the fused radar data features F R ; II. Adjust the YOLOv8 model based on the camera data structure: 1) Multi-scale feature fusion optimization: Based on the original multi-scale feature fusion of the YOLOv8 model, a dynamic scale selection mechanism is introduced to dynamically generate the selection probability vector through a reinforcement learning-based intelligent agent. , , the agent adjusts the probability according to the environmental feedback, and the feature map of the fused camera data is: ； Among them, F C is the fused camera data feature map, F C,si is the camera data feature map at the i-th scale, p Ci To select the camera data feature map F at the i-th scale C,si The probability of , N is the number of scales of the feature map; 2) Semantic-guided feature enhancement: The semantic segmentation information of the camera image is used to enhance the target detection performance. Let the semantic segmentation result of the camera image be S C , and combine it with the fused camera data feature map F C Through a gated fusion mechanism, a learnable gated vector is designed. , generated by convolution operation, H and W are the fused camera data feature maps F C The height and width of the fused camera data enhancement feature map are: ； Among them, F C,enhanced is the enhanced feature map of the fused camera data, and ⊙ represents element-wise multiplication.

4. The target detection method based on radar and camera data fusion according to claim 3, characterized in that: S4 uses a cross-modal feature fusion algorithm to fuse the target information detected by the radar with the camera image plane to obtain fused features, including: S41. Calculate the fused radar data feature F R Enhance the feature map F with the fused camera data C,enhanced The attention weight matrix between: ； Among them, A cross is the attention weight matrix, Q R =W Q F R , K C =W K F C,enhanced , W Q and W K are all learnable weight matrices, is the Softmax activation function; S42. Calculate the fusion feature by weighted summation: ； Among them, F fusion For fusion features.

5. The target detection method based on radar and camera data fusion according to claim 1, characterized in that: S51 combines the Swin Transformer and ConvNeXtV2 deep learning networks, uses a multimodal feature fusion method based on bilinear pooling to extract deep features, and determines the preliminary classification probability based on the deep features, including: S511, in order to maximize the balance between radar data features and camera data features, from the fusion feature F fusion Extract radar feature vector from input data , m is the number of radar feature vectors, and the Swin Transformer deep learning network is used to extract the remaining features F fusion - Extract camera feature vectors in R , n is the number of camera feature vectors, and the radar feature vector R and the camera feature vector C are linearly transformed: ； Among them, R trans 、C trans are the transformed radar feature vector and camera feature vector respectively, W R and W C are all learnable weight matrices, b R and b C is the bias term; S512, the transformed radar feature vector R trans , camera feature vector C trans Perform bilinear pooling to capture high-order interaction information between multimodal features and obtain a bilinear feature vector: ； Among them, F bilinear is the bilinear eigenvector, represents the outer product operation; S513, for the bilinear eigenvector F bilinear Perform dimensionality reduction to obtain a low-dimensional feature vector: ； Among them, F low is a low-dimensional feature vector, Represents vectorized operation, W red is the dimension reduction weight matrix, b red is the bias term; S514, use ConvNeXtV2 deep learning network to calculate the low-dimensional feature vector F low Perform deep feature extraction to obtain deep feature F deep ; S515, the depth feature F deep Input into a fully connected layer to get the preliminary classification probability: ； Among them, P pre is the initial classification probability, W pre is the weight matrix, b pre is the bias term, is the Softmax activation function.

6. The target detection method based on radar and camera data fusion according to claim 5, characterized in that: The ConvNeXtV2 deep learning network is improved on the basis of the original ConvNeXt deep learning network, introducing more efficient modules and optimization strategies. After passing through multiple ConvNeXtV2 blocks, the deep feature F is obtained. deep .

7. The target detection method based on radar and camera data fusion according to claim 5, characterized in that: In S52, the deep features and preliminary classification probabilities are input into the improved A2C reinforcement learning network for target classification, and the target classification results are obtained, including: S521, state definition: the depth feature F deep And the preliminary classification probability P pre splicing, as the state s=[P pre ;F deep ]; S522, Action Space: The action a of the agent is to calculate the initial classification probability P pre The adjustment strategy is achieved through the learnable transformation matrix W adj For the initial classification probability P pre Perform a linear transformation and add a bias term b adj , get the adjusted classification probability P adj : ； S523. Reward function: Considering the classification accuracy, classification uncertainty and action complexity, the reward function is designed: ； Among them, R(s,a) is the reward function, which represents the immediate reward obtained by taking action a in state s, Accuracy(P adj ) is the adjusted classification probability P adj The corresponding classification accuracy, Entropy (P adj ) is the adjusted classification probability P adj The corresponding classification uncertainty, Complexity(W adj ) is the transformation matrix W adj The corresponding action complexity, 、 and They are all weight coefficients, and the weight coefficients should be set reasonably 、 and , can balance the three factors of classification accuracy, classification uncertainty and action complexity, and guide the intelligent agent to learn the optimal adjustment strategy; S524. Improved A2C reinforcement learning algorithm used by the improved A2C reinforcement learning network: 1) Develop a strategic network , taking state s as input and outputting the probability of action a , is the parameter of the policy network. Under a given state s, according to the probability The sampled action is actually executed, and the above adjusted classification probability P adj That is, the result corresponding to the action actually performed; 2) The value function network is recorded as , takes state s as input and outputs the value estimate of state s, that is, the expectation of the long-term reward obtained by executing the optimal adjustment strategy starting from the current state, are the parameters of the value function network; 3) Policy network update: policy network parameters Updated by policy gradient, the calculation formula of policy gradient is: ； in, is the policy gradient, which represents the policy network parameters The update direction and size of is the objective function of the policy network, i.e., maximizing long-term rewards, is the expectation operator, s~p(s) means that the state s is sampled according to the state distribution p(s), a~ Indicates that action a is distributed according to probability Sampling obtained, Strategic Network The logarithmic probability of the policy network parameters The gradient of , which represents the policy network The output action probability is the policy network parameter sensitivity, R(s,a) is the reward function, is the value function, is the advantage function, which indicates the advantage of taking action a relative to the average action in state s and is used to guide the strategy network The update of the policy network Prefer to choose actions with positive advantages, that is, actions that can obtain higher than average rewards; 4) Value function network update: value function network parameters Update is performed by minimizing the mean square error loss, and the loss function is: ； in, is the network parameter with respect to the value function The loss function is is the maximum value function, which means the maximum value function obtained by taking all possible actions a' in state s'. State s' is the next state transferred to after executing action a. is a discount factor used to balance immediate rewards with future rewards. is the value function network's estimate of the value of state s, is the target value, is the predicted value; S525. The agent continuously interacts with the environment and adjusts the parameters of the policy network and the value function network according to the update rules of the policy network and the value function network based on the sampled state s, the executed action a, and the reward R(s,a). As training progresses, the policy network gradually learns to output the optimal action according to different states s, that is, it can obtain the adjusted classification probability P that maximizes the reward function R(s,a). adj , and finally according to the adjusted classification probability P adj Determine the target classification results.

Images