Multi-source fusion target detection and tracking system in complex scene

Abstract

The invention discloses a multi-source fusion target detection and tracking system in a complex scene. The multi-source fusion target detection and tracking system comprises a data receiving and collecting layer, a multi-source fusion target detection layer and a space-time fusion target tracking layer. The data receiving and collecting layer is responsible for collecting laser radar point cloud, millimeter wave radar point cloud and camera image data, making a complex scene data set and transmitting the complex scene data set to the multi-source fusion target detection layer; the multi-source fusion target detection layer performs fusion denoising and feature extraction on the complex scene data, inputs fusion features into a detection head to obtain a target detection result, and transmits the target detection result to the multi-source fusion target tracking layer; and the space-time fusion target tracking layer carries out trajectory prediction on the detection result and carries out space-time data association with a historical detection result so as to generate a target tracking trajectory. According to the complex scene multi-source fusion target detection and tracking system, the problem of fusion perception robustness of a vehicle-mounted sensor in a complex scene is mainly solved, and the capability of vehicle-mounted perception to adapt to the complex scene is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal environment perception of vehicle-mounted sensors, and specifically relates to a multi-source fusion target detection and tracking system in complex scenarios. Background Art

[0002] With the rapid development of fields such as autonomous driving, robotic navigation, and intelligent surveillance, object detection and tracking technologies in complex scenarios have become increasingly important. In these fields, systems must accurately identify and track objects such as vehicles, pedestrians, and obstacles in dynamic environments. However, due to environmental variability, such as rain, fog, dust, and low light conditions, a single sensor often struggles to provide reliable detection and tracking performance.

[0003] To overcome this challenge, researchers have begun exploring multi-source fusion techniques, integrating data from different sensors to improve the accuracy and robustness of target detection and tracking. These sensors include cameras, laser radar (LiDAR), and millimeter-wave radar, each of which has its own unique advantages and limitations. For example, cameras can provide rich texture information but are significantly affected by lighting conditions; LiDAR can provide accurate distance and velocity information but has limitations in texture recognition; and millimeter-wave radar has better penetration and is suitable for use in adverse weather conditions.

[0004] In a multi-source fusion system, data association is one of the core steps, which requires matching different observations of the same target in different time frames. This usually involves calculating the similarity between the detection result and each trajectory, and the calculation of the similarity can be based on appearance embedding, motion correlation, and position correlation. Appearance embedding, as a key feature of data association, can help the system capture key information in the image, such as color, texture, and shape, so as to achieve effective matching of the same target in different time frames. In addition, the Kalman filter (KF) plays an important role in target tracking. The KF is a recursive filter that can predict the motion of the target by estimating the system state and using the measurement value to correct the prediction. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-source fusion target detection and tracking system in complex scenarios to solve the problem of poor robustness of complex environment perception of vehicle-mounted sensors in the prior art.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A multi-source fusion target detection and tracking system in complex scenarios, the system comprising:

[0008] Data receiving and acquisition layer: This layer collects lidar point cloud, millimeter-wave radar point cloud, and camera image data for rainy and foggy scenes, dusty scenes, low-light scenes, and interference-free scenes through the lidar data receiving module, millimeter-wave radar data receiving module, and camera image receiving module. The complex scene dataset production module then provides data to the multi-source fusion target detection layer based on the training model.

[0009] Multi-source fusion target detection layer: This includes the multi-modal point cloud fusion denoising module, the image point cloud fusion denoising module, the bird's-eye view feature fusion module, and the multi-fusion detection head based on the heat map; it is used for denoising, feature fusion, and target detection of multi-modal data;

[0010] Spatiotemporal fusion target tracking layer: includes a spatiotemporal sequence data association module, a trajectory prediction module, and a trajectory update module; it is used to associate the detection results with historical trajectories and output continuous tracking trajectories through the Kalman filter.

[0011] Furthermore, the laser radar data receiving module in the data receiving and acquisition layer transmits point cloud data through the ETH aviation plug-in interface, the millimeter wave radar data receiving module transmits point cloud data through the CAN interface, and the camera image receiving module transmits YUV422 format images through the GMSL protocol;

[0012] The complex scene data set production module collects and labels multimodal data of rain and fog, dust, insufficient lighting and no interference scenes in training mode, and collects 5000-7000 frames for each scene.

[0013] Furthermore, a target detection and tracking method of a multi-source fusion target detection and tracking system in a complex scene has the following specific steps:

[0014] Step 1: Collect three types of data: lidar point cloud, millimeter-wave radar point cloud, and camera image data in different scenarios using three types of sensors;

[0015] Step 2: Perform noise segmentation on the collected data to obtain the denoised true value data, and annotate the category, size, position and speed data of the desired target, and integrate them into a multimodal dataset for training and testing;

[0016] Step 3: Use the Pointpillars network to extract the noisy lidar point cloud bird's-eye view features and the millimeter-wave radar point cloud bird's-eye view features respectively; convert the noise distribution into a Gaussian distribution through Gaussian reparameterization sampling, and use the denoiser combined with the millimeter-wave radar features to iteratively denoise and generate the denoised lidar point cloud features;

[0017] Step 4: Rasterize the millimeter-wave radar point cloud and project it onto an azimuth-altitude plane parallel to the image plane. Extract image features and millimeter-wave radar azimuth-altitude plane features. Align multi-level feature dimensions using a feature pyramid network, and fuse cross-modal features into the 3D query point using a deformable attention mechanism.

[0018] Step 5: Map the fused features to the bird's-eye view space, and the heat map-based detection head generates the target category, bounding box, pose, and velocity information;

[0019] Step 6: Determine whether it is in training mode. If so, input the training set true value label data and the results obtained by the detection network into the loss function, and further perform backpropagation to train the network model until it passes the test set verification; otherwise, output the detected bounding box and target speed and transmit them to the spatiotemporal fusion target tracking layer;

[0020] Step 7: Based on the target detection results, extract the target's appearance embedding features;

[0021] Step 8: Build a spatiotemporal data correlation matrix and match the current detection results with historical trajectories by weighted calculation of appearance embedding similarity, motion correlation, and position correlation.

[0022] Step 9: Use the extended Kalman filter to predict the target state, update the target trajectory based on the detection results, and output the continuous tracking results.

[0023] Furthermore, the specific steps of step 1 are:

[0024] Step 1.1: Use the lidar data receiving module, millimeter wave radar data receiving module, and camera image receiving module to receive environmental information from different sensors;

[0025] Step 1.2: Determine whether the system is in training mode. If not, directly transmit the three types of information received to the multi-source fusion target detection layer. If the system is in training mode, collect three types of data: lidar point cloud, millimeter-wave radar point cloud, and camera image data in different scenarios such as rain and fog, dust, insufficient lighting, and no interference. Collect 5,000 to 7,000 frames of data for each scenario.

[0026] Furthermore, the specific steps of step 3 are:

[0027] Step 3.1: Use the Pointpillars network to extract the bird's-eye view features F of the noisy lidar point cloud Lidar And the millimeter wave radar point cloud bird's eye view feature F Radar ;

[0028] Step 3.2: The original noise distribution δ is converted to originConvert to Gaussian distribution δ gaussian , calculated using the following formula:

[0029]

[0030] Among them, δ gaussian Conforms to Gaussian distribution ∈ obeys the standard normal distribution N(0,I); is a hyperparameter;

[0031] Step 3.3: Combine weather-robust mmWave radar point cloud features as conditions to guide the DiffusionTransformers denoiser D θ Iterative denoising is performed using the following formula:

[0032] F n+1 =D θ (Concat[F n ,F Radar ],n),n∈0,1,...,N

[0033] Among them, F n+1 Denotes denoising features, Concat denotes the millimeter wave radar feature F Radar and the nth denoised lidar point cloud feature F n Connect

[0034] Step 3.4: After N denoising steps, pass the Diffusion Transformer denoising device D θ Generate clear denoised lidar point cloud bird's-eye view features

[0035] Step 3.5: Determine whether it is training mode. If so, proceed to step 3.6. Otherwise, directly return to the clear lidar point cloud bird's-eye view feature after denoising.

[0036] Step 3.6: Calculate the loss of multimodal feature denoising, where the loss is calculated by the following formula:

[0037]

[0038] in, is the baseline feature extracted from the lidar point cloud after cleaning the weather noise, e is the number of training rounds, ψ is the loss rate, and weight(e) is the loss weight, which is calculated by the following formula:

[0039]

[0040] Where τ is the temperature parameter, is the offset, and the loss weight decreases nonlinearly with the number of training rounds to ensure that the model focuses on the feature denoising task in the early stage.

[0041] Furthermore, the specific steps of step 4 are:

[0042] Step 4.1: Represent the millimeter wave radar point cloud data as a raster network;

[0043] Step 4.2: Using the mmWave radar grid network, project the point cloud cube data onto the azimuth-altitude plane, which is parallel to the image plane to support fusion between the camera and mmWave radar.

[0044] Step 4.3: Extract image features F using the ResNet backbone network Camera , and millimeter-wave radar azimuth-altitude plane characteristics

[0045] Step 4.4: Align F using Feature Pyramid Network Camera 、 The dimension of the multi-level feature map;

[0046] Step 4.5: Initialize a set of query points in 3D space and project them onto the image plane and the bird's-eye view plane.

[0047] Step 4.6: Use the deformable attention mechanism to fuse the features of the projected clicks into the relevant query points.

[0048] Furthermore, the specific steps of step 5 are:

[0049] Step 5.1: Use the heatmap-based multi-source fusion detection head and regression head to retrieve the bounding box and target velocity from the queried features;

[0050] Step 5.2: Use the regressed bounding box position as the new query point;

[0051] Step 5.3: Update the features of the new query point in an iterative process to optimize the detected bounding box and target velocity.

[0052] Furthermore, when step 6 is in training mode, a one-to-many loss function is used for model training. The loss function includes focal loss and L1 regression loss. The loss weights are all set to 1, and the AdamW optimizer is used for optimization. The final loss function is expressed as follows:

[0053] L dec =L class +L bbox +L vel .

[0054] Furthermore, the specific steps of step 8 are:

[0055] Constructing a data correlation matrix This matrix is ​​composed of the appearance embedding similarity matrix A deep (T t ,D t ), motion correlation matrix A motion (T t ,D t ) and position correlation matrix A location (T t ,D t ) and is weighted according to the following formula:

[0056] A(T t ,D t )=w deep A deep (T t ,D t )+w motion A motion (T t ,D t )A loc (T t ,D t )

[0057] Among them, represents the target trajectory at time t; represents the target detection result at time t; w deep and w motion =1-w deep is a weight scalar; and only the motion correlation matrix is ​​the variable A motion (T t ,D t ), whose element a motion (τ t ,d t ) corresponds to a single trajectory τ t ∈T t and test results d t ∈D t The variable is calculated according to the following formula:

[0058] a motion (τ t ,d t )=w vel a centroid +(1-w vel )a pseudo

[0059] Among them, a centroid Represents the center of mass of the detected target; a pseudo Indicates the state difference of the detected target; w vel Represents a speed-related scalar weight, calculated according to the following formula:

[0060]

[0061] in, and Respectively represent τ t Velocity subsets in Kalman filter and detection results.

[0062] The beneficial effects of the present invention are:

[0063] The present invention integrates multi-source data from different sensors, including cameras, laser radars (LiDAR), and millimeter-wave radars (Radar), uses appearance embedding, motion correlation, and position correlation for data association, adopts an extended Kalman filter for target tracking, and combines deep learning technology for feature extraction and processing, thereby overcoming the limitations of a single sensor in complex environments, such as changes in lighting and severe weather conditions, thereby significantly improving the accuracy and robustness of target detection and tracking. The present invention utilizes the high-precision distance measurement capability of laser radars and the penetration of millimeter-wave radars, combined with the rich texture information of cameras, to achieve high-precision detection of targets, especially in severe weather conditions. It can also maintain high detection performance. In addition, by adopting a Kalman filter and a data association step, by calculating the similarity between the detection results and each trajectory, effective matching of the same target in different time frames is achieved, further improving the continuity and accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the structural composition of the present invention;

[0065] Figure 2 Schematic diagram for receiving and collecting sensor data;

[0066] Figure 3 Create module flow charts for complex scenario data sets at the data receiving and acquisition layer;

[0067] Figure 4 This is the flow chart of the multimodal point cloud fusion denoising module in the multi-source fusion target detection layer;

[0068] Figure 5 This is the flow chart of the image point cloud fusion denoising module in the multi-source fusion target detection layer;

[0069] Figure 6 This is the flow chart of the spatiotemporal fusion target tracking layer algorithm. DETAILED DESCRIPTION

[0070] The present invention will be further described below with reference to the accompanying drawings.

[0071] The present invention provides a multi-source fusion target detection and tracking system in complex scenes, based on Figure 1 , the system includes:

[0072] Data receiving and acquisition layer: Through three types of sensor receiving modules, three types of data are collected: lidar point cloud, millimeter wave radar point cloud and camera image data for rain and fog scenes, dust scenes, insufficient light scenes and interference-free scenes. The complex scene data set production module is used to provide corresponding data to the multi-source fusion target detection layer according to whether it is in training mode.

[0073] Specifically, it includes a lidar data receiving module, a millimeter-wave radar data receiving module, a camera image receiving module and a complex scene data set production module; the lidar data receiving module adopts the ETH aviation plug-in interface to realize real-time transmission of point cloud three-dimensional data to obtain three-dimensional environmental information; the millimeter-wave radar data receiving module adopts the CAN interface transmission protocol to realize millimeter-wave radar point cloud data transmission to obtain three-dimensional point clouds with strong penetration; the camera image receiving module adopts the GMSL transmission protocol to realize real-time transmission of YUV422 format images to enrich environmental semantic information; the complex scene data set production module is responsible for integrating and processing three types of data: lidar point cloud, millimeter-wave radar point cloud and camera image for rain and fog scenes, dust scenes, insufficient light scenes and interference-free scenes.

[0074] according to Figure 2 , the three types of sensors are installed at the front end of the autonomous driving vehicle to ensure that they have overlapping fields of view, that is, forward installation. Specifically, the lidar sensor is installed at the center of the front end of the vehicle, the millimeter-wave radar sensor is installed below the lidar, and the on-board camera is installed to the left of the millimeter-wave radar. This layout design ensures that there is sufficient overlapping field of view between the sensors, thereby covering a wider detection range and improving the accuracy of target detection and tracking. Through this installation method, data from different sensors can be effectively fused to achieve accurate detection and tracking of targets in complex scenarios.

[0075] Multi-source fusion target detection layer: includes fusion denoising and feature extraction of complex scene data, further inputs the fusion features into the detection head to obtain the target detection results, and transmits them to the multi-source fusion target tracking layer.

[0076] Specifically, it includes a multimodal point cloud fusion denoising module, an image point cloud fusion denoising module, a bird's-eye view feature fusion module and a multi-fusion detection head based on a heat map; the multimodal point cloud fusion denoising module is used to fuse millimeter-wave radar point cloud data with noisy lidar point cloud data to remove the influence of rainy and foggy weather and dust scenes on three-dimensional point cloud data, thereby obtaining high-precision three-dimensional spatial features; the image point cloud fusion denoising module uses the rich texture information of the camera to assist in the interpretation of radar data, and uses the precise distance and speed information of the radar to correct the noise and errors in the camera data to obtain semantically rich spatial features; the bird's-eye view feature fusion module is used to fuse the features extracted by the above two modules and map them to the bird's-eye view space; the multi-fusion detection head based on a heat map is used to perform target detection based on the generated bird's-eye view features, and generate target category, size and posture information.

[0077] Spatiotemporal fusion target tracking layer: The current detection results are used to predict the trajectory, and the spatiotemporal data is associated with the historical detection results to obtain the target tracking trajectory.

[0078] Specifically, it includes a spatiotemporal sequence data association module, a trajectory prediction module, and a trajectory update module; the spatiotemporal sequence data association module aims to accurately match the detected target with the existing trajectory. This module achieves this by calculating the similarity between the detection result and each trajectory. The similarity is based on appearance embedding, motion correlation, and position correlation; the trajectory prediction module predicts the state of the current time step based on the state estimate of the previous time step, which includes information such as the position and speed of the target; the trajectory update module uses new measurement data to update the predicted state estimate. In this module, the system compares the predicted state with the actual measured state and calculates the difference between them. Furthermore, the system adjusts the predicted state based on this difference to make it closer to the actual measured state.

[0079] according to Figure 3 The data receiving and collecting layer includes the following steps:

[0080] Step S101: Utilize the laser radar data receiving module, the millimeter wave radar data receiving module and the camera image receiving module to receive environmental information from different sensors.

[0081] Step S102: Determine whether it is in training mode; if it is in training mode, continue to execute step S103, otherwise the received three types of information are directly transmitted to the multi-source fusion target detection layer.

[0082] Step S103: Collect three types of data, namely, lidar point cloud, millimeter-wave radar point cloud, and camera image, in different scenarios such as rain and fog, dust, insufficient lighting, and no interference. Collect 5,000 to 7,000 frames of data for each scenario.

[0083] Step S104: performing noise segmentation processing on the collected point cloud and image data to obtain denoised true value data, and annotating information such as the category, size, position and speed of the desired target.

[0084] Step S105: Integrate the labeled data into a multimodal dataset for training and testing, which can fully cover target detection and tracking tasks under different environmental conditions.

[0085] Step S106: perform quality control and verification on the data set to ensure the accuracy and consistency of the data, and transmit the prepared data set to the multi-source fusion target detection layer for training.

[0086] according to Figure 4 The multi-modal point cloud fusion denoising module in the multi-source fusion target detection layer includes the following steps:

[0087] Step S201: Use the Pointpillars network to extract the noisy lidar point cloud bird's-eye view features F Lidar And the millimeter wave radar point cloud bird's eye view feature F Radar .

[0088] Step S202: The original noise distribution δ is converted to origin Convert to Gaussian distribution δ gaussian , calculated using the following formula:

[0089]

[0090] Among them, δ gaussian Conforms to Gaussian distribution ∈ obeys the standard normal distribution N(0,I); is a hyperparameter.

[0091] Step S203: Combine the weather-robust millimeter-wave radar point cloud features as conditions to guide the DiffusionTransformers denoiser D θ Predict denoising feature F n+1 , wherein the denoising feature F n+1 Calculated by the following formula:

[0092] F n+1 =D θ (Concat[F n ,F Radar],n), n∈0,1,…,N (2)

[0093] Among them, Concat means that the millimeter wave radar feature F Radar and the nth denoised lidar point cloud feature F n Connect them.

[0094] Step S204: After N denoising steps, the denoising is performed by the Diffusion Transformer denoising θ Generate clear denoised lidar point cloud bird's-eye view features

[0095] Step S205: Determine whether it is training mode, if yes, proceed to step S206, otherwise directly return to the clear lidar point cloud bird's eye view feature after denoising

[0096] Step S206: Calculate the loss of multimodal feature denoising, wherein the loss is calculated using the following formula:

[0097]

[0098] in, is the baseline feature extracted from the lidar point cloud after cleaning the weather noise, e is the number of training rounds, ψ is the loss rate, and weight(e) is the loss weight, which is calculated by the following formula:

[0099]

[0100] Where τ is the temperature parameter, is the offset, and the loss weight decreases nonlinearly with the number of training rounds to ensure that the model focuses on the feature denoising task in the early stage.

[0101] according to Figure 5 The image point cloud fusion denoising module in the multi-source fusion target detection layer includes the following steps:

[0102] Step S301: Representing the millimeter-wave radar point cloud data in a grid network;

[0103] Step S302: Using the millimeter-wave radar grid network, project the point cloud cube data onto the azimuth-altitude plane, which is parallel to the image plane to support fusion between the camera and the millimeter-wave radar;

[0104] Step S303: Extract image features F using the ResNet backbone network Camera , and millimeter-wave radar azimuth-altitude plane characteristics

[0105] Step S304: Align F using feature pyramid network Camera 、 The dimension of the multi-level feature map;

[0106] Step S305: Initialize a set of query points in the three-dimensional space and project them onto the image plane and the bird's-eye view plane;

[0107] Step S306: Using the deformable attention mechanism, the features of the projected click are fused into the relevant query points;

[0108] Step S307: using a multi-source fusion detection head and a regression head based on a heat map to retrieve a bounding box and target speed from the queried features;

[0109] Step S308: using the bounding box position obtained by regression as a new query point;

[0110] Step S309: updating the features of the new query point in an iterative process to optimize the detected bounding box and target speed;

[0111] Step S310: Determine whether it is training mode, if yes, proceed to step S311, otherwise output the detected bounding box and target speed, and transmit them to the spatiotemporal fusion target tracking layer;

[0112] Step S311: Use a one-to-many loss function to train the model. The loss function includes focal loss and L1 regression loss. The loss weights are all set to 1. The AdamW optimizer is used for optimization. The final loss function is expressed as follows:

[0113] L dec =L class +L bbox +L vel (5)

[0114] Among them, L dec represents the total loss of target detection; L class represents the target detection category loss; L bbox represents the 3D bounding box loss function; L vel represents the speed detection loss function.

[0115] according to Figure 6 The spatiotemporal fusion target tracking layer includes the following steps:

[0116] Step S401: Select the region of interest based on the bounding box obtained by the object detection layer, and use the CC-3DT network to extract the appearance embedding;

[0117] Step S402: Constructing a data association matrix This matrix is ​​composed of the appearance embedding similarity matrix A deep (Tt ,D t ), motion correlation matrix A motion (T t ,D t ) and position correlation matrix A location (T t ,D t ) and is weighted according to the following formula:

[0118] A(T t ,D t )=w deep A deep (T t ,D t )+w motion A motion (T t ,D t )A loc (T t ,D t ) (6)

[0119] Among them, represents the target trajectory at time t; represents the target detection result at time t; w deep and w motion =1-w deep is a weight scalar; and only the motion correlation matrix is ​​the variable A motion (T t ,D t ), whose element a motion (τ t ,d t ) corresponds to a single trajectory τ t ∈T t and test results d t ∈D t The variable is calculated according to the following formula:

[0120] a motion (τ t ,d t )=w vel a centroid +(1-w vel )a pseudo (7)

[0121] Among them, a centroid Represents the center of mass of the detected target; a pseudo Indicates the state difference of the detected target; w vel Represents a speed-related scalar weight, calculated according to the following formula:

[0122]

[0123] in, and Respectively represent τ t In the Kalman filter and the detection result d t speed subset.

[0124] Step S403: Using a Kalman filter for tracking. This process includes a trajectory prediction module and a trajectory module, and uses the speed output of the speed estimation and detection module to generate a tracking result.

[0125] The embodiments described in the patent of this invention are described one by one in a progressive manner. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part. For the disclosed embodiments, professionals and technicians in this field can implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0126] In summary, the present invention provides a multi-source fusion target detection and tracking system for complex scenarios. This system integrates data from different sensors, utilizes appearance embedding, motion correlation, and position correlation for data association, employs an extended Kalman filter for target tracking, and incorporates deep learning techniques for feature extraction and processing to improve the accuracy and robustness of target detection and tracking. Through detailed embodiments and technical solutions, the present invention demonstrates its feasibility and effectiveness in practical applications, and has broad application prospects and market potential.

[0127] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multi-source fusion target detection and tracking system in complex scenarios, characterized by: The system includes: Data receiving and acquisition layer: This layer collects lidar point cloud, millimeter-wave radar point cloud, and camera image data for rainy and foggy scenes, dusty scenes, low-light scenes, and interference-free scenes through the lidar data receiving module, millimeter-wave radar data receiving module, and camera image receiving module. The complex scene dataset production module then provides data to the multi-source fusion target detection layer based on the training model. Multi-source fusion target detection layer: This includes the multi-modal point cloud fusion denoising module, the image point cloud fusion denoising module, the bird's-eye view feature fusion module, and the multi-fusion detection head based on the heat map; it is used for denoising, feature fusion, and target detection of multi-modal data; Spatiotemporal fusion target tracking layer: includes a spatiotemporal sequence data association module, a trajectory prediction module, and a trajectory update module; it is used to associate the detection results with historical trajectories and output continuous tracking trajectories through the Kalman filter.

2. The multi-source fusion target detection and tracking system in complex scenarios according to claim 1, characterized in that: The laser radar data receiving module in the data receiving and acquisition layer transmits point cloud data through the ETH aviation plug-in interface, the millimeter wave radar data receiving module transmits point cloud data through the CAN interface, and the camera image receiving module transmits YUV422 format images through the GMSL protocol; The complex scene data set production module collects and labels multimodal data of rain and fog, dust, insufficient lighting and no interference scenes in training mode, and collects 5000-7000 frames for each scene.

3. The target detection and tracking method of the multi-source fusion target detection and tracking system in complex scenarios according to claim 1 is characterized by: The specific steps are as follows: Step 1: Collect three types of data: lidar point cloud, millimeter-wave radar point cloud, and camera image data in different scenarios using three types of sensors; Step 2: Perform noise segmentation on the collected data to obtain the denoised true value data, and annotate the category, size, position and speed data of the desired target, and integrate them into a multimodal dataset for training and testing; Step 3: Use the Pointpillars network to extract the noisy lidar point cloud bird's-eye view features and the millimeter-wave radar point cloud bird's-eye view features respectively; The noise distribution is converted to a Gaussian distribution through Gaussian reparameterization sampling, and the denoiser is used to iteratively denoise the millimeter-wave radar features to generate the denoised lidar point cloud features. Step 4: Rasterize the millimeter-wave radar point cloud and project it onto an azimuth-altitude plane parallel to the image plane. Extract image features and millimeter-wave radar azimuth-altitude plane features. Align multi-level feature dimensions using a feature pyramid network, and fuse cross-modal features into the 3D query point using a deformable attention mechanism. Step 5: Map the fused features to the bird's-eye view space, and the heat map-based detection head generates the target category, bounding box, pose, and velocity information; Step 6: Determine whether it is in training mode. If so, input the training set true value label data and the results obtained by the detection network into the loss function, and further perform backpropagation to train the network model until it passes the test set verification; otherwise, output the detected bounding box and target speed and transmit them to the spatiotemporal fusion target tracking layer; Step 7: Based on the target detection results, extract the target's appearance embedding features; Step 8: Build a spatiotemporal data correlation matrix and match the current detection results with historical trajectories by weighted calculation of appearance embedding similarity, motion correlation, and position correlation. Step 9: Use the extended Kalman filter to predict the target state, update the target trajectory based on the detection results, and output the continuous tracking results.

4. The multi-source fusion target detection and tracking system for complex scenarios according to claim 3 is characterized by: The specific steps of step 1 are: Step 1.1: Use the lidar data receiving module, millimeter wave radar data receiving module, and camera image receiving module to receive environmental information from different sensors; Step 1.2: Determine whether it is in training mode. If not, transmit the received three types of information directly to the multi-source fusion target detection layer; If it is in training mode, three types of data, namely lidar point cloud, millimeter-wave radar point cloud and camera image, are collected in different scenarios such as rain and fog, dust, insufficient light and no interference. 5,000 to 7,000 frames of data are collected for each scenario.

5. The multi-source fusion target detection and tracking system in complex scenarios according to claim 3 is characterized by: The specific steps of step 3 are: Step 3.1: Use the Pointpillars network to extract the bird's-eye view features F of the noisy lidar point cloud Lidar And the millimeter wave radar point cloud bird's eye view feature F Radar ; Step 3.2: The original noise distribution δ is converted to origin Convert to Gaussian distribution δ gaussian , calculated using the following formula: Among them, δ gaussian Conforms to Gaussian distribution ∈ obeys the standard normal distribution N(0,I); is a hyperparameter; Step 3.3: Combine weather-robust mmWave radar point cloud features as conditions to guide the DiffusionTransformers denoiser D θ Iterative denoising is performed using the following formula: F n+1 =D θ (Concat[F n ,F Radar ],n),n∈0,1,...,N Among them, F n+1 Represents the denoising feature, Concat represents the millimeter wave radar feature F Radar and the nth denoised lidar point cloud feature F n Connect Step 3.4: After N denoising steps, pass the Diffusion Transformer denoising device D θ Generate clear denoised lidar point cloud bird's-eye view features Step 3.5: Determine whether it is training mode. If so, proceed to step 3.

6. Otherwise, directly return to the clear lidar point cloud bird's-eye view feature after denoising. Step 3.6: Calculate the loss of multimodal feature denoising, where the loss is calculated by the following formula: in, is the baseline feature extracted from the lidar point cloud after cleaning the weather noise, e is the number of training rounds, ψ is the loss rate, and weight(e) is the loss weight, which is calculated by the following formula: Where τ is the temperature parameter, is the offset, and the loss weight decreases nonlinearly with the number of training rounds to ensure that the model focuses on the feature denoising task in the early stage.

6. The multi-source fusion target detection and tracking system in complex scenarios according to claim 3 is characterized by: The specific steps of step 4 are: Step 4.1: Represent the millimeter wave radar point cloud data as a raster network; Step 4.2: Using the mmWave radar grid network, project the point cloud cube data onto the azimuth-altitude plane, which is parallel to the image plane to support fusion between the camera and mmWave radar. Step 4.3: Extract image features F using the ResNet backbone network Camera , and millimeter-wave radar azimuth-altitude plane characteristics Step 4.4: Align F using Feature Pyramid Network Camera 、 The dimension of the multi-level feature map; Step 4.5: Initialize a set of query points in 3D space and project them onto the image plane and the bird's-eye view plane. Step 4.6: Use the deformable attention mechanism to fuse the features of the projected clicks into the relevant query points.

7. The multi-source fusion target detection and tracking system for complex scenarios according to claim 3 is characterized by: The specific steps of step 5 are: Step 5.1: Use the heatmap-based multi-source fusion detection head and regression head to retrieve the bounding box and target velocity from the queried features; Step 5.2: Use the bounding box position obtained by regression as the new query point; Step 5.3: Update the features of the new query point in an iterative process to optimize the detected bounding box and target velocity.

8. The multi-source fusion target detection and tracking system in complex scenarios according to claim 3 is characterized by: When step 6 is in training mode, a one-to-many loss function is used for model training. The loss function includes focus loss and L1 regression loss. The loss weights are all set to 1, and the AdamW optimizer is used for optimization. The final loss function is expressed as follows: L dec =L class +L bbox +L vel 。 9. The multi-source fusion target detection and tracking system in complex scenarios according to claim 3 is characterized by: The specific steps of step 8 are: Constructing a data correlation matrix This matrix is ​​composed of the appearance embedding similarity matrix A deep (T t ,D t ), motion correlation matrix A motion (T t ,D t ) and position correlation matrix A location (T t ,D t ) and is weighted according to the following formula: A(T t ,D t )=w deep A deep (T t ,D t )+w motion A motion (T t ,D t )A loc (T t ,D t ) Among them, represents the target trajectory at time t; represents the target detection result at time t; w deep and w motion =1-w deep is a weight scalar; and only the motion correlation matrix is ​​the variable A motion (T t ,D t ), whose element a motion (τ t ,d t ) corresponds to a single trajectory τ t ∈T t and test results d t ∈D t The variable is calculated according to the following formula: a motion (τ t ,d t )=w vel a centroid +(1-w vel )a pseudo Among them, a centroid Represents the center of mass of the detected target; a pseudo Indicates the state difference of the detected target; w vel Represents a speed-related scalar weight, calculated according to the following formula: in, and Respectively represent τ t In the Kalman filter and the detection result d t speed subset.

Images