Dynamic object identification and tracking method based on laser radar

Abstract

The invention discloses a dynamic object identification and tracking method based on a laser radar. The method comprises the following steps: S1, collecting data; s2, extracting features from the optical image data, extracting speed and distance information from the millimeter wave radar, and enhancing the expression ability of the features by using a deep learning method and the like; s3, fusing the point cloud data of the laser radar with the camera image, and further processing the fused data; s4, fusing a plurality of target lists to obtain reliable and real target information, and avoiding misinformation and missing detection; s5, comprehensively combining the decisions obtained by the multiple sensors by adopting a fusion strategy, and obtaining a fusion result; and S6, estimating and updating the state of the sensor carrier in real time by using a Bayesian filtering method. According to the invention, identification and tracking of the dynamic object are realized. The system has the beneficial effects that the sensing capability of the system is improved, false alarm and missing detection are reduced, and the reliability and the accuracy of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a dynamic object recognition and tracking method based on laser radar. Background Art

[0002] The application scenarios of dynamic object recognition and tracking are becoming more and more extensive, especially in the automatic driving of cars and automatic route planning of drones, which plays a very important role. In the existing technology, laser radar or millimeter wave radar is usually used in combination with optical image sensors and multi-data fusion systems to identify and track dynamic objects. However, the existing technology still has the following problems:

[0003] The detection capability of LiDAR will be greatly reduced under adverse weather conditions, such as heavy rain, snow, and dense fog. The amount of data generated by LiDAR is huge, and problems with data synchronization and timestamp alignment will occur during data fusion. In addition, millimeter-wave radar cannot observe Z coordinate data, only X, Y, and V. This limits its ability to recognize objects in three-dimensional space, especially the weak reflected waves for humans, and the low resolution for humans, making it impossible to effectively improve data fusion. Summary of the invention

[0004] The embodiment of the present invention provides a dynamic object recognition and tracking method based on laser radar to overcome the above-mentioned technical problems existing in the prior art.

[0005] A method for dynamic object recognition and tracking based on laser radar comprises the following steps:

[0006] S1: The laser radar sensor, millimeter wave radar sensor, and optical image sensor collect laser radar data, millimeter wave radar data, and optical image data respectively;

[0007] Preprocess all the sensor data collected, including data denoising, filtering, timestamp alignment, etc.

[0008] Perform sensor calibration to determine the coordinate relationship between sensors and the conversion relationship between them and the vehicle coordinate system;

[0009] S2: Extract edges, corners, etc. from optical image data, extract point cloud features from lidar data, extract speed and distance information from millimeter wave radar, and use deep learning and other methods to enhance the expression ability of features;

[0010] S3: Fuse the laser radar point cloud data with the camera image and further process the fused data, including data alignment and coordinate conversion;

[0011] S4: LiDAR sensors, millimeter-wave radar sensors, and optical image sensors extract target parameters and identify different targets to form a target list, and then fuse multiple target lists to obtain reliable and true target information to avoid false alarms and missed detections;

[0012] S5: Make preliminary decisions on the location, attributes and category of the target through optical image sensor data, and then use fusion strategies to comprehensively combine the decisions obtained by multiple sensors and obtain fusion results;

[0013] S6: Use the Bayesian filtering method to estimate the true state of the sensor carrier based on the observed data, and estimate and update the state of the sensor carrier in real time.

[0014] The present invention realizes the recognition and tracking of dynamic objects by comprehensively using the data of laser radar, millimeter wave radar and optical image sensor. Its beneficial effect is to improve the perception ability of the system, reduce false alarms and missed detections, and enhance the reliability and accuracy of the system.

[0015] Preferably, in S1, a deep learning model is used to denoise the data of the laser radar under bad weather conditions (such as heavy rain, snow, dense fog, etc.);

[0016] When training a deep learning model, an adversarial training method is used to conduct adversarial training between a generative model (G) and a discriminative model (D). The generative model generates denoised data, and the discriminative model determines the authenticity of the data.

[0017] Its loss function is:

[0018]

[0019] Among them, x represents the real lidar data;

[0020] z represents the noise data;

[0021] pdata(x) represents the distribution of real data;

[0022] pz(z) represents the distribution of noise data;

[0023] D(x) represents the output of the discriminant model, that is, the judgment result of the discriminator on the real data x;

[0024] G(z) represents the output of the generative model, that is, the fake data generated by the generator based on the noise data z;

[0025] We hope that the discriminator's judgment result on the real data is close to 1, that is, it is considered to be real data;

[0026] It is hoped that the discriminator's judgment result on the generated false data is close to 0, that is, it is considered to be false data.

[0027] In severe weather conditions, deep learning models are used to denoise lidar data and adversarial training methods are used to improve data quality. The beneficial effect is that high data accuracy and system performance can be maintained even in adverse environments.

[0028] Preferably, in step S1, a timestamp alignment method based on Kalman filtering is used to achieve time synchronization of lidar data, millimeter wave radar data and optical image data.

[0029] The Kalman filter model is:

[0030] x k|k-1 =F k x k-1|k-1 ;

[0031]

[0032] x k|k =x k|k-1 +K k (z k -H k x k|k-1 );

[0033] P k|k =(IK k H k ) k｜k-1 ;

[0034] Among them, x k｜k-1 and x k｜k is the state vector, which contains the timestamps of the data collected from each sensor (lidar, millimeter-wave radar, and optical image sensor);

[0035] x k｜k-1 is based on the state x at the previous moment k-1|k-1 And the state transfer matrix F k The predicted state at the current moment, which is specifically the timestamp of the current moment predicted based on the timestamp information of the previous moment;

[0036] X k|k It is the result of correcting the predicted state after taking into account the new observation data. Specifically, it is the timestamp of the current moment corrected according to the timestamp of the data actually received from the sensor.

[0037] F kIt is the state transfer matrix. This matrix describes how the state vector is transferred from one time step to the next time step. It specifically represents the prediction model of the timestamp. When the sensor data arrives at a fixed frequency, then F k To express this periodicity;

[0038] P k|k-1 is the prediction error covariance, P k|k is the updated error covariance. These two matrices describe the uncertainty of the state estimation, which specifically represents the uncertainty of the timestamp prediction and the uncertainty of the timestamp correction after considering the observed data.

[0039] Q k is the process noise covariance. This matrix describes the noise in the state transfer process, which specifically represents the statistical characteristics of the time jitter or delay in the process of sensor data generation and transmission.

[0040] H k It is the observation matrix. This matrix maps the observation values ​​to the state space. It specifically represents how to extract information for updating the timestamp estimate from the actual arrival timestamp of the sensor data.

[0041] K k is the Kalman gain, which determines the influence of new observations in updating the state estimate. It represents the weight given to new observations when updating the timestamp estimate.

[0042] Z k is the observation, which is the data actually received from the sensor, and it is the actual observation data used to update the timestamp estimate.

[0043] The present invention adopts a time stamp alignment method based on Kalman filtering to achieve time synchronization of different sensor data. Its beneficial effect is to improve the synchronization and accuracy of data processing and provide a reliable time reference for subsequent data analysis and processing.

[0044] Preferably, in step S2, a deep learning model is used to enhance the features of the laser radar point cloud data to extract richer spatial features;

[0045] When training the deep learning model, a 3D convolutional neural network (3D-CNN) is used to perform multi-layer convolution and pooling operations on the point cloud data.

[0046] The loss function is:

[0047]

[0048] Where L is the loss function, which is a measure of the difference between the model prediction and the true label;

[0049] yi is the true label of the i-th sample, which represents a specific feature or attribute in the point cloud data, specifically the semantic category of a certain area in the point cloud, such as pedestrians, vehicles, buildings, etc.;

[0050] is the predicted label of the i-th sample, that is, the feature or attribute predicted by the 3D-CNN model based on the input point cloud data;

[0051] N is the number of samples, that is, the total number of point cloud samples in the training dataset.

[0052] The present invention uses a deep learning model to enhance the features of LiDAR point cloud data and extract richer spatial features. Its beneficial effect is that it enhances the ability to recognize object features and improves the accuracy of target recognition.

[0053] Preferably, in step S3, a vision-lidar joint feature extraction method is used to deeply fuse the camera image and the lidar point cloud data;

[0054] A multimodal fusion network is used to perform multi-layer fusion operations on visual features and lidar features to generate joint features.

[0055] The present invention adopts a vision-lidar joint feature extraction method to deeply fuse camera images and lidar point cloud data. Its beneficial effect is that the accuracy and robustness of target recognition are improved through multimodal data fusion.

[0056] Preferably, the multimodal fusion network in S3 uses a two-stream network architecture to process image and point cloud data separately, and then fuse them in the deep layer of the network, wherein batch normalization and ReLU activation function are used in the network to enhance the expressive power of features, and early stopping and learning rate decay strategies are used to prevent overfitting.

[0057] The present invention uses a dual-stream network architecture for multimodal fusion, enhances feature expression capabilities, and adopts early stopping and learning rate decay strategies to prevent overfitting. Its beneficial effect is that it improves the training efficiency and generalization ability of the model and avoids the overfitting problem.

[0058] Preferably, in step S4, a multi-task learning method is used to simultaneously identify the location, attribute and category of the target;

[0059] When training a multi-task learning model, a shared feature extraction layer and multiple task branches are used to output the location, attributes, and category of the target respectively;

[0060] The loss function is:

[0061] L 总 =αL位置 +βL 属性 +γL 类别

[0062] Among them, L 总 is the total loss function, which is the weighted sum of position loss, attribute loss and category loss;

[0063] L 位置 is the position loss, which measures the difference between the target position predicted by the model and the true position;

[0064] L 属性 is the attribute loss, which measures the difference between the target attributes predicted by the model, including speed, size, shape, etc. and the true attributes;

[0065] L 类别 is the category loss, which measures the difference between the target category predicted by the model, including vehicles, pedestrians, bicycles, etc. and the true category;

[0066] α, β, γ are task weights, which correspond to the importance of position, attribute, and category tasks, respectively.

[0067] The present invention uses a multi-task learning method to simultaneously identify the location, attributes and category of a target. Its beneficial effect is that by sharing the feature extraction layer, the efficiency of the model is improved, and the results of multiple tasks can be output simultaneously, thereby enhancing the comprehensive recognition ability of the system.

[0068] Preferably, in step S6, a multi-sensor fusion method is used to estimate the state of the sensor carrier in real time based on the observed data, thereby improving the accuracy and robustness of the state estimation.

[0069] The fusion algorithm is:

[0070]

[0071] Among them, X k is the fused state estimate, which represents the estimate of the sensor carrier state after combining all sensor data at time k;

[0072] X k，i is the state estimate of the i-th sensor, which contains observation data from a lidar sensor, a millimeter-wave radar sensor, or an optical image sensor;

[0073] P k is the error covariance after fusion, which describes the uncertainty of the fusion state estimate;

[0074] P k，i is the error covariance of the ith sensor, which describes the uncertainty of the state estimate of the ith sensor;

[0075] w i is the weight of the i-th sensor, which reflects the contribution of the i-th sensor in the fusion process. The weight is determined based on the accuracy and reliability of the sensor;

[0076] M is the number of sensors, that is, the total number of sensors involved in state estimation. Here, lidar, millimeter-wave radar and optical image sensor are used, so M=3.

[0077] The present invention uses a multi-sensor fusion method to estimate the state of the sensor carrier in real time based on the observed data, which has the beneficial effect of improving the accuracy and robustness of the state estimation and enhancing the stability of the system.

[0078] Preferably, before S6, the method further includes:

[0079] Conditional generative adversarial networks (GANs) are used to generate environmental images directly from radar sensor data, making comprehensive use of all environmental features detected by the radar sensor. Then, the generated images are integrated with the optical images to perform S3 and evaluate the quality and availability of the environmental data after S1 fusion.

[0080] The present invention uses conditional generative adversarial networks (GANs) to directly generate environmental images from radar sensor data and integrate them with optical images. Its beneficial effect is that it comprehensively utilizes all environmental features detected by the radar sensor and improves the quality and availability of environmental data.

[0081] Preferably, in S5, it specifically includes the following steps:

[0082] S5.1: Use pre-trained deep learning models, including YOLO and SSD, to detect objects in optical image data, identify multiple categories of targets, and provide bounding box coordinates of the targets;

[0083] S5.2: Extract high-level features of detected targets to capture the shape and texture of the target;

[0084] S5.3: Use the shape and size characteristics of the target, combined with the known object dimensions, to estimate the speed and size of the target;

[0085] S5.4: Apply support vector machine (SVM) to the extracted features to determine the category of the target;

[0086] S5.5: Calculate the position of the target in the real world based on the bounding box coordinates of the target and the intrinsic parameters of the camera, and apply a perspective transformation to convert the pixel coordinates to the position in the world coordinate system;

[0087] S5.6: Dynamically assign weights based on sensor accuracy and reliability;

[0088] S5.7: Use the particle filter algorithm to select the fusion algorithm, and input the decision information of the optical image sensor and the decision result of the lidar sensor obtained in S5.4 and S5.5 into the fusion algorithm, wherein the decision information of the optical image sensor is used as the measurement update, and the other sensor data is used as the prediction step;

[0089] S5.8: Based on the fusion results, adjust sensor weights and fusion algorithm parameters to optimize decision accuracy, and design a feedback mechanism to compare the fusion results with actual observations or manually labeled data to evaluate and improve the fusion strategy.

[0090] The present invention improves the accuracy of target detection and recognition through a series of deep learning models and fusion algorithms, and dynamically adjusts sensor weights and fusion algorithm parameters to optimize decision accuracy. Its beneficial effect is that it achieves high-precision detection and recognition of targets and continuously optimizes system performance through feedback mechanisms.

[0091] Beneficial effects:

[0092] By comprehensively utilizing data from multiple sensors, high-precision recognition and tracking of dynamic objects is achieved. It improves data quality through deep learning models and adversarial training methods, especially for lidar data under severe weather conditions; uses Kalman filtering to achieve data time synchronization, enhancing the accuracy of data processing; uses deep learning models to enhance point cloud data features, improving the accuracy of target recognition; improves the accuracy and robustness of target recognition through multimodal data fusion and multi-task learning; estimates the state of sensor carriers in real time, enhancing the stability of the system; uses GANs to generate environmental images, improving the quality and availability of environmental data; and achieves high-precision detection and recognition of targets through a series of deep learning models and fusion algorithms, continuously optimizing system performance. These beneficial effects jointly improve the system's perception capabilities, reduce false alarms and missed detections, and enhance the reliability and accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0094] Figure 1 It is a flow chart of the steps of a dynamic object recognition and tracking method based on laser radar;

[0095] Figure 2 The flowchart of step S5 in a dynamic object recognition and tracking method based on laser radar. DETAILED DESCRIPTION

[0096] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0097] A method for dynamic object recognition and tracking based on laser radar comprises the following steps:

[0098] S1: The laser radar sensor, millimeter wave radar sensor, and optical image sensor collect laser radar data, millimeter wave radar data, and optical image data respectively;

[0099] Preprocess all the sensor data collected, including data denoising, filtering, timestamp alignment, etc.

[0100] Perform sensor calibration to determine the coordinate relationship between sensors and the conversion relationship between them and the vehicle coordinate system;

[0101] S2: Extract edges, corners, etc. from optical image data, extract point cloud features from lidar data, extract speed and distance information from millimeter wave radar, and use deep learning and other methods to enhance the expression ability of features;

[0102] S3: Fuse the laser radar point cloud data with the camera image and further process the fused data, including data alignment and coordinate conversion;

[0103] S4: LiDAR sensors, millimeter-wave radar sensors, and optical image sensors extract target parameters and identify different targets to form a target list, and then fuse multiple target lists to obtain reliable and true target information to avoid false alarms and missed detections;

[0104] S5: Make preliminary decisions on the location, attributes and category of the target through optical image sensor data, and then use fusion strategies to comprehensively combine the decisions obtained by multiple sensors and obtain fusion results;

[0105] S6: Use the Bayesian filtering method to estimate the true state of the sensor carrier based on the observed data, and estimate and update the state of the sensor carrier in real time. The Bayesian filtering method is a probabilistic algorithm based on the Bayesian theorem, which is used to estimate the state of random variables based on a series of incomplete and uncertain information.

[0106] It is understood that in the present invention, laser radar is a remote sensing technology that uses laser beams to measure the distance and speed of objects. It calculates the distance by emitting laser pulses and receiving the reflected light; millimeter wave radar is a radar system that uses electromagnetic waves in the millimeter wave band to detect and measure the distance, speed and angle of objects, and an optical image sensor is a sensor that can convert light signals into electrical signals, which is a camera and video surveillance system in this embodiment, used to capture images.

[0107] The present invention realizes the recognition and tracking of dynamic objects by comprehensively using the data of laser radar, millimeter wave radar and optical image sensor. Its beneficial effect is to improve the perception ability of the system, reduce false alarms and missed detections, and enhance the reliability and accuracy of the system.

[0108] Preferably, in S1, a deep learning model is used to denoise the data of the laser radar under bad weather conditions (such as heavy rain, snow, dense fog, etc.);

[0109] When training a deep learning model, an adversarial training method is used to conduct adversarial training between a generative model (G) and a discriminative model (D). The generative model generates denoised data, and the discriminative model determines the authenticity of the data.

[0110] Its loss function is:

[0111]

[0112] Among them, x represents the real lidar data;

[0113] z represents the noise data;

[0114] pdata(x) represents the distribution of real data;

[0115] p z (z) represents the distribution of noise data;

[0116] D(x) represents the output of the discriminant model, that is, the judgment result of the discriminator on the real data x;

[0117] G(z) represents the output of the generative model, that is, the fake data generated by the generator based on the noise data z;

[0118] We hope that the discriminator's judgment result on the real data is close to 1, that is, it is considered to be real data;

[0119] It is hoped that the discriminator's judgment result on the generated false data is close to 0, that is, it is considered to be false data.

[0120] In severe weather conditions, deep learning models are used to denoise lidar data and adversarial training methods are used to improve data quality. The beneficial effect is that high data accuracy and system performance can be maintained even in adverse environments.

[0121] Preferably, in step S1, a timestamp alignment method based on Kalman filtering is used to achieve time synchronization of lidar data, millimeter wave radar data and optical image data.

[0122] The Kalman filter model is:

[0123] x k|k-1 =F k x k-1|k-1 ;

[0124]

[0125] x k|k =x k|k-1 +K k (z k -H k x k|k-1 );

[0126] P k|k =(IK k H k ) k|k-1 ;

[0127] Among them, x k|k-1 and x k|k is the state vector, which contains the timestamps of the data collected from each sensor (lidar, millimeter-wave radar, and optical image sensor);

[0128] x k|k-1 is based on the state x at the previous moment k-1|k-1 And the state transfer matrix F k The predicted state at the current moment, which is specifically the timestamp of the current moment predicted based on the timestamp information of the previous moment;

[0129] x k|k It is the result of correcting the predicted state after taking into account the new observation data. Specifically, it is the timestamp of the current moment corrected according to the timestamp of the data actually received from the sensor.

[0130] F k It is the state transfer matrix. This matrix describes how the state vector is transferred from one time step to the next time step. It specifically represents the prediction model of the timestamp. When the sensor data arrives at a fixed frequency, then F k To express this periodicity;

[0131] Pk|k-1 is the prediction error covariance, P k|k is the updated error covariance. These two matrices describe the uncertainty of the state estimation, which specifically represents the uncertainty of the timestamp prediction and the uncertainty of the timestamp correction after considering the observed data.

[0132] Qk is the process noise covariance. This matrix describes the noise in the state transition process, which specifically represents the statistical characteristics of the time jitter or delay in the process of sensor data generation and transmission.

[0133] H k It is the observation matrix. This matrix maps the observation values ​​to the state space. It specifically represents how to extract information for updating the timestamp estimate from the actual arrival timestamp of the sensor data.

[0134] K k is the Kalman gain, which determines the influence of new observations in updating the state estimate. It represents the weight given to new observations when updating the timestamp estimate.

[0135] Z k is the observation, which is the data actually received from the sensor, and it is the actual observation data used to update the timestamp estimate.

[0136] The present invention adopts a time stamp alignment method based on Kalman filtering to achieve time synchronization of different sensor data. Its beneficial effect is to improve the synchronization and accuracy of data processing and provide a reliable time reference for subsequent data analysis and processing.

[0137] Preferably, in step S2, a deep learning model is used to enhance the features of the laser radar point cloud data to extract richer spatial features;

[0138] When training the deep learning model, a 3D convolutional neural network (3D-CNN) is used to perform multi-layer convolution and pooling operations on the point cloud data.

[0139] The loss function is:

[0140]

[0141] Where L is the loss function, which is a measure of the difference between the model prediction and the true label;

[0142] y i is the true label of the i-th sample, which represents a specific feature or attribute in the point cloud data, specifically the semantic category of a certain area in the point cloud, such as pedestrians, vehicles, buildings, etc.;

[0143] is the predicted label of the i-th sample, that is, the feature or attribute predicted by the 3D-CNN model based on the input point cloud data;

[0144] N is the number of samples, that is, the total number of point cloud samples in the training dataset.

[0145] The present invention uses a deep learning model to enhance the features of LiDAR point cloud data and extract richer spatial features. Its beneficial effect is that it enhances the ability to recognize object features and improves the accuracy of target recognition.

[0146] Preferably, in step S3, a vision-lidar joint feature extraction method is used to deeply fuse the camera image and the lidar point cloud data;

[0147] A multimodal fusion network is used to perform multi-layer fusion operations on visual features and lidar features to generate joint features.

[0148] The present invention adopts a vision-lidar joint feature extraction method to deeply fuse camera images and lidar point cloud data. Its beneficial effect is that the accuracy and robustness of target recognition are improved through multimodal data fusion.

[0149] Preferably, the multimodal fusion network in S3 uses a two-stream network architecture to process image and point cloud data separately, and then fuse them in the deep layer of the network, wherein batch normalization and ReLU activation function are used in the network to enhance the expressive power of features, and early stopping and learning rate decay strategies are used to prevent overfitting.

[0150] It should be noted that the two-stream network architecture is a deep learning network structure that has two parallel streams, which process different types of data, images and point clouds in this embodiment, and then fuse them in the deep layer of the network;

[0151] It should be noted that batch normalization is a technique used in deep learning to improve training speed and performance by normalizing the input of the layer to reduce internal covariate shift.

[0152] It should be noted that the ReLU activation function (RectifiedLinearUnitActivationFunction) is a commonly used activation function that sets all negative values ​​to zero and keeps positive values ​​unchanged, which helps solve the gradient disappearance problem.

[0153] It should be noted that Early Stopping is a technique to prevent overfitting by stopping training when the performance on the validation set no longer improves.

[0154] The present invention uses a dual-stream network architecture for multimodal fusion, enhances feature expression capabilities, and adopts early stopping and learning rate decay strategies to prevent overfitting. Its beneficial effect is that it improves the training efficiency and generalization ability of the model and avoids the overfitting problem.

[0155] Preferably, in step S4, a multi-task learning method is used to simultaneously identify the location, attribute and category of the target;

[0156] When training a multi-task learning model, a shared feature extraction layer and multiple task branches are used to output the location, attributes, and category of the target respectively;

[0157] The loss function is:

[0158] L 总 =αL 位置 +βL 属性 +γL 类别

[0159] Among them, L 总 is the total loss function, which is the weighted sum of position loss, attribute loss and category loss;

[0160] L 位置 is the position loss, which measures the difference between the target position predicted by the model and the true position;

[0161] L 属性 is the attribute loss, which measures the difference between the target attributes predicted by the model, including speed, size, shape, etc. and the true attributes;

[0162] L 类别 is the category loss, which measures the difference between the target category predicted by the model, including vehicles, pedestrians, bicycles, etc. and the true category;

[0163] α, β, γ are task weights, which correspond to the importance of position, attribute, and category tasks, respectively.

[0164] The present invention uses a multi-task learning method to simultaneously identify the location, attributes and category of a target. Its beneficial effect is that by sharing the feature extraction layer, the efficiency of the model is improved, and the results of multiple tasks can be output simultaneously, thereby enhancing the comprehensive recognition ability of the system.

[0165] Preferably, in step S6, a multi-sensor fusion method is used to estimate the state of the sensor carrier in real time based on the observed data, thereby improving the accuracy and robustness of the state estimation.

[0166] The fusion algorithm is:

[0167]

[0168] Among them, X k is the fused state estimate, which represents the estimate of the sensor carrier state after combining all sensor data at time k;

[0169] X k，i is the state estimate of the i-th sensor, which contains observation data from a lidar sensor, a millimeter-wave radar sensor, or an optical image sensor;

[0170] P k is the error covariance after fusion, which describes the uncertainty of the fusion state estimate;

[0171] P k，i is the error covariance of the ith sensor, which describes the uncertainty of the state estimate of the ith sensor;

[0172] w i is the weight of the i-th sensor, which reflects the contribution of the i-th sensor in the fusion process. The weight is determined based on the accuracy and reliability of the sensor;

[0173] M is the number of sensors, that is, the total number of sensors involved in state estimation. Here, lidar, millimeter-wave radar and optical image sensor are used, so M=3.

[0174] The present invention uses a multi-sensor fusion method to estimate the state of the sensor carrier in real time based on the observed data, which has the beneficial effect of improving the accuracy and robustness of the state estimation and enhancing the stability of the system.

[0175] Preferably, before S6, the method further includes:

[0176] Conditional generative adversarial networks (GANs) are used to generate environmental images directly from radar sensor data, making comprehensive use of all environmental features detected by the radar sensor. Then, the generated images are integrated with the optical images to perform S3 and evaluate the quality and availability of the environmental data after S1 fusion.

[0177] The present invention uses conditional generative adversarial networks (GANs) to directly generate environmental images from radar sensor data and integrate them with optical images. Its beneficial effect is that it comprehensively utilizes all environmental features detected by the radar sensor and improves the quality and availability of environmental data.

[0178] Preferably, in S5, it specifically includes the following steps:

[0179] S5.1: Use pre-trained deep learning models, including YOLO and SSD, to detect objects in optical image data, identify multiple categories of targets, and provide bounding box coordinates of the targets;

[0180] It should be noted that YOLO (You Only Look Once) is a real-time target detection system that solves target detection as a regression problem and can quickly and accurately detect targets in images; SSD (Single Shot MultiBox Detector) is a target detection framework that performs detection on feature maps of different scales to improve the detection capability of targets of different sizes;

[0181] S5.2: Extract high-level features of detected targets to capture the shape and texture of the target;

[0182] S5.3: Use the shape and size characteristics of the target, combined with the known object dimensions, to estimate the speed and size of the target;

[0183] S5.4: Apply support vector machine (SVM) to the extracted features to determine the category of the target;

[0184] S5.5: Calculate the position of the target in the real world based on the bounding box coordinates of the target and the intrinsic parameters of the camera, and apply a perspective transformation to convert the pixel coordinates to the position in the world coordinate system;

[0185] S5.6: Dynamically assign weights based on sensor accuracy and reliability;

[0186] S5.7: Use the particle filter algorithm to select the fusion algorithm, and input the decision information of the optical image sensor and the decision result of the lidar sensor obtained in S5.4 and S5.5 into the fusion algorithm, wherein the decision information of the optical image sensor is used as the measurement update, and the other sensor data is used as the prediction step;

[0187] It should be noted that particle filtering: This is a sequential Monte Carlo method used to estimate the state of a dynamic system, especially in nonlinear and non-Gaussian situations;

[0188] S5.8: Based on the fusion results, adjust sensor weights and fusion algorithm parameters to optimize decision accuracy, and design a feedback mechanism to compare the fusion results with actual observations or manually labeled data to evaluate and improve the fusion strategy.

[0189] The present invention improves the accuracy of target detection and recognition through a series of deep learning models and fusion algorithms, and dynamically adjusts sensor weights and fusion algorithm parameters to optimize decision accuracy. Its beneficial effect is that it achieves high-precision detection and recognition of targets and continuously optimizes system performance through feedback mechanisms.

[0190] The optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical scheme of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

[0191] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0192] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a single-chip microcomputer, a chip or a processor to execute all or part of the steps of the method described in each embodiment of the present application. In addition, the various different implementation methods of the embodiments of the present invention can also be arbitrarily combined, as long as they do not violate the ideas of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A method for dynamic object recognition and tracking based on laser radar, characterized in that: The steps include: S1: The laser radar sensor, millimeter wave radar sensor, and optical image sensor collect laser radar data, millimeter wave radar data, and optical image data respectively; Preprocess all the sensor data collected, including data denoising, filtering, and timestamp alignment; Perform sensor calibration to determine the coordinate relationship between sensors and the conversion relationship between them and the vehicle coordinate system; S2: Extract edges and corners from optical image data, extract point cloud features from lidar data, extract speed and distance information from millimeter wave radar, and use deep learning methods to enhance the expression ability of features; S3: Fuse the laser radar point cloud data with the camera image and further process the fused data, including data alignment and coordinate conversion; S4: LiDAR sensors, millimeter-wave radar sensors, and optical image sensors extract target parameters and identify different targets to form a target list, and then fuse multiple target lists to obtain reliable and true target information to avoid false alarms and missed detections; S5: Make preliminary decisions on the location, attributes and category of the target through optical image sensor data, and then use fusion strategies to comprehensively combine the decisions obtained by multiple sensors and obtain fusion results; S6: Use the Bayesian filtering method to estimate the true state of the sensor carrier based on the observed data, and estimate and update the state of the sensor carrier in real time.

2. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: In S1, a deep learning model is used to denoise the lidar data under adverse weather conditions; When training a deep learning model, an adversarial training method is used to conduct adversarial training between the generative model G and the discriminative model D. The generative model generates denoised data, and the discriminative model determines the authenticity of the data. Its loss function is: Among them, x represents the real lidar data; z represents the noise data; pdata(x) represents the distribution of real data; p z (z) represents the distribution of noise data; D(x) represents the output of the discriminant model, that is, the judgment result of the discriminator on the real data x; G(z) represents the output of the generative model, that is, the fake data generated by the generator based on the noise data z; We hope that the discriminator's judgment result on the real data is close to 1, that is, it is considered to be real data; It is hoped that the discriminator's judgment result on the generated false data is close to 0, that is, it is considered to be false data.

3. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: In step S1, a timestamp alignment method based on Kalman filtering is used to achieve time synchronization of lidar data, millimeter-wave radar data, and optical image data; The Kalman filter model is: x k|k-1 =F k × k-1 | k-1 ; x k|k =x k|k-1 +K k (z k -H k x k|k-1 ); P k|k =(I-K k H k )P k|k-1 ; Among them, x k|k-1 and x k|k is the state vector, which contains the timestamps of the data collected from various sensors, including lidar, millimeter-wave radar, and optical image sensor; x k|k-1 is based on the state x at the previous moment k-1|k-1 And the state transfer matrix F k The predicted state at the current moment, which is specifically the timestamp of the current moment predicted based on the timestamp information of the previous moment; x k|k It is the result of correcting the predicted state after taking into account the new observation data. Specifically, it is the timestamp of the current moment corrected according to the timestamp of the data actually received from the sensor. F k It is the state transfer matrix. This matrix describes how the state vector is transferred from one time step to the next time step. It specifically represents the prediction model of the timestamp. When the sensor data arrives at a fixed frequency, then F k To express this periodicity; P k|k-1 is the prediction error covariance, P k|k is the updated error covariance. These two matrices describe the uncertainty of the state estimate, which specifically represents the uncertainty of the timestamp prediction and the uncertainty of the timestamp correction after considering the observed data; Q k is the process noise covariance. This matrix describes the noise in the state transition process, which specifically represents the statistical characteristics of the time jitter or delay in the process of sensor data generation and transmission; H k The observation matrix maps the observations to the state space. It specifically represents how to extract information from the actual arrival timestamps of the sensor data to update the timestamp estimates. K k is the Kalman gain, which determines the influence of new observations in updating the state estimate. It represents the weight given to new observations when updating the timestamp estimate. Z k is the observation, which is the data actually received from the sensor, and it is the actual observation data used to update the timestamp estimate.

4. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: In step S2, a deep learning model is used to enhance the features of the lidar point cloud data and extract richer spatial features; When training the deep learning model, a 3D convolutional neural network is used to perform multi-layer convolution and pooling operations on the point cloud data; The loss function is: Where L is the loss function, which is a measure of the difference between the model prediction and the true label; y i is the true label of the i-th sample, which represents a specific feature or attribute in the point cloud data, specifically the semantic category of a certain area in the point cloud, such as pedestrians, vehicles, and buildings; is the predicted label of the i-th sample, that is, the feature or attribute predicted by the 3D-CNN model based on the input point cloud data; N is the number of samples, that is, the total number of point cloud samples in the training dataset.

5. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: In step S3, the camera image and lidar point cloud data are deeply fused by using the vision-lidar joint feature extraction method; A multimodal fusion network is used to perform multi-layer fusion operations on visual features and lidar features to generate joint features.

6. The method for dynamic object recognition and tracking based on laser radar according to claim 5, characterized in that: The multimodal fusion network in S3 uses a two-stream network architecture to process image and point cloud data separately, and then fuse them in the deep layer of the network. Batch normalization and ReLU activation function are used in the network to enhance the expressiveness of features, and early stopping and learning rate decay strategies are used to prevent overfitting.

7. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: In step S4, a multi-task learning method is used to simultaneously identify the location, attributes, and categories of the target; When training a multi-task learning model, a shared feature extraction layer and multiple task branches are used to output the location, attributes, and category of the target respectively; The loss function is: L 总 =αL 位置 +βL 属性 +γL 类别 Among them, L 总 is the total loss function, which is the weighted sum of position loss, attribute loss and category loss; L 位置 is the position loss, which measures the difference between the target position predicted by the model and the true position; L 属性 is the attribute loss, which measures the difference between the target attributes predicted by the model, including speed, size, shape, and the true attributes; L 类别 is the category loss, which measures the difference between the target category predicted by the model, including vehicles, pedestrians, and bicycles, and the true category; α, β, γ are task weights, which correspond to the importance of position, attribute, and category tasks, respectively.

8. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: In step S6, a multi-sensor fusion method is used to estimate the state of the sensor carrier in real time based on the observed data to improve the accuracy and robustness of the state estimation; The fusion algorithm is: Among them, X k is the fused state estimate, which represents the estimate of the sensor carrier state after combining all sensor data at time k; x k，i is the state estimate of the i-th sensor, which contains observation data from a lidar sensor, a millimeter-wave radar sensor, or an optical image sensor; P k is the error covariance after fusion, which describes the uncertainty of the fusion state estimate; P k，i is the error covariance of the ith sensor, which describes the uncertainty of the state estimate of the ith sensor; w i is the weight of the i-th sensor, which reflects the contribution of the i-th sensor in the fusion process. The weight is determined based on the accuracy and reliability of the sensor; M is the number of sensors, that is, the total number of sensors involved in state estimation. Here, lidar, millimeter-wave radar and optical image sensor are used, so M=3.

9. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: Prior to S6, this also included: Conditional generative adversarial networks (GANs) are used to generate environmental images directly from radar sensor data, making comprehensive use of all environmental features detected by the radar sensor. Then, the generated images are integrated with the optical images to perform S3 and evaluate the quality and availability of the environmental data after S1 fusion.

10. The method for dynamic object recognition and tracking based on laser radar according to claim 1, characterized in that: In S5, it specifically includes the following steps: S5.1: Use pre-trained deep learning models, including YOLO and SSD, to detect objects in optical image data, identify multiple categories of targets, and provide bounding box coordinates of the targets; S5.2: Extract high-level features of detected targets to capture the shape and texture of the target; S5.3: Use the shape and size characteristics of the target, combined with the known object dimensions, to estimate the speed and size of the target; S5.4: Apply support vector machine (SVM) to the extracted features to determine the category of the target; S5.5: Calculate the position of the target in the real world based on the bounding box coordinates of the target and the intrinsic parameters of the camera, and apply a perspective transformation to convert the pixel coordinates to the position in the world coordinate system; S5.6: Dynamically assign weights based on sensor accuracy and reliability; S5.7: Use the particle filter algorithm to select the fusion algorithm, input the decision information of the optical image sensor obtained in S5.4 and S5.5 and the decision result of the lidar sensor into the fusion algorithm, wherein the decision information of the optical image sensor is used as the measurement update, and the other sensor data is used as the prediction step; S5.8: Based on the fusion results, adjust sensor weights and fusion algorithm parameters to optimize decision accuracy, and design a feedback mechanism to compare the fusion results with actual observations or manually labeled data to evaluate and improve the fusion strategy.

Images