Intelligent driving environment adaptive system and method based on multi-sensor fusion

Abstract

The application belongs to the technical field of intelligent driving, and discloses an intelligent driving environment adaptive system and method based on multi-sensor fusion, which comprises the following steps: collecting driving images, average vehicle speed and road types; obtaining camera cleaning state evaluation values and image feature data from the driving images; fusing the driving images, laser radar point clouds and millimeter wave radar echoes to generate comprehensive target detection result sets and extract density feature data; obtaining consistency indexes based on cross-sensor target matching and generating conflict measurement values; inputting the cleaning state evaluation values, image feature data, vehicle speed, road types and density feature data into a comprehensive weight model to obtain a candidate weight vector; combining the conflict measurement values to constrain and optimize the candidate weight to obtain a target weight vector and update the three-sensor fusion weight; and the application realizes adaptive updating of the multi-sensor fusion weight, reduces false detection and missed detection and estimation drift, and improves the stability and safety of intelligent driving environment perception.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and more specifically, to an intelligent driving environment adaptive system and method based on multi-sensor fusion. Background Technology

[0002] With the development of intelligent driving technology, vehicle environmental perception systems typically use multiple sensors such as cameras, lidar, and millimeter-wave radar to collaboratively perceive the vehicle's surrounding environment and use the perception results as input for subsequent path planning and control decisions. Since different sensors vary in terms of target recognition, ranging, speed measurement, and environmental adaptability, multi-sensor fusion has become a common technical approach to improve the integrity and reliability of environmental perception.

[0003] However, during actual vehicle operation, the environmental perception input is not constant. On the one hand, changes in road type, vehicle speed, target distribution, lighting, occlusion interference, and lens contamination can cause variations in the detection quality of different sensors at any given moment. On the other hand, the position, speed, or target recognition results output by different sensors for the same external target may also become inconsistent with changes in the scene. In other words, multi-sensor fusion processing faces a dynamic input process that continuously changes with the driving process and whose reliability fluctuates constantly.

[0004] Existing multi-sensor fusion methods typically employ fixed fusion weights or adjust the fusion weights based on a single scene factor. While this approach can merge multi-source information, it struggles to continuously transmit environmental changes, sensor state changes, and differences in multi-source detection results to the fusion weight determination process. When the output quality of a particular sensor deteriorates or significant deviations emerge between multi-source detection results, existing solutions may still continue to perform subsequent fusion calculations according to the existing weight relationships. This results in unreliable inputs not being effectively mitigated before fusion, and multi-source differences not being constrained in a timely manner at the weight level.

[0005] In the above situation, if the output of low-reliability sensors continues to participate in the fusion processing, or if the inconsistency between the multi-source detection results continues to be used in subsequent calculations based on the original allocation relationship, it is easy to cause false detection and missed detection of external targets, as well as amplification of position and velocity estimation deviations. This can further lead to increased fluctuations in environmental perception results and decreased output stability in complex scenarios. Therefore, the existing technology lacks a processing method that can continuously adjust the fusion weights and suppress the amplification of abnormal inputs under the condition that environmental changes and differences in multi-source detection results coexist.

[0006] Therefore, how to adaptively adjust the multi-sensor fusion weights to match the current perception state in response to changes in environmental conditions, changes in sensor output quality, and inconsistencies in multi-source detection results during intelligent driving, in order to improve the stability and reliability of environmental perception results under complex road conditions, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies where the fusion weights are difficult to continuously adjust in response to changes in environmental conditions, sensor output quality, and differences in multi-source detection results during multi-sensor fusion, resulting in the difficulty in timely mitigation of low-reliability inputs, the continued amplification of inconsistent information from multiple sources, and a decline in the stability and reliability of environmental perception results, this application provides an intelligent driving environment adaptive system and method based on multi-sensor fusion. This system and method are designed for the environmental perception processing of intelligent driving vehicles under complex road conditions. Taking into account the detection results from multiple sensors such as cameras, LiDAR, and millimeter-wave radar, and combining current environmental state information, target detection result statistics, and differences between multi-source detection results, the fusion weights are adaptively determined and dynamically adjusted to improve the accuracy, consistency, stability, and reliability of environmental perception results in complex scenarios.

[0008] Firstly, this application proposes an adaptive method for intelligent driving environments based on multi-sensor fusion, including:

[0009] Collect driving images, average driving speed, and road type during the intelligent driving process of the vehicle;

[0010] The driving images are input into the cleaning status assessment model to obtain the camera cleaning status assessment value, and the driving images are subjected to feature extraction to obtain image feature data;

[0011] Based on driving images, 3D point cloud data obtained from LiDAR scanning, and echo data obtained from millimeter-wave radar scanning, a comprehensive target detection result set is generated; density feature data is then generated based on the comprehensive target detection result set.

[0012] Cross-sensor target matching is performed on the comprehensive target detection result set to obtain a cross-sensor matching triplet set. The sensor consistency index of each cross-sensor matching triplet is calculated, and a conflict metric is generated based on the sensor consistency index. The sensor consistency index includes a position deviation index and a velocity deviation index.

[0013] The comprehensive weight model is based on the camera cleanliness status assessment values, image feature data, average driving speed, and road type and density feature data. The output includes candidate weight vectors for camera, lidar, and millimeter-wave radar.

[0014] The objective function for weight optimization is constructed based on the conflict metric and candidate weight vectors. The objective weight vector is obtained by solving under preset constraints, and the weights of the camera, LiDAR and millimeter-wave radar are updated based on the objective weight vector.

[0015] Furthermore, methods for constructing a weight optimization objective function and solving for the target weight vector include:

[0016] The weight vector to be solved is composed of camera weights, LiDAR weights, and millimeter-wave radar weights. A candidate consistency penalty term is constructed between the weight vector to be solved and the candidate weight vector, a smoothing penalty term is constructed between the weight vector to be solved and the historical weight vector, and a conflict penalty term is constructed between the weight vector to be solved and the preset equilibrium reference vector, combined with the conflict metric. The weight optimization objective function is obtained by weighted summation of each penalty term. Under the constraints that the weights are non-negative, the sum of the weights is 1, the weights are within the preset upper and lower bounds, and the change in weights relative to the historical weights does not exceed the preset change threshold, the weight optimization objective function is minimized to obtain the target weight vector containing the target weights of the camera, LiDAR, and millimeter-wave radar.

[0017] Furthermore, the method for obtaining the cross-sensor matching triplet set includes:

[0018] Based on the sensor source, the comprehensive target detection result set is divided into camera target list, lidar target list and millimeter-wave radar target list;

[0019] Using the lidar target list as the matching benchmark, for each lidar target entry, candidate entries with position distances less than the target matching distance threshold and velocity differences less than the velocity matching threshold are selected from the millimeter-wave radar target list and the camera target list, respectively. The matching cost is calculated and the entry with the minimum cost is selected as the corresponding matching entry. The lidar target entries, millimeter-wave radar matching entries, and camera matching entries are constructed into a cross-sensor matching triplet set.

[0020] Furthermore, the method for calculating the sensor consistency index for each cross-sensor matching triplet and generating a conflict metric based on the sensor consistency index includes:

[0021] The spatial distance and velocity difference between the lidar reference target entry and the camera matching entry and the millimeter-wave radar matching entry are calculated respectively. The larger value of the two sets of spatial distances is determined as the position deviation index, and the larger value of the two sets of velocity differences is determined as the velocity deviation index. The position deviation index and velocity deviation index are subjected to dimension unification and normalization processing, and then weighted and summed according to the preset weight coefficients to obtain the conflict measurement value.

[0022] Furthermore, the comprehensive target detection result set includes a camera target detection result set, a lidar target detection result set, and a millimeter-wave radar target detection result set; the method for obtaining the camera target detection result set includes:

[0023] The driving image is input into the target detection network to obtain a two-dimensional detection box and a detection confidence score. The detection confidence score is used as the first target confidence score parameter. The corresponding LiDAR point cloud is projected onto the image plane, outlier removal is performed on the point cloud falling within the two-dimensional detection box, and the three-dimensional centroid is obtained. The three-dimensional centroid is transformed into the vehicle coordinate system as the first target position parameter. The target positions at adjacent time points are correlated with trajectories, and the first target velocity parameter is calculated. The first target confidence score, the first target position parameter, and the first target velocity parameter constitute the camera target detection result set.

[0024] Furthermore, the method for obtaining the set of lidar target detection results includes:

[0025] Perform effective detection spatial range filtering and voxel downsampling on the 3D point cloud data, and perform ground segmentation and Euclidean clustering on the downsampled point cloud to obtain a list of point cloud clusters;

[0026] Point cloud clusters with fewer points than the minimum point threshold are filtered out. For the remaining point cloud clusters, the three-dimensional centroid is calculated as the position parameter of the second target. The confidence parameter of the second target is calculated based on the reasonableness of the point cloud cluster's point count and bounding box size.

[0027] The position parameters of the second target at the current time are associated with the position parameters of the second target at the previous time according to the nearest neighbor distance, and the velocity parameters of the second target are calculated based on the position difference and time interval between adjacent times.

[0028] The second target's position parameters, second target's velocity parameters, and second target's confidence parameters constitute the set of lidar target detection results.

[0029] Furthermore, the method for obtaining the millimeter-wave radar target detection result set includes:

[0030] Range-dimensional Fast Fourier Transform and Doppler-dimensional Fast Fourier Transform are performed on the echo data to obtain the range-velocity spectrum;

[0031] In the range-velocity spectrum, the background noise is estimated based on the power of the reference cell and a detection threshold is generated. Cells whose power is greater than the detection threshold are identified as candidate detection cells.

[0032] Angle estimation is performed on the candidate detection units and combined with distance information to transform to the vehicle coordinate system to obtain the position parameters of the third target. The velocity parameters of the third target are calculated based on the Doppler frequency.

[0033] Candidate detection units are clustered according to spatial proximity, and the confidence parameter of the third target is calculated based on the signal-to-noise ratio, detection amplitude, and number of consecutive hits.

[0034] The third target's position parameters, velocity parameters, and confidence parameters constitute the set of millimeter-wave radar target detection results.

[0035] Furthermore, the method for obtaining the density feature data includes:

[0036] Based on the comprehensive target detection result set, extract the lidar target point cloud set corresponding to each external target, calculate the surface smoothness, reflectivity, volume, height, contour scale and reflection intensity statistics, construct the target recognition feature vector and input it into the target recognition model to obtain the pedestrian or vehicle target type;

[0037] Under highways and rural roads, the scanning unit area is calculated based on the millimeter-wave radar field of view and maximum effective scanning distance. Under urban expressways, residential roads, and mountain roads, the scanning unit area is calculated based on the lidar horizontal field of view and maximum effective scanning distance. The number of vehicles and pedestrians in the scanning area is counted, and then divided by the scanning unit area to obtain the vehicle density and pedestrian density, which constitute density feature data.

[0038] Secondly, this application proposes an intelligent driving environment adaptive system based on multi-sensor fusion, and implements the intelligent driving environment adaptive method based on multi-sensor fusion, including:

[0039] The driving information collection module is used to collect driving images, average driving speed, and road type during the intelligent driving process of the vehicle;

[0040] The image feature extraction module is used to input driving images into the cleanliness assessment model to obtain camera cleanliness assessment values, and to extract features from the driving images to obtain image feature data.

[0041] The multi-source target detection module generates a comprehensive target detection result set based on driving images, 3D point cloud data obtained from LiDAR scanning, and echo data obtained from millimeter-wave radar scanning; and generates density feature data based on the comprehensive target detection result set.

[0042] The collision measurement module performs cross-sensor target matching on the comprehensive target detection result set to obtain a set of cross-sensor matching triples. It calculates the sensor consistency index for each cross-sensor matching triple and generates a collision measurement value based on the sensor consistency index. The sensor consistency index includes a position deviation index and a velocity deviation index.

[0043] The comprehensive weight generation module inputs the comprehensive weight model based on the camera cleanliness status assessment values, image feature data, average driving speed, road type and density feature data, and outputs a candidate weight vector including camera candidate weights, lidar candidate weights and millimeter-wave radar candidate weights.

[0044] The weight optimization solution module constructs a weight optimization objective function based on the conflict metric and candidate weight vectors, solves for the target weight vector under preset constraints, and updates the weights of the camera, lidar, and millimeter-wave radar based on the target weight vector.

[0045] The technical effects and advantages of this application regarding the intelligent driving environment adaptive system and method based on multi-sensor fusion are as follows:

[0046] This application addresses the problems of fixed fusion weights being difficult to adapt to environmental changes and being easily influenced by low-reliability outputs when sensors degrade. It jointly models camera cleanliness assessment values ​​with image feature data, average vehicle speed, road type, and vehicle and pedestrian densities obtained from comprehensive target detection results. It directly outputs a candidate weight vector consisting of candidate weights for cameras, LiDAR, and millimeter-wave radar. This allows the initial weight values ​​to adaptively adjust with rain, fog, lens contamination, road type changes, and traffic density variations. This enables the full utilization of semantic information from cameras, spatial accuracy from LiDAR, and long-range speed measurement advantages from millimeter-wave radar under different operating conditions, reducing the impact of single sensor performance fluctuations on the fusion results.

[0047] Furthermore, this application uses cross-sensor target matching as the source of consistency constraints, quantifies position deviation and velocity deviation into conflict metrics, and enables the degree of inconsistency of multi-source outputs to have a calculable and comparable scalar representation. This suppresses the amplification effect of false detection, missed detection and position and velocity estimation drift on the fusion results at the source of conflict, thereby improving the robustness and safety boundary of environmental perception output.

[0048] Regarding the weighting mechanism, this application introduces the candidate weight vector and conflict metric into the weighting optimization objective function, and introduces the historical weights from the previous fusion time to form a smoothing constraint. Under the constraint that the weights are non-negative and the sum of the camera target weight, lidar target weight, and millimeter-wave radar target weight is 1, the target weight vector is solved. At the same time, a preset change threshold is used to limit the maximum change in weights between adjacent fusion times, thereby achieving a more conservative and stable weight allocation when the conflict intensifies, and maintaining continuous weight evolution when the conflict weakens. Ultimately, a closed-loop weight adaptive strategy of prior prediction, conflict suppression, and smooth update is realized, which significantly improves the stability, availability, and reliability of driving decisions in complex intelligent driving environments.

[0049] In summary, this application achieves adaptive updating of multi-sensor fusion weights by generating candidate weights driven by environmental factors and optimizing conflict metric constraints. This significantly reduces false detections, missed detections, and estimation drift during multi-sensor fusion, thereby improving the stability and safety of perception in complex intelligent driving environments. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of an intelligent driving environment adaptive system based on multi-sensor fusion according to Embodiment 1 of this application;

[0051] Figure 2 This is a flowchart of the intelligent driving environment adaptive method based on multi-sensor fusion in Embodiment 2 of this application;

[0052] Figure 3 Flowchart of the method for obtaining density feature data;

[0053] Figure 4 The flowchart shows the method for solving the target weight vector. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] Example 1:

[0056] Please see Figure 1 As shown in the figure, the intelligent driving environment adaptive system based on multi-sensor fusion described in this embodiment includes a driving information acquisition module, an image feature extraction module, a multi-source target detection module, a matching conflict measurement module, a comprehensive weight generation module, and a weight optimization solution module. Each module is connected by wires and / or wirelessly to realize data transmission.

[0057] The driving information collection module is used to collect driving images, average driving speed, and road type during the intelligent driving process of the vehicle.

[0058] It should be noted that the driving information acquisition module is used to simultaneously acquire driving images, average driving speed, and road type during intelligent vehicle driving, forming the basic input required for subsequent cleanliness assessment, target detection, and adaptive weight optimization. Driving images are acquired by the onboard camera, reading continuous frames from the camera data stream at a preset sampling frequency, and adding a timestamp and camera intrinsic / extrinsic parameter identifiers to each frame for alignment with other sensor data. Average driving speed is acquired from the vehicle speed signal. The driving information acquisition module reads the instantaneous vehicle speed values ​​output by wheel speed sensors, vehicle speed controllers, or integrated navigation odometers from the onboard network, and performs filtering and statistical summarization within a sliding time window to obtain the average driving speed. The sliding time window can be set according to the sampling period or a fixed duration, thus balancing real-time performance and noise resistance. Road type is obtained from road diagnostic results. The driving information acquisition module determines the road category by reading road attribute fields from a high-precision map or fusing lane line or curb detection results with positioning information, obtaining road type identifiers such as highways, urban roads, rural roads, and ramps, which are used to achieve adaptive adjustment for different road scenarios in subsequent weight optimization.

[0059] The image feature extraction module is used to input driving images into a cleanliness assessment model to obtain camera cleanliness assessment values, and to extract features from the driving images to obtain image feature data. The image feature data includes brightness data, contrast data, color saturation data, and light source direction.

[0060] It should be noted that the cleanliness assessment model is used to quantitatively evaluate the cleanliness of vehicle-mounted camera lenses, outputting a camera cleanliness assessment value. This value is used to adaptively adjust camera reliability in subsequent weight setting and fusion decision-making. The camera cleanliness assessment value is a normalized parameter used to quantify the cleanliness of the vehicle-mounted camera lens, ranging from 0 to 1. A higher value indicates a cleaner camera lens, and less impact from dust, water droplets, mud, oil, and fogging on the driving image. A lower value indicates a higher degree of lens contamination, and a greater risk of occlusion, blurring, abnormal brightness, or texture loss in the driving image. The camera cleanliness assessment value is output by the cleanliness assessment model based on the driving image. The cleanliness assessment model extracts image degradation features such as blurring degree, local occlusion ratio, edge sharpness, abnormal brightness distribution, and texture integrity from the driving image to comprehensively score the current camera imaging quality.

[0061] For example, when the camera lens is unobstructed and the image edges are clear, the camera cleanliness assessment value can be output as 0.90 to 1.00; when the lens has a small amount of dust or raindrops but the target outline can still be stably identified, the camera cleanliness assessment value can be output as 0.60 to 0.89; when the lens has obvious stains, raindrop coverage, or fogging that reduces the reliability of target detection, the camera cleanliness assessment value can be output as 0.00 to 0.59. By setting this parameter, this application can transform the impact of camera lens contamination on visual perception reliability into a calculable numerical input, and input it into the comprehensive weight model. This allows the subsequent candidate weight vector generation process to adaptively adjust the camera weights according to the actual imaging quality of the camera, thereby reducing the risk of false detections, missed detections, and fusion result shifts caused by lens contamination.

[0062] Furthermore, to improve the adaptability of the cleanliness assessment model to different lens contamination patterns and road conditions, preprocessing is performed on the driving image set before model training. The preprocessing includes: unifying the size of the original driving images, normalizing pixels, and screening sample quality. For example, the input images can be scaled to 224×224 pixels or 256×256 pixels, and the red, green, and blue channel values ​​can be normalized to the range of 0 to 1. Abnormal samples with severe exposure anomalies, incorrect timestamps, or inconsistencies with labeled values ​​are removed to ensure consistency between the training samples and the labeled values.

[0063] To enhance the model's generalization ability to complex degradation scenarios such as lens-attached dust, raindrops, mud, and partial occlusion, data augmentation can be performed on driving images during the training set construction phase. Data augmentation methods include brightness perturbation, contrast perturbation, slight rotation, scaling, blur perturbation, and partial occlusion perturbation. For example, the brightness perturbation coefficient can be set to 0.7 to 1.3, the contrast perturbation coefficient to 0.8 to 1.2, the rotation angle to -5° to 5°, the scaling ratio to 0.9 to 1.1, the Gaussian blur kernel standard deviation to 0.5 to 2.0, and the proportion of the partially occluded region to the image area to 1% to 15%. Through these processing steps, the model can not only learn the pollution features in real-world samples but also learn the imaging degradation patterns under varying pollution levels.

[0064] The cleanliness assessment model can employ a multi-layer convolutional feature extraction structure and a regression output structure. For example, the model includes a first, second, third, and fourth convolutional layer group connected sequentially. Each convolutional layer group includes convolution operations, normalization processing, and non-linear activation processing, followed by pooling to extract lens contamination texture, blurred edge features, occluded area morphology features, and global sharpness distribution features layer by layer. The kernel size of each convolutional layer group can be set to 3×3, and the number of output channels can be set to 32, 64, 128, and 256 respectively. Max pooling can be used, and the pooling window size can be set to 2×2. After global averaging of the feature map output from the fourth convolutional layer group, it is input into a fully connected layer, outputting a single scalar as the camera cleanliness assessment value. This single scalar can be mapped to a range of 0 to 1; a larger value indicates a cleaner lens, and a smaller value indicates more obvious lens contamination.

[0065] During the training phase, driving images from the training set are used as model input, and corresponding camera cleanliness assessment values ​​are used as supervision labels. Parameter updates are performed using regression. The training loss can be the mean squared error loss, or a weighted combination of the mean squared error loss and the mean absolute error loss, to balance overall fitting stability and robustness to outliers. For example, the mean squared error loss weight in the total loss can be set to 0.7, and the mean absolute error loss weight can be set to 0.3. Model parameter updates can use an adaptive moment estimation optimization method. The initial learning rate can be set to 0.001, the weight decay coefficient can be set to 0.0001, the batch size can be set to 16, 32, or 64, and the number of training epochs can be set to 50 to 150. To avoid oscillations in the later stages of training, the learning rate can be reduced by a preset decay coefficient (e.g., 0.1) when the validation loss no longer decreases for several consecutive epochs.

[0066] To ensure model training stability and prevent overfitting, the training set can be further divided into a validation set, and the validation loss can be calculated after each training epoch. When the validation loss does not decrease for 10 consecutive epochs, or when the decrease in validation loss is lower than a preset convergence threshold, the model is considered to have reached convergence and training can be stopped; the convergence threshold can be set to 0.0001. Simultaneously, random deactivation can be implemented before or after the fully connected layer, with the random deactivation ratio set to 0.2 to 0.5, to reduce the risk of overfitting the model to local texture noise. After training, the model parameters corresponding to the minimum validation loss are saved as fixed parameters for the deployment phase.

[0067] During the model performance evaluation phase, the mean squared error, mean absolute error, and correlation coefficient between the predicted evaluation values ​​and the manually labeled evaluation values ​​can be calculated based on the test set to verify the degree of agreement between the model output and the manually labeled results. If the mean squared error on the test set is lower than the preset mean squared error threshold and the correlation coefficient is higher than the preset correlation threshold, then the trained cleanliness assessment model is considered to meet the deployment requirements.

[0068] It should be noted that during the model performance evaluation phase, the mean squared error threshold and correlation threshold are predetermined based on the annotation scale of the cleanliness status assessment values, the consistency of manual annotation, and the statistical results of test set errors. Specifically, after the camera cleanliness status assessment values ​​are normalized to the range of 0 to 1, multiple annotators independently score the same test sample and calculate the average absolute difference between the manually annotated assessment values ​​as an acceptable annotation error reference value. Then, the upper limit of the allowable absolute error is determined based on the maximum scoring deviation allowed by the business, and the square of the upper limit of the allowable absolute error is used as the mean squared error threshold.

[0069] The correlation threshold is predetermined based on the consistency between the manually labeled results. Specifically, a predetermined number of driving images are extracted from the test set as consistency evaluation samples; for example, 300 driving images are extracted. Five labelers independently score the camera cleanliness status of each driving image under the same scoring rules, with scores ranging from 0 to 1. For each labeler, their scores for all consistency evaluation samples are arranged into a scoring sequence in sample order. The Pearson correlation coefficient between any two labelers' scoring sequences is calculated, resulting in 10 pairwise correlation coefficients. The average of these 10 pairwise correlation coefficients yields the manually labeled consistency correlation coefficient. The correlation threshold is obtained by subtracting a predetermined safety margin from the manually labeled consistency correlation coefficient. This safety margin is used to prevent the model from being misjudged as meeting deployment requirements simply because it only reaches the boundary value of manually labeled consistency.

[0070] During deployment, the solidified model parameters are written to the vehicle's storage unit. At runtime, the image feature extraction module calls the cleanliness assessment model to perform inference calculations on real-time acquired single-frame driving images or multiple consecutive frames of driving images within a sliding time window. If a continuous multi-frame input method is used, a moving average or weighted average can be applied to the assessment values ​​output from each frame to obtain the current camera cleanliness assessment value, reducing assessment fluctuations caused by occasional noise in a single frame. For example, the sliding time window length can be set to 3 to 10 frames. After obtaining the camera cleanliness assessment value, this value, along with brightness data, contrast data, color saturation data, and light source direction, is input into a subsequent comprehensive weight model as a quantitative basis for adaptive adjustment of camera reliability.

[0071] After training, the model parameters are fixed and deployed in the cleaning diagnosis module. During runtime, the driving images collected in real time are input into the cleaning status assessment model, and the camera cleaning status assessment value is output. When lens contamination causes the image quality to deteriorate, the assessment value is reduced accordingly, thus providing a quantifiable input basis for the subsequent comprehensive weight model and weight optimization solution, avoiding the problem of unstable weight setting or failure to conform to the actual imaging quality caused by relying solely on empirical thresholds.

[0072] It's important to note that in intelligent driving systems, the cleanliness of camera lenses has a significant impact on camera weighting. Lens cleanliness directly affects image quality and obstacle detection capabilities, thus requiring real-time monitoring. For example, when dust, rainwater, mud, or other debris adheres to the lens, the captured driving image will be blurry, distorted, or partially obscured, preventing accurate obstacle detection and affecting the accuracy of the obstacle perception system, ultimately impacting safety during intelligent driving.

[0073] The methods for obtaining the brightness data and color saturation include:

[0074] S101: The initial value of n is preset to 1, and the value of n ranges from 1 to N, where N is the number of pixels in the driving image.

[0075] S102: Obtain the red channel value, green channel value, and blue channel value of the nth pixel from the driving image.

[0076] S103: Normalize the red channel value, green channel value, and blue channel value to obtain the normalized red channel value, green channel value, and blue channel value.

[0077] The method for obtaining the normalized red channel value, green channel value, and blue channel value includes:

[0078] ;

[0079] ;

[0080] ;

[0081] in, This is the red channel value of the nth pixel after normalization. This represents the green channel value of the nth pixel after normalization. This is the blue channel value of the nth pixel after normalization.

[0082] S104: Use a grayscale conversion formula to convert the normalized red, green, and blue channel values ​​into the grayscale value of the nth pixel, and add the grayscale value to the grayscale value set; calculate the color saturation of the nth pixel based on the normalized red, green, and blue channel values, and add the color saturation to the color saturation set; compare the grayscale value with a preset grayscale value threshold, and if the grayscale value is greater than or equal to the grayscale value threshold, mark the pixel corresponding to the nth grayscale value as a highlight pixel and add the highlight pixel to the highlight pixel set.

[0083] It should be noted that the grayscale threshold is used to filter bright pixels from the driving image. These bright pixels are used to subsequently construct bright areas and calculate the light source direction. Since the red, green, and blue channel values ​​in this invention have been normalized, the grayscale value calculated based on the normalized channel values ​​ranges from 0 to 1, and the grayscale threshold is also set within this range. The grayscale threshold is not arbitrarily set but is predetermined based on the grayscale distribution of bright and non-bright areas in historical driving image samples.

[0084] Specifically, driving image samples covering scenarios such as daytime, nighttime, backlighting, tunnel exits, and vehicle headlight illumination are pre-collected. High-brightness areas in the samples, such as sunlight spots, headlight spots, and areas with strong reflections, are labeled, while unlabeled roads, vehicles, pedestrians, and background areas are designated as non-high-brightness areas. The grayscale distribution of pixels in both high-brightness and non-high-brightness areas is statistically analyzed. The lower quantile of the grayscale value in the high-brightness area is used as the lower limit reference value, and the higher quantile of the grayscale value in the non-high-brightness area is used as the upper limit reference value for the background area. The median value between these two values ​​is then set as the grayscale threshold. For example, when the lower quantile of the grayscale value in the high-brightness area is 0.78 and the higher quantile of the grayscale value in the non-high-brightness area is 0.62, the grayscale threshold can be set to 0.70. During online processing, when the grayscale value of a pixel is greater than or equal to 0.70, that pixel is identified as a high-brightness pixel and added to the high-brightness pixel set.

[0085] The method for obtaining the grayscale value of the nth pixel includes:

[0086] ;

[0087] in, Let be the grayscale value of the nth pixel.

[0088] It should be noted that the grayscale value is used to convert the color pixels of the driving image into a single-channel intensity representation for subsequent image quality evaluation and feature extraction. The driving image is an RGB format image, and each pixel is composed of values ​​from the red, green, and blue channels, with channel values ​​typically ranging from 0 to 255 integers. To make the grayscale value more closely approximate the human eye's perception of the differences in the brightness contribution of different colors, this application uses a weighted calculation method of the brightness components to obtain the grayscale value of the nth pixel, specifically... .

[0089] It should be noted that the coefficients 0.299, 0.587, and 0.114 in the grayscale conversion formula are preset brightness weighting coefficients, used to characterize the differences in the contribution of the red, green, and blue channels to pixel brightness. Since the human eye is sensitive to different colors at different brightness levels, with the green channel being the most sensitive, followed by the red channel, and relatively less sensitive to the blue channel, when converting an RGB color image to a grayscale image, a simple averaging of the three channels is not used. Instead, a weighted calculation is performed based on the contribution ratio of each channel to perceived brightness. In this application, the weight of the green channel is set to 0.587, the weight of the red channel is set to 0.299, and the weight of the blue channel is set to 0.114, so that the grayscale values ​​can more closely approximate the actual impact of target contours, road surface brightness variations, and light intensity on visual perception in driving images.

[0090] Furthermore, the sum of the three coefficients 0.299, 0.587, and 0.114 is 1. Therefore, when the normalized red, green, and blue channel values ​​are all within the range of 0 to 1, the calculated grayscale value remains within the range of 0 to 1. When the unnormalized channel values ​​are within the range of 0 to 255, the calculated grayscale value also remains within the corresponding grayscale level. Through the above weight settings, pixel brightness information can be stably extracted under complex lighting conditions such as backlighting, shadows, headlight illumination, and strong reflections. This avoids underestimating the contribution of the green channel to image brightness or overestimating the impact of blue channel noise on grayscale values ​​using simple averaging, thus providing a stable grayscale basis for subsequent high-brightness pixel selection, brightness data calculation, contrast data calculation, and light source direction determination.

[0091] The method for obtaining the color saturation of the nth pixel includes:

[0092] ;

[0093] in, Let be the color saturation of the nth pixel.

[0094] S105: Let n = n + 1. If n is less than or equal to N, then execute S102 to S104. If n is greater than N, then execute S106.

[0095] S106: Calculate the average gray value in the gray value set, where the average gray value is the brightness data; calculate the average color saturation in the color saturation set, where the average color saturation is the color saturation data.

[0096] The method for calculating the average grayscale value includes:

[0097] ;

[0098] in, This represents the average grayscale value.

[0099] The method for calculating the average color saturation includes:

[0100] ;

[0101] in, This represents the average color saturation.

[0102] It's important to note that if the brightness data of the driving image is too high (i.e., the image is very bright), it will be overexposed, losing detail and reducing image clarity. Conversely, if the brightness data is too low (i.e., the image is very dark), it will be underexposed, reducing edge sharpness and making it difficult to discern object details, thus also reducing image clarity. Image clarity affects the accuracy of obstacle recognition, so brightness data must be considered. In intelligent driving and camera systems, color saturation helps determine the quality of camera images. Color saturation reflects the vibrancy of colors in an image; lower saturation indicates insufficient light and less vivid colors, reducing the color difference between obstacles and the background, making it difficult for the camera to distinguish objects or obstacles. Cameras are more likely to recognize obstacles in high-saturation environments.

[0103] The method for obtaining the contrast data includes:

[0104] ;

[0105] in, For contrast data, This refers to brightness data. It's important to note that in intelligent driving systems, contrast data helps determine whether a camera can clearly identify obstacles. For example, very low contrast data indicates a lack of clear light and dark contrast in the driving image, meaning the camera is operating in complex environments such as low light or heavy fog, resulting in blurred obstacle outlines and an inability to clearly identify obstacles. Conversely, moderate or high contrast data indicates clear bright and dark areas in the image, sharp obstacle outlines, and a strong obstacle recognition capability from the camera.

[0106] The method for obtaining the direction of the light source includes:

[0107] Edge detection technology is used to construct a bright region from a set of bright pixels. Image processing techniques are then used to construct the minimum circumcircle of the bright region. The coordinates of the center of the minimum circumcircle are used as the coordinates of the light source center of the bright region, denoted as the first coordinate. The center position of the driving image is used as the center coordinate of the driving image, denoted as the second coordinate. When obtaining the light source direction based on the first and second coordinates, the horizontal coordinate difference is obtained by subtracting the horizontal coordinate of the first coordinate from the horizontal coordinate of the second coordinate. Then, the vertical coordinate difference is obtained by subtracting the vertical coordinate of the first coordinate from the vertical coordinate of the second coordinate. The horizontal and vertical coordinate differences are used as inputs for a two-parameter arctangent operation to obtain the initial direction angle radian value of the light source relative to the center of the driving image. The initial direction angle radian value is multiplied by 180 and divided by pi to convert it into an angle value. Then, 360 is added to the angle value and a modulo operation is performed on 360 to obtain the light source direction with a value range of 0° to 360°.

[0108] The light source direction is used to characterize the directional angle of the light source center relative to the center of the driving image. Under the condition that the camera is installed and calibrated and the camera optical axis is aligned with the vehicle's forward axis, the light source direction serves as an approximate representation of the light source's forward orientation relative to the vehicle. Pi is a constant, approximately equal to 3.141592654. When the first coordinate coincides with the second coordinate, or when the difference between the lateral and longitudinal coordinates is less than a preset threshold, the light source direction is recorded as 0°.

[0109] It should be noted that the preset threshold is a center offset judgment threshold, used to determine whether the coordinate offset between the center of the highlighted area and the center of the driving image can be ignored. Specifically, the first coordinate is the coordinate of the light source center obtained after processing the set of highlighted pixels through region connectivity and the minimum circumcircle, and the second coordinate is the center coordinate of the driving image; the absolute values ​​of the horizontal and vertical coordinate differences between the first and second coordinates are calculated respectively. When the absolute value of the horizontal coordinate difference is not greater than the center offset judgment threshold, and the absolute value of the vertical coordinate difference is not greater than the center offset judgment threshold, it is determined that the center of the light source and the center of the driving image are approximately coincident, and the light source direction is recorded as 0°, thereby avoiding unstable jumps in the light source direction angle caused by small pixel disturbances near the center.

[0110] The center offset judgment threshold can be preset based on the driving image resolution, camera installation calibration error, and center detection error in historical samples. For example, when the driving image resolution is 1920×1080 pixels, the center offset judgment threshold can be set to 10 pixels; when the image coordinates have been normalized to the range of 0 to 1, the center offset judgment threshold can be set to 0.01. Furthermore, the threshold can also be determined based on calibration samples: collect historical driving image samples where the light source is near the image center, count the lateral and vertical offsets of the center of the highlighted area relative to the image center, and use the larger of the 95th percentile values ​​of the two types of offsets as the center offset judgment threshold.

[0111] It's important to note that the direction of the light source has a significant impact on camera imaging. Changes in the light source's position directly affect image quality and target recognition accuracy, especially under backlighting and sidelighting conditions. When the light source direction is close to 0°, it indicates that the light source is located above the center of the driving image and close to the area in front of the vehicle. In this case, corresponding to a forward backlighting scene, the driving image is prone to overexposure, glare, and blurred object outlines, increasing the difficulty of recognizing obstacles and vehicles ahead. When the light source direction is in the range of (0°, 90°), it indicates that the light source is located to the left front of the camera; when the light source direction is in the range of [270°, 360°), it indicates that the light source is located to the right front of the camera. Under sidelighting conditions, obstacles or the sides of vehicles are prone to forming shadow areas, which reduces the camera's recognition capability, but it can still provide some effective environmental information.

[0112] Based on the aforementioned light source direction, brightness data, contrast data, color saturation data, and camera cleanliness status assessment values ​​can be combined to quantify the reliability of the camera under the current operating conditions. This provides input for the subsequent weight setting and weight optimization solution of the comprehensive weight model, enabling the multi-sensor fusion results to more accurately reflect the changes in the camera's sensing capabilities under different lighting conditions.

[0113] The multi-source target detection module generates a comprehensive target detection result set based on driving images, 3D point cloud data obtained from LiDAR scanning, and echo data obtained from millimeter-wave radar scanning; density feature data is then generated based on this comprehensive target detection result set. The comprehensive target detection result set includes camera target detection result sets, LiDAR target detection result sets, and millimeter-wave radar target detection result sets; each target detection result set in the comprehensive target detection result set contains target position parameters, target velocity parameters, and target confidence parameters. The density feature data includes vehicle density and pedestrian density.

[0114] The method for obtaining the comprehensive target detection result set includes:

[0115] The driving image is input into the target detection network to obtain a two-dimensional detection box and detection confidence score for each external target; external targets refer to the detected targets in the intelligent driving environment. The coordinates of the LiDAR point cloud corresponding to the driving image are transformed and projected onto the image plane to obtain the corresponding pixel positions of the LiDAR point cloud in the driving image. For each two-dimensional detection box, the point cloud set that falls within the two-dimensional detection box after projection is filtered, outlier removal is performed on the point cloud set, the three-dimensional centroid of the point cloud set is calculated and transformed to the vehicle coordinate system, and the three-dimensional centroid is used as the first target position parameter of the external target. Trajectory association is performed on the external target positions at adjacent time points, and the first target velocity parameter is calculated based on the trajectory position difference and time interval between adjacent time points. The detection confidence score is used as the first target confidence parameter, and together with the first target position parameter and the first target velocity parameter, a camera target detection result set is constructed.

[0116] When performing outlier removal on point cloud sets falling within the 2D detection frame, the point cloud sets are first uniformly converted to the vehicle coordinate system, and the number of points in the point cloud set is counted. When the number of points is less than the minimum point count threshold of the point cloud cluster, the point cloud set is determined to lack effective 3D support and is not included in the 3D centroid calculation. For point cloud sets with the required number of points, a preset number of neighboring points are found with each point as the center, the average spatial distance from the point to the neighboring points is calculated, and the overall mean and standard deviation of the average spatial distance of all points in the point cloud set are calculated. When the average spatial distance of a point is greater than the result of the overall mean plus the outlier determination coefficient multiplied by the standard deviation of the distance, the point is determined to be an isolated outlier and deleted.

[0117] Furthermore, to reduce the impact of background points in the 2D detection frame on the target centroid, after deleting isolated outliers, a depth consistency screening is performed on the remaining point cloud set. Specifically, the median distance of the remaining point cloud set in the forward direction of the vehicle is calculated, and the forward distance of each point is compared with the median distance; if the difference in forward distance is greater than a preset depth discretization threshold, the corresponding point is identified as a background outlier and deleted. After the above outlier removal is completed, the 3D centroid is calculated based on the remaining valid point cloud set, and the 3D centroid is used as the target position parameter of the external target, thereby avoiding the offset of target position estimation caused by isolated noise points, road surface reflection points, and background points within the detection frame.

[0118] It should be noted that the object detection network is a deep learning model used to automatically identify external targets in intelligent driving environments from driving images and output detection results. The model input is a single frame of driving image, and the model output is a number of two-dimensional detection boxes for external targets and their corresponding detection confidence scores. The two-dimensional detection boxes describe the spatial occupancy of external targets on the image plane, providing the position and scale information of external targets in the image in the form of rectangular boxes. The detection confidence scores quantify the reliability of correctly detecting external targets, reflecting the probability or reliability score of the model that the detection result is a real external target. The object detection network can adopt a single-stage or two-stage detection structure with convolutional neural networks as the backbone. Through supervised training on large-scale labeled datasets, the model learns the discriminative features of external targets such as texture, contour, and shape under different lighting, weather, occlusion, and distance conditions, thereby enabling the localization of image regions of external targets in real-time driving images.

[0119] During the training phase, a training set of driving images is pre-constructed, containing multiple frames of driving images and corresponding external target bounding boxes for each frame. These boxes are provided manually or semi-automatically and indicate the true location range of each external target within the image. The target detection network is trained using the driving images as input and the external target bounding boxes as supervision signals. By minimizing the localization error loss between the predicted detection boxes and the bounding boxes, and the target existence loss between the predicted and labeled results, the network parameters gradually converge to a configuration capable of simultaneously performing "target localization" and "target existence determination." Since the network has learned the difference features between external targets and background regions, as well as the regression rules of target boundaries, during training, when the driving images are input into the target detection network during the inference phase, the network can output a two-dimensional detection box for each external target, along with a corresponding detection confidence score. This confidence score characterizes the degree to which the region corresponding to the detection box represents a real external target. By employing a detection confidence threshold to filter low-confidence results, false detections can be further suppressed, and the stability of subsequent cross-sensor matching and fusion processing can be improved.

[0120] In the vehicle coordinate system, the 3D point cloud data acquired by the LiDAR is filtered for a preset effective detection space range to obtain effective point clouds. Voxel downsampling is then performed on the effective point clouds to obtain downsampled point clouds. Ground segmentation is performed on the downsampled point clouds to obtain non-ground point sets, and Euclidean clustering is performed on the non-ground point sets to obtain a list of point cloud clusters. For each point cloud cluster, clusters with fewer points than the minimum point count threshold are removed, resulting in retained point cloud clusters. The 3D centroid of the retained point cloud clusters is calculated and used as the second target position parameter. Simultaneously, the second target confidence parameter is calculated based on the reasonableness of the point cluster count and bounding box size. The current second target position parameter is associated with the previously output second target position parameter using nearest neighbor distance. Targets with an association distance less than a preset target matching distance threshold are identified as the same external target. For the associated external targets, the second target velocity parameter is calculated based on the target position difference and time interval between adjacent time points. The second target position parameter, second target velocity parameter, and second target confidence parameter are encapsulated to form a LiDAR target detection result set.

[0121] It should be noted that the preset effective detection space range is limited to the forward distance range, lateral distance range, and height range in the vehicle coordinate system, and only point cloud points falling within the aforementioned forward distance range, lateral distance range, and height range are retained.

[0122] For example, in intelligent driving application scenarios on urban roads and highways, taking the vehicle's driving direction as the forward axis of the vehicle coordinate system, the left and right directions as the lateral axis, and the vertical direction as the height axis, this application can set the forward distance range from 0 meters to 80 meters under normal driving conditions to cover the main following vehicles, cutting-in vehicles, and pedestrians in the current lane and adjacent lanes; the lateral distance range is set from -12 meters to +12 meters to cover the current lane, one adjacent lane on each side, and lateral targets near the road edge; the height range is set from -1 meter to +4 meters, where -1 meter to 0 meters is used to accommodate height deviations caused by road surface undulations and installation errors, and 0 meters to 4 meters is used to cover the main height of typical targets such as cars, trucks, and pedestrians and suppress the interference of non-critical high-altitude point clouds such as elevated signs and tree canopies on clustering.

[0123] For example, in high-speed or closed-road cruising scenarios, to detect vehicles and obstacles further ahead, the forward distance range can be extended to 0 to 150 meters, the lateral distance range can be maintained at -10 to +10 meters to focus on targets within the lane and reduce roadside clutter, and the height range can be maintained at -1 to +4 meters to stably cover external targets such as vehicles and pedestrians. In low-speed congested or park road scenarios, to improve the detection accuracy of dense targets at close range and reduce the amount of invalid point cloud computing, the forward distance range can be reduced to 0 to 50 meters, the lateral distance range can be set to -15 to +15 meters to cover more frequent lateral pedestrians and non-motorized vehicles, and the height range can be set to -1 to +3 meters to further suppress high-altitude background point clouds. Through the above quantifiable spatial range settings, the point cloud screening process can be executed with clear parameters in different application scenarios, thereby providing a stable and reproducible data foundation for subsequent point cloud clustering, target position and velocity estimation, and density feature calculation.

[0124] For example, to suppress the impact of isolated noise points and sparse clutter on clustering results, the minimum number of points in a point cloud cluster can be set to 30. When a point cloud cluster contains fewer than 30 points, it is determined that the cluster does not meet the minimum geometric support condition for external targets and is therefore removed. For scenarios where point clouds of distant targets are sparser, this threshold can be lowered to 20 points without changing the removal rules to improve the retention rate of distant targets. For scenarios where rain, fog, dust, or reflected clutter lead to an increase in noise clusters, this threshold can be raised to 40 points to enhance noise resistance and reduce false detections.

[0125] The target matching distance threshold refers to the maximum distance threshold used to determine whether two targets belong to the same object across multiple frames of sensor data. For example, the target matching distance threshold is adjusted based on environmental characteristics and vehicle driving status in different driving scenarios. In urban road scenarios, due to the short distance between targets and low vehicle speed, the target matching distance threshold is set to 2 meters to ensure accurate matching of targets in adjacent frames at low speeds, avoiding false associations. In highway scenarios, due to the long distance between vehicles and high vehicle speed, the target matching distance threshold can be increased to 5 meters to ensure accurate matching of targets ahead with targets in the current frame at higher speeds, avoiding false matching caused by high target speeds. In special environments such as tunnels or complex weather conditions, the accuracy of target recognition may be affected; to improve the stability of target matching, the target matching distance threshold can be appropriately increased to 6 meters. The target matching distance threshold can be flexibly adjusted according to these different driving scenarios to ensure high accuracy and high reliability of target detection.

[0126] It should be noted that, to ensure the feasibility and consistency of trajectory association between external targets at adjacent time points, this application adopts the nearest neighbor distance rule when associating external targets at adjacent time points. The "nearest neighbor distance" refers to the Euclidean distance calculated in the vehicle coordinate system between the position parameters of the second external target at the current time and the position parameters of the second external target at the previous time point. The target pair with the smallest Euclidean distance among the target pairs satisfying the candidate constraints is selected as the same external target. The candidate constraints are used to avoid erroneous association in dense target scenarios, specifically including: the positional distance between the external target at the current time point and the external target at the previous time point is less than the target matching distance threshold, and the velocity difference between the external target at the current time point and the external target at the previous time point is less than the velocity matching threshold; when a previous external target has already been associated, an occupancy mark is set for that previous external target, preventing it from participating in the nearest neighbor selection of other external targets at the current time point, thus forming a one-to-one association relationship. When there is no external target in the previous time step that meets the candidate constraint conditions, the external target in the current time step is determined as a newly appeared target and its trajectory is initialized; when the external target in the previous time step is not associated with any target in the current time step, the external target in the previous time step is recorded as short-term lost and its trajectory state is retained within a preset number of retention frames to improve the association stability under occlusion and short-term missed detection conditions.

[0127] Methods for calculating the confidence parameters of the second objective based on the reasonableness of point cloud cluster points and bounding box size include:

[0128] For retained point cloud clusters, the number of points in the cluster is mapped to a point score: when the number of points in a cluster reaches the saturation value, the point score is 1; when the number of points in a cluster is between the minimum point count threshold and the saturation value, the point score increases linearly and is limited to between 0 and 1. For example, if the minimum point count threshold is 30 and the saturation value is 120, then the point score is 0 when the number of points in a cluster is 30, 1 when the number of points in a cluster is 120, and 0.5 when the number of points in a cluster is 75.

[0129] The bounding box size rationality score is calculated by: calculating the 3D bounding box of the point cloud cluster in the vehicle coordinate system, obtaining the bounding box length, bounding box width, and bounding box height; and pre-setting the effective size range of the external target, including the lower and upper limits of length, width, and height. The size rationality score is calculated using the method of "full marks for within the range, proportional penalty for exceeding the limits": when the bounding box length falls within the preset length range, the length score is 1; when the bounding box length is less than the lower length limit, the length score is the bounding box length divided by the lower length limit; and when the bounding box length is greater than the upper length limit, the length score is the upper length limit divided by the bounding box length. The bounding box width score and bounding box height score are calculated according to the same rules. The three scores are multiplied together to obtain the size rationality score, which is limited to between 0 and 1. For example, if the length ranges from 0.3 meters to 12 meters, the width ranges from 0.3 meters to 4 meters, and the height ranges from 0.3 meters to 4 meters, then when the length of the enclosure box is 6 meters, the width is 1.8 meters, and the height is 1.6 meters, all three items score 1; when the height of the enclosure box is 6 meters, the height score is 4 / 6, thus the overall size reasonableness score decreases as the degree of deviation decreases.

[0130] The second target confidence parameter is jointly determined by the point count score and the size reasonableness score, and is expressed as a product with a limit between 0 and 1. The product form ensures that the confidence level will decrease in two cases: insufficient point cloud clusters leading to unreliable geometric support, and bounding box sizes that are significantly inconsistent with the road participant scale. This suppresses noisy and anomalous clusters from entering the subsequent cross-sensor matching and conflict measurement calculation process without introducing additional classification information.

[0131] The range spectrum is obtained by performing a range-dimensional Fast Fourier Transform (FFT) on the echo data acquired by the millimeter-wave radar. The range spectrum is then subjected to a Doppler-dimensional FFT along a slow time series to obtain the range-velocity spectrum. The range-dimensional FFT is performed on the sequence of sampling points within a single transmission cycle, while the Doppler-dimensional FFT is performed on a slow time series consisting of multiple transmission cycles.

[0132] It should be noted that in millimeter-wave radar systems, a slow time series refers to a set of echo sampling sequences obtained for the same range cell over multiple consecutive transmission cycles within the same observation period. Specifically, millimeter-wave radar typically transmits multiple frequency-modulated pulses at a fixed repetition period, also known as multiple transmission cycles, and a range spectrum is obtained for each transmission cycle. Arranging the complex echo values ​​corresponding to the 1st, 2nd, and subsequent transmission cycles for the same range cell in chronological order constitutes the slow time series for that range cell. Due to the relative radial motion between the target and the radar, the phase of this slow time series changes regularly with the transmission cycle. This phase change corresponds to Doppler frequency information. Therefore, performing a Doppler-dimensional fast Fourier transform on the slow time series yields the velocity distribution for that range cell, which is then combined with the range dimension result to form a range-velocity spectrum for subsequent candidate detection cell extraction and third target velocity parameter calculation.

[0133] In the distance-velocity spectrum, a protection unit and a reference unit are defined with the unit to be detected as the center. The reference unit is the neighboring unit excluding the protection unit. The background noise estimate is obtained by averaging the power of the reference unit. The background noise estimate is multiplied by the threshold coefficient to obtain the detection threshold. When the power of the unit to be detected is greater than the detection threshold, it is determined as a candidate detection unit.

[0134] It should be noted that, during the extraction of candidate detection units from the range-velocity spectrum, to avoid the impact of target energy leakage on background noise statistics, this application sets up a protection unit around the unit to be detected and removes the protection unit from the noise estimation range. Specifically, a protection unit range is defined in both the range and velocity dimensions, centered on the unit to be detected. Units within the protection unit range do not participate in background noise estimation. Subsequently, a reference unit range is defined outside the protection unit. Units within the reference unit range serve as background noise statistical samples, and the power of the reference unit is averaged to obtain the estimated background noise value. A detection threshold is calculated based on the estimated background noise value and a preset threshold coefficient. The power of the unit to be detected is compared with the detection threshold. When the power of the unit to be detected is greater than the detection threshold, the unit to be detected is identified as a candidate detection unit. By setting up protection units and excluding their participation in noise estimation, it is possible to suppress the entry of target main lobe or side lobe energy into noise statistics, which could lead to an increase in the threshold, thereby reducing the risk of missed detection and improving the stability of candidate detection unit extraction.

[0135] Furthermore, the threshold coefficient is used to amplify the background noise estimate obtained from the reference units to a detection threshold, which is equal to the background noise estimate multiplied by the threshold coefficient. The threshold coefficient is set in relation to the preset false alarm probability and the number of reference units, enabling control of the false alarm rate of candidate detection units at a fixed probability level. Taking a power spectrum detection with 32 reference units as an example, when the preset false alarm probability is set to 0.1%, the threshold coefficient is approximately 7.71; when the preset false alarm probability is set to 0.01%, the threshold coefficient is approximately 10.67; and when the preset false alarm probability is set to 0.1%, the threshold coefficient is approximately 13.86. In engineering implementation, the system can fix the number of reference units at 32 and configure the threshold coefficient to 10.67 to meet the detection requirement of a false alarm probability of 0.01%, thereby stably outputting candidate detection units on the range-velocity spectrum and reducing false detections caused by random noise.

[0136] Angle estimation is performed on candidate detection units to obtain azimuth and elevation angles, and the range, azimuth, and elevation angles are transformed into the vehicle coordinate system to obtain the third target position parameters. Radial velocity is calculated based on the Doppler frequency corresponding to the candidate detection units, and the radial velocity is decomposed into the vehicle coordinate system by combining the azimuth and elevation angles to obtain the third target velocity parameters. Candidate detection units are clustered according to spatial proximity, and the third target position parameters are stabilized by the cluster centroid. The third target confidence parameters are calculated based on the signal-to-noise ratio, detection amplitude, and number of consecutive hits. The third target position parameters, third target velocity parameters, and third target confidence parameters are encapsulated to form a set of millimeter-wave radar target detection results.

[0137] It should be noted that the millimeter-wave radar target confidence parameter is used to quantify the degree to which the candidate target is a real external target and the measurement is reliable. This application limits the target confidence parameter to the range of 0 to 1 and calculates it based on the signal-to-noise ratio, detection amplitude, and number of consecutive hits corresponding to the same candidate detection unit. In specific implementation, candidate detection units are first extracted from the range-velocity spectrum and the background noise estimate calculated by the reference unit is obtained. The ratio of the candidate detection unit power to the background noise estimate is converted into a signal-to-noise ratio. Scoring begins when the signal-to-noise ratio is not less than 6 dB, and the scoring saturates to 1 when the signal-to-noise ratio reaches 18 dB. When the signal-to-noise ratio is between 6 dB and 18 dB, it is linearly mapped to the range of 0 to 1, and the score for the signal-to-noise ratio below 6 dB is set to 0.

[0138] Subsequently, the detection amplitude score is calculated, which is determined based on the amplitude gain of the candidate detection unit's amplitude relative to the estimated background noise amplitude. Specifically, scoring begins when the amplitude gain relative to the estimated background noise is not less than 3 dB, saturates to 1 when the amplitude gain reaches 15 dB, and is linearly mapped to 0 to 1 when the amplitude gain is between 3 dB and 15 dB. Detection amplitudes below 3 dB are set to 0. Next, the consecutive hit count score is calculated. The multi-source target detection module maintains a millimeter-wave radar target trajectory table, associating the current candidate target with the target from the previous time step according to a target matching distance threshold. Successful association increments the consecutive hit count for that target; unsuccessful association sets the consecutive hit count to 1. The score is set to 0 when the consecutive hit count is 1, saturates to 1 when the consecutive hit count reaches 5, and is linearly mapped to 0 to 1 when the consecutive hit count is between 1 and 5.

[0139] Finally, the three scores are weighted and summed according to fixed weights to obtain the third target confidence parameter, where the signal-to-noise ratio score has a weight of 0.4, the detection amplitude score has a weight of 0.4, and the consecutive hit count score has a weight of 0.2. The calculation result is truncated to the range of 0 to 1 and output. When the third target confidence parameter is lower than 0.3, the corresponding target is removed from the millimeter-wave radar target detection result set to suppress false detections caused by random noise and occasional clutter.

[0140] The camera target detection result set, lidar target detection result set, and millimeter-wave radar target detection result set are uniformly formatted and summarized to generate a comprehensive target detection result set.

[0141] like Figure 3 As shown, the method for obtaining the density feature data includes:

[0142] R external targets are identified based on the comprehensive target detection result set, and the corresponding target point cloud set is extracted for each external target from the lidar 3D point cloud data. The surface smoothness, reflectivity, volume, height, contour scale and reflection intensity statistics of each external target are calculated based on the target point cloud set and constructed into a target recognition feature vector.

[0143] It should be noted that, for each point in the target point cloud set, its neighborhood point set is calculated and a local plane is fitted. The residuals from each point to the local plane are statistically analyzed, and the average residual is taken as the surface roughness. Surface smoothness is then defined as a preset constant minus the surface roughness, thus achieving a definite mapping where the smaller the residual, the smoother the surface. Reflectivity and reflection intensity statistics are obtained from the point cloud intensity field. Specifically, the intensity values ​​of the target point cloud set are normalized, and the mean intensity, variance intensity, maximum intensity, and intensity quantile are calculated. The mean intensity is used as the reflectivity index, and the mean intensity and variance intensity statistics are combined as the reflection intensity statistics output. Height is obtained from the difference between the maximum and minimum values ​​of the target point cloud set in the height direction of the vehicle coordinate system. Contour scale is obtained from the projection range of the target point cloud set in the forward and lateral directions of the vehicle coordinate system. Specifically, the maximum projection distance minus the minimum projection distance in the forward direction is calculated as the length scale, and the maximum projection distance minus the minimum projection distance in the lateral direction is calculated as the width scale. The length and width scales are output as the contour scale. The volume is calculated from a 3D bounding box. Specifically, a 3D axis-aligned bounding box is fitted based on the target point cloud point set. The length, width, and height of the bounding box are read, and the product of the three is used as the volume feature.

[0144] The target recognition feature vectors corresponding to R external targets are input into the target recognition model to obtain the target types corresponding to the R external targets; the target types include pedestrians and vehicles; the R target types include One vehicle and A pedestrian, and The sum equals R. and All are positive integers greater than 0; to ensure that the discrimination rule is determined and reproducible, when the target recognition model outputs a vehicle, the external target is included in the vehicle count; when the target recognition model outputs a pedestrian, the external target is included in the pedestrian count; when the target recognition model outputs neither a vehicle nor a pedestrian, the external target is not included in the density statistics.

[0145] It should be noted that the target recognition model maps the target recognition feature vectors corresponding to external targets to target types, enabling the differentiation and counting of vehicle and pedestrian targets during density feature data calculation. The target recognition feature vector is obtained by concatenating the surface smoothness, reflectivity, volume, height, contour scale, and reflectivity statistics corresponding to the same external target in a preset order. Before being input into the model, it undergoes dimensional unification and normalization to eliminate the impact of differences in feature value ranges on model training stability. The output of the target recognition model is a target type label, which includes at least two categories: vehicles and pedestrians. Model training is performed using supervised learning, enabling the model to learn the correspondence between point cloud geometry and reflectivity characteristics and target categories.

[0146] Furthermore, the training method for the target recognition model includes training data construction, feature preprocessing, model training, model validation, and model deployment. The target recognition model receives target recognition feature vectors corresponding to external targets and outputs the target type corresponding to the external targets. Target types include vehicles, pedestrians, and other targets; when the target recognition model outputs "vehicle," the corresponding external target is included in the vehicle count; when the target recognition model outputs "pedestrian," the corresponding external target is included in the pedestrian count; when the target recognition model outputs "other targets," the corresponding external target is not included in the density statistics.

[0147] In the training data construction step, multi-scene LiDAR 3D point cloud data are pre-collected, and the true category labels of external targets are acquired simultaneously. The collected scenes include urban roads, highways, rural roads, residential roads, and mountain roads. For each frame of 3D point cloud data, external target point cloud sets are extracted following an effective spatial filtering, clustering segmentation, and bounding box fitting process consistent with online processing. Then, for each external target point cloud set, surface smoothness, reflectivity, volume, height, contour length, contour width, mean reflection intensity, variance of reflection intensity, and maximum reflection intensity are calculated to form the target recognition feature vector corresponding to the external target. For example, the target recognition feature vector can be set as a 9-dimensional vector, with the 9 features arranged sequentially in the order of surface smoothness, reflectivity, volume, height, contour length, contour width, mean reflection intensity, variance of reflection intensity, and maximum reflection intensity.

[0148] During the training sample selection process, to reduce the impact of abnormal point clouds and mislabeled data on model training, only external target samples with at least 20 point clouds are retained, while abnormal samples with abnormal bounding box sizes, missing reflection intensity, and inconsistent category labels are removed. To avoid bias in model training towards the category with a larger number of samples, balanced sampling is performed on vehicle samples, pedestrian samples, and other target samples. For example, the number of samples from the three categories can be constructed in a 1:1:1 ratio to build the training dataset; when the original sample size is unbalanced, undersampling or oversampling can be used for adjustment.

[0149] In the feature preprocessing step, the target recognition feature vectors undergo dimensional unification and normalization. For example, the mean and standard deviation of each feature dimension can be calculated based on the training set, and standardized by subtracting the mean and then dividing by the standard deviation. For feature dimensions with a standard deviation less than a preset lower limit, the standard deviation is replaced with 1 to avoid division by zero. After training, the mean and standard deviation parameters corresponding to each feature dimension are saved along with the target recognition model parameters to ensure that the same feature transformation method is used in the online inference stage and the training stage.

[0150] In the model training step, a multilayer perceptron classification network is selected as the target recognition model. The target recognition model includes an input layer, hidden layers, and an output layer. The input layer receives a 9-dimensional target recognition feature vector, the hidden layers extract the nonlinear combination relationship between different features, and the output layer outputs the probabilities of vehicles, pedestrians, and other targets. For example, the hidden layers can be set to 3 layers, with the first hidden layer having 64 neurons, the second hidden layer having 32 neurons, and the third hidden layer having 16 neurons. The activation function of each hidden layer is a linear rectified function, and random deactivation is set after the hidden layers, with the random deactivation ratio set to 0.2 to 0.4. The output layer has 3 neurons, and the output layer uses a normalized exponential function to obtain the probability values ​​corresponding to the 3 types of targets.

[0151] During the training phase, the training samples are divided into a training set, a validation set, and a test set. For example, the proportions of the training, validation, and test sets can be set to 70%, 15%, and 15%, respectively. During training, the target type label is used as a supervision signal, the cross-entropy loss function is used to measure the difference between the model's output probability and the true label, and an adaptive moment estimation optimization method is used to update the model parameters. For example, the initial learning rate can be set to 0.001, the batch size can be set to 64 or 128, the weight decay coefficient can be set to 0.0005, and the number of training epochs can be set to 80 to 150 epochs. To improve training stability, the learning rate can be multiplied by 0.5 for decay if the validation set loss does not decrease for 8 consecutive epochs, and the minimum learning rate can be set to 0.00001.

[0152] During model convergence control, the validation set loss and validation set classification accuracy are recorded after each training epoch. The model is considered convergent and training is stopped when the validation set loss does not decrease for 12 consecutive epochs, or when the difference between the current validation set loss and the loss of the most recent epoch is less than 0.0001. After training, the model parameters corresponding to the minimum validation set loss are saved as the fixed parameters of the target recognition model.

[0153] In the model validation step, classification accuracy, vehicle recall, pedestrian recall, and macro-average F1 score are calculated based on the test set to verify the object recognition model's ability to distinguish between vehicle and pedestrian targets. For example, if the test set classification accuracy is not lower than 0.90, the vehicle recall is not lower than 0.88, the pedestrian recall is not lower than 0.85, and the macro-average F1 score is not lower than 0.86, the object recognition model can be considered to meet the deployment requirements. If the test results do not meet the preset requirements, the process returns to the training data construction step or the model training step, and the sample ratio, number of network layers, or training parameters are readjusted before retraining.

[0154] In the model deployment step, the solidified target recognition model parameters, mean parameters of each feature dimension, and standard deviation parameters of each feature dimension are written into the vehicle-side storage unit. During runtime, for each external target point cloud set, the target recognition feature vector is calculated according to the feature extraction order consistent with the training phase. Then, standardization processing is performed according to the saved mean and standard deviation parameters, and the results are input into the target recognition model, outputting vehicle probability, pedestrian probability, and other target probabilities. The category with the highest probability is taken as the target type of the current external target, and vehicle density and pedestrian density statistics are performed accordingly. This ensures that the generation process of density feature data has a clear and reproducible training path and online inference path.

[0155] The statistical region is adaptively selected based on road type, and the scanning unit area is calculated. When the road type is a highway or rural road, the field of view and maximum effective scanning distance from the millimeter-wave radar's factory parameters are read, and the scanning unit area is calculated based on a sector area, only counting external targets within the sector area. When the road type is an urban expressway, residential road, or mountain road, the horizontal field of view and maximum effective scanning distance from the lidar's factory parameters are read, and the scanning unit area is calculated based on a sector area, only counting external targets within the sector area. After defining the statistical region, the vehicle count is divided by the scanning unit area to obtain the vehicle density, and the pedestrian count is divided by the scanning unit area to obtain the pedestrian density. The vehicle density and pedestrian density together constitute density feature data and are output for subsequent comprehensive weight model weight adaptive adjustment in scenarios with dense traffic participants.

[0156] It should be noted that the coverage areas of lidar and millimeter-wave radar differ when calculating vehicle and pedestrian density. Therefore, when scanning a unit area, it is necessary to determine the specific road type to improve the accuracy and effectiveness of the calculation. LiDAR has higher resolution and accuracy, suitable for detecting the precise shape and position of surrounding objects. LiDAR typically has a narrower field of view, but provides more detailed information within that range, making it more suitable for urban expressways, residential roads, and mountain roads. In contrast, millimeter-wave radar has a wider coverage area and is suitable for long-range detection, making it more suitable for highways and rural roads.

[0157] The conflict measurement module performs cross-sensor target matching on the comprehensive target detection result set to obtain a cross-sensor matching triplet set, calculates the sensor consistency index for each cross-sensor matching triplet, and generates a conflict measurement value based on the sensor consistency index; the sensor consistency index includes position deviation index and velocity deviation index.

[0158] The method for obtaining the cross-sensor matching triplet set includes:

[0159] The comprehensive target detection results are split into camera target lists, lidar target lists, and millimeter-wave radar target lists based on the sensor source.

[0160] It should be noted that the comprehensive target detection result set is a summary of the target detection outputs of each sensor within the fusion time. The fusion time is the time reference point for the unified alignment and fusion processing of multi-sensor data. The set contains multiple target entries, each corresponding to a detected external target in the intelligent driving environment, such as a vehicle ahead, a vehicle in an adjacent lane, or a pedestrian. To facilitate subsequent cross-sensor target matching and consistency calculation, this application writes a sensor source identifier for each target entry when generating it, indicating that the target entry was generated by a camera, LiDAR, or millimeter-wave radar. Based on the sensor source identifier, the matching conflict measurement module groups the target entries in the comprehensive target detection result set according to their source, forming a camera target list, a LiDAR target list, and a millimeter-wave radar target list. Each list is a set of multiple external target entries detected by the corresponding sensor within the same fusion time. "Multiple" in the list refers to multiple target entries rather than multiple sensor devices, thus enabling the subsequent matching process to be performed under the unified caliber of using LiDAR target entries as the reference target entries and camera and millimeter-wave radar target entries as target entries to be matched.

[0161] The lidar target list is set as the matching benchmark list, and the lidar target entries in the lidar target list are traversed one by one; the lidar target entry currently being traversed is recorded as the lidar benchmark target entry. For each lidar benchmark target entry, candidate millimeter-wave radar target entries are filtered from the millimeter-wave radar target list. The filtering conditions are that the position distance between the target position parameters is less than the target matching distance threshold and the velocity difference between the target velocity parameters is less than the velocity matching threshold. The matching cost is calculated for the candidate millimeter-wave radar target entries that meet the filtering conditions. The matching cost is obtained by weighting the position distance and velocity difference according to preset weights, and the millimeter-wave radar target entry with the smallest matching cost is selected as the millimeter-wave radar matching entry corresponding to the lidar benchmark target entry. When there is no candidate millimeter-wave radar target entry that meets the filtering conditions, a millimeter-wave radar unmatched mark is recorded.

[0162] The same rules are used to filter and determine the camera matching entries for the LiDAR reference target entries in the camera target list; when there are no candidate camera target entries that meet the filtering conditions, a camera unmatched mark is recorded.

[0163] The method for filtering and determining matching camera entries for the LiDAR baseline target entry in the camera target list using the same rules includes: reading the target position parameters and target velocity parameters of the current LiDAR baseline target entry, and iterating through each camera target entry in the camera target list; for each camera target entry, calculating the position distance and velocity difference between the camera target entry and the LiDAR baseline target entry, and comparing the position distance and velocity difference with the target matching distance threshold and velocity matching threshold, respectively; when the position distance is less than the target matching distance threshold and the velocity difference is less than the velocity matching threshold, adding the camera target entry to the candidate camera set; when the position distance does not meet the threshold or the velocity difference does not meet the threshold, directly excluding the camera target entry from the candidate set to reduce the probability of cross-target misassociation. A matching cost is calculated for each candidate camera entry in the candidate camera set, and the matching cost is obtained by weighting the position distance and velocity difference according to the matching cost weight coefficient.

[0164] The module selects the candidate camera entry with the lowest matching cost from the candidate camera set as the camera matching entry and sets an occupancy mark for this camera matching entry so that it will not be repeatedly matched by other lidar reference target entries at the same fusion time, thereby avoiding the confusion of matching relationships caused by "one-to-many" or "many-to-one". When multiple candidate camera entries have the same or very close matching costs, the module prioritizes the candidate camera entry with the higher target confidence parameter as the camera matching entry to improve the reliability of the matching results.

[0165] Each lidar target entry in the lidar target list is traversed one by one. For each lidar target entry, corresponding millimeter-wave radar and camera matching entries are determined in both the millimeter-wave radar and camera target lists. Centered on the current lidar target entry, the lidar target entry, the millimeter-wave radar matching entry, and the camera matching entry are combined and encapsulated into a cross-sensor matching triplet. This triplet represents the correspondence of the same external target in the outputs of the three types of sensors. Through this construction method, each lidar target entry in the lidar target list generates a corresponding cross-sensor matching triplet. All cross-sensor matching triplets are then combined into a set. This provides a one-to-one input unit for subsequent calculations of position deviation, velocity deviation, and collision metrics based on the cross-sensor matching triplets.

[0166] The method for calculating the matching cost includes:

[0167] ;

[0168] in, For matching cost, The location cost weighting coefficient, For speed cost weighting coefficient, Let Euclidean distance be the distance between the position parameters of the two targets. The absolute value of the difference between the two target velocity parameters. The location scale parameter can be set to a target matching distance threshold. For the velocity scale parameter, a velocity matching threshold can be selected. and The sum is 1. The formula for calculating the matching cost is a normalized and then weighted form.

[0169] For example, in urban road scenarios with low speeds and dense target density, to suppress misassociations of adjacent targets, stronger positional constraints are configured, taking... Equal to 2m, Equals 3m / s, and takes It is 0.7. The value is 0.3. In high-speed cruising scenarios on highways, the target distance is larger and the speed difference is more discriminative, so we take... Equal to 5m, It equals 8 m / s, and takes It is 0.5. The value is 0.5. In scenarios where echo quality fluctuations cause position jitter, such as rain, fog, and tunnels, to improve matching recall and utilize speed stability, the value is... Equal to 6m It equals 10 m / s, and takes It is 0.4. The value is 0.6. The above weighting coefficients can be written into the configuration table according to road type and average driving speed, so that the matching cost calculation has a clear formula and parameter examples.

[0170] It should be noted that the speed matching threshold is used to limit the speed consistency requirements when matching targets across sensors. Specifically, a target is allowed to enter the candidate matching set only when the absolute value of the difference between the target speed parameter of the lidar reference target entry and the target speed parameter of the candidate sensor target entry is less than the speed matching threshold, thereby reducing the probability of different targets being misassociated. The speed scalar in the vehicle coordinate system is used as the comparison caliber. The speed scalar refers to the magnitude of the target speed parameter. In low-speed driving scenarios on urban roads, considering the effects of sensor speed measurement noise and short-term acceleration and deceleration, the speed matching threshold is set to 3 meters per second to allow reasonable speed fluctuations of the same target in adjacent frames while suppressing mismatches of different targets. In high-speed cruising scenarios on highways, the relative speed of the target changes more significantly and the radar speed measurement is more stable. The speed matching threshold is set to 8 meters per second to cover the speed differences of the same vehicle during lane changes, acceleration, and following, and to maintain stable matching. In scenarios with degraded echo quality, such as tunnels and rain / fog, the speed matching threshold is set to 10 meters per second to improve the matching recall rate and avoid matching failures due to speed measurement jitter. After establishing the correspondence between the speed matching threshold and road type and vehicle speed range, it can be written into the configuration table as a preset parameter of the matching conflict measurement module, thereby ensuring that the cross-sensor target matching rule has clear data examples and reproducible execution criteria.

[0171] Methods for calculating the sensor consistency index for each cross-sensor matching triplet and generating conflict metrics based on the sensor consistency index include:

[0172] It should be noted that the conflict metric is a numerical parameter used to quantify the degree of inconsistency in the detection results of different sensors for the same external target within the same fusion time. The conflict metric is generated based on the position deviation and velocity deviation indices in the cross-sensor matching triplet, and its value range can be normalized to 0 to 1. The closer the conflict metric is to 0, the more consistent the position and velocity parameters output by the camera, LiDAR, and millimeter-wave radar for the same external target, and the lower the degree of conflict in the multi-source detection results. Conversely, the closer the conflict metric is to 1, the greater the difference in the position and velocity parameters output by different sensors, and the higher the degree of conflict in the multi-source detection results. The conflict metric is used as a real-time constraint parameter in the subsequent weight optimization objective function, enabling the weight optimization process to reduce the pulling effect of abnormal output from a single sensor on the fusion result when cross-sensor output conflict increases, and to improve the stability of the multi-sensor fusion sensing results.

[0173] The spatial distance between the lidar reference target entry and the camera matching entry is taken as the first position deviation, and the spatial distance between the lidar reference target entry and the millimeter-wave radar matching entry is taken as the second position deviation. The maximum value between the first and second position deviations is taken as the position deviation index. The velocity difference between the lidar reference target entry and the camera matching entry is taken as the first velocity deviation, and the velocity difference between the lidar reference target entry and the millimeter-wave radar matching entry is taken as the second velocity deviation. The maximum value between the first and second velocity deviations is taken as the velocity deviation index.

[0174] The position deviation index and velocity deviation index are unified in terms of dimensions and normalized in terms of amplitude to obtain normalized position deviation index and normalized velocity deviation index; both normalized position deviation index and normalized velocity deviation index fall within [0,1]; the normalized position deviation index and normalized velocity deviation index are weighted and summed according to preset weight coefficients to obtain the conflict metric value, thereby realizing the unified quantification of "position inconsistency and velocity inconsistency". The larger the conflict metric value, the stronger the cross-sensor conflict.

[0175] The method for calculating the conflict metric includes:

[0176] ;

[0177] in, For conflict metrics, and For the corresponding weighting coefficients, For location weighting coefficients, For speed weighting coefficients, and The sum of them is 1, which is... As a positional deviation index, As a speed deviation index, These are preset position scale parameters. This refers to a preset velocity scale parameter. For example, in the implementation, A target matching distance threshold can be set. A speed matching threshold is acceptable.

[0178] It should be noted that in the calculation of the conflict metric, the position weight coefficient and the velocity weight coefficient are used to reflect the difference in the importance of current position consistency and current velocity consistency to the cross-sensor conflict judgment. Combining the multi-sensor fusion and adaptive weight update link of this application, the weight coefficients can be configured in a scalable manner according to road type, average driving speed, and density feature data: In urban road low-speed and target-dense scenarios, the distance between external targets is small and occlusion is frequent, making position errors more likely to cause false associations and misjudgments. Therefore, a position weight coefficient of 0.7 and a speed weight coefficient of 0.3 are adopted to make the conflict metric more sensitive to position deviation. In high-speed cruising scenarios on highways, the distance between targets is large and relative speed changes are more discriminative. A position weight coefficient of 0.5 and a speed weight coefficient of 0.5 are adopted to balance the contributions of position consistency and speed consistency. In scenarios with visual degradation or increased point cloud jitter, such as rain, fog, and tunnels, position measurement fluctuations increase while millimeter-wave radar speed measurement is more stable. To improve the stability of conflict discrimination, a position weight coefficient of 0.4 and a speed weight coefficient of 0.6 are adopted to make the conflict metric more dependent on speed consistency and suppress false triggering caused by position noise. The above weight coefficients can be written into a configuration table and correspond to road type and average driving speed, thus providing clear and implementable parameter examples for the conflict metric calculation rules.

[0179] The conflict metric is directly related to the technical problem addressed in this application. This application targets adaptive scenarios for intelligent driving environments using multi-sensor fusion. The core technical problem is that different sensors may produce inconsistent detection results under conditions such as lens contamination, rain / fog obstruction, strong glare, sparse point clouds, and radar multipath propagation. If fixed weights are still used for fusion, the fusion result is easily influenced by low-reliability sensors, leading to false detections, missed detections, or deviations in target position and velocity estimation, thereby reducing the stability and safety of environmental perception. The conflict metric, by simultaneously quantifying position and velocity deviation indicators, transforms the degree of inconsistency across sensors into an optimizable scalar input. This allows the weight optimization objective function to explicitly constrain moments of strong conflict as a penalty term, thereby reducing the fusion weights of inconsistent sensors when conflict increases and restoring weight balance when conflict decreases. Ultimately, this achieves adaptive adjustment of multi-sensor fusion weights under different road types and environmental interference conditions, improving the reliability and robustness of the fused perception output.

[0180] The comprehensive weight generation module inputs the comprehensive weight model based on the camera cleanliness status assessment values, image feature data, average driving speed, and road type and density feature data, and outputs a candidate weight vector including camera candidate weights, lidar candidate weights, and millimeter-wave radar candidate weights.

[0181] The integrated weight model is used to predict candidate weight vectors for cameras, LiDAR, and millimeter-wave radar under different road types, vehicle speeds, traffic densities, and sensor states, thus providing initial weights for subsequent weight optimization. The inputs to the integrated weight model are the camera cleanliness assessment values, image feature data, average driving speed, and road type and density feature data corresponding to the same fusion time. The output is a candidate weight vector containing candidate weights for cameras, LiDAR, and millimeter-wave radar. All three candidate weights are non-negative and sum to 1.

[0182] The training method for the comprehensive weight model includes the following steps: training sample construction, training label construction, feature preprocessing, model training, model validation, and model deployment.

[0183] In the training sample construction step, multi-scene, multi-sensor synchronous data is pre-collected, covering scenarios such as sunny days, nighttime, backlighting, rain and fog, tunnels, congestion, highway cruising, rural roads, urban expressways, residential roads, and mountain roads. For each fusion time point, input features are generated following the same path as online processing: the driving image is input into the cleanliness assessment model to obtain camera cleanliness assessment values; feature extraction is performed on the driving image to obtain brightness data, contrast data, color saturation data, and light source direction; average driving speed is obtained from vehicle network statistics; road type is encoded as a fixed-dimensional discrete feature; and vehicle density and pedestrian density are calculated based on the comprehensive object detection result set to form density feature data. This forms the input feature vector for the comprehensive weight model.

[0184] To avoid numerical jumps in the light source direction between the 0° and 360° neighborhoods, the light source direction is converted from an angle value into a sine and cosine component before being written into the input feature vector during training sample construction. For example, the input feature vector of the comprehensive weight model can be set as a 14-dimensional vector, with the 14 features being, in order: camera cleanliness assessment value, brightness data, contrast data, color saturation data, sine component of the light source direction, cosine component of the light source direction, average driving speed, highway coding bit, rural road coding bit, urban expressway coding bit, residential road coding bit, mountain road coding bit, vehicle density, and pedestrian density.

[0185] In the training label construction step, an optimal candidate weight vector is generated as a supervision label for each fusion time step. The optimal candidate weight vector is obtained through the offline evaluation process. At the same fusion time step, the target detection results of the camera, LiDAR, and millimeter-wave radar are associated with the offline reference ground truth, respectively. The reliability scores of the camera, LiDAR, and millimeter-wave radar are obtained according to the reliability score calculation method. Then, the three reliability scores are normalized to obtain the initial weight vector. Under the condition that each weight is non-negative and the sum of the three is 1, the candidate weight combinations are traversed with a step size of 0.02. Offline fusion evaluation is performed on each candidate weight combination to calculate the mean position error, mean velocity error, false negative rate, false positive rate, and false critical event rate after fusion, and an offline evaluation function is constructed. For example, the offline evaluation function can be set as follows: the weight of the position error term is 0.25, the weight of the velocity error term is 0.15, the weight of the missed detection term is 0.30, the weight of the false detection term is 0.15, the weight of the pseudo-critical event term is 0.10, and the weight of the initial weight deviation penalty term is 0.05. The candidate weight combination with the smallest value of the offline evaluation function is selected as the optimal candidate weight vector corresponding to the fusion moment, so that the training label simultaneously reflects the environmental conditions, sensor reliability and offline fusion effect.

[0186] In the feature preprocessing step, continuous features in the input feature vector undergo dimensional unification and normalization. For example, the mean and standard deviation of each feature dimension—including camera cleanliness assessment values, brightness data, contrast data, color saturation data, sine component of light source direction, cosine component of light source direction, average driving speed, vehicle density, and pedestrian density—can be calculated based on the training set. Standardization is performed by subtracting the mean and then dividing by the standard deviation. For feature dimensions with a standard deviation less than 0.000001, the standard deviation is replaced with 1 to avoid division by zero. Road types are encoded using one-hot encoding and do not undergo repeated standardization. After training, the mean and standard deviation parameters corresponding to each continuous feature are saved along with the parameters of the comprehensive weight model to ensure consistent feature transformation between online inference and training phases.

[0187] In the model training step, a multilayer perceptron regression network is selected as the comprehensive weight model. The comprehensive weight model includes an input layer, hidden layers, and an output layer. The input layer receives a 14-dimensional input feature vector; the hidden layers are used to extract the nonlinear mapping relationship between the environmental state and the sensor weight allocation; the output layer is used to output the candidate weights for the camera, LiDAR, and millimeter-wave radar. For example, the hidden layers can be set to 3 layers, with 64 neurons in the first hidden layer, 32 neurons in the second hidden layer, and 16 neurons in the third hidden layer; each hidden layer uses a linear rectified activation function, and random deactivation is set after the first and second hidden layers, with a random deactivation ratio of 0.2. The output layer has 3 neurons and uses a normalized exponential function, so that the three output candidate weights are all non-negative and their sum is 1.

[0188] During the training phase, the optimal candidate weight vector is used as the supervision label, and the combined loss function is employed to update the parameters of the integrated weight model. For example, the combined loss function consists of a weighted average squared error loss and a relative entropy loss, with the average squared error loss weighted at 0.7 and the relative entropy loss weighted at 0.3. The average squared error loss constrains the numerical deviation between the predicted candidate weight vector and the optimal candidate weight vector, while the relative entropy loss constrains the distributional difference between the predicted candidate weight vector and the optimal candidate weight vector. The model parameters are updated using an adaptive moment estimation optimization method, with an initial learning rate of 0.001, a weight decay coefficient of 0.0005, a batch size of 128 samples, and 100 to 150 training epochs.

[0189] In terms of training set partitioning, all training samples are divided into a training set, a validation set, and a test set. For example, the training set accounts for 70%, the validation set for 15%, and the test set for 15%. During training, the validation set loss is calculated after each training epoch. If the validation set loss does not decrease for eight consecutive epochs, the current learning rate is multiplied by 0.5 to perform decay, with the minimum learning rate set to 0.00001. If the validation set loss does not decrease for 12 consecutive epochs, or the difference between the current validation set loss and the previous epoch's validation set loss is less than 0.0001, the model is considered to have reached convergence and training is stopped. After training, the model parameters corresponding to the minimum validation set loss are saved as the fixed parameters of the comprehensive weight model.

[0190] In the model validation step, the prediction error of the candidate weight vector and the degree of improvement in offline fusion performance are calculated based on the test set. For example, the mean absolute error of the 3D candidate weight vector can be used as the weight prediction metric, and the mean fusion position error, mean fusion speed error, fusion false negative rate, and fusion false positive rate after applying the predicted candidate weights can be used as task metrics. When the mean absolute error of the weights on the test set is not higher than 0.05, and the mean fusion position error, mean fusion speed error, fusion false negative rate, and fusion false positive rate all decrease compared to the fixed weight fusion scheme, the comprehensive weight model is deemed to meet the deployment requirements. If the preset requirements are not met, the process returns to the training label construction step or the model training step, readjusts the offline evaluation function coefficients, hidden layer size, or training parameters, and then retrains.

[0191] In the model deployment step, the solidified integrated weight model parameters, the mean parameters and standard deviation parameters corresponding to each continuous feature, and the road type coding rules are written into the vehicle-side storage unit. During runtime, the camera cleanliness assessment values, image feature data, average driving speed, road type, and density feature data at the current fusion moment are used to construct an input feature vector in the same order as during the training phase. After standardization, the vector is input into the integrated weight model, outputting a candidate weight vector composed of camera candidate weights, LiDAR candidate weights, and millimeter-wave radar candidate weights. The candidate weight vector is then input into the subsequent weight optimization solution process, participating together with the conflict metric in the solution of the target weight vector, thereby enabling the integrated weight model to have a clear and reproducible training path and online inference path.

[0192] An example of the formula for calculating the reliability score is as follows:

[0193] ;

[0194] in, The value of u ranges from 1 to 3, corresponding to the reliability score. For example, This can represent the camera's reliability score. This can represent the reliability score of the lidar. It can represent the reliability score of millimeter-wave radar. This represents the mean of the corresponding positional error. This represents the average speed error. The corresponding false negative rate, The corresponding false detection rate, This corresponds to the pseudo-critical event rate. This is the position error penalty coefficient. This is the speed error penalty coefficient. This is the penalty coefficient for missed detection. This is the false detection penalty coefficient. The penalty coefficient for pseudo-critical events. When, it indicates that the corresponding sensor has the highest reliability at the current moment. When the value is 0, it indicates that the corresponding sensor has the lowest reliability at the current moment.

[0195] For example, Take 2m; Take 3 m / s; penalty coefficient Set the penalty coefficient to 0.25. Set the penalty coefficient to 0.15. Set the penalty coefficient to 0.35. Set the penalty coefficient to 0.15. The value is set to 0.10 to reflect the logic of setting the penalty for missed detection as the highest, followed by positional deviation, then speed deviation and false detection, and false critical events as an additional penalty.

[0196] The following data example illustrates how rain-soaked roads and water droplets adhering to lenses can degrade camera quality: Statistics on camera performance within key sensing areas are collected. 1.8m 2.0m / s 0.22 0.08 Substituting 0.12 into the above parameters, the camera reliability score is approximately 0.574; at the same time, the lidar's... 0.35m 0.8 m / s 0.06 0.03 The value is 0.02. Substituting these values ​​into the above parameters, the reliability score of the lidar is approximately 0.889; at the same time, the millimeter-wave radar... 0.75m 0.5m / s 0.09 0.04 The value is 0.03. Substituting the above parameters, we get a millimeter-wave radar reliability score of 0.841.

[0197] The above examples illustrate the penalty setting in this application where errors in the critical perception area have a greater impact on security: when camera contamination leads to an increase in the false negative rate and false critical event rate, the camera reliability score is significantly reduced, thereby generating training supervision signals for the comprehensive weight model that reduce the candidate weights of cameras and increase the candidate weights of LiDAR and millimeter-wave radar. Similarly, in scenarios where tunnel multipath leads to an increase in the false negative rate of millimeter-wave radar, the false negative rate and false critical event rate of millimeter-wave radar will increase the penalty term and lower the reliability score. The trained comprehensive weight model can then learn the weight allocation rules that suppress millimeter-wave radar weights and depend on point clouds and images under such conditions, so that the candidate weight vector directly corresponds to the technical problem that this application aims to solve: inconsistent multi-source outputs leading to unstable fusion and an increase in false negatives and false negatives.

[0198] The weight optimization solution module constructs a weight optimization objective function based on the conflict metric and candidate weight vectors, solves for the target weight vector under preset constraints, and updates the weights of the camera, lidar, and millimeter-wave radar based on the target weight vector.

[0199] It should be noted that the weight optimization solution module combines the candidate weight vector output by the comprehensive weight model with the conflict metric value output by the matching conflict metric module at the fusion time, constructs the weight optimization objective function, and solves it under constraints to obtain the target weight vector, thereby updating the fusion weights of the camera, LiDAR, and millimeter-wave radar. The inputs to the weight optimization solution module include the candidate weight vector, the conflict metric value, and the historical weight vector from the previous fusion time; the output of the weight optimization solution module is the target weight vector, which includes the target weights of the camera, LiDAR, and millimeter-wave radar.

[0200] like Figure 4 As shown, the method for constructing a weight optimization objective function based on conflict metrics and candidate weight vectors, and solving for the target weight vector under preset constraints, includes:

[0201] A weight vector to be solved is constructed using camera weights, LiDAR weights, and millimeter-wave radar weights. The squared L2 norm between the weight vector to be solved and the candidate weight vectors is used as a candidate consistency penalty term. The squared L2 norm between the weight vector to be solved and the historical weight vector at the previous fusion time is used as a smoothing penalty term. The squared L2 norm between the weight vector to be solved and the preset equilibrium reference vector, multiplied by a conflict metric, is used as a conflict penalty term. The candidate consistency penalty term, smoothing penalty term, and conflict penalty term are weighted and summed according to their corresponding preset coefficients to obtain the weight optimization objective function. Under the constraints that the weights are non-negative, the sum of the three weights is 1, each weight is within a preset upper and lower bound, and the change in each weight relative to the historical weight does not exceed a preset change threshold, the weight optimization objective function is minimized to obtain the target weight vector. A constrained convex optimization solver can be used to minimize the objective function.

[0202] It should be noted that the non-negative weights and the sum of the weights of the three are 1 means that the target weights of the camera, the LiDAR, and the millimeter-wave radar are non-negative and the sum of the weights of the three is 1.

[0203] A preset change threshold is used to limit the maximum change in weights between adjacent fusion moments, avoiding jitter in the fusion output caused by drastic weight jumps triggered by short-term noise, instantaneous occlusion, or occasional false detections. For example, when the fusion period is 100ms, this application can set the camera weight change threshold to 0.10, the LiDAR weight change threshold to 0.10, and the millimeter-wave radar weight change threshold to 0.10. That is, for any sensor, the absolute value of the difference between the target weight at the current fusion moment and the historical weight at the previous fusion moment does not exceed 0.10. For example, if the weight vector at the previous fusion moment is [0.30, 0.45, 0.25], then the possible ranges for the camera weight at the next fusion moment are [0.2, 0.4], the possible ranges for the LiDAR weight are [0.35, 0.55], and the possible ranges for the millimeter-wave radar weight are [0.15, 0.35]. The final target weight vector is obtained under the constraint that the weights are non-negative and the sum of the three weights is 1. Furthermore, in scenarios where the conflict metric is large and the weights require a faster response, the change threshold can be relaxed to 0.15; in high-speed cruise scenarios where the conflict metric is small and stability is emphasized, the change threshold can be tightened to 0.05. By providing data examples and value ranges for the above change thresholds, the weight smoothing constraint of this application has a clear and implementable parameter caliber, thus supporting the full disclosure of the specification.

[0204] The method for constructing the weight optimization objective function is as follows:

[0205] ;

[0206] in, Optimize the objective function for the weights. Let be the weight vector to be solved. For candidate weight vectors, The historical weight vector is the target weight vector obtained by weight optimization at the previous fusion time and actually used for multi-sensor fusion. This target weight vector is cached and read as a reference value for smoothing constraints at the current fusion time. For candidate consistency penalty items, To smooth out the penalty term, This is a conflict penalty item. This is a preset equilibrium reference vector, used to pull the weights towards a more balanced distribution when there is significant conflict, preventing a single sensor from dominating fusion and amplifying errors when multiple sources are inconsistent. For example, it is set as a vector with equal weights for the camera, LiDAR, and millimeter-wave radar. . The candidate consistency coefficient, For smoothing coefficients, This serves as the baseline coefficient for conflict equilibrium.

[0207] It should be noted that, in setting the coefficients of the objective function for weight optimization, this application divides the coefficients into two categories: preset basic coefficients and real-time dynamic coefficients. This ensures that the weight solution is both configurable and can adaptively adjust to changes in the environment. The preset basic coefficients include candidate consistency coefficients, smoothing coefficients, and conflict balancing benchmark coefficients. These are used to characterize the relative importance of the solved weights being close to the candidate weight vector, the solved weights being close to the historical weight vector at the previous fusion time, and the suppression of excessive weights of a single sensor when conflicts exist. The preset basic coefficients can be written into a configuration table and can be selected by looking up the table according to road type, average driving speed, and density characteristic data, thereby providing a clear and reproducible parameter caliber for the weight update strategy in different scenarios. For example, in urban road scenarios with low speeds and high vehicle and pedestrian density, a candidate consistency coefficient of 1.0, a smoothing coefficient of 0.6, and a conflict equilibrium baseline coefficient of 0.8 are used to emphasize that weight changes are controlled in congested scenarios and quickly tend towards conservative allocation when inconsistencies occur. In high-speed cruising scenarios on highways, a candidate consistency coefficient of 1.2, a smoothing coefficient of 0.4, and a conflict equilibrium baseline coefficient of 0.6 are used to improve the following of the prior output of the integrated weight model and maintain moderate smoothness. In scenarios with fluctuating perception quality such as rain, fog, backlight, or tunnels, a candidate consistency coefficient of 0.8, a smoothing coefficient of 0.5, and a conflict equilibrium baseline coefficient of 1.0 are used to enhance the constraint of the conflict equilibrium term on the solution results and suppress the abnormal dominance of a single sensor in the fusion output.

[0208] Real-time dynamic coefficients are introduced by conflict metrics, which are generated by normalizing and weighting position and velocity deviation indices, with values ​​limited to the range of 0 to 1. The conflict equilibrium term in the weight optimization objective function participates in the solution by multiplying the conflict equilibrium benchmark coefficient by the conflict metric, so that the actual strength of the conflict equilibrium term is automatically adjusted according to the real-time conflict level. When the conflict metric is close to 0, the contribution of the conflict equilibrium term approaches 0, and the solution weight is mainly determined by the candidate consistency penalty term and the smoothing penalty term, thereby keeping the weight close to the candidate weight and evolving smoothly. When the conflict metric increases, the contribution of the conflict equilibrium term increases synchronously, pushing the solution weight closer to the preset equilibrium reference vector, so as to reduce the risk that the fusion result will be dragged by the excessive weight of a single sensor when the multi-source output is inconsistent. Through the coefficient setting method that combines the preset grading of basic coefficients with the dynamic modulation of conflict metrics, this application realizes the configurability, reproducibility and scene adaptation of the weight optimization solution process.

[0209] It should be noted that the objective function for weight optimization in this application stems from the core contradiction of multi-sensor fusion in complex road environments. Specifically, on the one hand, the comprehensive weight generation module outputs candidate weight vectors based on camera cleanliness assessment values, image feature data, average driving speed, and road type and density feature data. These candidate weight vectors can reflect the prior reliability trends of each sensor under different scenarios. On the other hand, the matching conflict measurement module calculates position deviation and speed deviation indices based on cross-sensor target matching and generates conflict measurement values. These conflict measurement values ​​can reflect the consistency of the outputs of each sensor at the current fusion moment. When the environment changes abruptly or the sensors are interfered with, the candidate weight vectors may lag behind the real-time conflict state. If candidate weights are directly used for fusion, they are easily influenced by an unreliable sensor when the conflict is large, leading to false detections, missed detections, or positioning speed estimation errors. Therefore, this application incorporates both the prior validity of candidate weights and the real-time constraint of conflict measurement into the weight solution, and introduces weight continuity constraints to suppress temporal jitter, forming an implementable optimization solution framework.

[0210] To achieve the above objectives, this application constructs a weight vector to be solved using camera weights, LiDAR weights, and millimeter-wave radar weights. First, a candidate consistency penalty term is introduced as a basic term to constrain the solved weights from deviating from the candidate weight vector output by the comprehensive weight model. The candidate consistency penalty term uses the L2 squared form to measure the difference between the solved weight vector and the candidate weight vector. This is because the L2 squared form can unify the deviations of the three weights into a scalar penalty, and applies a stronger penalty to weight components with larger deviations. This ensures that the solved result follows the candidate weights when there is no significant conflict, thereby inheriting the comprehensive weight model's ability to utilize prior information such as road type, vehicle speed, density, and cleanliness.

[0211] Secondly, this application introduces a smoothing penalty term to characterize the temporal continuity requirement of the weights. Weights are coefficients used in multi-source fusion. If there are drastic jumps between adjacent fusion moments, even if the fusion result of a single frame is reasonable, it will cause the fusion output to jitter in the temporal dimension, affecting trajectory correlation and decision stability. Therefore, the smoothing penalty term uses the squared L2 norm to measure the difference between the weight vector to be solved and the historical weight vector at the previous fusion moment, ensuring continuous weight updates. This, combined with weight change threshold constraints, suppresses sudden weight changes caused by occasional false detections, short-term occlusion, or instantaneous noise, thereby improving the stability and usability of the fusion results.

[0212] Furthermore, this application introduces a conflict metric into the conflict penalty term to achieve an adaptive mechanism where a more conservative weight allocation is needed as the conflict intensifies. The conflict metric is obtained by normalizing and weighting the position deviation index and velocity deviation index, reflecting the degree of inconsistency across sensor outputs. When the conflict metric is large, it indicates that at least one sensor output deviates significantly from other sensors. If the weights are still allowed to be excessively concentrated on a single sensor, the impact of this deviation on the fusion result will be amplified. To address this, this application sets an equilibrium reference vector as a conservative weight allocation benchmark under conflict conditions and uses the L2 squared value to measure the difference between the weight vector to be solved and the equilibrium reference vector. This difference is then multiplied by the conflict metric as a dynamic modulation factor, so that the conflict penalty term has almost no effect when the conflict metric is close to 0, but becomes significantly stronger as the conflict metric increases. This drives the solution weights toward the equilibrium reference vector and reduces the risk of single-sensor dominating the fusion. This design directly addresses the technical problem of this application: when inconsistencies are caused by sensor contamination, rain or fog, tunnel multipath, or dense targets, an increase in the conflict metric triggers a more conservative weight allocation, suppressing the influence of unreliable outputs on the fusion result.

[0213] Based on the above three types of requirements, this application forms a weight optimization objective function by weighting and summing the candidate consistency penalty term, smoothing penalty term, and conflict penalty term according to preset coefficients. The objective function is then minimized under constraints such as non-negative weights, a weight sum of 1, upper and lower bounds of weights, and weight change thresholds, resulting in a target weight vector that satisfies the probability allocation form and possesses temporal continuity. Through this derivation process, this application structurally achieves a unified approach of "prior weight prediction—real-time conflict constraint—temporal smoothing control," enabling weight updates to utilize prior information such as cleanliness, vehicle speed, road type, and density, while also suppressing cross-sensor inconsistencies in real time, thereby improving the robustness and security of multi-sensor fusion perception output.

[0214] Example 2:

[0215] Please see Figure 2 As shown, this embodiment provides an adaptive method for intelligent driving environment based on multi-sensor fusion, including:

[0216] Collect driving images, average driving speed, and road type during the intelligent driving process of the vehicle.

[0217] The driving images are input into the cleanliness assessment model to obtain the camera cleanliness assessment value, and the driving images are then used to extract features to obtain image feature data.

[0218] Based on driving images, 3D point cloud data obtained from LiDAR scanning, and echo data obtained from millimeter-wave radar scanning, a comprehensive target detection result set is generated; density feature data is generated based on the comprehensive target detection result set.

[0219] Cross-sensor target matching is performed on the comprehensive target detection result set to obtain a cross-sensor matching triplet set. The sensor consistency index of each cross-sensor matching triplet is calculated, and a conflict metric is generated based on the sensor consistency index. The sensor consistency index includes a position deviation index and a velocity deviation index.

[0220] The comprehensive weight model is based on the input of camera cleanliness assessment values, image feature data, average driving speed, and road type and density feature data. The output includes candidate weight vectors for camera, lidar, and millimeter-wave radar.

[0221] The objective function for weight optimization is constructed based on the conflict metric and candidate weight vectors. The objective weight vector is obtained by solving under preset constraints, and the weights of the camera, LiDAR and millimeter-wave radar are updated based on the objective weight vector.

[0222] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0223] Finally: The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An adaptive method for intelligent driving environment based on multi-sensor fusion, characterized in that, include: Collect driving images, average driving speed, and road type during the intelligent driving process of the vehicle; The driving images are input into the cleanliness assessment model to obtain the camera cleanliness assessment value, and the driving images are feature extracted to obtain image feature data; Based on driving images, 3D point cloud data obtained from LiDAR scanning, and echo data obtained from millimeter-wave radar scanning, a comprehensive target detection result set is generated; density feature data is then generated based on the comprehensive target detection result set. Cross-sensor target matching is performed on the comprehensive target detection result set to obtain a cross-sensor matching triplet set. The sensor consistency index of each cross-sensor matching triplet is calculated, and a conflict metric is generated based on the sensor consistency index. The sensor consistency index includes a position deviation index and a velocity deviation index. The comprehensive weight model is based on the camera cleanliness status assessment values, image feature data, average driving speed, and road type and density feature data. The output includes candidate weight vectors for camera, lidar, and millimeter-wave radar. The objective function for weight optimization is constructed based on the conflict metric and candidate weight vectors. The objective weight vector is obtained by solving under preset constraints, and the weights of the camera, LiDAR and millimeter-wave radar are updated based on the objective weight vector.

2. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 1, characterized in that, Methods for constructing a weight optimization objective function and solving for the objective weight vector include: The weight vector to be solved is composed of camera weights, LiDAR weights, and millimeter-wave radar weights. A candidate consistency penalty term is constructed between the weight vector to be solved and the candidate weight vector, a smoothing penalty term is constructed between the weight vector to be solved and the historical weight vector, and a conflict penalty term is constructed between the weight vector to be solved and the preset equilibrium reference vector, combined with the conflict metric. The weight optimization objective function is obtained by weighted summation of each penalty term. Under the constraints that the weights are non-negative, the sum of the weights is 1, the weights are within the preset upper and lower bounds, and the change in weights relative to the historical weights does not exceed the preset change threshold, the weight optimization objective function is minimized to obtain the target weight vector containing the target weights of the camera, LiDAR, and millimeter-wave radar.

3. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 1, characterized in that, The method for obtaining the cross-sensor matching triplet set includes: Based on the sensor source, the comprehensive target detection result set is divided into camera target list, lidar target list and millimeter-wave radar target list; Using the lidar target list as the matching benchmark, for each lidar target entry, candidate entries with position distances less than the target matching distance threshold and velocity differences less than the velocity matching threshold are selected from the millimeter-wave radar target list and the camera target list, respectively. The matching cost is calculated and the entry with the minimum cost is selected as the corresponding matching entry. The lidar target entries, millimeter-wave radar matching entries, and camera matching entries are constructed into a cross-sensor matching triplet set.

4. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 3, characterized in that, Methods for calculating the sensor consistency index for each cross-sensor matching triplet and generating conflict metrics based on the sensor consistency index include: The spatial distance and velocity difference between the lidar reference target entry and the camera matching entry and the millimeter-wave radar matching entry are calculated respectively. The larger value of the two sets of spatial distances is determined as the position deviation index, and the larger value of the two sets of velocity differences is determined as the velocity deviation index. The position deviation index and velocity deviation index are subjected to dimension unification and normalization processing, and then weighted and summed according to the preset weight coefficients to obtain the conflict measurement value.

5. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 1, characterized in that, The comprehensive target detection result set includes the camera target detection result set, the lidar target detection result set, and the millimeter-wave radar target detection result set; The method for obtaining the set of camera target detection results includes: The driving image is input into the target detection network to obtain a two-dimensional detection box and a detection confidence score. The detection confidence score is used as the first target confidence score parameter. The corresponding LiDAR point cloud is projected onto the image plane, outlier removal is performed on the point cloud falling within the two-dimensional detection box, and the three-dimensional centroid is obtained. The three-dimensional centroid is transformed into the vehicle coordinate system as the first target position parameter. The target positions at adjacent time points are correlated with trajectories, and the first target velocity parameter is calculated. The first target confidence score, the first target position parameter, and the first target velocity parameter constitute the camera target detection result set.

6. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 5, characterized in that, The method for obtaining the set of laser radar target detection results includes: Perform effective detection spatial range filtering and voxel downsampling on the 3D point cloud data, and perform ground segmentation and Euclidean clustering on the downsampled point cloud to obtain a list of point cloud clusters; Point cloud clusters with fewer points than the minimum point threshold are filtered out. For the remaining point cloud clusters, the three-dimensional centroid is calculated as the position parameter of the second target. The confidence parameter of the second target is calculated based on the reasonableness of the point cloud cluster's point count and bounding box size. The position parameters of the second target at the current time are associated with the position parameters of the second target at the previous time according to the nearest neighbor distance, and the velocity parameters of the second target are calculated based on the position difference and time interval between adjacent times. The second target's position parameters, second target's velocity parameters, and second target's confidence parameters constitute the set of lidar target detection results.

7. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 5, characterized in that, The method for obtaining the set of millimeter-wave radar target detection results includes: Range-dimensional Fast Fourier Transform and Doppler-dimensional Fast Fourier Transform are performed on the echo data to obtain the range-velocity spectrum; In the range-velocity spectrum, the background noise is estimated based on the power of the reference cell and a detection threshold is generated. Cells whose power is greater than the detection threshold are identified as candidate detection cells. Angle estimation is performed on the candidate detection units and combined with distance information to transform to the vehicle coordinate system to obtain the position parameters of the third target. The velocity parameters of the third target are calculated based on the Doppler frequency. Candidate detection units are clustered according to spatial proximity, and the confidence parameter of the third target is calculated based on the signal-to-noise ratio, detection amplitude, and number of consecutive hits. The third target's position parameters, velocity parameters, and confidence parameters constitute the set of millimeter-wave radar target detection results.

8. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 1, characterized in that, The method for obtaining the density feature data includes: Based on the comprehensive target detection result set, extract the lidar target point cloud set corresponding to each external target, calculate the surface smoothness, reflectivity, volume, height, contour scale and reflection intensity statistics, construct the target recognition feature vector and input it into the target recognition model to obtain the pedestrian or vehicle target type; Under highways and rural roads, the scanning unit area is calculated based on the millimeter-wave radar field of view and maximum effective scanning distance. Under urban expressways, residential roads, and mountain roads, the scanning unit area is calculated based on the lidar horizontal field of view and maximum effective scanning distance. The number of vehicles and pedestrians in the scanning area is counted, and then divided by the scanning unit area to obtain the vehicle density and pedestrian density, which constitute density feature data.

9. The intelligent driving environment adaptive method based on multi-sensor fusion according to claim 1, characterized in that, The image feature data includes brightness data, contrast data, color saturation data, and light source direction; the method for acquiring the image feature data includes: The system acquires and normalizes the red, green, and blue channel values ​​of each pixel in the driving image. Based on the normalized channel values, it calculates the pixel's grayscale value and color saturation, identifying pixels with grayscale values ​​not less than a preset grayscale threshold as highlight pixels. The system calculates the mean of the grayscale value set as brightness data and the mean of the color saturation set as color saturation data, and obtains contrast data based on the grayscale value set. Edge detection is performed on the highlight pixel set to construct highlight regions and a minimum circumcircle. The light source direction is calculated based on the center position of the circle and the center position of the driving image.

10. An intelligent driving environment adaptive system based on multi-sensor fusion, implementing the intelligent driving environment adaptive method based on multi-sensor fusion as described in any one of claims 1-9, characterized in that, include: The driving information collection module is used to collect driving images, average driving speed, and road type during the intelligent driving process of the vehicle; The image feature extraction module is used to input driving images into the cleanliness assessment model to obtain camera cleanliness assessment values, and to extract features from the driving images to obtain image feature data. The multi-source target detection module generates a comprehensive target detection result set based on driving images, 3D point cloud data obtained from LiDAR scanning, and echo data obtained from millimeter-wave radar scanning; and generates density feature data based on the comprehensive target detection result set. The conflict measurement module performs cross-sensor target matching on the comprehensive target detection result set to obtain a set of cross-sensor matching triples. It calculates the sensor consistency index for each cross-sensor matching triple and generates a conflict measurement value based on the sensor consistency index. The sensor consistency index includes a position deviation index and a velocity deviation index. The comprehensive weight generation module inputs the comprehensive weight model based on the camera cleanliness status assessment values, image feature data, average driving speed, road type and density feature data, and outputs a candidate weight vector including camera candidate weights, lidar candidate weights and millimeter-wave radar candidate weights. The weight optimization solution module constructs a weight optimization objective function based on the conflict metric and candidate weight vectors, solves for the target weight vector under preset constraints, and updates the weights of the camera, lidar, and millimeter-wave radar based on the target weight vector.

Images