Adaptive weight fusion method and system based on mutual verification of heterogeneous data

Abstract

The invention provides an adaptive weight fusion method based on mutual verification of heterogeneous data. The method comprises the following steps: (1) starting a system, initializing parameters of each module, acquiring millimeter-wave radar data and visual data through a data acquisition and preprocessing module, and preprocessing the millimeter-wave radar data and the visual data to extract target and environment related information, performing space-time alignment on the two types of preprocessed data based on sensor calibration information and timestamps to generate synchronous data; (2) sequentially executing signal-level mutual verification, feature-level mutual verification and decision-level mutual verification on the synchronous data by utilizing a multi-level mutual verification module, and respectively generating a signal-level mutual verification score, a feature-level mutual verification score and a decision-level mutual verification score; (3) calculating a normalized weight through an adaptive weight calculation module; and (4) fusing the two types of sensor data at a signal level, a feature level and a decision level according to the normalized weight by using a multi-level fusion module to generate enhanced environmental representation data, unified target feature representation and final fusion decision output.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a multi-sensor data fusion method, in particular to an adaptive weight fusion method and system based on heterogeneous data mutual verification. BACKGROUND

[0002] In the fields of automatic driving, intelligent monitoring, unmanned aerial vehicle navigation, etc. that require high-precision environmental perception, millimeter wave radar and visual sensors have complementary performance (millimeter wave radar has strong anti-adverse weather capability and can accurately obtain target motion information, and visual sensors can provide rich target appearance and scene details), and their data fusion technology is a key means to improve the reliability of the perception system.

[0003] However, the existing millimeter wave radar and visual signal fusion system still has significant technical bottlenecks, which are difficult to meet the perception needs in complex and variable scenes, and the specific manifestations are as follows: Traditional fusion strategies have poor adaptability: Most systems use fixed weight or preset rule fusion methods, and the weight distribution cannot be dynamically adjusted according to the actual scene (such as sunny / rainy day, backlight / normal light) or the real-time state of the sensor (such as radar interference by clutter, blurred visual image). For example, in a rainy day scene, the visual sensor loses image details due to water on the lens, but if the fixed weight (visual weight higher than radar) used in sunny days is still used, the fusion result accuracy will be greatly reduced, and even target mis-detection and missed detection will occur.

[0004] Dynamic weight methods lack mutual verification mechanisms: A few existing technologies attempt to introduce dynamic weight adjustment, but only rely on single sensor data quality evaluation (such as radar point cloud density, visual image clarity) or predefined scene rules, without establishing mutual verification mechanisms between heterogeneous sensors. This single-dimensional evaluation cannot accurately determine the credibility of the sensor in the current scene — for example, although the radar detects “target motion signals”, it cannot verify through vision whether the signal is a real target (such as a false radar reflection caused by a roadside tree swaying), which easily leads to weight distribution deviation.

[0005] The fusion architecture is single and lacks cooperation: Existing technologies mostly focus on single-level fusion (such as only signal-level point cloud and image feature fusion, or only decision-level target category fusion), ignoring the “signal-level - feature-level - decision-level” whole-process fusion architecture design, and not establishing a unified adaptive weight strategy at each level. This fragmented fusion mode cannot fully utilize the performance advantages of the two types of sensors, for example, if signal-level fusion does not adjust the weight in combination with the subsequent feature-level mutual verification results, it will cause the bias of the basic data to be passed to the subsequent decision-making link, reducing the overall perception accuracy of the system.

[0006] System fault tolerance and weak generalization ability: the existing fusion system lacks effective response mechanisms for sensor abnormalities (such as temporary failure of radar, visual sensor being blocked), and does not introduce historical scene experience learning ability - in unknown or rare scenes (such as sudden fog, non-standard obstacles), it is unable to optimize the current weight allocation through the decision data of similar historical scenes, resulting in poor system generalization ability and difficulty in maintaining stable performance.

[0007] In summary, the existing millimeter wave radar and visual data fusion technology has obvious deficiencies in adaptability, reliability evaluation, fusion architecture integrity and fault tolerance, and there is an urgent need for a technical solution that can realize sensor mutual verification, dynamic weight adjustment and full-level fusion collaboration to break through the above technical bottlenecks. SUMMARY

[0008] The present application aims to solve the above-mentioned defects of the existing millimeter wave radar and visual data fusion technology, and the specific purposes include: establishing a multi-level mutual verification and correction mechanism between heterogeneous sensor data to improve the reliability of the fused data; designing an adaptive weight adjustment algorithm based on scene conditions, sensor state and historical experience to ensure the performance stability of the system in complex environments; constructing a unified multi-level fusion architecture of "signal level - feature level - decision level", and applying adaptive weight strategy at each level to improve the overall perception accuracy of the system; enhancing the fault tolerance of the system to single sensor abnormalities, reducing the negative impact of abnormal data on the fusion results, and improving the robustness of the system.

[0009] To this end, according to one aspect of the present disclosure, an adaptive weight fusion method based on mutual verification of heterogeneous data is provided, characterized by the following steps: (1) the system is started and the parameters of each module are initialized, millimeter wave radar data and visual data are collected through the data acquisition and preprocessing module, the millimeter wave radar data and visual data are preprocessed to extract target and environment related information, and the preprocessed two types of data are spatio-temporally aligned based on sensor calibration information and time stamp to generate synchronized data; (2) the multi-level mutual verification module is used to sequentially perform signal level mutual verification, feature level mutual verification and decision level mutual verification on the synchronized data, respectively generating signal level mutual verification score, feature level mutual verification score and decision level mutual verification score; (3) the adaptive weight calculation module is used to calculate the normalized weight; and (4) the multi-level fusion module is used to fuse the two types of sensor data at the signal level, feature level and decision level according to the normalized weight, respectively, to generate enhanced environment representation data, unified target feature representation and final fusion decision output.

[0010] According to the adaptive weight fusion method based on mutual verification of heterogeneous data of the present disclosure, the preprocessing of the millimeter wave radar data in step (1) includes point cloud filtering, clustering and target extraction, wherein the point cloud filtering adopts statistical filtering or voxel filtering, the clustering adopts DBSCAN algorithm or K-means algorithm, and the target extraction filters target point cloud clusters based on the clustering results; the preprocessing of the visual data includes image enhancement, target detection and feature extraction, wherein the image enhancement adopts histogram equalization or Retinex algorithm, the target detection adopts YOLO algorithm or Faster R-CNN algorithm, and the feature extraction adopts HOG feature extraction algorithm or CNN feature extraction algorithm.

[0011] According to the adaptive weight fusion method based on mutual verification of heterogeneous data of the present disclosure, in step (2), the signal level mutual verification is used to evaluate the quality of the original data of the two types of sensors and the consistency of target detection coverage, the feature level mutual verification is used to evaluate the consistency of the target features extracted by the two types of sensors and the motion state estimation, and the decision level mutual verification is used to evaluate the consistency of the independent decision results of the two types of sensors and the historical decision correlation.

[0012] According to the adaptive weight fusion method based on mutual verification of heterogeneous data of the present disclosure, the specific process of the signal level mutual verification in step (2) includes: evaluating the millimeter wave radar point cloud density, signal-to-noise ratio and RCS value distribution, evaluating the visual image definition, contrast and lighting condition, and then calculating the target detection consistency score in the coverage area of the two types of sensor data; wherein the visual image definition is evaluated by the mean or entropy value of the gray gradient, the contrast is evaluated by the standard deviation or variance of the gray value, the lighting condition is evaluated by the average gray value or brightness component, and the target detection consistency score is quantified by the spatial position overlap of the target.

[0013] According to the adaptive weight fusion method based on mutual verification of heterogeneous data of the present disclosure, the specific process of the feature level mutual verification in step (2) includes: calculating the consistency score of the position, size and outline difference of the same target detected by the two types of sensors, calculating the consistency score of the speed and direction estimation results of the moving target, and forming a feature level mutual verification matrix recording the consistency score of each feature dimension; wherein the position difference is quantified by the distance between the target center coordinates, the size difference is quantified by the parameter deviation of the target bounding box, the outline difference is quantified by the outline distance metric, and the speed and direction difference is quantified by the speed vector similarity and size error.

[0014] According to the adaptive weight fusion method based on heterogeneous data mutual verification of the disclosure, the specific process of the decision-level mutual verification in step (2) includes: comparing the category matching rate of the independent decision results of the two types of sensors, combining the decision accuracy rate in the same or similar historical scenes, calculating the similarity of the current scene and the historical scene, and then generating a decision-level mutual verification score by weighted sum of the category matching rate, the historical decision accuracy rate and the scene similarity; wherein the scene similarity is quantified by the distance or similarity of the scene features.

[0015] According to the adaptive weight fusion method based on heterogeneous data mutual verification of the disclosure, in step (3), the basic weight is calculated based on the inherent performance characteristics of the sensor and the current environmental conditions, and then the mutual verification coefficient is calculated in combination with the signal-level mutual verification score, the feature-level mutual verification score and the decision-level mutual verification score, and the historical score is calculated based on the historical decision accuracy rate, the scene similarity and the time decay factor, and then the final weight is calculated according to the basic weight, the mutual verification coefficient and the historical score, and the final weight is normalized.

[0016] According to the adaptive weight fusion method based on heterogeneous data mutual verification of the disclosure, in step (4), the signal-level fusion adopts a weighted average method or a Kalman filtering algorithm, and the millimeter wave radar point cloud data and the visual image features are fused according to the signal-level normalized weight; the feature-level fusion adopts feature splicing or attention mechanism, and the target detection, classification and attribute features are fused according to the feature-level normalized weight; the decision-level fusion adopts a voting method or D-S evidence theory, and the independent decision results of the two types of sensors are fused according to the decision-level normalized weight.

[0017] According to the adaptive weight fusion method based on heterogeneous data mutual verification of the disclosure, it further includes: (5) the output result optimization module is used for abnormal processing, result consistency verification and performance evaluation on the final fusion decision output, and the optimized final result is output, and the performance evaluation result is fed back to the adaptive weight calculation module, which is used for optimizing subsequent weight calculation.

[0018] According to another aspect of the present disclosure, a system of any of the adaptive weight fusion methods based on mutual verification of heterogeneous data is also provided, characterized in that it comprises a data acquisition and preprocessing module, a multi-level mutual verification module, an adaptive weight calculation module, a multi-level fusion module and an output result optimization module; the data acquisition and preprocessing module is used to acquire millimeter wave radar data and visual data, perform preprocessing and time-space alignment; the multi-level mutual verification module is used to perform signal level, feature level and decision level mutual verification and generate mutual verification scores of each level; the adaptive weight calculation module is used to calculate basic weights, mutual verification coefficients, historical scores and normalized weights; the multi-level fusion module is used to perform weighted fusion at the signal level, the feature level and the decision level; the output result optimization module is used to perform abnormality processing, consistency verification and performance evaluation on the fusion result, and feed back the evaluation result to the adaptive weight calculation module.

[0019] By adopting the present disclosure, the system environment and state adaptability can be improved. The present disclosure can dynamically adapt to different scene conditions and real-time states of sensors through the "multi-level mutual verification + adaptive weight adjustment" mechanism. For different environments such as sunny / rainy days, backlight / normal light, and sensor performance fluctuations such as radar jamming interference and visual image blur, the system can adjust the fusion weights of radar and vision based on the mutual verification results and environmental characteristics (such as increasing the radar weight in rainy days and optimizing the vision weight in sunny days), avoiding the performance drop problem of traditional fixed weight schemes in complex environments, and ensuring that the perception system maintains stable output in diversified scenes. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, together with the description.

[0021] Figure 1 Shown is a flowchart of an adaptive weight fusion method based on mutual verification of heterogeneous data according to an exemplary embodiment of the present disclosure.

[0022] Figure 2 Shown is a flowchart of an adaptive weight fusion method based on mutual verification of heterogeneous data according to another exemplary embodiment of the present disclosure.

[0023] Figure 3 Shown is a flowchart of an adaptive weight fusion method based on mutual verification of heterogeneous data according to another exemplary embodiment of the present disclosure.

[0024] Figure 4 Shown is a flowchart of an adaptive weight fusion method based on mutual verification of heterogeneous data according to another exemplary embodiment of the present disclosure.

[0025] Figure 5 FIG. 6 illustrates a flow chart of a method for adaptive weight fusion based on mutual verification of heterogeneous data according to an example embodiment of the present disclosure.

[0026] Figure 6 FIG. 6 illustrates a flow chart of a method for adaptive weight fusion based on mutual verification of heterogeneous data according to an example embodiment of the present disclosure.

[0027] Figure 7 FIG. 6 illustrates a flow chart of a method for adaptive weight fusion based on mutual verification of heterogeneous data according to an example embodiment of the present disclosure.

[0028] Figure 8 FIG. 6 illustrates a flow chart of a method for adaptive weight fusion based on mutual verification of heterogeneous data according to an example embodiment of the present disclosure.

[0029] Figure 9 FIG. 6 illustrates a flow chart of a method for adaptive weight fusion based on mutual verification of heterogeneous data according to an example embodiment of the present disclosure.

[0030] Figure 10 FIG. 6 illustrates a flow chart of a method for adaptive weight fusion based on mutual verification of heterogeneous data according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The example embodiments will be described in detail herein with reference to the attached drawings. The description of the example embodiments is intended to apply to various alternative embodiments as well. It is to be understood that other equipment and processes can be utilized, and changes can be made without departing from the scope of the present disclosure. The following detailed description is not to be taken in a limiting sense, and the scope of the present disclosure is defined only by the appended claims.

[0032] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] It should be understood that, although the terms first, second, third, etc. can be employed in this disclosure to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information. For example, first can also be referred to as second, and vice versa, without departing from the scope of the disclosure. Depending on the context, the word "if' as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0034] To enable those skilled in the art to better understand the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and specific embodiments.

[0035] The following describes the specific embodiments of the adaptive weight fusion method and system based on mutual verification of heterogeneous data according to the typical application scenarios (such as automatic driving environment perception scenarios), to ensure that those skilled in the art can understand and reproduce the technical solutions of the present application according to the following content.

[0036] Figure 1 As shown is a flowchart of an adaptive weight fusion method based on mutual verification of heterogeneous data according to an exemplary embodiment of the present disclosure, including step 100, in which the system is started and the parameters of each module are initialized, millimeter wave radar data and visual data are collected through the data collection and preprocessing module, the millimeter wave radar data and visual data are respectively preprocessed to extract target and environment related information, and the two types of preprocessed data are spatio-temporally aligned based on sensor calibration information and timestamps to generate synchronized data.

[0037] According to an embodiment of the present application, in the system initialization and data processing stage, after the system is started, the basic parameter initialization of each functional module (such as sensor communication parameters, data processing threshold, etc.) is first completed. Through a dedicated data collection and preprocessing module, the original data output by the millimeter wave radar and the image data output by the visual sensor (such as a camera) are synchronously acquired; the two types of data are respectively preprocessed - the radar data preprocessing focuses on extracting the spatial distribution and motion related information of the target in the environment, and the visual data preprocessing focuses on extracting the appearance, category and scene environment information of the target; then, based on the pre-acquired sensor calibration information (such as the relative position and angle relationship of the radar and the camera) and the timestamps generated by the data, the spatio-temporal alignment operation is performed on the two types of preprocessed data, to ensure that the two types of data correspond to the same physical scene in the spatial dimension and remain synchronized in the time dimension, and finally generate synchronized data that can be used for subsequent processing.

[0038] Figure 1The flowchart according to the method of the present disclosure is shown, and further comprises step 200, wherein the multi-level mutual verification module is used to sequentially perform signal level mutual verification, feature level mutual verification and decision level mutual verification on the synchronization data, to generate signal level mutual verification scores, feature level mutual verification scores and decision level mutual verification scores respectively According to one embodiment of the present disclosure, in the multi-level mutual verification stage, the multi-level mutual verification module is used to perform hierarchical verification on the synchronization data: first, signal level mutual verification is performed to verify the reliability of the two types of data from the perspective of sensor raw data quality; then, feature level mutual verification is performed to verify the consistency from the perspective of target features and motion states extracted from the data; finally, decision level mutual verification is performed to verify the matching degree from the perspective of the decision results independently generated by the two types of sensors, and for each level of mutual verification process, mutual verification scores are generated respectively to represent the consistency of the data at this level.

[0039] Figure 1 The flowchart according to the method of the present disclosure is shown, and further comprises step 300, wherein the normalization weight is calculated by the adaptive weight calculation module.

[0040] According to one embodiment of the present disclosure, in the adaptive weight calculation stage, the adaptive weight calculation module is used to calculate the weight that can be used for data fusion, in combination with the aforementioned multi-level mutual verification scores, and factors such as the performance characteristics of the sensor itself and the influence of the current environment on the sensor, and to normalize the weight to ensure that the total weight meets the logical requirements of fusion calculation (such as the total weight being 1).

[0041] Figure 1 The flowchart according to the method of the present disclosure is shown, and further comprises step 400, wherein the multi-level fusion module is used to fuse the two types of sensor data at the signal level, feature level and decision level respectively according to the normalization weight, to generate enhanced environment representation data, unified target feature representation and final fusion decision output.

[0042] According to one embodiment of the present disclosure, in the multi-level fusion stage, the multi-level fusion module is used to fuse the millimeter wave radar data and the visual data at the signal level, feature level and decision level respectively according to the aforementioned normalization weight: signal level fusion aims to generate more comprehensive environment representation data, feature level fusion aims to form unified target feature description, and decision level fusion aims to output the final fusion decision result with higher precision, so as to realize the complementary advantages of the two types of heterogeneous data.

[0043] Figure 2The diagram shown is a flowchart of an adaptive weight fusion method based on heterogeneous data mutual verification according to another exemplary embodiment of this disclosure. In step 100, the preprocessing of the millimeter-wave radar data includes point cloud filtering, clustering, and target extraction. The point cloud filtering adopts statistical filtering or voxel filtering, the clustering adopts the DBSCAN algorithm or the K-means algorithm, and the target extraction selects target point cloud clusters based on the clustering results. The preprocessing of the visual data includes image enhancement, target detection, and feature extraction. The image enhancement adopts histogram equalization or the Retinex algorithm, the target detection adopts the YOLO algorithm or the Faster R-CNN algorithm, and the feature extraction adopts the HOG feature extraction algorithm or the CNN feature extraction algorithm.

[0044] According to one embodiment of the present invention, the preprocessing of millimeter-wave radar data and visual data is specifically described herein.

[0045] According to one embodiment of the present invention, the above-mentioned preprocessing operations are performed in the following manner: (a) Millimeter-wave radar data preprocessing Three operations are performed sequentially on the raw point cloud data output by the radar: 1. Point cloud filtering: Statistical filtering or voxel filtering methods are used to remove noise points in the point cloud (such as environmental interference clutter, sensor noise, etc.) and retain the effective point cloud corresponding to the real target. 2. Clustering: Using the DBSCAN or K-means algorithm, the filtered effective point cloud is divided into multiple clusters according to its spatial distribution characteristics. Each cluster corresponds to a potential physical target (such as a vehicle, pedestrian, or obstacle). 3. Target Extraction: Based on the above clustering results, and combined with the typical characteristics of the target (such as the volume, shape, and spatial location of the point cloud cluster), target point cloud clusters that meet the actual application requirements are selected (such as excluding point cloud clusters that are too small or have no practical significance), thus completing the target extraction of radar data.

[0046] (ii) Visual data preprocessing Three operations are performed sequentially on the raw image data output by the camera: 1. Image Enhancement: Histogram equalization or Retinex algorithm is used to improve the visual effect of the image, such as enhancing image details in backlit and low-contrast scenes, correcting image deviations caused by uneven lighting, and ensuring that targets in the image are clearly distinguishable. 2. Target Detection: Using the YOLO algorithm or Faster R-CNN algorithm, targets (such as vehicles, pedestrians, and traffic signs) are identified and located from the enhanced image, and the location range of the target in the image (such as bounding box) and the preliminary judgment of the target category are output. 3. Feature extraction: using HOG feature extraction algorithm or CNN feature extraction algorithm to extract features representing target characteristics (such as target contour features, texture features, color features, etc.) from the detected target area, providing a basis for subsequent feature-level cross-checking and fusion.

[0047] Figure 3 is a flow chart of an adaptive weight fusion method based on heterogeneous data cross-checking according to another exemplary embodiment of the present disclosure, wherein in step 200, the signal-level cross-checking is used to evaluate the quality and consistency of target detection of two types of sensor raw data, the feature-level cross-checking is used to evaluate the consistency of target features and motion state estimation extracted by two types of sensors, and the decision-level cross-checking is used to evaluate the consistency and historical decision correlation of independent decision results of two types of sensors.

[0048] According to an embodiment of the present disclosure, according to the evaluation direction of each level of the above-mentioned multi-level cross-checking, the following logic is executed: 1) Purpose of signal-level cross-checking: focusing on the "quality reliability" and "detection coverage consistency" of two types of sensor raw data: on the one hand, evaluating the integrity, signal-to-noise ratio of millimeter wave radar raw data (such as point cloud), and basic quality indicators such as the clarity of visual raw image and the rationality of lighting; on the other hand, verifying whether the number of targets detected by two types of sensors in the same space range and the target coverage range match, to ensure that the data for subsequent processing has basic reliability.

[0049] 2) Purpose of feature-level cross-checking: focusing on the "target feature consistency" and "motion state estimation consistency" extracted by two types of sensors: for the same physical target, verifying whether the target spatial features (such as position, size) extracted by radar and the target appearance features (such as contour, color) extracted by vision match; at the same time, verifying whether the target motion state (such as speed, motion direction) calculated by radar based on point cloud and the target motion state estimated by vision based on image sequence are consistent, to ensure that the feature level of two types of data has a fusion basis.

[0050] 3) Purpose of decision-level cross-checking: focusing on the "consistency" and "historical correlation" of independent decision results of two types of sensors: on the one hand, verifying whether the decision results (such as target category determination, scene risk level) independently generated by radar and vision for the same target or scene match; on the other hand, combining the decision accuracy of two types of sensors in the same or similar scenes in history to evaluate the credibility of the current decision result, to ensure that the fusion at the decision level has reasonableness.

[0051] Figure 4is a flow chart of an adaptive weight fusion method based on heterogeneous data mutual verification according to another exemplary embodiment of the present disclosure, wherein the specific process of the signal level mutual verification in step 200 includes: evaluating the millimeter wave radar point cloud density, signal-to-noise ratio and RCS value distribution, evaluating the visual image definition, contrast and lighting condition, and then calculating the target detection consistency score in the coverage area of the two types of sensor data; wherein the visual image definition is evaluated by the mean or entropy value of the gray gradient, the contrast is evaluated by the standard deviation or variance of the gray value, the lighting condition is evaluated by the average gray value or brightness component, and the target detection consistency score is quantified by the spatial position overlap of the target.

[0052] According to an embodiment of the present application, the specific implementation process of the above signal level mutual verification is executed in the following manner: 1) Millimeter wave radar data quality evaluation The radar data quality is evaluated from three dimensions: first, the point cloud density is evaluated to determine whether the number of effective point clouds in a unit space range is sufficient to represent the target; second, the signal-to-noise ratio is evaluated to determine the proportion of effective signals and noise in the radar received signal, reflecting the data anti-interference ability; third, the RCS value distribution is evaluated to determine whether the target radar reflection cross section characteristics corresponding to the point cloud conform to the characteristics of typical targets (such as the RCS value range of vehicles and pedestrians is different), and to exclude abnormal reflection targets.

[0053] 2) Visual data quality evaluation The image data quality is evaluated from three dimensions: first, the image definition is evaluated by indicators such as mean or entropy value of gray gradient to determine whether the target edges and details in the image are clear; second, the image contrast is evaluated by indicators such as standard deviation or variance of gray value to determine whether the difference between light and dark areas in the image is reasonable, avoiding target blurring due to too low contrast; third, the lighting condition is evaluated by indicators such as average gray value or brightness component to determine whether the image has extreme lighting conditions such as over-bright (such as backlight) or over-dark (such as no lighting at night), ensuring that the influence of lighting on target detection is within an acceptable range.

[0054] 3) Target detection consistency score Based on the spatial positions of the targets detected by the two types of sensors, the target detection consistency score is generated by quantifying the degree of overlap of the target spatial positions (such as the proportion of the projection overlap area of the target in space), the higher the degree of overlap, the more consistent the detection of the same target by the two types of sensors, the higher the score; otherwise, the score is lower, reflecting the possibility of data deviation.

[0055] Figure 5Fig. 2 shows a flow chart of a method for adaptive weight fusion based on heterogeneous data mutual verification according to another example embodiment of the present disclosure, wherein the specific process of the feature-level mutual verification in step 200 includes: calculating consistency scores for position, size, and contour differences of the same target detected by two types of sensors, calculating consistency scores for speed and direction estimation differences of the moving target, and forming a feature-level mutual verification matrix recording the consistency scores of each feature dimension; wherein the position difference is quantified by the distance of the target center coordinates, the size difference is quantified by the parameter deviation of the target bounding box, and the contour difference is quantified by the contour distance metric, and the speed and direction difference are quantified by the speed vector similarity and size error.

[0056] According to an embodiment of the present disclosure, the specific implementation process of the feature-level mutual verification is performed in the following manner: 1) Static feature consistency evaluation For the same target detected by two types of sensors, calculate consistency scores from three static feature dimensions: first, the position difference, which quantifies the deviation of target position positioning by radar and vision through the spatial distance of target center coordinates; second, the size difference, which quantifies the difference in target size description by two types of data through the parameter deviation (such as length, width, and height) of the target bounding box; third, the contour difference, which quantifies the difference in target shape description by two types of data through the contour distance metric (such as the spatial distance of target contour key points).

[0057] 2) Dynamic feature consistency evaluation For moving targets, calculate consistency scores from the motion state dimension: quantify the difference in target motion state (speed and motion direction) estimation by two types of sensors through the speed vector similarity (such as the included angle of the speed direction) and the speed size error (such as the difference between the radar measured speed and the vision estimated speed).

[0058] 3) Generation of feature-level mutual verification matrix Organize the consistency scores of each dimension of the static and dynamic features into a feature-level mutual verification matrix, each element of the matrix corresponds to a consistency score of a feature dimension, which directly reflects the matching of two types of data in each feature dimension and provides a detailed basis for subsequent weight calculation.

[0059] Figure 6Fig. 2 shows a flow chart of a self-adaptive weight fusion method based on heterogeneous data mutual verification according to another exemplary embodiment of the present disclosure, wherein the specific process of the decision-level mutual verification in step 200 includes: comparing the category matching rate of the independent decision results of the two types of sensors, combining the decision accuracy in the same or similar historical scenarios to calculate the similarity between the current scenario and the historical scenario, and then generating a decision-level mutual verification score by weighted summation of the category matching rate, the historical decision accuracy and the scenario similarity; wherein the scenario similarity is quantified by the distance or similarity of the scenario features.

[0060] According to an embodiment of the present application, the specific implementation process of the above decision-level mutual verification is executed in the following manner: 1) Category matching rate calculation The independent decision results (such as target categories: "car", "pedestrian", "non-motor vehicle") generated by the millimeter wave radar and the visual sensor for the same target or scenario are compared, and the proportion of the number of the same decision results of the two types of sensors in the total number of decisions, i.e. the category matching rate, is calculated. The higher the matching rate, the stronger the preliminary consistency of the decisions of the two types of sensors.

[0061] 2) Historical decision accuracy reference The decision accuracy of the two types of sensors in the same or similar scenarios in the system historical operation data (such as the decision accuracy of the radar on the vehicle category and the decision accuracy of the visual on the vehicle category in the historical "urban road straight" scenario) is retrieved as a reference basis for the current decision credibility. The higher the historical accuracy, the higher the potential credibility of the current decision.

[0062] 3) Scenario similarity calculation The key features of the current scenario (such as weather conditions, road types, traffic flow, target types) are extracted and compared with the features of the historical scenarios, and the similarity between the current scenario and the historical scenario is quantified by the distance or similarity of the scenario features (such as the cosine similarity of the feature vector). The higher the similarity, the greater the reference value of the historical decision accuracy to the current decision.

[0063] 4) Generation of decision-level mutual verification score Reasonable weights are given to the above three indexes of category matching rate, historical decision accuracy and scenario similarity (such as assigning weights according to the influence of each index on the decision credibility), and the decision-level mutual verification score is generated by weighted summation, which comprehensively reflects the consistency and credibility of the decision results of the two types of sensors.

[0064] Figure 7Fig. 3 shows a flow chart of a self-adaptive weight fusion method based on heterogeneous data mutual verification according to another example embodiment of the present disclosure, wherein in step 300, 300-1 calculates a basic weight based on inherent performance characteristics of sensors and current environmental conditions, and then calculates a mutual verification coefficient in combination with the signal-level mutual verification score, the feature-level mutual verification score and the decision-level mutual verification score, and further calculates a historical score based on historical decision accuracy, scene similarity and a time decay factor, and then calculates an optimal weight according to the basic weight, the mutual verification coefficient and the historical score, and normalizes the optimal weight.

[0065] According to an embodiment of the present disclosure, the specific implementation process of the above adaptive weight calculation is executed in the following manner: 1) Basic weight calculation Based on inherent performance characteristics of sensors (e.g., radar is superior to vision in anti-interference ability in bad weather, and vision is superior to radar in target category recognition accuracy) and current environmental conditions (e.g., the influence of sunny day, rainy day, night, etc. on the two types of sensors is different), the initial basic weight is calculated first, which reflects the "inherent advantage ratio" of the sensor in the current environment, such as the basic weight of vision is higher than that of radar in sunny day, and the basic weight of radar is higher than that of vision in rainy day.

[0066] 2) Mutual verification coefficient calculation In combination with the signal-level mutual verification score, the feature-level mutual verification score and the decision-level mutual verification score generated in the previous step, reasonable weights are assigned to the three types of scores (e.g., the decision-level mutual verification has a greater impact on the fusion result, and the weight is higher), and the mutual verification coefficient is calculated by weighted calculation, which reflects the "consistency degree" of the two types of data at present. The higher the mutual verification coefficient is, the more matched the two types of data are, and the weight distribution can be more balanced; otherwise, it needs to be inclined to the sensor with more reliable data.

[0067] 3) Historical score calculation Based on the historical decision accuracy (e.g., the matching rate of the fusion decision and the real scene in a period of time), the similarity between the current scene and the historical scene (referring to the scene similarity calculation method of claim 6), and the time decay factor (e.g., the reference value of recent historical data is higher than that of long-term data, and the decay factor reduces the weight of long-term data), the historical score is calculated, which reflects the "reference value" of the historical data to the current weight. The higher the historical score is, the more the current weight can refer to the historical optimal weight distribution logic.

[0068] 4) Optimal weight and normalization The final weight is calculated by a reasonable mathematical model (such as multiplication or weighted summation) based on the above basic weight, mutual verification coefficient and historical score; then the final weight is normalized to ensure that the total weight of the two types of sensors meets the fusion calculation requirements (such as the total is 1), avoiding the deviation of the fusion result caused by the difference in weight value.

[0069] Figure 8 Fig. 4 shows a flow chart of an adaptive weight fusion method based on mutual verification of heterogeneous data according to another example embodiment of the present disclosure. In step 400, the signal-level fusion adopts a weighted average method or a Kalman filter algorithm to fuse the millimeter wave radar point cloud data and visual image features according to the signal-level normalized weight; the feature-level fusion adopts feature splicing or an attention mechanism to fuse the target detection, classification and attribute features according to the feature-level normalized weight; and the decision-level fusion adopts a voting method or D-S evidence theory to fuse the independent decision results of the two types of sensors according to the decision-level normalized weight.

[0070] According to an embodiment of the present disclosure, the above multi-level fusion is implemented in the following manner: (I) Signal-level fusion The weighted average method or Kalman filter algorithm is adopted to fuse the millimeter wave radar point cloud data and visual image features according to the signal-level normalized weight generated by claim 7: If the weighted average method is adopted, the contribution proportion of the two types of data is allocated according to the weight, the spatial information of the radar point cloud is combined with the pixel features of the visual image to generate environment representation data with wider coverage and richer details (such as containing both target spatial position and appearance details); If the Kalman filter algorithm is adopted, the noise in the two types of data is suppressed by using the anti-interference characteristic of the filter to generate more stable environment representation data, which is suitable for dynamic scenes (such as fast target motion).

[0071] (II) Feature-level fusion The feature splicing or attention mechanism is adopted to fuse the target features extracted by the two types of sensors according to the feature-level normalized weight: If the feature splicing is adopted, the target spatial features (such as position and speed) extracted by the radar and the target appearance features (such as contour and color) extracted by the visual are spliced into a unified feature vector according to the weight, forming a comprehensive target feature representation; If the attention mechanism is adopted, the more reliable feature dimensions in the two types of data are focused on according to the weight (such as giving higher attention weight to the radar speed feature when it is more reliable), the contribution of effective features is strengthened, and the influence of biased features is weakened, generating a target feature representation with higher accuracy.

[0072] (III) Decision-level fusion The independent decision results of the two types of sensors are fused according to the decision-level normalized weight by using the voting method or the D-S evidence theory: If the voting method is used, the decision results of the two types of sensors are counted according to the weight (for example, the decision of the sensor with a higher weight accounts for a higher proportion), and the decision supported by the majority is taken as the final fusion decision. If the D-S evidence theory is used, the decision results of the two types of sensors are taken as “evidence”, the confidence of each decision result is calculated according to the weight, and the decision with the highest confidence is selected as the final output, which is suitable for the scene where the decision result is uncertain (for example, the target category is fuzzy).

[0073] Figure 9 As shown in the flow chart of the adaptive weight fusion method based on mutual verification of heterogeneous data according to another example embodiment of the present disclosure, one embodiment of the present disclosure further includes step 500 of performing abnormality processing, result consistency verification and performance evaluation on the final fusion decision output by the output result optimization module, outputting the optimized final result, and feeding back the performance evaluation result to the adaptive weight calculation module for optimizing subsequent weight calculation.

[0074] According to an embodiment of the present application, the above step (5) can be performed in the following manner: 1) Abnormality processing Abnormality detection is performed on the generated final fusion decision output: for example, data that obviously do not conform to physical laws in the fusion result (for example, sudden large jumps in target speed, target position exceeding the reasonable spatial range), or results that deviate significantly from the historical trajectory; for the detected abnormal results, a reasonable correction method (for example, historical trajectory prediction correction, reference to more reliable sensor data correction) is used to ensure the reasonableness of the output result.

[0075] 2) Result consistency verification The consistency of the optimized fusion result with the original data of the two types of sensors and the historical fusion result is verified: for example, whether the features (position, size, category) of the fusion target match the core features of the original data of the sensors, and whether the time sequence of the fusion result is smoothly connected with the historical trajectory; if there is inconsistency, the previous steps (such as mutual verification and weight calculation) are traced back to find the deviation reason, and the result logic is ensured to be self-consistent.

[0076] 3) Performance evaluation and feedback The performance of the optimized fusion result is evaluated: for example, based on high-precision reference data (such as lidar data, manually annotated data), the accuracy and error range of the fusion result are evaluated; the performance evaluation result is fed back to the adaptive weight calculation step, which is used to adjust the parameters of subsequent weight calculation (such as correcting the distribution ratio of the basic weight, adjusting the weight of the mutual verification coefficient), realizing iterative optimization of weight calculation, and improving the accuracy of the subsequent fusion result.

[0077] Figure 10 Fig. 1 is a block diagram of a system for an adaptive weight fusion method based on mutual verification of heterogeneous data according to an exemplary embodiment of the present disclosure, which includes a data acquisition and preprocessing module 10, a multi-level mutual verification module 20, an adaptive weight calculation module 30, a multi-level fusion module 40, and an output result optimization module 50; the data acquisition and preprocessing module 10 is used to acquire millimeter wave radar data and visual data, perform preprocessing and time-space alignment; the multi-level mutual verification module 20 is used to perform signal level, feature level and decision level mutual verification and generate mutual verification scores of each level; the adaptive weight calculation module 30 is used to calculate basic weights, mutual verification coefficients, historical scores and normalized weights; the multi-level fusion module 40 is used to perform weighted fusion at the signal level, feature level and decision level; the output result optimization module 50 is used to perform abnormal processing, consistency verification and performance evaluation on the fusion result, and feed back the evaluation result to the adaptive weight calculation module.

[0078] According to an embodiment of the present application, the system of the present disclosure operates in the following structure and function: (I) System module composition The system includes five core functional modules: a data acquisition and preprocessing module 10, a multi-level mutual verification module 20, an adaptive weight calculation module 30, a multi-level fusion module 40, and an output result optimization module 50, each module realizes data interaction through a data interface (such as Ethernet, CAN bus), forming a complete data flow closed loop.

[0079] (II) Core functions of each module 1) Data acquisition and preprocessing module 10 Responsible for acquiring millimeter wave radar data and visual data, performing filtering, target detection, feature extraction operations on the two types of data through the built-in preprocessing algorithm (for details, refer to the preprocessing), and completing time-space alignment based on sensor calibration information and time stamp, outputting synchronous data to the multi-level mutual verification module and the multi-level fusion module.

[0080] 2) Multi-level mutual verification module 20 The synchronization data output by the data acquisition and preprocessing module is received, and the signal level, feature level and decision level mutual verification are sequentially performed according to the mutual verification logic, the mutual verification scores of each level are generated, and the scores are output to the adaptive weight calculation module.

[0081] 3) adaptive weight calculation module 30 The mutual verification scores output by the multi-level mutual verification module are received, the preset sensor performance parameters, the current environment information and the historical operation data are combined, the basic weight, the mutual verification coefficient and the historical score are calculated according to the weight calculation logic, the normalized weight is obtained, and the normalized weight is output to the multi-level fusion module.

[0082] 4) multi-level fusion module 40 The synchronization data of the data acquisition and preprocessing module and the normalized weight of the adaptive weight calculation module are received, the fusion operation is performed at the signal level, the feature level and the decision level respectively according to the fusion logic, the final fusion decision output is generated, and the final fusion decision output is transmitted to the output result optimization module.

[0083] 5) output result optimization module 50 The fusion decision output of the multi-level fusion module is received, the abnormality processing, the result consistency verification and the performance evaluation are performed according to the optimization logic, the optimized final result (such as being sent to an automatic driving decision system or a monitoring terminal) is output, and the performance evaluation result is fed back to the adaptive weight calculation module, so as to optimize the subsequent weight calculation and realize the iterative improvement of the system performance.

[0084] In summary, the system can stably realize the fusion method through the cooperative work of the above five modules, is suitable for the scenes such as automatic driving, intelligent monitoring, unmanned aerial vehicle navigation and the like which need heterogeneous sensor data fusion, has strong environmental adaptability and result reliability.

[0085] The terms "about" and "approximately" can be used to mean within ±20% of the target size in some embodiments, within ±10% of the target size in some embodiments, within ±5% of the target size in some embodiments, and also within ±2% of the target size in some embodiments. The terms "about" and "approximately" can include the target size. The technical solutions described herein can be implemented as a method, where at least one embodiment has been provided. The acts performed as part of the method can be ordered in any suitable way. Accordingly, embodiments can be constructed where acts are performed in an order different than illustrated, which can include performing some acts simultaneously, even though shown as being performed sequentially in illustrative embodiments. Further, the method can include more acts than those shown in some embodiments, and fewer acts than shown in other embodiments. While at least one illustrative embodiment of the application is described herein, various changes, modifications, and improvements can readily occur to those skilled in the art. Such changes, modifications, and improvements are intended to be within the spirit and scope of the application. Accordingly, the foregoing description is by way of example only and is not intended to be limiting. The application is limited only as defined in the following claims and the equivalents thereto. The exemplary embodiments of this disclosure are specifically illustrated and described herein. But, it will be appreciated that the disclosure is not limited to the specific configurations, arrangements or implementations described herein; rather, the disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An adaptive weight fusion method based on mutual verification of heterogeneous data, characterized in that, The method comprises the following steps: (1) The system is started and the parameters of each module are initialized, millimeter wave radar data and visual data are collected through the data acquisition and preprocessing module, the millimeter wave radar data and visual data are respectively preprocessed to extract target and environment related information, and the preprocessed two types of data are spatio-temporally aligned based on sensor calibration information and time stamps to generate synchronized data; (2) The multi-level mutual verification module is used to sequentially perform signal level mutual verification, feature level mutual verification and decision level mutual verification on the synchronized data to generate signal level mutual verification scores, feature level mutual verification scores and decision level mutual verification scores respectively; (3) The normalized weight is calculated by the adaptive weight calculation module; and (4) The multi-level fusion module is used to fuse the two types of sensor data at the signal level, feature level and decision level according to the normalized weight to generate enhanced environment representation data, unified target feature representation and final fusion decision output.

2. The method of claim 1, wherein, The preprocessing of the millimeter wave radar data in step (1) includes point cloud filtering, clustering and target extraction, wherein the point cloud filtering adopts statistical filtering or voxel filtering, the clustering adopts DBSCAN algorithm or K-means algorithm, and the target extraction filters target point cloud clusters based on the clustering results; the preprocessing of the visual data includes image enhancement, target detection and feature extraction, wherein the image enhancement adopts histogram equalization or Retinex algorithm, the target detection adopts YOLO algorithm or Faster R-CNN algorithm, and the feature extraction adopts HOG feature extraction algorithm or CNN feature extraction algorithm.

3. The method of claim 1, wherein the self-adaptive weight fusion method based on heterogeneous data mutual verification is characterized in that In step (2), the signal level mutual verification is used to evaluate the quality of the original data of the two types of sensors and the consistency of target detection coverage, the feature level mutual verification is used to evaluate the consistency of the target features and motion state estimation extracted by the two types of sensors, and the decision level mutual verification is used to evaluate the consistency of the independent decision results of the two types of sensors and the historical decision correlation.

4. The method of claim 2, wherein, The specific process of the signal level mutual verification in step (2) includes: evaluating the millimeter wave radar point cloud density, signal-to-noise ratio and RCS value distribution, evaluating the visual image sharpness, contrast and lighting conditions, and then calculating the target detection consistency score in the coverage area of the two types of sensor data; wherein the visual image sharpness is evaluated by the mean or entropy of the gray gradient, the contrast is evaluated by the standard deviation or variance of the gray value, the lighting condition is evaluated by the average gray value or brightness component, and the target detection consistency score is quantified by the spatial position overlap of the target.

5. The method of claim 2, wherein, The specific process of the feature level mutual verification in step (2) includes: calculating the consistency score of the position, size and contour difference of the same target detected by the two types of sensors, calculating the consistency score of the speed and direction estimation results of the moving target, and forming a feature level mutual verification matrix recording the consistency score of each feature dimension; wherein the position difference is quantified by the distance between the target center coordinates, the size difference is quantified by the parameter deviation of the target bounding box, the contour difference is quantified by the contour distance metric, and the speed and direction difference is quantified by the speed vector similarity and size error.

6. The method of claim 1, wherein, The specific process of the decision-level mutual verification in step (2) includes: comparing the category matching rate of the independent decision results of the two types of sensors, combining the decision accuracy in the same or similar historical scenes, calculating the similarity of the current scene and the historical scene, and then generating a decision-level mutual verification score by weighted summation of the category matching rate, the historical decision accuracy and the scene similarity; wherein the scene similarity is quantified by distance or similarity measurement of scene features.

7. The method of claim 1, wherein, In step (3), the base weight is calculated based on the inherent performance characteristics of the sensor and the current environmental conditions, and then the mutual verification coefficient is calculated by combining the signal-level mutual verification score, the feature-level mutual verification score and the decision-level mutual verification score, and the historical score is calculated based on the historical decision accuracy, the scene similarity and the time decay factor, and then the final weight is calculated according to the base weight, the mutual verification coefficient and the historical score, and the final weight is normalized.

8. The method of claim 1, wherein, In step (4), the signal-level fusion adopts a weighted average method or a Kalman filtering algorithm to fuse the millimeter wave radar point cloud data and the visual image features according to the signal-level normalized weight; the feature-level fusion adopts feature splicing or attention mechanism to fuse the target detection, classification and attribute features according to the feature-level normalized weight; and the decision-level fusion adopts a voting method or D-S evidence theory to fuse the independent decision results of the two types of sensors according to the decision-level normalized weight.

9. The method of claim 1, wherein Further comprising: (5) The output result optimization module is used for performing abnormal processing, result consistency verification and performance evaluation on the final fusion decision output, outputting the optimized final result, and feeding back the performance evaluation result to the adaptive weight calculation module for optimizing subsequent weight calculation.

10. A system for implementing the self-adaptive weight fusion method based on mutual verification of heterogeneous data according to any one of claims 1-9, characterized in that, The data acquisition and preprocessing module is used for acquiring millimeter wave radar data and visual data, and performing preprocessing and time-space alignment; the multi-level mutual verification module is used for performing signal-level, feature-level and decision-level mutual verification and generating mutual verification scores of each level; the adaptive weight calculation module is used for calculating base weight, mutual verification coefficient, historical score and normalized weight; the multi-level fusion module is used for performing weighted fusion at the signal level, the feature level and the decision level; and the output result optimization module is used for performing abnormal processing, consistency verification and performance evaluation on the fusion result, and feeding back the evaluation result to the adaptive weight calculation module. The data acquisition and preprocessing module is used for acquiring millimeter wave radar data and visual data, and performing preprocessing and time-space alignment; the multi-level mutual verification module is used for performing signal-level, feature-level and decision-level mutual verification and generating mutual verification scores of each level; the adaptive weight calculation module is used for calculating base weight, mutual verification coefficient, historical score and normalized weight; the multi-level fusion module is used for performing weighted fusion at the signal level, the feature level and the decision level; and the output result optimization module is used for performing abnormal processing, consistency verification and performance evaluation on the fusion result, and feeding back the evaluation result to the adaptive weight calculation module.

Images