Distributed multi-source heterogeneous sensor detection data fusion method and system

Abstract

The invention relates to the field of vehicle safety, in particular to a distributed multi-source heterogeneous sensor detection data fusion method and system. The method comprises the following steps: firstly, carrying out hard and soft failure detection on original data through wavelet analysis and federated filtering to realize source data purification; then, carrying out cross-sensor consistency check on types, distances and speeds, and eliminating isolated features; and finally, constructing a feature information table by taking the distance as a benchmark, realizing accurate association between the multi-dimensional features and the target through norm difference calculation, and classifying for weighted fusion. Therefore, a progressive processing framework of validity judgment-consistency check-association fusion is formed, and the accuracy of data processing of the multi-source sensing system in scenes such as automatic driving and the like is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of vehicle safety, specifically to a method and system for fusing detection data from distributed multi-source heterogeneous sensors. Background Technology

[0002] Environmental perception, as a core component of systems such as autonomous driving and intelligent transportation, heavily relies on real-time data provided by multi-source heterogeneous sensors, including visual sensors (cameras) and radar sensors (LiDAR, millimeter-wave radar). By fusing this data, precise status, distance, and category information of traffic targets can be effectively obtained, providing crucial information for higher-level decision-making and control. However, the complex and ever-changing real-world driving environment poses significant challenges to the reliability, accuracy, and environmental adaptability of multi-sensor data fusion technology.

[0003] Currently, common multi-sensor data fusion schemes often employ fixed sensor combinations (such as LiDAR and camera combinations, millimeter-wave radar and camera combinations, etc.) and integrate data based on predetermined rules or algorithms. These methods typically focus on directly fusing data from similar sensors or calibrated data, for example, estimating the target's position and velocity using methods such as Kalman filtering and Bayesian estimation. However, these mainstream methods generally treat the validity and consistency of sensor data as ideal prerequisites, and their fusion logic is relatively mechanical, lacking an active mechanism for in-depth analysis and filtering of the data quality itself. This leads to the following technical problems: 1. Lack of intelligent judgment ability on the validity of sensor feature data. The system cannot effectively identify failure feature data caused by sensor momentary failure, signal interference or performance degradation, causing these abnormal data to directly enter the fusion process, becoming the main source of false detection, missed detection and result deviation.

[0004] 2. Lack of accurate association ability between multi-source features and target entities. Existing methods can usually only verify the consistency of similar features (such as distance) across different sensors, but cannot build a cross-dimensional matching knowledge system. This can easily lead to the problem of "association confusion" where multiple features are incorrectly matched to a single target, or a single feature is randomly matched to multiple targets. As a result, the fused target information cannot accurately reflect the real physical world.

[0005] It is evident that the aforementioned existing technologies are unable to effectively identify data sources and cannot achieve a distributed data fusion method that intelligently associates features with targets, thereby reducing the robustness and accuracy of perception systems in complex scenarios. Summary of the Invention

[0006] To address the technical problem of poor detection performance caused by the lack of real-time evaluation of sensor data validity, this invention provides a method for determining the validity of detection data from distributed multi-source heterogeneous sensors. Based on this validity determination method, to further address the technical problem of poor detection performance caused by the inaccurate association of corresponding data for the same target, this invention also provides a method for associating distributed multi-source heterogeneous sensor data. Based on the aforementioned validity determination method and data association method, this invention further provides a method and system for fusion of distributed multi-source heterogeneous sensor data.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for determining the validity of detection data from distributed multi-source heterogeneous sensors includes the following steps: Wavelet packet decomposition is performed on the detection time series data collected by the target detection sensor to obtain multiple frequency band signals. The energy value of each frequency band signal is calculated, and the detection time series data whose energy value changes do not exceed a set threshold are identified as hard effective feature data. The federated filter is used to perform state estimation on the detection time series data collected by the target detection sensor, and the cumulative error between the state estimate and the actual measurement value is calculated. The detection time series data with a continuous or decreasing cumulative error are identified as soft effective feature data. Detection time-series data that are simultaneously labeled as both hard and soft effective feature data are determined to be valid data.

[0008] As a further technical solution of the present invention: the energy value change does not exceed a set threshold, specifically expressed as follows: ; And / or: ; In the formula, E(t-1) represents the energy value at time t-1, and E(t) represents the energy value at time t; E th Indicates the set energy threshold; ΔE th This indicates the set threshold for the rate of change of energy.

[0009] As a further technical solution of the present invention: the cumulative error remains unchanged or decreases specifically means the following: ; ; ; In the formula, γ N This represents the cumulative error between the state estimate and the actual measurement up to time N. Indicates γ N The rate of change; γ N-1This represents the cumulative error between the estimated state value and the actual measured value up to time N-1. Indicates γ N-1 The rate of change; x k This represents the actual measured value at time k. This represents the state estimate at time k-1 for time k.

[0010] As a further technical solution of the present invention: the target detection sensor includes a camera, a lidar and a millimeter-wave radar; the target's attribute characteristics include target type, target distance and target speed; the detection time series data includes type feature data, distance feature data and speed feature data.

[0011] A method for associating distributed, multi-source, heterogeneous sensor data includes the following steps: A method for judging the validity of detection data from distributed multi-source heterogeneous sensors is adopted to judge the validity of the detection time series data collected by each target detection sensor, thereby obtaining the valid data of each target detection sensor. For the same type of feature data in the effective data of each target detection sensor, the consistency of target type, target distance and target speed are checked respectively, and isolated feature data that fail to pass cross-validation are removed; Based on the consistency test results, target detection sensors whose distance feature data is determined to be valid data are selected, and a set of valid sensors is constructed. For each target detection sensor in the set of valid sensors, an information table of target type-velocity-distance is established for each individual target detected. The norm difference of similar feature attribute data between information tables from different target detection sensors is calculated, and targets corresponding to information tables with norm differences less than a set norm threshold are associated as the same target.

[0012] As a further technical solution of the present invention: the target type consistency check includes the following steps: The point cloud contour of the target type detected by the lidar is projected onto the image coordinate system of the camera to obtain the projected contour. Normalize the target contour and the projected contour in the camera image respectively; Calculate the sum of squared errors between the two contours, and then calculate the similarity probability based on this. When the similarity probability is greater than the set type similarity threshold, it is determined that the type characteristics of the LiDAR and the camera are consistent, and the target type data detected by the LiDAR and the camera are consistent feature data.

[0013] As a further technical solution of the present invention, the distance consistency check includes the following steps: Acquire distance measurements from the camera, lidar, and millimeter-wave radar respectively; Construct a distance matching matrix between the camera, LiDAR, and millimeter-wave radar, where matrix element a ij This represents the matching result between a certain distance measurement value of target detection sensor i and a certain distance measurement value of target detection sensor j. The matching result is determined by whether the absolute value of the difference between the two distance measurements is less than a set distance error threshold. If it is less than the threshold, the two distance measurements are considered to support each other; otherwise, they are not considered to support each other. For any distance measurement, if it is supported by distance measurements from at least two other target detection sensors in the matching matrix, then that distance measurement is determined to be consistent feature data.

[0014] As a further technical solution of the present invention, the speed consistency verification includes the following steps: Acquire velocity measurements from lidar and millimeter-wave radar; Based on the assumption of a normal distribution of velocity measurements from two target detection sensors, calculate its probability density function; Calculate the confidence measure of the overlapping region of two probability density functions; When the confidence measure is greater than the set velocity consistency threshold, the velocity characteristics of the lidar and the millimeter-wave radar are determined to be consistent, and the corresponding velocity characteristic data is considered consistent characteristic data.

[0015] A distributed multi-source heterogeneous sensor data fusion method includes fusing various feature data of information tables associated with the same target through a distributed multi-source heterogeneous sensor data association method according to their categories, and assigning the fusion result to the target to obtain the target's final attribute feature data; for distance features and velocity features, a weighted average fusion algorithm is used for fusion; for type features, a confidence-weighted voting fusion algorithm is used for fusion.

[0016] A distributed multi-source heterogeneous sensor data fusion system includes: The validity assessment module is configured to execute a method for assessing the validity of data detected by distributed multi-source heterogeneous sensors, and to complete the data validity assessment. The data association module is configured to execute a distributed multi-source heterogeneous sensor data association method to complete the data association. The data fusion module is configured as a distributed multi-source heterogeneous sensor data fusion method to complete the data fusion.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. By introducing a "hard valid" criterion based on wavelet packet decomposition, abrupt changes in the signal frequency domain can be sensitively captured, identifying sudden abnormal data. Simultaneously, a "soft valid" criterion based on federated filtering can effectively monitor the cumulative trend of state estimation errors, thereby identifying progressive abnormal data caused by slow sensor performance degradation or calibration offsets. Through a logical AND operation of the "hard valid" and "soft valid" features, the final validity of the data is comprehensively determined, providing a foundation for subsequent targeted data processing and fault diagnosis. This significantly improves the input quality and reliability of the fusion system from the data source, providing high-quality, high-reliability input for the entire data fusion system and fundamentally reducing fusion bias, false positives, and false negatives caused by data quality issues.

[0018] 2. By integrating pre-emptive validity judgment and multi-dimensional consistency verification, and introducing a target association criterion based on information table norm difference calculation, we have achieved accurate and efficient mapping from raw data to target attribution. This effectively filters out invalid and inconsistent data, significantly improves the accuracy and reliability of feature-target association in multi-source heterogeneous sensor environments, and lays a solid foundation for subsequent fusion. Attached Figure Description

[0019] Figure 1 This is a flowchart of the fusion method in this invention.

[0020] Figure 2 This is a flowchart of the validity determination process in this invention.

[0021] Figure 3 This is a flowchart of the data association process in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1As shown, this invention provides a complete distributed multi-source heterogeneous sensor data fusion technology solution, the core of which lies in constructing a three-level progressive processing architecture. First, an abnormal data is identified and eliminated from the source through a dual validity judgment method based on wavelet packet decomposition and federated filtering, ensuring input quality. Second, an association method including three types of consistency checks—type, distance, and velocity—is proposed, achieving precise binding of multi-source features to the target through information table norm difference calculation. Based on this, a data fusion method based on weighted averaging and confidence voting is further defined, ultimately outputting a highly reliable target state description. Furthermore, the above methods are implemented as a fusion system containing corresponding modules, forming a complete protection system for the method and application products.

[0024] In this embodiment, the target detection sensors are commonly used in vehicles, such as cameras, lidar, and millimeter-wave radar. The three types of sensors work together to collect time-series detection data of targets in the traffic environment. The target attribute features extracted from these data include target type, target distance, and target speed. The corresponding time-series detection data are divided into type feature data, distance feature data, and speed feature data.

[0025] I. Data Validity Assessment

[0026] This section aims to extract target attribute features from time-series data collected by multiple sensors. It employs a dual validity detection mechanism based on time-series wavelet analysis and federated filtering to initially eliminate invalid features and ensure the reliability of the data foundation for subsequent processing.

[0027] 1. Timing detection data

[0028] Acquire time-series detection data from cameras, lidar, and millimeter-wave radar, and extract target type, distance, and velocity features from the time-series detection data.

[0029] Type: The LiDAR scans the rear of the vehicle, acquires point cloud data, and then performs clustering processing. The clustered point cloud contours are projected onto a two-dimensional plane to obtain clear vehicle rear contour features, which serve as the core basis for type feature data and provide a foundation for subsequent target type determination.

[0030] Distance: Combining the distance detection principle of radar sensors and the imaging principle of cameras, a geometric model is constructed to align the world coordinate system and the image coordinate system, enabling the estimation of the vehicle's true distance. Assume P(X w ,Y w Z w Let P be any point in the world coordinate system. Alignment of the coordinate axes is achieved using a rotation matrix R, and alignment of the origin of the coordinate system is achieved using a translation matrix T. The transformation process satisfies: ; In the above formula, Zc This represents the depth value in the camera coordinate system (i.e., the distance of a point in the world coordinate system along the camera's optical axis, usually in meters). u and v are the coordinates of point P projected onto the image, and u0 and v0 are the coordinates of the principal point of the image (the intersection of the camera's optical axis and the image plane, usually the center pixel coordinates of the image, in pixels); f x f y K1 represents the scale factor in the u and v directions of the pixel coordinate system, K2 is the camera's intrinsic parameter matrix, and K3 is the camera's extrinsic parameter matrix. This model enables coordinate transformation, accurately estimates the true distance to the target, and generates distance feature data. A fixed format for representing a zero vector and homogeneous coordinate transformation.

[0031] Speed: Based on the Doppler frequency shift principle, millimeter-wave radar obtains the radial relative speed V of the vehicle. R as follows: ; In the formula, f h λ is the radar's operating center frequency; λ is the wavelength.

[0032] LiDAR cannot directly obtain vehicle speed information, but it has high distance detection accuracy and can calculate the vehicle's relative speed by differentiating the distance from the sampling time. Let the radar sampling time be t. sample The velocity at time k+1 is calculated as follows: V R (k+1): ; In the formula, S(K+1) represents the target distance measured by the lidar at time k+1. S(K) represents the target distance measured by the lidar at time k.

[0033] 2. Wavelet Analysis

[0034] Wavelet analysis approximates signals by scaling (scale factor a) and shifting (shift factor b) the wavelet basis functions. For discrete signals acquired by target detection sensors, such as mechanical vibration signals acquired by vibration sensors with a preset sampling frequency, after discretization, the Mallat algorithm is used to implement fast wavelet transform, and the scaling coefficient c is used to approximate the signal. j and wavelet coefficients d j Complete the multi-scale decomposition of the signal.

[0035] Given that conventional multi-resolution analysis only performs recursive decomposition on low-frequency signals, resulting in insufficient resolution for high-frequency signals and difficulty in identifying extremely high-frequency interference signals, this embodiment introduces wavelet packet analysis to achieve refined decomposition of high-frequency signals. Wavelet packets reconstruct the multi-frequency space into orthogonal subspaces. Orthogonal decomposition of each subspace is achieved through a dual-scale equation: ; u n (t): Orthogonal basis functions at the nth scale, which can be scaling functions or wavelet functions, and are the basic basis for multiresolution analysis.

[0036] u 2n (t), u 2n+1 (t): Higher-scale basis functions generated by the nth-scale basis functions, corresponding to orthogonal bases after scale refinement, thereby improving resolution.

[0037] 2t - k: Scale and translation transformation terms, 2t performs scale refinement, and -k performs translation of basis functions.

[0038] h(k): Low-pass filter coefficients, corresponding to the refinement coefficients of the scaling function, used to generate "approximate components" of the signal.

[0039] g(k): High-pass filter coefficients, corresponding to the refinement coefficients of the wavelet function, used to generate the "detail components" of the signal.

[0040] Z: The summation variable covers all integers and is the global summation constraint for the two-scale equation.

[0041] The decomposition algorithm is as follows: ; : Discrete wavelet packet coefficients corresponding to the k-th discrete data at the j-th scale and the n-th wavelet packet subspace.

[0042] : Wavelet packet coefficients of the (j+1)th scale and the 2nth subspace obtained by decomposing the l-th discrete data from the j-th scale coefficients.

[0043] : Wavelet packet coefficients of the (j+1)th scale and (2n+1)th subspace obtained by decomposing the l-th discrete data from the j-th scale coefficients.

[0044] h k-2l : Low-pass filter coefficients corresponding to wavelet packet decomposition.

[0045] g k-2l : High-pass filter coefficients corresponding to wavelet packet decomposition.

[0046] j: The scale of wavelet packet decomposition.

[0047] Accordingly, the reconstruction algorithm is: ; : Discrete wavelet packet coefficients corresponding to the l-th discrete data at the j-th scale and the n-th wavelet packet subspace.

[0048] H l-2k : Low-pass filter coefficients corresponding to wavelet packet decomposition.

[0049] g l-2k : High-pass filter coefficients corresponding to wavelet packet decomposition.

[0050] After performing j-level wavelet packet decomposition on the vibration, displacement, and other signals detected by the sensor, the energy of each frequency band can be calculated using the square of the Euclid norm: ; The total energy of the original signal S is expressed as the square of the L2 norm of the signal.

[0051] : Discrete wavelet packet coefficients corresponding to the nth wavelet packet subspace at the j-th scale (l is the index of the coefficient, corresponding to the discrete sampling point of the signal); the value of n ranges from 0 to 2. j -1, corresponding to 2 at the j-th scale. j A space of orthogonal wavelet packets (the wavelet packet at scale j will divide the signal into 2... j (A number of non-overlapping orthogonal frequency bands).

[0052] : The squared L2 norm of the coefficients in the nth wavelet packet subspace at the j-th scale, representing the energy of the signal in the corresponding frequency band of that subspace.

[0053] In the above formula, each term corresponds to the energy signal at different frequencies. Based on the actual detection data collected by the vibration sensor, the multi-scale orthogonal decomposition of the wavelet packet analysis described above achieves refined feature extraction. When a sudden disappearance or appearance of energy value in a certain frequency band is detected, it is determined to be a hard failure feature; conversely, it is determined to be a hard effective feature, and the hard effective feature data is output. In other words, detection time-series data with energy value changes not exceeding a set threshold are identified as hard effective feature data. The energy value change not exceeding the set threshold is specifically expressed as follows: ; And / or: ; In the formula, E(t-1) represents the energy value at time t-1, and E(t) represents the energy value at time t; E th Indicates the set energy threshold; ΔE th This indicates the set threshold for the rate of change of energy.

[0054] 3. Federated Filtering

[0055] The federated filter employs a two-stage filtering architecture consisting of a main filter and sub-filters to fuse the feature parameters of multiple sensors. The sub-filters receive the feature parameters output by the sensors, independently calculate local state estimates and local covariance matrices, and then transmit the calculation results to the main filter for global fusion to obtain the globally optimal estimate.

[0056] Global State Estimation The value is calculated using the following formula: ; W i : The weighting coefficient of the i-th sub-sensor.

[0057] N: The number of sub-sensors / sub-filters participating in the fusion.

[0058] The local state estimate of the i-th sub-sensor is the state result obtained by the sensor through independent measurement and calculation.

[0059] Under ideal conditions where the sampled feature parameters are free from noise interference, the local state estimation of the sub-filter output. As an unbiased estimate, the global state estimate also satisfies unbiasedness, and its error covariance is minimized.

[0060] Based on the property of unbiased estimation, the global estimation error... The formula for calculating the covariance is: ; P g : The covariance matrix of the global state estimation is a symmetric positive definite matrix.

[0061] E{·}: Mathematical expectation operator, used to calculate the statistical average of the random variables within the brackets, here quantifying the statistical characteristics of the estimation error.

[0062] X: The actual value of the state.

[0063] T: Matrix transpose flag.

[0064] To ensure the effectiveness of the local estimates of the sub-filters as a basis for subsequent fusion, significant outliers need to be removed through feature parameter preprocessing. To minimize the global estimation error covariance, the optimal weighting matrix is ​​solved using the following formula: ; ; P ii : The local covariance matrix of the i-th sub-sensor, reflecting the uncertainty of the estimation result of the sensor alone.

[0065] The inverse matrix of the local covariance (information matrix) is positively correlated with the estimation accuracy of the sensor.

[0066] Based on the conservation of information, the noise is allocated to the sub-filter and the main filter according to the allocation coefficients, which satisfy the following: ; β i The information allocation coefficient of the i-th sub-filter represents the proportion of information allocated to that sub-filter. β m The information allocation coefficient of the main filter represents the proportion of information allocated to the main filter.

[0067] Based on the above allocation coefficients, the allocation method between the noise matrix and the state estimate is as follows: ; ; Q: The system noise covariance matrix reflects the uncertainty of system noise during the state estimation process.

[0068] P: State estimation covariance matrix, reflecting the uncertainty of the state estimation result.

[0069] To monitor the soft failure state of sensor characteristic parameters, a soft failure confidence factor is introduced. , representing the cumulative error between the state value and the estimated value up to time k: ; ; In the formula, γ N This represents the cumulative error between the state estimate and the actual measurement up to time N. Indicates γ N The rate of change; γ N-1 This represents the cumulative error between the estimated state value and the actual measured value up to time N-1. Indicates γ N-1 The rate of change; x k This represents the actual measured value at time k. This represents the state estimate at time k-1 for time k.

[0070] If the derivative of the confidence factor satisfies continuity This indicates that the characteristic parameters within the time series have accumulated errors, which is judged as a soft failure feature; otherwise, it is a soft effective feature; that is, the accumulated error remains unchanged or decreases ( The detection time series data is identified as soft effective feature data.

[0071] 4. Validity judgment

[0072] like Figure 2 As shown, features exhibiting either hard or soft failures are considered anomalous and do not participate in the fusion process; features without either failure are considered valid and participate in the fusion process. In other words, data is only considered valid and enters the subsequent processing flow if it is simultaneously identified as both hard and soft valid feature data; otherwise, it is considered anomalous data and is discarded.

[0073] As shown in the analysis in sections 1-4 of this part, firstly, three key attribute features—target type, distance, and velocity—are extracted from the raw data collected by multiple sensors such as cameras, lidar, and millimeter-wave radar. Then, a dual-path parallel validity detection mechanism is introduced: on the one hand, wavelet analysis based on time series is used to identify "hard failures" caused by transient interference or faults by monitoring abrupt changes in signal frequency domain energy; on the other hand, a federated filtering algorithm is used to detect "soft failures" caused by slow performance degradation by tracking the cumulative trend of state estimation errors. Finally, through a logical AND operation on the "hard validity" and "soft validity" features, only data that passes both tests is considered valid, systematically filtering out invalid and abnormal information at the source, laying a reliable foundation for subsequent high-quality data fusion.

[0074] II. Data Association

[0075] like Figure 3 As shown, this section aims to perform cross-sensor cross-validation on feature data that has passed the validity judgment, so as to ensure the synergy and reliability of multi-source data in describing the same physical target, and ultimately achieve accurate association between features and targets.

[0076] The core logic of cross-validation is as follows: for any feature data (such as a distance value, a velocity value, or a type determination) detected by any target detection sensor, its reliability is confirmed by comparing and logically judging the detection results of the same feature data from at least one other independent target detection sensor. If the feature data cannot be sufficiently supported by data from other target detection sensors, it is considered isolated feature data and is discarded. This invention designs targeted cross-validation mechanisms based on the different physical characteristics and sensor capabilities of the three types of features: type, distance, and velocity. Target type cross-validation: The lidar and the camera are compared pairwise through contour projection and similarity calculation. When the calculated similarity probability is greater than the set type similarity threshold, the test is passed and the lidar and the camera are consistent with each other in terms of the type feature data of the current target. Otherwise, the test is failed and the corresponding type feature data is considered inconsistent and will be removed as isolated data in subsequent steps.

[0077] Target velocity cross-validation: The lidar and millimeter-wave radar are compared pairwise using the probability density function overlap. If the calculated confidence measure is greater than the set velocity consistency threshold, the test is passed and the velocity feature data of the lidar and millimeter-wave radar for the current target are considered to be consistent. Otherwise, the test is failed and the corresponding velocity feature data is considered inconsistent and will be removed as isolated data in subsequent steps.

[0078] Target distance cross-validation: This is achieved by constructing a matching matrix and performing majority support judgments among the camera, LiDAR, and millimeter-wave radar (i.e., a distance value must be supported by at least two other target detection sensors). For any target detection sensor (such as a camera) measuring the distance to a specific target, its position within the matching matrix is ​​statistically analyzed. The number of "supports" obtained (i.e., the number of values ​​in the matrix corresponding to that value). The number of matching results. If the number is greater than or equal to 2 (i.e., supported by at least two other different sensors), the test is passed and the distance measurement is considered consistent feature data; if the number of supports is less than 2, the test is failed and the distance measurement is considered isolated data and will be discarded in subsequent steps.

[0079] 1. Type consistency check

[0080] The consistency between the target texture contour from the high-definition camera and the equivalent viewpoint contour from the LiDAR is verified. The rotation matrix R is obtained through calibration. L→C Translation matrix B L→C The three-dimensional coordinates P of the lidar L Projected into image coordinates P C : ; Furthermore, the image contour set and point cloud contour features are normalized: ; : The normalized value of the lidar-camera feature parameters corresponding to the i-th target, with a value range of [0,1]; the closer the value is to 1, the higher the matching degree between the lidar and camera features of the target.

[0081] : The original feature parameters of the lidar-camera corresponding to the i-th target.

[0082] : The minimum value among the original feature parameters of n targets to be matched.

[0083] : The maximum value among the original feature parameters of n targets to be matched.

[0084] Calculate the sum of squared errors ( : The lidar feature parameter vector corresponding to the i-th target, which is usually the contour coordinates and feature values ​​of the lidar point cloud projected onto the camera coordinate system; The camera feature parameter vector corresponding to the i-th target (usually the contour coordinates and feature values ​​of the target in the camera image) is introduced, along with a similarity probability R. 2 Conduct the inspection: ; In the above formula, , representing the sum of squares of the total deviations. The value, between 0 and 1, represents the probability of similarity between the data; the higher the value, the higher the similarity. When the similarity probability is greater than the set type similarity threshold, it is determined that the type characteristics of the LiDAR and the camera are consistent, and the target type data detected by the LiDAR and the camera are considered to have consistent feature data.

[0085] 2. Speed ​​Consistency Check

[0086] Based on the two-dimensional normal probability density function, the velocity consistency between lidar and millimeter-wave radar is tested. and These are velocity detection values ​​from lidar and millimeter-wave radar, respectively. The velocity of the target in the x-direction as measured by the lidar; : The velocity of the target in the y-direction as measured by the lidar; : The velocity of the target measured by millimeter-wave radar in the x-direction; : The target's velocity measured by millimeter-wave radar in the y-direction. (Note: The last part is a typological unit and doesn't need to be translated.) , , , The confidence measure for the calculated region is : ; : The lidar velocity probability density function corresponding to the i-th target, x l y l These are the x and y components of the lidar velocity.

[0087] : The millimeter-wave radar velocity probability density function corresponding to the i-th target, x r y r These are the x and y components of the velocity of the millimeter-wave radar.

[0088] ; : The standard deviation of the velocity x-direction component of the i-th target lidar, reflecting the degree of dispersion of the velocity measurement in that direction.

[0089] : The standard deviation of the y-direction component of the velocity of the i-th target lidar.

[0090] The correlation coefficient of the velocity components in the x and y directions of a lidar system reflects the degree of linear correlation between the velocities in the two directions.

[0091] ; : The standard deviation of the x-direction component of the velocity of the i-th target millimeter-wave radar.

[0092] : The standard deviation of the y-direction component of the millimeter-wave radar velocity of the i-th target.

[0093] : Correlation coefficients of the x and y components of velocity in millimeter-wave radar.

[0094] Representation function and Overlapping parts The larger the value, the greater the overlap, indicating that the target parameters are closer and the consistency is better. Conversely, when... When the confidence metric approaches 0, it indicates that the overlap ratio is close to zero, the velocity deviation is significant, and the consistency is poor. When the confidence metric is greater than the set velocity consistency threshold, it is determined that the velocity characteristics of the lidar and the millimeter-wave radar are consistent, and the corresponding velocity feature data is considered consistent feature data.

[0095] 3. Distance Consistency Check

[0096] By performing pairwise checks on the target distance values ​​of the camera (id=1), lidar (id=2), and millimeter-wave radar (id=3), the distance consistency test values ​​can form a matrix V. ss As shown below: ; in, This represents the matching matrix between sensor i and sensor j. Define i=j, Considering that the sensor may detect multiple targets, the expanded matrix A is as follows: ; Where, matrix element a ij This represents the matching result between a certain distance measurement value of target detection sensor i and a certain distance measurement value of target detection sensor j. m and k represent the number of targets detected by sensors i and j, respectively. This represents the distance matching result for the sensor target ID. When multiple sensors detect the same target, the distance error should be very small. Based on the error distribution pattern, a distance consistency check threshold is set as follows: ,definition: ; like If the distance detection of sensors i and j is inconsistent, then the corresponding distance measurements are not mutually supportive; conversely, if If a distance measurement is found to be consistent with other distance measurements, then the corresponding distance measurements are considered to be mutually supportive. For any distance measurement, if it is supported by distance measurements from at least two other target detection sensors in the matching matrix, then that distance measurement is determined to be consistent feature data.

[0097] The three consistency verification methods in this section aim to ensure the synergy and reliability of multi-source sensor data through distributed verification. For the three types of features that have passed validity checks—type, distance, and velocity—dedicated verification mechanisms are designed: type features are compared by the projection similarity between the lidar point cloud contour and the camera image contour; distance features are judged by constructing a cross-sensor matching matrix and calculating mutual support; and velocity features are judged based on the confidence measure calculated using the overlap of the probability density function. The core logic is to eliminate isolated feature data that fail cross-validation, thereby selecting a mutually corroborating and consistently reliable feature set in both spatial and temporal dimensions, laying the foundation for subsequent accurate association.

[0098] 4. Target association

[0099] Given that distance information is ubiquitous in various sensors and has relatively high accuracy, this embodiment chooses distance features as the association benchmark. Specifically, based on the results of the aforementioned distance consistency test, target detection sensors whose distance feature data are determined to be consistent (i.e., supported by at least two other sensors) are selected to form an effective sensor set for subsequent association.

[0100] By leveraging the commonalities in distance features acquired by various target detection sensors (cameras, LiDAR, millimeter-wave radar), and through distance consistency checks, a set of sensors with valid distance data is selected. For these selected valid sensors, internal associations are established based on their multi-dimensional single-target features, matching "type-distance" and "velocity-distance" correspondences to form single-sensor feature association pairs. An information representation system (information table) is then established based on these valid features. ; In the formula, U is the set of targets detected by the sensor. A=C∪D, where C is the set of conditional attributes, including type, speed, and distance; and D is the set of result attributes, i.e., target identifier. V represents a set of attribute values. aThe range of values ​​for 'a' represents the attribute feature value corresponding to the target, encompassing the collective measured values ​​of various features. f: U×A→V represents the information function, mapping the correspondence between the target and attribute values.

[0101] Let the datasets collected by lidar, millimeter-wave radar, and camera be respectively... (The set of certain feature data (such as range and velocity components) of the lidar corresponding to the 1st to the lth targets) (The set of millimeter-wave radar characteristic data corresponding to the 1st to the mth targets) (The set of similar camera features corresponding to the 1st to the kth targets), construct a multi-dimensional feature data information table; It should be noted that since the camera cannot obtain vehicle speed and the millimeter-wave radar cannot identify the type, the camera's speed attribute value is set to 0 and the millimeter-wave radar's type attribute value is set to 0 in the information table, and this type of zero value does not participate in the subsequent norm difference calculation.

[0102] Knowledge representations with multi-dimensional feature information can be derived. as follows: ; U: “Domain of discourse”, which is the set of all targets to be matched.

[0103] "Indistinguishable relation" indicates that in the condition attribute set Under (such as type, distance, speed), the attribute values ​​between data are completely consistent.

[0104] : indicates "for conditional attribute sets" Any attribute 'a' in the dataset.

[0105] : These represent attribute a in the lidar data. Camera data Millimeter-wave radar data The values ​​on attribute a are equal. If they are equal, it means that the values ​​of the three types of sensor data are completely consistent on attribute a.

[0106] Considering that cameras and millimeter-wave radars can only acquire two types of features, feature consistency checks need to be performed between each pair of sensors, such as the target feature knowledge representation formulas for lidar and cameras. As shown below: ; R 12 A specific attribute (or a subset of attributes).

[0107] In attribute R 12Under the constraints, the domain of discourse U is divided into multiple equivalence classes. Each equivalence class corresponds to a set of "LiDAR-camera feature data", representing different candidate targets.

[0108] The core matching rule is: if they are the same target, they must satisfy the distance norm of the matching parameters with the same features. It is 0, that is: ; The subsets that match in the knowledge table must satisfy this formula, and the feature data of the same type that match must be assigned the same value. A distance norm of zero is the optimal choice, but it is often difficult to achieve in actual detection. Therefore, a norm threshold is set (of course, the norm threshold can also be set to 0). When the distance norm is less than the norm threshold, the corresponding targets are the same target.

[0109] Norm determination method 1 (optimal selection): Calculate separately and determine independently.

[0110] Calculate a norm difference for type, speed, and distance, resulting in three values; an independent norm threshold needs to be set for each feature. Only when all three differences are less than their respective norm thresholds are they considered the same target.

[0111] Norm determination method two (second best): comprehensive calculation and unified determination

[0112] The effective differences of the three features are combined to form a multidimensional feature difference vector. Then, the comprehensive norm of this vector is calculated, requiring only a comprehensive norm threshold to be set. If the comprehensive norm is less than the comprehensive norm threshold, the features are determined to be the same target.

[0113] Based on the actual situation, one of the above norm judgment methods is adopted. By setting a norm threshold, when the norm difference between two information tables is less than the norm threshold, the corresponding targets are determined to be the same target, thus completing the accurate association between multi-sensor features and targets; if the norm difference is greater than or equal to the threshold, they are determined to be different targets, and other subsequent processing is performed separately.

[0114] The high-confidence distance features obtained after the consistency check in this section serve as the association benchmark. An information table covering attributes such as type, distance, and speed is established, and a knowledge representation system is used to standardize the feature data from different sensors. Based on this, the norm difference of similar feature attribute values ​​between different information tables is calculated, and combined with spatial proximity and physical logic constraints, multi-dimensional features describing the same entity are clustered and bound to the same target. This step ultimately outputs a structured set of target associations, fundamentally avoiding mismatches and confusion between features and targets, and achieving semantic-level integration from data to objects.

[0115] III. Data Fusion

[0116] The various feature data of each information table associated with the same target through the above association method are fused according to category, and the fusion result is assigned to the target to obtain the final attribute feature data of the target.

[0117] For distance and velocity features, a weighted average fusion algorithm is used for fusion; for type features, a confidence-weighted voting fusion algorithm is used for fusion.

[0118] IV. Fusion System

[0119] The fusion system can be deployed on the in-vehicle computing platform of intelligent driving vehicles, achieving highly reliable environmental perception through various modules. In terms of hardware, the fusion system can integrate multiple in-vehicle sensors (such as forward-facing cameras, solid-state LiDAR, and millimeter-wave radar) as data sources, and rely on in-vehicle domain controllers or high-performance embedded processors as the computing core.

[0120] 1. Validity Judgment Module

[0121] The validity assessment module is configured to receive raw time-series data collected by each sensor in real time and execute the validity assessment method described above.

[0122] The validity judgment module runs in parallel with the energy mutation detection logic based on wavelet packet decomposition and the cumulative error trend analysis logic based on federated filtering. It performs dual judgments on the input data as "hard valid" and "soft valid", and outputs a set of feature data marked as "valid".

[0123] 2. Data Association Module

[0124] The data association module receives "valid" feature data from the validity judgment module and executes the data association method described above.

[0125] First, cross-sensor consistency checks are performed on the type, distance, and velocity features in the valid data to eliminate isolated features. Then, a multi-dimensional feature information table is constructed for each detection target based on the check results. Finally, by calculating the norm difference of similar feature attribute values ​​between different information tables, feature data that meet the matching conditions are associated and bound to the same target.

[0126] 3. Data Fusion Module

[0127] The data fusion module receives the target association result output by the data association module and executes the data fusion method described above.

[0128] The data fusion module uses corresponding fusion strategies (such as weighted averaging of distance and speed features and confidence-weighted voting for type features) to calculate the final fusion attributes of the target (such as location, speed, type, and confidence) for various feature data associated with the same target, and outputs a structured target state list.

[0129] In actual operation, the three modules above process data sequentially in a pipeline manner. Time-series detection data flows through the validity judgment module for purification; the resulting valid data is then sent to the data association module for organization and attribution determination. The generated target association clusters are finally handed over to the data fusion module for information aggregation and status generation. Efficient data transmission between modules is achieved through shared memory or message queues, and consistent data processing timing is ensured through unified timestamp management.

[0130] Through the modular and process-oriented implementation described above, the fusion system achieves fully automated processing of multi-source heterogeneous sensor data, from "ensuring effectiveness" to "precise correlation" and then to "optimal fusion," ultimately outputting high-quality and highly reliable environmental perception information, providing a solid foundation for intelligent driving decision-making and control.

[0131] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for determining the validity of data detected by distributed multi-source heterogeneous sensors, characterized in that, Includes the following steps: Wavelet packet decomposition is performed on the detection time series data collected by the target detection sensor to obtain multiple frequency band signals. The energy value of each frequency band signal is calculated, and the detection time series data whose energy value changes do not exceed a set threshold are identified as hard effective feature data. The federated filter is used to perform state estimation on the detection time series data collected by the target detection sensor, and the cumulative error between the state estimate and the actual measurement value is calculated. The detection time series data with a continuous or decreasing cumulative error are identified as soft effective feature data. Detection time-series data that are simultaneously labeled as both hard and soft effective feature data are determined to be valid data.

2. The method for determining the validity of distributed multi-source heterogeneous sensor detection data according to claim 1, characterized in that, The energy value change not exceeding the set threshold is specifically expressed as follows: ； And / or: ； In the formula, E(t-1) represents the energy value at time t-1, and E(t) represents the energy value at time t; E th Indicates the set energy threshold; ΔE th This indicates the set threshold for the rate of change of energy.

3. The method for determining the validity of distributed multi-source heterogeneous sensor detection data according to claim 1 or 2, characterized in that, The cumulative error remains constant or decreases as specifically expressed as follows: ； ； ； In the formula, γ N This represents the cumulative error between the state estimate and the actual measurement up to time N. Indicates γ N The rate of change; γ N-1 This represents the cumulative error between the estimated state value and the actual measured value up to time N-1. Indicates γ N-1 The rate of change; x k This represents the actual measured value at time k. This represents the state estimate at time k-1 for time k.

4. The method for determining the validity of distributed multi-source heterogeneous sensor detection data according to claim 3, characterized in that, Target detection sensors include cameras, lidar, and millimeter-wave radar; target attribute characteristics include target type, target distance, and target velocity; detection time-series data includes type characteristic data, distance characteristic data, and velocity characteristic data.

5. A method for data association of distributed multi-source heterogeneous sensors, characterized in that, Includes the following steps: The validity judgment method for distributed multi-source heterogeneous sensor detection data as described in any one of claims 1-4 is used to judge the validity of the detection time series data collected by each target detection sensor to obtain the valid data of each target detection sensor. For the same type of feature data in the effective data of each target detection sensor, the consistency of target type, target distance and target speed are checked respectively, and isolated feature data that fail to pass cross-validation are removed; Based on the consistency test results, target detection sensors whose distance feature data is determined to be valid data are selected, and a set of valid sensors is constructed; for each target detection sensor in the set of valid sensors, an information table of target type-velocity-distance is established for each individual target detected by the target detection sensor. Calculate the norm difference between similar feature attribute data from information tables of different target detection sensors, and associate the targets corresponding to information tables whose norm difference is less than a set norm threshold as the same target.

6. The distributed multi-source heterogeneous sensor data association method according to claim 5, characterized in that, The target type consistency check includes the following steps: The point cloud contour of the target type detected by the lidar is projected onto the image coordinate system of the camera to obtain the projected contour. Normalize the target contour and the projected contour in the camera image respectively; Calculate the sum of squared errors between the two contours, and then calculate the similarity probability based on this. When the similarity probability is greater than the set type similarity threshold, it is determined that the type characteristics of the LiDAR and the camera are consistent, and the target type data detected by the LiDAR and the camera are consistent feature data.

7. The distributed multi-source heterogeneous sensor data association method according to claim 5, characterized in that, Distance consistency checks include the following steps: Acquire distance measurements from the camera, lidar, and millimeter-wave radar respectively; Construct a distance matching matrix between the camera, LiDAR, and millimeter-wave radar, where matrix element a ij This represents the matching result between a certain distance measurement value of target detection sensor i and a certain distance measurement value of target detection sensor j. The matching result is determined by whether the absolute value of the difference between the two distance measurements is less than a set distance error threshold. If it is less than the threshold, the two distance measurements are considered to support each other; otherwise, they are not considered to support each other. For any distance measurement, if it is supported by distance measurements from at least two other target detection sensors in the matching matrix, then that distance measurement is determined to be consistent feature data.

8. The distributed multi-source heterogeneous sensor data association method according to claim 5, characterized in that, Speed ​​consistency verification includes the following steps: Acquire velocity measurements from lidar and millimeter-wave radar; Based on the assumption of a normal distribution of velocity measurements from two target detection sensors, calculate its probability density function; Calculate the confidence measure of the overlapping region of two probability density functions; When the confidence measure is greater than the set velocity consistency threshold, the velocity characteristics of the lidar and the millimeter-wave radar are determined to be consistent, and the corresponding velocity characteristic data is considered consistent characteristic data.

9. A distributed multi-source heterogeneous sensor data fusion method, characterized in that, This includes fusing various feature data from information tables associated with the same target using the distributed multi-source heterogeneous sensor data association method as described in any one of claims 5-8, and assigning the fusion result to the target to obtain the target's final attribute feature data; for distance features and velocity features, a weighted average fusion algorithm is used for fusion. For type features, a confidence-weighted voting fusion algorithm is used for fusion.

10. A distributed multi-source heterogeneous sensor data fusion system, characterized in that, include: The validity judgment module is configured to execute the validity judgment method for distributed multi-source heterogeneous sensor detection data as described in any one of claims 1-4, and to complete the data validity judgment; The data association module is configured to execute the distributed multi-source heterogeneous sensor data association method as described in any one of claims 5-8 to complete the data association. The data fusion module is configured to execute the distributed multi-source heterogeneous sensor data fusion method as described in claim 9 to complete the data fusion.

Images