A heterogeneous sensor information fusion method and device

Abstract

This invention discloses a method and apparatus for heterogeneous sensor information fusion. The method includes the following steps: S01. Acquiring data from two heterogeneous sensors and performing temporal and spatial registration to obtain registered data; S02. Based on the joint distribution of measurement information from the two heterogeneous sensors, acquiring measurement information from the two heterogeneous sensors at the same time from the registered data and performing correlation pairing to obtain correlated observation pairs; S03. Based on the joint distribution relationship between the measurement information and noise variance of the two heterogeneous sensors, identifying suspect measurement pairs from the correlated observation pairs, and fusing the measurement information of the suspect measurement pairs to obtain fused suspect measurement pairs; S04. Performing decision-level fusion identification based on the fused suspect measurement pairs to obtain the final identification result. This invention can realize heterogeneous sensor expression fusion and has the advantages of simple implementation, high fusion efficiency, good adaptability to complex scenarios, and strong anti-interference.

Description

Technical Field

[0001] This invention relates to the field of sensor information fusion technology, and in particular to a method and apparatus for heterogeneous sensor information fusion. Background Technology

[0002] Different types of sensors have different advantages. For example, radar can operate in all weather conditions, is small in size, and can measure speed and distance, while infrared sensors are less affected by lighting conditions and have accurate angle measurements. However, each individual sensor is limited by its own performance and function, resulting in relatively simple operating modes and target information acquisition. For instance, millimeter-wave radar produces sparse point cloud data, has many false alarms, and is susceptible to interference; visible light cameras are greatly affected by weather and lighting conditions; and infrared sensors have a small monitoring range and are greatly affected by temperature. Therefore, a single sensor is difficult to adapt to today's increasingly complex usage requirements. Thus, fusing information from different sensors, based on single-sensor detection, helps improve recognition accuracy and adaptability to complex environments.

[0003] Heterogeneous sensors detect different types of information. For example, millimeter-wave radar typically detects angles, distances, and velocities, while infrared imaging sensors typically obtain information such as position and angle within image pixels. These are sensor detection information with different dimensions, making direct information fusion impossible. Therefore, existing multi-sensor information fusion methods usually perform matching at the data association level. For instance, a common method for fusing millimeter-wave radar and infrared camera information involves first transforming the spatial coordinates of the two sensors, then mapping radar points to the infrared image pixel coordinate system, and finally performing a simple matching between the radar points and infrared image targets. However, this type of method struggles to represent and fuse information from heterogeneous sensors and often fails to achieve good results in complex environments. For example, millimeter-wave radar suffers from a high rate of false alarms, while infrared images have relatively simple texture and detail information. This can lead to matching infrared targets with radar false alarm points, or even the absence of infrared targets to match the actual radar targets.

[0004] Chinese patent application CN114994655A discloses a method for combined tracking of radar and infrared points based on AdaBoost. This method uses hypothetical tracks as training samples for machine learning, trains an AdaBoost classifier to classify hypothetical tracks as real or fake, and updates the combined tracks through filtering. However, this method relies on the AdaBoost classifier to classify real and fake points / tracks and on offline labeling of samples, thus requiring a large amount of prior information. This not only makes implementation complex but also results in poor adaptability to complex scenarios. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: in view of the technical problems existing in the prior art, the present invention provides a heterogeneous sensor information fusion method and device that can realize heterogeneous sensor expression fusion, which is simple to implement, has high fusion efficiency, good adaptability to complex scenarios, and strong anti-interference.

[0006] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0007] A method for information fusion from heterogeneous sensors, comprising the following steps:

[0008] Step S01. Acquire the data detected by the two heterogeneous sensors and perform time and space registration to obtain the registered data;

[0009] Step S02. Based on the joint distribution of the measurement information of the two heterogeneous sensors, obtain the measurement information of the two heterogeneous sensors at the same time from the registered data and perform correlation pairing to obtain the correlation observation pair;

[0010] Step S03. Based on the joint distribution relationship between the measurement information of the two heterogeneous sensors and the noise variance, identify the suspected measurement pair from the associated observation pair, and fuse the measurement information of the suspected measurement pair to obtain the fused suspected measurement pair;

[0011] Step S04. Perform decision-level identification based on the fused suspect measurement pairs to obtain the final identification result.

[0012] Furthermore, the measurement information includes azimuth and pitch angles. In step S02, a first statistical decision quantity T is constructed by using the azimuth and pitch angles obtained from two heterogeneous sensors, which follows a chi-square distribution with 2 degrees of freedom. When the first statistical decision quantity T is greater than a preset threshold λ, it is determined that the measurement values ​​of the two heterogeneous sensors to be judged are related observation pairs from the same location; otherwise, it is determined that they are not related observation pairs from the same location. The preset first threshold λ is set according to the chi-square distribution with 2 degrees of freedom.

[0013] Furthermore, the expression for calculating the first statistical decision quantity T is as follows:

[0014]

[0015] in, and These are the azimuth angle measurements obtained from the first sensor and the second sensor, respectively. and The pitch angle measurements obtained from the first and second sensors are σ and σ, respectively. Rθ , σ represents the variance of the azimuth and elevation angle measurements obtained from the first sensor. Iθ , These are the variances of the azimuth and pitch angle measurements obtained from the second sensor, respectively. The first and second sensors are heterogeneous sensors.

[0016] Furthermore, in step S03, based on the joint distribution relationship between the measurement information of the heterogeneous sensors and the noise variance, the suspected measurement pairs are identified from the associated observation pairs. This includes: using the azimuth and elevation angles obtained from the two heterogeneous sensors, as well as the noise variance of the azimuth angle measurement and the noise variance of the elevation angle measurement from the two heterogeneous sensors, to construct a chi-square distribution with 2 degrees of freedom. The second statistical decision measure, α, is the probability of misclassifying two different target observations as the same target observation. When the second statistical decision measure is greater than a preset second threshold... The measurement values ​​of the two heterogeneous sensors to be judged are determined to be a suspected measurement pair, wherein the preset second threshold is... Based on the chi-square distribution with 2 degrees of freedom The settings are obtained.

[0017] Furthermore, the expression for calculating the second statistical decision measure is as follows:

[0018]

[0019] in, This represents the second statistical decision quantity between the i-th measurement information from the first sensor and the j-th measurement information from the second sensor at time k. These represent the azimuth angle observation of the first sensor, the j-th azimuth angle observation of the second sensor, the i-th elevation angle observation of the first sensor, and the j-th elevation angle observation of the second sensor at time k, respectively. These are the azimuth measurement noise variance of the first sensor, the pitch measurement noise variance of the first sensor, the azimuth measurement noise variance of the second sensor, and the pitch measurement noise variance of the second sensor, respectively. The first sensor and the second sensor are heterogeneous sensors.

[0020] Furthermore, in step S03, the azimuth angle of the suspected measurement pair is weighted by the azimuth angle measurement noise variance of the two heterogeneous sensors to obtain the fused azimuth angle, and the elevation angle of the suspected measurement pair is weighted by the elevation angle measurement noise variance of the two heterogeneous sensors to obtain the fused elevation angle.

[0021] Furthermore, the fused azimuth angle and fused elevation angle are calculated using the following formulas:

[0022]

[0023]

[0024] in, Let v be the fused variance angle and fused pitch angle at time k, respectively. These represent the azimuth angle observation of the first sensor, the j-th azimuth angle observation of the second sensor, the i-th elevation angle observation of the first sensor, and the j-th elevation angle observation of the second sensor at time k, respectively. These are the azimuth measurement noise variance of the first sensor, the pitch measurement noise variance of the first sensor, the azimuth measurement noise variance of the second sensor, and the pitch measurement noise variance of the second sensor, respectively. The first sensor and the second sensor are heterogeneous sensors.

[0025] Furthermore, in step S04, the evidence theory method is used for decision-level identification. In this step, the confidence of the target category in the identification results obtained by the two sensors is used as evidence. The mutual support and conflict intensity between the information from the two heterogeneous sensors are calculated to measure the contribution of different sensor information to the final fused information. Then, the information from the two heterogeneous sensors is weighted according to the mutual support and conflict intensity to obtain the fusion result.

[0026] Furthermore, the steps for decision-level identification using evidence theory methods include:

[0027] S401. Parameter initialization: Take the target category confidence in the recognition results obtained by the two heterogeneous sensors as evidence, set the basic probability allocation function for any evidence, and calculate the conflict intensity value and mutual support value between each evidence.

[0028] S402. Conflict detection: Determine whether the mutual support value is greater than a preset threshold. If it is, fuse the evidence to obtain the current fusion confidence and proceed to step S404; otherwise, proceed to step S403.

[0029] S403. Calculate the overall distance between the two heterogeneous sensors and the fused evidence from the previous moment, and select the sensor with the smaller distance to the fused evidence from the previous moment as the evidence for the current period.

[0030] S404. Calculate the total support of all evidence for each evidence based on evidence obtained in the current period and multiple previous historical periods;

[0031] S405. Calculate a weight value based on the total support of all evidence to each other to adjust the weighted evidence m. WAE ;

[0032] S406. Calculate the weighted modified evidence m. WAE The fusion confidence of each category is obtained by multiple fusions, and the category corresponding to the highest confidence is determined as the final target category.

[0033] A heterogeneous sensor information fusion device, the device comprising:

[0034] The spatiotemporal registration module is used to acquire data detected by two heterogeneous sensors and perform time and space registration to obtain registered data.

[0035] The fusion detection module is used to obtain the measurement information of the two heterogeneous sensors at the same time from the registered data based on the joint distribution state of the measurement information of the two heterogeneous sensors, and perform correlation pairing to obtain the correlation observation pair;

[0036] The fusion tracking module is used to determine the suspected measurement pair from the associated observation pair based on the distribution relationship between the measurement information and noise variance of the two heterogeneous sensors, and to fuse the measurement information of the suspected measurement pair to obtain the fused suspected measurement pair.

[0037] The fusion identification module is used to perform decision-level identification based on the fused suspect measurement pairs to obtain the final identification result;

[0038] Alternatively, the device may include a processor and a memory for storing a computer program, and the processor for executing the computer program to perform the methods described above.

[0039] Compared with existing technologies, the advantages of this invention are as follows: This invention performs temporal and spatial registration of data detected by two heterogeneous sensors. During the fusion detection process, it uses a data association method to pair the target points based on the joint distribution of the target points from the two sensors, thus separating the measurements generated by the target from those generated by clutter. This allows for simple and efficient fusion of information from heterogeneous distributed sensors at the representation level. During the fusion tracking process, the measurement information of the associated observation pair is further fused based on the joint distribution relationship between the measurement information of the two heterogeneous sensors and the noise variance, effectively improving the matching accuracy. Ultimately, through decision-level recognition, accurate identification results can be obtained, significantly improving recognition performance. Attached Figure Description

[0040] Figure 1 This is a schematic diagram illustrating the overall implementation process of the radar sensor and infrared imaging sensor information fusion method in this embodiment.

[0041] Figure 2 This is a schematic diagram illustrating the implementation process of the radar sensor and infrared imaging sensor information fusion method in this embodiment.

[0042] Figure 3 This is a detailed flowchart illustrating the implementation process of fusion tracking using the adaptive variance weighted fusion method in this embodiment.

[0043] Figure 4This is a schematic diagram illustrating the implementation process of decision-level fusion identification using the evidence theory approach in this embodiment. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0045] Heterogeneous sensors are sensors with different structures that use different detection mechanisms to detect targets, such as radar sensors and infrared imaging sensors. Fusion of information from two heterogeneous sensors can effectively utilize the advantages of both, acquiring target information from different dimensions, and changing operating modes according to the environment and scene, thus improving recognition performance in complex environments. For example, fusing information from a radar sensor and an infrared imaging sensor provides highly complementary information. The infrared imaging sensor can acquire precise azimuth and elevation angles, geometric shape, and other apparent information of the infrared target, while the radar can acquire information such as the target's distance, velocity, acceleration, radar cross-section (RCS), and azimuth and elevation angles. This information together constitutes the target's multidimensional feature information. Furthermore, radar operates in an active mode and excels at detecting moving targets, but is significantly affected by electronic interference and reflected signals, while the infrared imaging sensor operates in a passive mode and is unaffected by electronic interference. The complementarity of these two sensors effectively improves environmental adaptability and recognition performance.

[0046] Although heterogeneous sensors have different detection mechanisms, the clutter detected by them is spatially randomly distributed. This means that the distribution of clutter points detected by the two sensors at the same time is generally different, while the target point distribution is relatively stable. In other words, the distribution of target points detected by heterogeneous sensors differs significantly from the distribution of clutter. Utilizing these distribution characteristics, this invention performs temporal and spatial registration of the data detected by two heterogeneous sensors. During the fusion detection process, it uses a data association method to pair (suspicious target points) based on the joint distribution of target points from both sensors. This separates the measurements generated by the target from those generated by the clutter, enabling simple and efficient fusion of information from heterogeneous distributed sensors at the representation level. During the fusion tracking process, the measurement information of the associated observation pair is further fused based on the joint distribution relationship between the measurement information and noise variance of the two heterogeneous sensors. This effectively improves the matching accuracy, resulting in precise identification results after decision-level recognition, thus significantly enhancing recognition performance.

[0047] The present invention will be specifically described below using the example of information fusion between radar sensor and infrared imaging sensor.

[0048] like Figure 1 , 2As shown, the detailed steps for information fusion between the radar sensor and the infrared imaging sensor in this embodiment include:

[0049] S01. Spatiotemporal registration: Acquire data detected by radar sensors (such as millimeter-wave radar) and infrared imaging sensors, perform temporal and spatial registration, and obtain registered data.

[0050] Radar and infrared imaging sensors are two sensors with different operating modes and significantly different characteristics. For example, their installation positions often differ, and their sampling frequencies are usually different, inevitably resulting in spatial deviations in the detected target information. Therefore, during information fusion, it is necessary to spatially and temporally register and align the observation models and measurement data from different sensors, calibrating the target information obtained by both sensors to the same spatial coordinate system. Spatial registration is the process of unifying the observation data acquired by different sensors into a single reference coordinate system.

[0051] Considering that the installation positions of millimeter-wave radar and infrared imaging sensors are relatively similar and the target information obtained by the two sensors is in the same reference coordinate system, this embodiment obtains the spatial difference between the two sensors only through static testing, and then spatially calibrates the information of the two sensors by transformation and translation.

[0052] Because millimeter-wave radar and infrared imaging sensors have different sampling frequencies and typically begin detecting targets at different times, a suitable reference time point needs to be selected to synchronize the observation data obtained from the millimeter-wave radar and infrared imaging sensors before fusing the information from the two sensors. In this embodiment, the time registration step specifically involves interpolating and extrapolating the target data collected by each sensor within the same time period to estimate the data at each moment within two observation periods, thereby achieving temporal synchronization of the two sensor information.

[0053] When the initial sampling times of the millimeter-wave radar and the infrared imaging sensor are the same, since the sampling frequency of the infrared imaging sensor is usually higher than that of the millimeter-wave radar, in order to avoid the loss of infrared sensor information, this embodiment specifically synchronizes the millimeter-wave radar data with the infrared data through interpolation and extrapolation; when the initial sampling times of the millimeter-wave radar and the infrared imaging sensor are different, it can be assumed that the first sampled data of the infrared imaging sensor was sampled at the same time as the most recent sampled data of the millimeter-wave radar. The slight error caused by this assumption is within the acceptable range.

[0054] Taking time registration in a specific application example, when using linear interpolation to implement interpolation and extrapolation, assume x R1 x is the value measured by the radar sensor at time t1. R2 x is the estimated value of the radar sensor at time t2. R3This is the measurement value taken by the radar sensor at time t3. Since the interval between t1 and t3 is very short, it can be approximated as a linear change by adjusting x. R1 and x R3 By performing linear interpolation, the estimated value of the radar sensor at time t2 can be obtained:

[0055]

[0056] This allows for the alignment of radar sensor data with lower sampling frequencies with infrared imaging sensor data.

[0057] For spatiotemporal registration between other heterogeneous sensors, the same principle as described above can be used, or other spatiotemporal registration methods can be used according to actual needs.

[0058] S02. Fusion detection: Based on the joint distribution of measurement information from two heterogeneous sensors, the measurement information of the two heterogeneous sensors at the same time is obtained from the registered data and associated and paired to obtain associated observation pairs.

[0059] After spatiotemporal registration, to fully utilize the information obtained from two heterogeneous distributed sensors, this embodiment uses a data association method to fuse and detect the measurement information from the two sensors. The measurement information specifically includes azimuth and elevation angles. A first statistical decision quantity T, following a chi-square distribution with two degrees of freedom, is constructed using the azimuth and elevation angles obtained from the radar sensor and infrared imaging sensor. When the first statistical decision quantity T is greater than a preset threshold λ, the measurement values ​​from the radar sensor and the infrared sensor are determined to be a pair of correlated observations from the same location; otherwise, they are determined not to be a pair of correlated observations from the same location. The preset first threshold λ is set according to the chi-square distribution with two degrees of freedom. The detailed process of constructing the first statistical decision quantity T in this embodiment is as follows:

[0060] First, at time k, acquire radar measurements, including the azimuth, elevation, and range of the suspected points detected by the radar. These measurements are the sum of the true values ​​and measurement noise. Assume that the measurement noise is independent and follows a normal distribution with a mean of zero. At the same time, acquire infrared imaging sensor measurements, including the azimuth and elevation of the suspected points detected by the infrared imaging sensor. Again, these measurements are the sum of the true values ​​and measurement noise, and it is assumed that the measurement noise is independent and follows a normal distribution with a mean of zero.

[0061] Assume the suspicious point at time k consists of a real target and several clutter points. Due to the different structures and detection mechanisms of radar and infrared imaging sensors, and the random spatial distribution of clutter, the clutter point distributions obtained by the two sensors at the same time are generally different, while the target point remains relatively stable. Therefore, a statistic that follows a standard normal distribution can be constructed:

[0062]

[0063]

[0064] in, and The azimuth measurements are obtained from millimeter-wave radar and infrared imaging sensors, respectively. and The elevation angle measurements σ are obtained from millimeter-wave radar and infrared imaging sensors, respectively. Rθ ,σ Iθ , denoted as , respectively, are the variances of the azimuth and elevation angle measurements of the millimeter-wave radar / infrared imaging sensor, and N(0,1) is a standard normal distribution.

[0065] As mentioned above, the joint distribution of measurement information from radar and infrared imaging sensors both follow a standard normal distribution, while clutter does not possess this characteristic. Therefore, it is possible to distinguish between genuine target points and clutter. Furthermore, using equations (2) and (3), a statistical decision quantity following a chi-square distribution with two degrees of freedom can be constructed using the azimuth and elevation angles obtained from the radar and infrared imaging sensors to determine whether a point is a suspicious target. This statistical decision quantity is the first statistical decision quantity T, and its calculation expression is as follows:

[0066]

[0067] in, and These are the azimuth angle measurements obtained from the radar sensor and the infrared sensor, respectively. and The elevation angle measurements σ are obtained from the radar sensor and the infrared sensor, respectively. Rθ , σ represents the variance of the azimuth and elevation angle measurements obtained from the radar sensor. Iθ , These represent the variances of the azimuth and elevation angle measurements obtained from the infrared sensors, respectively.

[0068] Based on the fact that the suspicious target points and clutter of the two sensors follow different joint distributions, the measurements generated by the target and the measurements generated by the clutter are first separated, and then the suspicious points generated by the two sensors are fused, which can effectively improve the matching accuracy and thus improve the final recognition performance.

[0069] Since the measurement values ​​generated by the target follow a chi-square distribution, while the measurement values ​​generated by clutter do not follow this distribution, this embodiment can first set the quantiles, and then obtain the first threshold λ when the degree of freedom is 2, based on the characteristics of the chi-square distribution. Then, the first statistical decision quantity T calculated by equation (4) can be used to determine whether the measurement values ​​of the radar sensor and the infrared sensor are suspicious target points from the same location, thereby realizing the association pairing and obtaining the associated observation pair. That is, when the first statistical decision quantity T is greater than the first threshold λ, it is determined that the measurement values ​​of the radar sensor to be judged and the measurement values ​​of the infrared sensor are associated observation pairs from the same location; otherwise, it is determined that they are not associated observation pairs from the same location.

[0070] In a specific application embodiment, the detailed steps for achieving fusion detection of the measurement values ​​from the radar sensor and the infrared sensor are as follows:

[0071] Step S21. Set the first threshold λ: First, set the quantiles, and obtain the value of the first threshold λ when the degrees of freedom are 2;

[0072] Step S22. Calculate the first statistical decision T for different suspicious target points according to formula (4).

[0073] Step S23. Compare the first statistic T with the first threshold λ. If T < λ, determine that the two current measurement values ​​are from the same suspicious target point. If T > λ, determine that the two current measurement values ​​are from different suspicious points.

[0074] Step S24. Traverse all combinations of suspicious target points in the current moment using the millimeter-wave radar and infrared sensor, and repeat step S23 until all suspicious target points have been traversed;

[0075] Step S25. If the measurement values ​​of the two sensors match, that is, they are suspicious target points from the same location, then record their location and proceed to step S3; otherwise, continue to the next moment of detection.

[0076] This embodiment achieves fusion detection by recognizing that the target point and clutter detected by the sensor follow different joint distributions. This effectively distinguishes the real target point from the clutter, thereby significantly improving the accuracy of point association matching.

[0077] S03. Fusion Tracking: Based on the joint distribution relationship between the measurement information and noise variance of the radar sensor and the infrared imaging sensor, suspect measurement pairs are identified from the associated observation pairs, and the measurement information of the suspect measurement pairs is fused to obtain the fused suspect measurement pairs.

[0078] Radar can provide not only high-precision range values ​​for targets, but also relatively low-precision elevation and azimuth angle measurements. Infrared imaging sensors can provide high-precision elevation and azimuth angle measurements. This embodiment combines the high-precision range values ​​measured by radar and the high-precision angle values ​​measured by infrared imaging sensors through fusion tracking, so as to improve the accuracy of target state estimation under certain conditions.

[0079] Based on the matching results of the fusion detection, this embodiment specifically employs a variance adaptive fusion tracking method to achieve fusion tracking. It uses a weighted "point-to-point" association logic, that is, weighting is performed based on the variance of the measurement noise. The weighting object is the weighted fusion of the original target state measurement values ​​(points) output by the sensor, followed by filtering to obtain a more accurate description of the target state. Figure 3 As shown.

[0080] In this embodiment, based on the joint distribution relationship between the measurement information and noise variance of the radar sensor and the infrared imaging sensor, the suspected measurement pairs are identified from the associated observation pairs. Specifically, this involves constructing a chi-square distribution with 2 degrees of freedom using the azimuth and elevation angles obtained from the radar sensor and the infrared imaging sensor, as well as the azimuth and elevation angle measurement noise variances from the radar sensor and the infrared imaging sensor. The second statistical decision measure, α, is the probability of misclassifying two different target observations as the same target observation. When the second statistical decision measure is greater than a preset second threshold... The two heterogeneous sensor measurements to be judged are determined to be a suspected measurement pair, with a preset second threshold. Based on the chi-square distribution with 2 degrees of freedom The settings are obtained.

[0081] The specific steps for constructing the aforementioned second statistical decision quantity are as follows:

[0082] set up These represent the radar's i-th azimuth observation at time k, the infrared imaging sensor's j-th azimuth observation, the radar's i-th elevation observation, and the infrared imaging sensor's j-th elevation observation, respectively. These are the noise variances for radar azimuth measurement, radar elevation measurement, infrared sensor azimuth measurement, and infrared sensor elevation measurement, respectively. Assuming that the observation noise from both the millimeter-wave radar and the infrared imaging sensor is white noise, therefore:

[0083]

[0084]

[0085]

[0086]

[0087] Therefore, the following hypothesis is established:

[0088]

[0089] The conditions under which H0 is true are:

[0090]

[0091] As shown in equation (6), the joint distribution of the measurement information and noise variance of the radar sensor and infrared sensor follows a normal distribution. Therefore, the second statistical decision quantity can be constructed according to the same principle as the first statistical decision quantity T. 2 (), as shown in equation (7), the second statistical decision quantity It follows a chi-square distribution with 2 degrees of freedom.

[0092]

[0093] Where α is the probability of misclassifying two different target observations as the same target observation. The second statistical decision quantity represents the relationship between the measurement information of the i-th radar sensor and the measurement information of the j-th infrared sensor at time k. These represent the i-th azimuth angle observation of the radar sensor, the j-th azimuth angle observation of the infrared sensor, the i-th elevation angle observation of the radar sensor, and the j-th elevation angle observation of the infrared sensor at time k, respectively. These are the azimuth measurement noise variance of the radar sensor, the elevation measurement noise variance of the radar sensor, the azimuth measurement noise variance of the infrared sensor, and the elevation measurement noise variance of the infrared sensor, respectively.

[0094] Therefore, the condition for 0 to be true is:

[0095]

[0096] By using the radar sensor and infrared imaging sensor observation pair that satisfy the above equation (8) as the suspected measurement pair, the target of interest can be found from the suspected target point pairs detected by fusion detection through the joint distribution relationship between the sensor measurement information and noise variance, and further weighted fusion can be performed.

[0097] In this embodiment, the azimuth angle of the suspected measurement pair obtained above is weighted by the noise variance of the azimuth angle measurement of the radar sensor and the infrared sensor to obtain the fused azimuth angle. The elevation angle of the suspected measurement pair obtained above is weighted by the noise variance of the elevation angle measurement of the two heterogeneous sensors to obtain the fused elevation angle.

[0098] The fused azimuth and fused elevation angles are calculated using the following formulas:

[0099]

[0100]

[0101] in, and These are the fused variance angle and fused pitch angle at time k, respectively.

[0102] In this embodiment, by weighting the variance of the suspected measurement according to equations (9) and (10), the suspected spurious observation can be obtained. Since the variance of sensor information with higher reliability contributes more to the fusion process, adaptive fusion of observation pairs can be achieved by weighting the fusion based on variance. Figure 3 As shown, after obtaining the fused angle value, the extended Kalman filter algorithm is used to update the trajectory, which can provide a more accurate description of the target state, effectively improve the matching accuracy, and further enhance the fusion effect.

[0103] S04. Fusion Identification: Decision-level identification is performed based on the fused suspect measurement pairs to obtain the final identification result.

[0104] Decision-level recognition involves fusing the preliminary recognition results from radar and infrared imaging sensors to determine the final outcome. Target information generated by millimeter-wave radar and infrared imaging sensors belongs to different dimensions, and the measurements from the two sensors have different units of measurement. This embodiment specifically uses an evidence theory approach for decision-level recognition. Specifically, the target category confidence scores from the recognition results obtained by the two sensors are used as evidence to convert the confidence scores of target information detected by different sensors into support scores. The mutual support and conflict strength between the radar and infrared imaging sensor information are calculated to measure the contribution of different sensor information to the final fused information. The fusion result is then obtained by weighting the two heterogeneous sensor information based on the mutual support and conflict strength. This process eliminates uncertainties in sensor information, distinguishes between "unknown" and "uncertain," and effectively handles conflicting information. This allows for the effective fusion of heterogeneous sensor information, obtaining the optimal information fusion scheme to improve recognition performance. Furthermore, it requires less prior knowledge, has more flexible application conditions, and can further enhance the environmental adaptability of the recognition system.

[0105] In this embodiment, during target fusion recognition, through steps S01 to S03, two sensors detect the target respectively. The radar detects the target's position, distance, speed, and other information, while the infrared imaging sensor detects the target's azimuth, pixel position in the image, and apparent contour. Matching and recognition are performed based on the information obtained from each sensor to obtain a preliminary recognition result. Based on this, step S04 uses an evidence theory method to further perform decision-level fusion processing on the recognition result. The implementation principle of the evidence theory method includes:

[0106] First, define the recognition frame Θ, and let m be a 2 Θ →A fundamental probability assignment (BPA) function for [0,1] satisfies:

[0107]

[0108] Where A represents the event, Let m represent the empty set. The corresponding trust function (Be l) for m is defined as:

[0109]

[0110] The trust function represents the overall level of trust in A, and the likelihood function (Pl) of A is defined as:

[0111]

[0112] This indicates that the level of trust in A is not negated.

[0113] Suppose two pieces of evidence and a sum for the same element O in the identification framework. i The trust intervals are [Bel(O i )1,Pl(O i )1] and [Bel(O i )2,Pl(O i )2], its distance can be expressed as:

[0114]

[0115] The distance of the confidence interval reflects both the difference in confidence between two pairs of evidence and the difference in their uncertainty; it is a weighted representation of confidence and uncertainty.

[0116] Based on the distance metric of the trust intervals of the same element on the identification framework between two pieces of evidence, let the overall distance between the two pieces of evidence be:

[0117]

[0118] Where n represents the number of elements in the identification frame, and C is the normalization factor, which is a constant.

[0119] The overall distance K between pieces of evidence reflects the conflict between them. When two pieces of evidence are completely identical, the conflict is zero. The greater the difference between the evidence, the greater the conflict. When the conflict between the evidence exceeds a certain level, sensor feature measurement information is used to determine its reliability, and then the confidence level of the evidence is corrected to eliminate the conflict problem.

[0120] like Figure 4 As shown, the detailed steps of using the evidence theory method for decision-level fusion identification in this embodiment include:

[0121] S401. Parameter Initialization: Using the target category confidence scores from the recognition results obtained by the two sensors as evidence, a basic probability assignment function is set for any evidence, and the conflict intensity value K(m) between each piece of evidence is calculated. R ,m I ) and mutual support value C(m R ,m I )=1-K(m R ,m I );

[0122] S402. Collision Detection: Compare the collision strength value with the threshold τ. If K(m) R ,m I If K(m) ≤ τ, it indicates that the sensor evidence has high mutual support. The evidence is fused to obtain the current fusion confidence m(t), and the process proceeds to step S404. R ,m I If the evidence is greater than τ, it indicates a strong conflict, and the process proceeds to step S403.

[0123] S403. Calculate the overall distance between each sensor and the fused evidence from the previous time step, and select the sensor with the smaller distance to the fused evidence from the previous time step as the evidence for the current period, that is, select the sensor with the smaller conflict with the previous time step as the confidence level for the current period. The calculation expression is as follows:

[0124]

[0125] S404. Calculate the total support of all evidence for each evidence based on the evidence obtained in the current period and the previous r-1 historical periods;

[0126] First, by combining the evidence obtained from the current period and the previous r-1 historical periods, the mutual support matrix is ​​calculated:

[0127]

[0128] in, C ij Let ∈[0,1], i,j∈{1,2,…,r} be the mutual support between pieces of evidence, and let C be the support between them.ij =C ji ,∑C ij =1.

[0129] In a mutual support matrix, all elements in a row represent the set of support levels of all pieces of evidence for a given piece of evidence. Therefore, the sum of all elements in a row represents the total support levels of all pieces of evidence for a given piece of evidence. For evidence m... i The total support of all evidence is The greater the overall support of an evidence, the more evidence it is supported by, and therefore the greater its credibility. When multiple pieces of evidence are involved in the fusion, it should receive greater weight.

[0130] S405. Calculate a weighted value based on the total support of all evidence for each piece of evidence, and then adjust the weighted evidence m accordingly. WAE .

[0131] The specific fusion weight for each piece of evidence is defined as follows:

[0132]

[0133] And calculate the weighted combination correction evidence based on the weights:

[0134] S406. Calculate the weighted modified evidence m WAE The fusion confidence of each category is obtained by multiple fusions, and the category corresponding to the highest confidence is determined as the final target category.

[0135] This embodiment achieves information fusion from heterogeneous sensors by sequentially performing spatiotemporal registration, fusion detection, fusion tracking, and fusion recognition. First, spatial and temporal registration and alignment are performed on the observation models and measurement data of different sensors, along with association of target information within the same field of view. After spatiotemporal registration, fusion detection comprehensively considers potential clutter from each sensor and makes logical judgments based on a first statistical decision quantity T for the detection results of different sensors, achieving association and pairing. Based on the results of fusion detection, fusion tracking performs fusion tracking on the measurement data of different sensors under tracking conditions according to a second statistical decision quantity. 2 The system performs correlation analysis to identify targets of interest and then uses adaptive variance-weighted fusion to fuse target state information, eliminating the effects of measurement errors, clutter, false alarms, and interference. Finally, the fusion recognition extracts the associated target characteristic information and achieves decision-level fusion recognition through evidence theory methods, thereby reducing the impact of scene clutter and interference on target recognition and improving target recognition accuracy.

[0136] In addition to achieving information fusion between radar sensors and infrared imaging sensors, this invention can also be applied to achieving information fusion between other heterogeneous and dissimilar sensors.

[0137] The heterogeneous sensor information fusion device in this embodiment includes:

[0138] The spatiotemporal registration module is used to acquire data detected by two heterogeneous sensors and perform time and space registration to obtain registered data.

[0139] The fusion detection module is used to obtain the measurement information of the two heterogeneous sensors at the same time from the registered data according to the distribution state of the measurement information of the two heterogeneous sensors, and perform correlation pairing to obtain the correlation observation pair;

[0140] The fusion tracking module is used to determine the suspected measurement pair from the associated observation pair based on the distribution relationship between the measurement information and noise variance of the two heterogeneous sensors, and to fuse the measurement information of the suspected measurement pair to obtain the fused suspected measurement pair.

[0141] The fusion identification module is used to perform decision-level fusion identification based on the fused suspect measurement pairs to obtain the final identification result.

[0142] The heterogeneous sensor information fusion device in this embodiment corresponds one-to-one with the heterogeneous sensor information fusion method in the above embodiment, and will not be described in detail here.

[0143] In another embodiment, the heterogeneous sensor information fusion device of the present invention may further include a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method as described above.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for information fusion from heterogeneous sensors, characterized in that the steps include... include: Step S01. Acquire the data detected by the two heterogeneous sensors and perform time and space registration to obtain the registered data; Step S02. Based on the joint distribution of the measurement information of the two heterogeneous sensors, obtain the measurement information of the two heterogeneous sensors at the same time from the registered data and perform correlation pairing to obtain the correlation observation pair; Step S03. Based on the joint distribution relationship between the measurement information of the two heterogeneous sensors and the noise variance, identify the suspected measurement pair from the associated observation pair, and fuse the measurement information of the suspected measurement pair to obtain the fused suspected measurement pair. The suspected measurement pair is the measurement value of the two heterogeneous sensors in the associated observation pair that satisfies the joint distribution. Step S04. Perform decision-level identification based on the fused suspect measurement pairs to obtain the final identification result.

2. The heterogeneous sensor information fusion method according to claim 1, characterized in that, The measurement information includes azimuth and elevation angles. In step S02, the azimuth and elevation angles obtained by two heterogeneous sensors are used to construct a first statistical decision quantity T that follows a chi-square distribution with 2 degrees of freedom. When the first statistical decision quantity T is greater than a preset first threshold... When the measurement values ​​of the two heterogeneous sensors to be judged are determined to be a correlated observation pair from the same location, otherwise they are determined not to be a correlated observation pair from the same location, wherein the preset first threshold is... It is obtained based on a chi-square distribution with 2 degrees of freedom.

3. The heterogeneous sensor information fusion method according to claim 2, characterized in that, The expression for calculating the first statistical decision value T is: ， in, and These are the azimuth angle measurements obtained from the first sensor and the second sensor, respectively. and The values ​​are the pitch angle measurements obtained from the first sensor and the second sensor, respectively. These are the variances of the azimuth and elevation angle measurements obtained from the first sensor, respectively. These are the variances of the azimuth and pitch angle measurements obtained from the second sensor, respectively. The first and second sensors are heterogeneous sensors.

4. The heterogeneous sensor information fusion method according to claim 1, characterized in that, In step S03, based on the joint distribution relationship between the measurement information of the heterogeneous sensors and the noise variance, the suspected measurement pairs are identified from the associated observation pairs. This includes: using the azimuth and elevation angles obtained from two heterogeneous sensors, and constructing the noise variances of the azimuth and elevation angle measurements from the two heterogeneous sensors to follow a chi-square distribution with 2 degrees of freedom. The second statistical judgment quantity, To calculate the probability of misclassifying two different target observations as the same target observation, when the second statistical decision quantity is greater than a preset second threshold... The measurement values ​​of the two heterogeneous sensors to be judged are determined to be a suspected measurement pair, wherein the preset second threshold is... Based on the chi-square distribution with 2 degrees of freedom The settings are obtained.

5. The heterogeneous sensor information fusion method according to claim 4, characterized in that, The expression for calculating the second statistical decision measure is: in, express Time of the first The first sensor measurement information and the first j The second statistical decision quantity between the measurement information of the second sensor. They are respectively The first sensor at time 1 One azimuth angle observation, the second sensor's first Azimuth angle observation, the first sensor's first... The first elevation angle observation and the second sensor's first Observation at each pitch angle, These are the azimuth measurement noise variance of the first sensor, the pitch measurement noise variance of the first sensor, the azimuth measurement noise variance of the second sensor, and the pitch measurement noise variance of the second sensor, respectively. The first sensor and the second sensor are heterogeneous sensors.

6. The heterogeneous sensor information fusion method according to any one of claims 1 to 5, characterized in that, In step S03, the azimuth angle of the suspected measurement pair is weighted by the azimuth angle measurement noise variance of the two heterogeneous sensors to obtain the fused azimuth angle. The elevation angle of the suspected measurement pair is weighted by the elevation angle measurement noise variance of the two heterogeneous sensors to obtain the fused elevation angle.

7. The heterogeneous sensor information fusion method according to claim 6, characterized in that, The fused azimuth and fused elevation angles are calculated using the following formulas: , , in, and They are respectively The variance angle and pitch angle after fusion at each moment. They are respectively The first sensor at time 1 One azimuth angle observation, the second sensor's first Azimuth angle observation, the first sensor's first... The first elevation angle observation and the second sensor's first Observation at each pitch angle, These are the azimuth measurement noise variance of the first sensor, the pitch measurement noise variance of the first sensor, the azimuth measurement noise variance of the second sensor, and the pitch measurement noise variance of the second sensor, respectively. The first sensor and the second sensor are heterogeneous sensors.

8. The heterogeneous sensor information fusion method according to any one of claims 1 to 5, characterized in that, In step S04, the evidence theory method is used for decision-level identification. The target category confidence in the identification results obtained by the two sensors is used as evidence. The mutual support and conflict intensity between the two heterogeneous sensor information are calculated to measure the contribution of different sensor information to the final fused information. Then, the two heterogeneous sensor information are weighted according to the mutual support and conflict intensity to obtain the fusion result.

9. The heterogeneous sensor information fusion method according to claim 8, characterized in that, The steps for decision-level fusion identification using evidence theory methods include: S401. Parameter initialization: Take the target category confidence in the recognition results obtained by the two heterogeneous sensors as evidence, set the basic probability allocation function for any evidence, and calculate the conflict intensity value and mutual support value between each evidence. S402. Conflict detection: Determine whether the mutual support value is greater than a preset threshold. If so, fuse the evidence to obtain the current fusion confidence and then proceed to step S404; otherwise, proceed to step S403. S403. Calculate the overall distance between the two heterogeneous sensors and the fused evidence from the previous moment, and select the sensor with the smaller distance to the fused evidence from the previous moment as the evidence for the current period. S404. Calculate the total support of all evidence for each evidence based on evidence obtained in the current period and multiple previous historical periods; S405. Calculate a weighted value based on the total support of all evidence to each other to adjust the evidence accordingly. ; S406. Calculate the weighted modified evidence. The fusion confidence of each category is obtained by multiple fusions, and the category corresponding to the highest confidence is determined as the final target category.

10. A heterogeneous sensor information fusion device, characterized in that, The device includes: The spatiotemporal registration module is used to acquire data detected by two heterogeneous sensors and perform time and space registration to obtain registered data. The fusion detection module is used to obtain the measurement information of the two heterogeneous sensors at the same time from the registered data based on the joint distribution state of the measurement information of the two heterogeneous sensors, and perform correlation pairing to obtain the correlation observation pair; The fusion tracking module is used to determine the suspected measurement pair from the associated observation pair based on the distribution relationship between the measurement information and noise variance of the two heterogeneous sensors, and to fuse the measurement information of the suspected measurement pair to obtain the fused suspected measurement pair. The suspected measurement pair is the measurement values ​​of the two heterogeneous sensors in the associated observation pair that satisfy the joint distribution. The fusion identification module is used to perform decision-level identification based on the fused suspect measurement pairs to obtain the final identification result; Alternatively, the device may include a processor and a memory for storing a computer program, and the processor for executing the computer program to perform the method as described in any one of claims 1 to 9.

Images