Multi-sensor decision-level information fusion method based on sentence vector and category similarity
By constructing a sentence corpus and training a sentence vector model to obtain a category similarity matrix, and combining DS evidence theory and piecewise function to correct the similarity matrix, the problem of high-conflict evidence in multi-sensor information fusion is solved, and more accurate decision-level information fusion is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAN MODERN CONTROL TECH RES INST
- Filing Date
- 2025-04-23
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies struggle to effectively handle evidence fusion problems with high conflict levels in multi-sensor information fusion, and they neglect the correlation and similarity between different target categories.
A multi-sensor decision-level information fusion method based on sentence vectors and category similarity is adopted. By constructing a sentence corpus and training a sentence vector semantic representation model, a category similarity matrix is obtained. The similarity matrix is then corrected using a piecewise function and fused with DS evidence theory. A non-maximum suppression strategy is used to determine the target category.
It effectively solved the problem of evidence fusion with high degree of conflict, obtained more accurate and reliable multi-sensor decision-level information fusion results, and improved the accuracy and reliability of information fusion.
Smart Images
Figure CN120654170B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information processing technology, and specifically to a multi-sensor decision-level information fusion method based on sentence vectors and category similarity. Background Technology
[0002] Multi-sensor information fusion, also known as data fusion or knowledge fusion, refers to the comprehensive processing of information, data, and knowledge from multiple sensors or data sources to obtain more comprehensive, accurate, and reliable conclusions. Multi-sensor information fusion has broad application prospects in fields such as medical diagnosis, intelligent transportation, security and protection, finance and economics, industrial control, and weapon systems, and has attracted widespread attention. Based on the fusion granularity, multi-sensor information fusion can be divided into signal-level (pixel-level), feature-level, and decision-level fusion.
[0003] The existing technologies, such as "A Dempster. Upper and Lower Probabilities Induced by A Multi-valued Mapping. Annals of Mathematical Statistics, 1967, 38: 325-339." and "A Mathematical Theory of Evidence" (G. Shafer. A Mathematical Theory of Evidence. Princeton, USA: Princeton University Press, 1976), propose the Dempster-Shafer (DS) evidence theory, which is a decision-level multi-sensor information fusion method based on the confidence function and uncertainty reasoning. This method is simple in principle and highly practical, and has been applied to the Baidu Apollo autonomous driving platform. However, it also suffers from the problem of exploding solution space dimensions and has difficulty handling evidence fusion problems with high degree of conflict.
[0004] In addition, existing technologies such as "Dempster-Shafer Framework and New Combination Rules", "A New Synthesis Formula Based on Evidence Theory", "An Improved Method of DS Evidence Reasoning", "Evidence Combination Method Based on Cosine Angle", and "Advanced Application of DSmT Algorithm in Information Fusion" mainly solve the problem of evidence fusion with high degree of conflict by relying on the numerical differences in the probability of different evidence sources to assign each target category. They ignore the correlation and similarity between different target categories. For example, the correlation and similarity between the two categories "tank" and "armored vehicle" are very high, while the similarity between the two categories "tank" and "aircraft" is relatively low. Summary of the Invention
[0005] The purpose of this invention is to provide a multi-sensor decision-level information fusion method based on sentence vectors and category similarity, which can efficiently handle evidence fusion problems with high degree of conflict and obtain more accurate and reliable multi-sensor decision-level information fusion results.
[0006] To achieve the above objectives, the present invention employs the following technical solution:
[0007] Multi-sensor decision-level information fusion methods based on sentence vectors and category similarity include:
[0008] Acquire the target recognition results from multiple different sensors;
[0009] Construct a sentence corpus and train a sentence vector semantic representation model;
[0010] Select an encoding template, concatenate each category with the encoding template to form a sentence, and encode the sentence using a trained sentence vector semantic representation model to obtain the sentence's encoding vector; for different category combinations, use the semantic similarity of the encoding vectors as the category similarity to obtain the target's category similarity matrix;
[0011] The category similarity matrix is corrected using a piecewise function;
[0012] Based on the corrected category similarity matrix, target recognition results from any two sensors are fused, and the fused result is then fused with the target recognition result from another sensor. This method is used to fuse all target recognition results sequentially, and the target category is determined based on the final fusion result using a non-maximum suppression strategy.
[0013] Furthermore, in the target identification results of the various different sensors, each identification result includes a probability that the target belongs to a different category.
[0014] Furthermore, a sentence corpus is constructed and a sentence vector semantic representation model is trained, including:
[0015] Randomly initialize the parameters of the sentence vector semantic representation model;
[0016] Construct a sentence corpus for object recognition, and use the sentences in the sentence corpus as the original samples;
[0017] For each original sample, construct positive samples and negative samples; the original sample and each positive sample form a positive sample pair, and the original sample and each negative sample form a negative sample pair;
[0018] The original sample, positive sample, and negative sample are encoded using a sentence vector semantic representation model to obtain the corresponding encoded vectors.
[0019] A sentence vector semantic representation model is trained using positive and negative sample pairs; the InfoNCE loss function is calculated during training; and the cosine similarity between samples contained in the positive and negative sample pairs is used as the semantic similarity in the InfoNCE loss function.
[0020] The parameters of the sentence vector semantic representation model are updated using stochastic gradient descent until the training termination condition is met, and the trained sentence vector semantic representation model is then saved.
[0021] Furthermore, the method for constructing positive samples is through synonym replacement, sentence-end punctuation change, and multilingual back translation; the method for constructing negative samples is to randomly sample other sentences from a corpus of sentences.
[0022] Furthermore, the category similarity matrix is corrected using a piecewise function, specifically as follows:
[0023] The following piecewise function is used to correct the category similarity matrix, so that the corrected category similarity sim(C) i C j The value of ) is between 0.0 and 1.0:
[0024]
[0025] Among them, sim(C i C j ) represents two categories C i and C j Category similarity.
[0026] Furthermore, when fusing the target recognition results of the two sensors, different target categories are first selected, the basic fusion probability and the probability allocation weight of the categories are calculated, and the allocation fusion probability of the categories is determined; all category combinations are traversed in turn, and the total fusion probability of each category is determined based on all the allocation fusion probabilities of each category; finally, the total fusion probability of all categories is normalized to obtain the fusion probability vector, which is used as the information fusion result of the selected two sensors.
[0027] Furthermore, the information fusion result of the two sensors is regarded as the target recognition result of a single virtual sensor. The target recognition result of a single virtual sensor is fused with that of another sensor using the method of fusing the target recognition results of the two sensors. Similarly, the target recognition results of other sensors are fused using the same method, and the target category is determined based on the final fusion result.
[0028] Furthermore, the target category is determined based on the final fusion result, specifically using a non-maximum suppression strategy to decide the target category:
[0029]
[0030] Where C Target For the target category obtained from the decision, P Radar_SAR_Infrared_Lidar (C i C represents the final fusion result. i For the i-th category.
[0031] A terminal device includes a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, it implements the multi-sensor decision-level information fusion method based on sentence vectors and category similarity.
[0032] A computer-readable storage medium storing a computer program; when executed by a processor, the computer program implements the multi-sensor decision-level information fusion method based on sentence vectors and category similarity.
[0033] Compared with the prior art, the present invention has the following technical features:
[0034] By leveraging intelligent technology and employing artificial intelligence models to better characterize the correlation and similarity between different categories, a new set of fusion reasoning rules based on the target category similarity matrix is proposed on the basis of DS evidence theory. This can better solve the problem of evidence fusion with high degree of conflict and obtain more accurate and reliable multi-sensor decision-level information fusion results. Attached Figure Description
[0035] Figure 1 This is a flowchart of the present invention;
[0036] Figure 2 This is a flowchart of the training process for the sentence vector semantic representation model.
[0037] Figure 3 Flowchart for calculating the target category similarity matrix;
[0038] Figure 4 A detailed flowchart for fusing information from any two sensors;
[0039] Figure 5 This is a flowchart of multi-sensor information fusion. Detailed Implementation
[0040] This invention provides a multi-sensor decision-level information fusion method based on sentence vectors and category similarity. First, the similarity of target categories is modeled as the semantic similarity of corresponding sentences, constructing a category similarity matrix, which is then corrected using a piecewise function. Next, based on DS evidence theory, the category similarity matrix is introduced, and a new set of fusion rules is proposed. This method can efficiently handle evidence fusion problems with high conflict levels, obtaining more accurate and reliable multi-sensor decision-level information fusion results. The specific implementation steps of this invention are as follows:
[0041] Step 1: Obtain the target identification results from multiple different sensors; each identification result includes the probability that the target belongs to a different category.
[0042] In one embodiment of the present invention, the various sensors include a two-dimensional active phased array radar, a synthetic aperture radar, an infrared sensor, and a lidar sensor, etc.; the category to which the target belongs is represented as: {C1, C2, C3, ..., C n}, where C n This represents the nth target category; the target identification results of each sensor are as follows:
[0043] (1) For the range information acquired by the two-dimensional active phased array radar, its one-dimensional range image information is obtained, and features such as aspect ratio, waveform entropy, first moment, second moment, third moment, first center distance, second center distance, and third center moment are extracted. The target is then classified using a support vector machine model to obtain the target recognition result:
[0044] P Radar =[P Radar (C1),P Radar (C2),P Radar (C3),…,P Radar (C n )]
[0045] Among them, P Radar (C i ) indicates that the target identified by the two-dimensional active phased array radar belongs to the i-th category C. i The probability of satisfies the normalization condition:
[0046] (2) For radar images acquired by synthetic aperture radar, denoising, deblurring, de-occlusion, and image super-resolution preprocessing are performed using a diffusion model. Target detection and recognition are then performed using the DINO target detection model based on the Transformer structure to obtain the target recognition results:
[0047] P SAR =[P SAR (C1),P SAR (C2),PSAR (C3),…,P SAR (C n )]
[0048] Among them, P SAR (C i ) indicates that the target identified by synthetic aperture radar belongs to the i-th category C. i The probability of satisfies the normalization condition:
[0049] (3) For infrared images acquired by infrared sensors, noise reduction, deblurring, de-occlusion, and image super-resolution preprocessing are performed using the Diffusion Model. Target detection and recognition are then performed using the DINO target detection model based on the Transformer structure to obtain the target recognition results.
[0050] P Infrared =[P Infrared (C1),P Infrared (C2),P Infrared (C3),…,P Infrared (C n )]
[0051] Among them, P Infrared (C i ) indicates that the target identified by the infrared sensor belongs to the i-th category C. i The probability of satisfies the normalization condition:
[0052] (4) For the 3D point cloud data acquired by the lidar sensor, outlier filtering algorithms such as ROR, SOR, DROR, and DSOR are used for noise reduction preprocessing. Then, the PV-RCNN 3D point cloud target detection model is used for target detection and recognition to obtain the target recognition results:
[0053] P Lidar =[P Lidar (C1),P Lidar (C2),P Lidar (C3),…,P Lidar (C n )]
[0054] Among them, P Lidar (C i ) indicates that the target identified by the infrared sensor belongs to the i-th category C. i The probability of satisfies the normalization condition:
[0055] Step 2: Construct a sentence corpus and train a sentence vector semantic representation model.
[0056] Step 21: Randomly initialize the parameters of the sentence vector semantic representation model; wherein the sentence vector semantic representation model adopts an existing model, such as the SIMCSE model.
[0057] Step 22: Construct a sentence corpus for target recognition. The corpus should contain at least a million sentences. The sources of the sentences in the corpus should include official documents, academic monographs, journal and conference papers, newspapers and magazines, the Internet, etc. These sentences will be used as the original samples.
[0058] Step 23: For each original sample, data augmentation is performed through methods such as synonym replacement, sentence-end punctuation change, and multilingual back translation to obtain positive samples. Each positive sample and the original sample constitute a positive sample pair.
[0059] Step 24: For each original sample, randomly sample several other sentences from the sentence corpus as negative samples; each negative sample and the original sample form a negative sample pair.
[0060] Step 25: Encode the original sample, positive sample, and negative sample using the sentence vector semantic representation model to obtain the encoded vectors s, s + ,s - .
[0061] Step 26: Train the sentence vector semantic representation model using positive and negative sample pairs; calculate the InfoNCE loss function during training:
[0062]
[0063] Wherein, cos_sim(s,s) + ) represents the cosine similarity between the encoding vectors of the original sample and the positive sample, and is used to characterize semantic similarity; Let represent the encoding vector of the j-th negative sample, M represent the number of negative samples, e is the natural constant, and τ is the temperature factor;
[0064]
[0065] The purpose of the InfoCSE loss function is to bring the vector representation of positive sample pairs closer together and push away the vector representation of negative sample pairs, so that the model can better reflect the semantic information of the sentence.
[0066] Step 27: Update the parameters of the sentence vector semantic representation model using stochastic gradient descent until the training termination condition is met, and save the trained sentence vector semantic representation model; for a new sentence, input the sentence into the trained sentence vector semantic representation model to obtain the encoding vector of the sentence.
[0067] Step 3: Select an encoding template, concatenate each category with the encoding template to form a sentence, and encode the sentence using the trained sentence vector semantic representation model to obtain the sentence encoding vector; for different category combinations, use the semantic similarity of the encoding vector as the category similarity to obtain the target category similarity matrix.
[0068] Step 31: Select an encoding template, which is user-defined; combine each category with the encoding template to form a sentence; taking the category "tank" as an example, the sentence formed by combining it with the encoding template is "the target may be a tank"; where "the target may be" is the encoding template; the encoding template is used together with the category to form sentences that describe different attributes or information of the target.
[0069] Step 32: For each sentence composed of a category and an encoding template, obtain the corresponding encoding vector using the trained sentence vector semantic representation model; where category C i The corresponding encoded vector is represented as s i ;
[0070] Step 33, for category C i and C j , and represent its corresponding encoding vector s i and s j The cosine similarity is used as their class similarity:
[0071] sim(C i C j ) = cos_sim(s i ,s j )
[0072] Step 34: Calculate the category similarity of all category combinations in turn to obtain the category similarity matrix of the target.
[0073] Step 4: Correct the category similarity matrix using a piecewise function.
[0074] The category similarity matrix is corrected using a piecewise function; for any two categories C i and C j Category similarity sim(C i C j The category similarity value calculated in step 3, i.e., the i-th row and j-th column of the category similarity matrix, should be between -1.0 and 1.0. However, to meet the requirements of the decision-level information fusion process, it is necessary to correct it using the following piecewise function so that the corrected category similarity sim(C) is within the range of -1.0 to 1.0. i C j The value of ) is between 0.0 and 1.0:
[0075]
[0076] By applying the above formula to correct all category similarities sequentially, we obtain the corrected category similarity matrix.
[0077] Step 5: Based on the corrected category similarity matrix, select any two sensors to fuse the target recognition results, and then fuse the fused result with the target recognition result of another sensor; use this method to fuse all target recognition results in sequence, and determine the target category based on the final fusion result using a non-maximum suppression strategy.
[0078] Step 51: When fusing the target recognition results of the two sensors, first select different target categories, calculate the basic fusion probability and the probability allocation weight of the categories, and determine the allocation fusion probability of the categories; then iterate through all category combinations in turn, and determine the total fusion probability of each category based on all the allocation fusion probabilities of each category; finally, normalize the total fusion probability of all categories to obtain the fusion probability vector, which is used as the information fusion result of the selected two sensors.
[0079] (1) Fusion of target identification results from two-dimensional active phased array radar and synthetic aperture radar; if the target identified by the two-dimensional active phased array radar is category C i The probability is P Radar (C i The synthetic aperture radar identifies the target as category C. j The probability is P SAR (C j The basic fusion probability is calculated as follows:
[0080] P Fusion (C i C j )=(P Radar (C i )+λsim(C i C j )P SAR (C j ))(P SAR (C j )+λsim(C i C j )P Radar (C i ))
[0081] Where λ is a pre-defined parameter factor, representing the importance of category similarity information.
[0082] (2) If the two-dimensional active phased array radar identifies the target as category C j The probability is P Radar (C jThe synthetic aperture radar identifies the target as category C. i The probability is P SAR (C i The basic fusion probability is calculated as follows:
[0083] P′ Fusion (C j C i )=(P Radar (C j )+λsim(C i C j )P SAR (C i ))(P SAR (C i )+λsim(C i C j )P Radar (C j ))
[0084] (3) Calculate category C for the cases in steps (1) and (2) respectively. i Probability allocation weight:
[0085]
[0086] (4) Calculate category C in steps (1) and (2) respectively. i Allocation fusion probability:
[0087] P Fusion1 (C i |C i C j )=α(C i )P Fusion (C i C j )
[0088] P Fusion2 (C i |C j C i )=β(C i )P′ Fusion (C j C i )
[0089] (5) Take all category combinations in turn and calculate the corresponding allocation and fusion probability;
[0090] (6) Calculate the total fusion probability for category C. i The total fusion probability is:
[0091]
[0092] (7) Normalize the total fusion probability of all categories to obtain the fusion probability vector, which serves as the information fusion result of the two-dimensional active phased array radar and the synthetic aperture radar:
[0093]
[0094] Step 52: Treat the information fusion result of the two sensors as the target recognition result of a single virtual sensor, and use the above method of fusing the target recognition results of the two sensors to fuse the target recognition results of a single virtual sensor with those of another sensor; and so on, fuse the target recognition results of other sensors in the same way, and determine the target category based on the final fusion result.
[0095] The obtained fusion result P Radar_SAR (C i The result is considered as the target recognition result of a single virtual sensor. Using the same method as in step 51, the target recognition result of the single virtual sensor is fused with the target recognition result of the infrared sensor to obtain a new fusion result P. Radar_SAR_Infrared (C i ).
[0096] The obtained fusion result P Radar_SAR_Infrared (C i The result P is considered as the target recognition result of a single virtual sensor. Using the same method as in step 51, the target recognition result of the single virtual sensor is fused with the target recognition result of the lidar sensor to obtain a new fusion result P. Radar_SAR_Infrared_Lidar (C i ).
[0097] Finally, regarding the final fusion result P Radar_SAR_Infrared_Lidar (C i The non-maximum suppression strategy is used to determine the target category.
[0098]
[0099] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A multi-sensor decision-level information fusion method based on sentence vectors and category similarity, characterized in that, include: Acquire the target recognition results from multiple different sensors; Construct a sentence corpus and train a sentence vector semantic representation model; Select an encoding template, concatenate each category with the encoding template to form a sentence, and encode the sentence using a trained sentence vector semantic representation model to obtain the sentence's encoding vector; for different category combinations, use the semantic similarity of the encoding vectors as the category similarity to obtain the target's category similarity matrix; The category similarity matrix is corrected using a piecewise function, including: The following piecewise function is used to correct the category similarity matrix, resulting in a corrected category similarity matrix. The value ranges from 0.0 to 1.0: in, Representing two categories C i and C j The category similarity is calculated using cosine similarity. Based on the corrected category similarity matrix, target recognition results from any two sensors are fused, and the fused result is then fused with the target recognition result from another sensor. This method is used to fuse all target recognition results sequentially, and the target category is determined based on the final fusion result using a non-maximum suppression strategy.
2. The multi-sensor decision-level information fusion method based on sentence vectors and category similarity according to claim 1, characterized in that, In the target identification results of the various different sensors, each identification result includes the probability that the target belongs to a different category.
3. The multi-sensor decision-level information fusion method based on sentence vectors and category similarity according to claim 1, characterized in that, Constructing a sentence corpus and training a sentence vector semantic representation model includes: Randomly initialize the parameters of the sentence vector semantic representation model; Construct a sentence corpus for object recognition, and use the sentences in the sentence corpus as the original samples; For each original sample, construct positive samples and negative samples; the original sample and each positive sample form a positive sample pair, and the original sample and each negative sample form a negative sample pair; The original sample, positive sample, and negative sample are encoded using a sentence vector semantic representation model to obtain the corresponding encoded vectors. A sentence vector semantic representation model is trained using positive and negative sample pairs; the InfoNCE loss function is calculated during training; and the cosine similarity between samples contained in the positive and negative sample pairs is used as the semantic similarity in the InfoNCE loss function. The parameters of the sentence vector semantic representation model are updated using stochastic gradient descent until the training termination condition is met, and the trained sentence vector semantic representation model is then saved.
4. The multi-sensor decision-level information fusion method based on sentence vectors and category similarity according to claim 3, characterized in that, The method for constructing positive samples is through synonym replacement, sentence-end punctuation change, and multilingual back translation; the method for constructing negative samples is to randomly sample other sentences from a corpus of sentences.
5. The multi-sensor decision-level information fusion method based on sentence vectors and category similarity according to claim 1, characterized in that, When fusing the target recognition results of two sensors, different target categories are first selected, the basic fusion probability and the probability allocation weight of each category are calculated, and the category allocation fusion probability is determined. The process iterates through all category combinations and determines the total fusion probability based on the assigned fusion probabilities for each category. Finally, the total fusion probabilities of all categories are normalized to obtain the fusion probability vector, which serves as the information fusion result of the two selected sensors.
6. The multi-sensor decision-level information fusion method based on sentence vectors and category similarity according to claim 5, characterized in that, The information fusion result of the two sensors is regarded as the target recognition result of a single virtual sensor. The target recognition results of the single virtual sensor are fused with those of the other sensor using the same method. The target category is determined based on the final fusion result.
7. The multi-sensor decision-level information fusion method based on sentence vectors and category similarity according to claim 1, characterized in that, The target category is determined based on the final fusion result, specifically using a non-maximum suppression strategy to decide the target category: in The target category obtained for decision-making, For the final fusion result, For the first There are several categories.
8. A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, When the processor executes the computer program, it implements the multi-sensor decision-level information fusion method based on sentence vectors and category similarity as described in any one of claims 1-7.
9. A computer-readable storage medium storing a computer program; characterized in that, When the computer program is executed by the processor, it implements the multi-sensor decision-level information fusion method based on sentence vectors and category similarity as described in any one of claims 1-7.