Multi-sensor decision-level information fusion method based on sentence vector and category similarity

Abstract

The invention discloses a multi-sensor decision-level information fusion method based on sentence vector and category similarity. The method comprises the following steps: acquiring recognition results of a plurality of different sensors on a target; constructing a sentence corpus and training a sentence vector semantic representation model; selecting a coding template, splicing each category with the coding template to form a sentence, and obtaining a coding vector of the sentence by utilizing the trained sentence vector semantic representation model; for different category combinations, using semantic similarity of the coding vectors as category similarity to obtain a category similarity matrix of the target; correcting the category similarity matrix by using a piecewise function; based on the corrected category similarity matrix, target recognition results of any two sensors are selected to be fused, and a fusion result is fused with a target recognition result of the other sensor; through the method, fusion of all target identification results is completed in sequence, and a target category is determined through a non-maximum suppression strategy based on a final fusion result.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a multi-sensor decision-level information fusion method based on sentence vectors and category similarity. Background Art

[0002] Multi-sensor information fusion, also known as data fusion or knowledge fusion, involves the integrated processing of information, data, and knowledge from multiple sensors or data sources to derive more comprehensive, accurate, and reliable conclusions. Multi-sensor information fusion has broad application prospects in fields such as medical diagnosis, smart transportation, security and protection, finance and economics, industrial control, and weapon systems, and has garnered widespread attention. Multi-sensor information fusion can be categorized by fusion granularity: signal level (pixel level), feature level, and decision level.

[0003] In the existing technology, "A Dempster. Upper and Lower Probabilities Induced by A Multi-valued Mapping. Annals of Mathematical Statistics, 1967, 38: 325-339." and "A Mathematical Theory of Evidence" (G Shafer. A Mathematical Theory of Evidence. Princeton, USA: Princeton University Press, 1976) proposed the DS (Dempster-Shafer, DS) evidence theory, which is a decision-level multi-sensor information fusion method based on trust function and uncertainty reasoning. This method is simple in principle and highly practical. It has been applied to Baidu's Apollo autonomous driving platform, but it also has the problem of solution space dimensionality explosion and difficulty in handling the problem of high-conflict evidence fusion.

[0004] In addition, the existing technologies such as "Dempster-Shafer Framework and New Combination Rules", "A New Synthesis Formula Based on Evidence Theory", "An Improved Method for DS Evidence Reasoning", "Evidence Combination Method Based on Angle Cosine", and "Advanced Application of DSmT Algorithm in Information Fusion" mainly solve the problem of high-conflict evidence fusion based on the numerical differences in the allocation probabilities of different evidence sources to each target category, but ignore the correlation and similarity between different target categories themselves. For example, the correlation and similarity between "tanks" and "armored vehicles" are very high, while the similarity between "tanks" and "aircraft" is relatively low. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-sensor decision-level information fusion method based on sentence vector and category similarity, which can efficiently handle the problem of high-conflict evidence fusion and obtain more accurate and reliable multi-sensor decision-level information fusion results.

[0006] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0007] The multi-sensor decision-level information fusion method based on sentence vector and category similarity includes:

[0008] Obtain target recognition results from multiple different sensors;

[0009] Build a sentence corpus and train a sentence vector semantic representation model;

[0010] Select an encoding template, concatenate each category with the encoding template to form a sentence, and use the trained sentence vector semantic representation model to encode the sentence to obtain the encoding vector of the sentence; for different category combinations, use the semantic similarity of the encoding vector as the category similarity to obtain the target category similarity matrix;

[0011] Use piecewise function to modify the category similarity matrix;

[0012] Based on the corrected category similarity matrix, the target recognition results of any two sensors are selected for fusion, and the fusion result is then fused with the target recognition result of another sensor; this method completes the fusion of all target recognition results in sequence, and the target category is determined based on the final fusion result through the non-maximum suppression strategy.

[0013] Furthermore, among the recognition results of the target by the multiple different sensors, each recognition result includes a probability that the target belongs to a different category.

[0014] Furthermore, we construct a sentence corpus and train a sentence vector semantic representation model, including:

[0015] Randomly initialize the parameters of the sentence vector semantic representation model;

[0016] Construct a sentence corpus in the target recognition domain. The sentences in the sentence corpus will be used as original samples.

[0017] For each original sample, construct positive samples and negative samples; the original sample and each positive sample constitute a positive sample pair, and each negative sample constitutes a negative sample pair;

[0018] The original sample, positive sample and negative sample are encoded and represented by the sentence vector semantic representation model to obtain the corresponding encoding vector;

[0019] The sentence vector semantic representation model is trained using positive sample pairs and negative sample pairs; the InfoNCE loss function is calculated during the training process; wherein the cosine similarity between the samples contained in the positive sample pairs and the negative sample pairs is used as the semantic similarity in the InfoNCE loss function;

[0020] The parameters of the sentence vector semantic representation model are updated by the stochastic gradient descent method until the training termination condition is met, and the trained sentence vector semantic representation model is saved.

[0021] Furthermore, the method of constructing positive samples is through synonym replacement, sentence-end punctuation change, and multilingual back-translation; the method of constructing negative samples is to randomly sample other sentences in a number of sentence corpora.

[0022] Furthermore, the category similarity matrix is ​​modified using a piecewise function, specifically:

[0023] The following piecewise function is used to modify the category similarity matrix so that the modified category similarity sim(C i ,C j ) takes values ​​between 0.0 and 1.0:

[0024]

[0025] Among them, sim(C i ,C j ) represents two categories C i and C j The category similarity.

[0026] Furthermore, when fusing the target recognition results of the two sensors, different target categories are first selected, the basic fusion probability and the probability distribution ratio of the category are calculated, and the category distribution fusion probability is determined; all category combinations are traversed in turn, and the total fusion probability is determined based on all the distribution fusion probabilities of each category; finally, the total fusion probability of all categories is normalized to obtain the fusion probability vector, which is used as the information fusion result of the two selected sensors.

[0027] Furthermore, the information fusion results of the two sensors are regarded as the recognition results of the target by a single virtual sensor. The target recognition results of the single virtual sensor are fused with those of another sensor using the method of fusing the target recognition results of the two sensors mentioned above. Similarly, the recognition results of the targets of other sensors are fused using the same method, and the category of the target is determined based on the final fusion result.

[0028] Furthermore, the target category is determined based on the final fusion result, and the non-maximum suppression strategy is used to decide the target category:

[0029]

[0030] Among them C Target is the target category obtained by decision, P Radar_SAR_Infrared_Lidar (C i ) is the final fusion result, C i is the i-th category.

[0031] A terminal device includes a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, the multi-sensor decision-level information fusion method based on sentence vector and category similarity is implemented.

[0032] A computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the multi-sensor decision-level information fusion method based on sentence vectors and category similarity is implemented.

[0033] Compared with the prior art, the present invention has the following technical features:

[0034] Through intelligent empowerment, an artificial intelligence model is used to better characterize the correlation and similarity between different categories. On the basis of DS evidence theory, a new set of fusion inference rules based on the target category similarity matrix is ​​proposed, which can better solve the problem of evidence fusion with high conflict degree and obtain more accurate and reliable multi-sensor decision-level information fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the present invention;

[0036] Figure 2 Flowchart for sentence vector semantic representation model training;

[0037] Figure 3 Flowchart for calculating target category similarity matrix;

[0038] Figure 4 Detailed flow chart for information fusion between any two sensors;

[0039] Figure 5 This is the multi-sensor information fusion flow chart. DETAILED DESCRIPTION

[0040] The present invention provides a multi-sensor decision-level information fusion method based on sentence vectors and category similarity. First, the similarity of the target category is modeled as the semantic similarity of the corresponding sentences, a category similarity matrix is ​​constructed, and then modified by a piecewise function. Then, based on the DS evidence theory, the category similarity matrix is ​​introduced, and a new set of fusion rules is proposed. This method can efficiently handle the fusion problem of high-conflict evidence and obtain more accurate and reliable multi-sensor decision-level information fusion results. The specific implementation steps of the present invention are as follows:

[0041] Step 1: Obtain target recognition results from multiple different sensors; each recognition result includes the probability that the target belongs to a different category.

[0042] In one embodiment of the present invention, the multiple sensors include two-dimensional active phased array radar, synthetic aperture radar, infrared sensor and lidar sensor, etc. The category to which the target belongs is represented as: {C1, C2, C3, ..., C n}, where C n Indicates the nth target category; the recognition results of each sensor for the target are as follows:

[0043] (1) For the range information obtained by the two-dimensional active phased array radar, obtain its one-dimensional range image information, extract the aspect ratio, waveform entropy, first-order moment, second-order moment, third-order moment, first-order center distance, second-order center distance, third-order center moment and other features, and classify the target through the support vector machine model to obtain the target recognition result:

[0044] P Radar =[P Radar (C1),P Radar (C2),P Radar (C3),…,P Radar (C n )]

[0045] Among them, P Radar (C i ) indicates that the target identified by the two-dimensional active phased array radar belongs to the i-th category C i The probability of , which satisfies the normalization condition:

[0046] (2) For radar images acquired by synthetic aperture radar, denoising, deblurring, deocclusion, and image super-resolution preprocessing are performed using a diffusion model. Target detection and recognition are performed using the DINO target detection model based on the Transformer structure to obtain the target recognition results:

[0047] P SAR =[P SAR (C1),P SAR (C2),PSAR (C3),…,P SAR (C n )]

[0048] Among them, P SAR (C i ) indicates that the target identified by the synthetic aperture radar belongs to the i-th category C i The probability of , which satisfies the normalization condition:

[0049] (3) For the infrared image acquired by the infrared sensor, denoising, deblurring, deocclusion and image super-resolution preprocessing are performed through the Diffusion Model diffusion model, and target detection and recognition are performed through the DINO target detection model based on the Transformer structure to obtain the target recognition result:

[0050] P Infrared =[P Infrared (C1),P Infrared (C2),P Infrared (C3),…,P Infrared (C n )]

[0051] Among them, P Infrared (C i ) indicates that the target identified by the infrared sensor belongs to the i-th category C i The probability of , which satisfies the normalization condition:

[0052] (4) For the radar three-dimensional point cloud data obtained by the lidar sensor, outlier filtering algorithms such as ROR, SOR, DROR, and DSOR are used for denoising preprocessing, and then the PV-RCNN three-dimensional point cloud target detection model is used for target detection and recognition to obtain the target recognition result:

[0053] P Lidar =[P Lidar (C1),P Lidar (C2),P Lidar (C3),…,P Lidar (C n )]

[0054] Among them, P Lidar (C i ) indicates that the target identified by the infrared sensor belongs to the i-th category C i The probability of , which satisfies the normalization condition:

[0055] Step 2: Build a sentence corpus and train a sentence vector semantic representation model.

[0056] Step 21, randomly initialize the parameters of the sentence vector semantic representation model; wherein the sentence vector semantic representation model adopts an existing model, for example, it can be a SIMCSE model.

[0057] Step 22: Build a sentence corpus in the target recognition field. The number of sentences in the corpus should be at least one million. The sources of the sentences in the corpus should include official documents, academic monographs, journals and conference papers, newspapers and magazines, the Internet, etc. These sentences will serve as original samples.

[0058] In step 23, for each original sample, data enhancement is performed by replacing synonyms, changing end-of-sentence punctuation, and multilingual back-translation to obtain a positive sample. Each positive sample and the original sample constitute a positive sample pair.

[0059] Step 24: For each original sample, randomly sample several other sentences in the sentence corpus as negative samples; each negative sample and the original sample constitute a negative sample pair.

[0060] Step 25: Encode the original sample, positive sample and negative sample through the sentence vector semantic representation model to obtain the encoding vector s, s + ,s - .

[0061] Step 26: Use the positive sample pairs and negative sample pairs to train the sentence vector semantic representation model; calculate the InfoNCE loss function during the training process:

[0062]

[0063] Among them, cos_sim(s,s + ) represents the cosine similarity between the encoding vectors of the original sample and the positive sample, which is used to characterize the semantic similarity; represents the encoding vector of the jth negative sample, M represents the number of negative samples; e is a natural constant, and τ is a temperature factor;

[0064]

[0065] The purpose of the InfoCSE loss function is to bring the vector representations of positive sample pairs closer and push the vector representations of negative sample pairs further apart, so that the model can better reflect the semantic information of the sentence.

[0066] Step 27, update the parameters of the sentence vector semantic representation model by stochastic gradient descent method until the training termination condition is met, and save the trained sentence vector semantic representation model; for a new sentence, input the sentence into the trained sentence vector semantic representation model to obtain the encoding vector of the sentence.

[0067] Step 3: Select a coding template, concatenate each category with the coding template into a sentence, encode the sentence using the trained sentence vector semantic representation model, and obtain the encoding vector of the sentence; for different category combinations, use the semantic similarity of the encoding vector as the category similarity to obtain the target category similarity matrix.

[0068] Step 31, select a coding template, where the coding template is user-defined; combine each category with the coding template to form a sentence; taking the category "tank" as an example, the sentence formed by combining it with the coding template is "the target may be a tank"; where "the target may be" is the coding template; the coding template is used together with the category to form a sentence for describing different attributes or information of the target.

[0069] Step 32: For each sentence composed of a category and a coding template, the corresponding coding vector is obtained using the trained sentence vector semantic representation model; i The corresponding encoding vector is represented as s i ;

[0070] Step 33, for Category C i and C j , and its corresponding encoding vector is represented as s i and s j The cosine similarity of is used as their category similarity:

[0071] sim(C i ,C j )=cos_sim(s i ,s j )

[0072] Step 34 , calculate the category similarities of all category combinations in sequence to obtain the category similarity matrix of the target.

[0073] Step 4: Use the piecewise function to modify the category similarity matrix.

[0074] The category similarity matrix is ​​modified using a piecewise function; for any two categories C i and C j The category similarity sim(C i ,C j ), that is, the i-th row and j-th column in the category similarity matrix, the category similarity calculated by step 3 should be between -1.0 and 1.0; however, in order to meet the requirements of the decision-level information fusion process, it is necessary to use the following piecewise function to correct it so that the corrected category similarity sim(C i ,C j ) takes values ​​between 0.0 and 1.0:

[0075]

[0076] All category similarities are corrected in turn using the above formula to obtain a corrected category similarity matrix.

[0077] In step 5, based on the corrected category similarity matrix, the target recognition results of any two sensors are selected for fusion, and the fusion result is then fused with the target recognition result of another sensor; this method is used to complete the fusion of all target recognition results in sequence, and the target category is determined based on the final fusion result using the non-maximum suppression strategy.

[0078] In step 51, when fusing the target recognition results of the two sensors, first select different target categories, calculate the basic fusion probability and the probability distribution ratio of the category, and determine the category distribution fusion probability; then traverse all category combinations in turn, and determine the total fusion probability based on all the distribution fusion probabilities of each category; finally, normalize the total fusion probability of all categories to obtain a fusion probability vector as the information fusion result of the two selected sensors.

[0079] (1) Fusion of the target recognition results of the two-dimensional active phased array radar and the synthetic aperture radar; if the two-dimensional active phased array radar identifies the target as category C i The probability is P Radar (C i ), the synthetic aperture radar identifies the target as category C j The probability is P SAR (C j ), calculate the basic fusion probability as:

[0080] P Fusion (C i ,C j )=(P Radar (C i )+λsim(C i ,C j )P SAR (C j ))(P SAR (C j )+λsim(C i ,C j )P Radar (C i ))

[0081] Where λ is a pre-set parameter factor, representing the importance of category similarity information.

[0082] (2) If the 2D active phased array radar identifies the target as category C j The probability is P Radar (C j), the synthetic aperture radar identifies the target as category C i The probability is P SAR (C i ), calculate the basic fusion probability as:

[0083] P′ Fusion (C j ,C i )=(P Radar (C j )+λsim(C i ,C j )P SAR (C i ))(P SAR (C i )+λsim(C i ,C j )P Radar (C j ))

[0084] (3) Calculate the category C in step (1) and step (2) respectively i The probability distribution ratio of:

[0085]

[0086] (4) Calculate the category C in the case of step (1) and step (2) respectively i The distribution fusion probability of :

[0087] P Fusion1 (C i |C i ,C j )=α(C i )P Fusion (C i ,C j )

[0088] P Fusion2 (C i |C j ,C i )=β(C i )P′ Fusion (C j ,C i )

[0089] (5) Take all category combinations in turn and calculate the corresponding allocation fusion probability;

[0090] (6) Calculate the total fusion probability, for category C i , the total fusion probability is:

[0091]

[0092] (7) The total fusion probability of all categories is normalized to obtain the fusion probability vector, which is used as the information fusion result of the two-dimensional active phased array radar and synthetic aperture radar:

[0093]

[0094] In step 52, the information fusion results of the two sensors are regarded as the target recognition results of a single virtual sensor. The target recognition results of the single virtual sensor are fused with the target recognition results of another sensor using the method for fusing the target recognition results of the two sensors mentioned above. Similarly, the target recognition results of other sensors are fused using the same method, and the target category is determined based on the final fusion result.

[0095] The fusion result P Radar_SAR (C i ) is regarded as the recognition result of the target by a single virtual sensor, and the recognition result of the target by the single virtual sensor is fused with the recognition result of the target by the infrared sensor using the same method as step 51 to obtain a new fusion result P Radar_SAR_Infrared (C i ).

[0096] The fusion result P Radar_SAR_Infrared (C i ) is regarded as the recognition result of the target by a single virtual sensor, and the recognition result of the target by the single virtual sensor is fused with the recognition result of the target by the laser radar sensor using the same method as step 51 to obtain a new fusion result P Radar_SAR_Infrared_Lidar (C i ).

[0097] Finally, for the final fusion result P Radar_SAR_Infrared_Lidar (C i ), using non-maximum suppression strategy to decide the target category:

[0098]

[0099] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multi-sensor decision-level information fusion method based on sentence vector and category similarity, characterized by: include: Obtain target recognition results from multiple different sensors; Build a sentence corpus and train a sentence vector semantic representation model; Select an encoding template, concatenate each category with the encoding template to form a sentence, and use the trained sentence vector semantic representation model to encode the sentence to obtain the encoding vector of the sentence; for different category combinations, use the semantic similarity of the encoding vector as the category similarity to obtain the target category similarity matrix; Use piecewise function to modify the category similarity matrix; Based on the corrected category similarity matrix, the target recognition results of any two sensors are selected for fusion, and the fusion result is then fused with the target recognition result of another sensor; this method completes the fusion of all target recognition results in sequence, and the target category is determined based on the final fusion result through the non-maximum suppression strategy.

2. The multi-sensor decision-level information fusion method based on sentence vector and category similarity according to claim 1 is characterized in that Among the recognition results of the target by the multiple different sensors, each recognition result includes a probability that the target belongs to a different category.

3. The multi-sensor decision-level information fusion method based on sentence vector and category similarity according to claim 1 is characterized in that: Build a sentence corpus and train a sentence vector semantic representation model, including: Randomly initialize the parameters of the sentence vector semantic representation model; Construct a sentence corpus in the target recognition domain. The sentences in the sentence corpus will be used as original samples. For each original sample, construct positive samples and negative samples; the original sample and each positive sample constitute a positive sample pair, and each negative sample constitutes a negative sample pair; The original sample, positive sample and negative sample are encoded and represented by the sentence vector semantic representation model to obtain the corresponding encoding vector; The sentence vector semantic representation model is trained using positive sample pairs and negative sample pairs; the InfoNCE loss function is calculated during the training process; wherein the cosine similarity between the samples contained in the positive sample pairs and the negative sample pairs is used as the semantic similarity in the InfoNCE loss function; The parameters of the sentence vector semantic representation model are updated by the stochastic gradient descent method until the training termination condition is met, and the trained sentence vector semantic representation model is saved.

4. The multi-sensor decision-level information fusion method based on sentence vector and category similarity according to claim 3 is characterized in that The method of constructing positive samples is through synonym replacement, sentence-end punctuation change, and multilingual back-translation; the method of constructing negative samples is to randomly sample other sentences in a number of sentence corpora.

5. The multi-sensor decision-level information fusion method based on sentence vector and category similarity according to claim 1 is characterized in that The category similarity matrix is ​​modified using a piecewise function, specifically: The following piecewise function is used to modify the category similarity matrix so that the modified category similarity sim(C i ,C j ) takes values ​​between 0.0 and 1.0: Among them, sim(C i ,C j ) represents two categories C i and C j The category similarity.

6. The multi-sensor decision-level information fusion method based on sentence vector and category similarity according to claim 1 is characterized in that When fusing the target recognition results of two sensors, first select different target categories, calculate the basic fusion probability and the probability distribution ratio of the category, and determine the distribution fusion probability of the category; All category combinations are traversed in turn, and the total fusion probability is determined based on all the assigned fusion probabilities of each category; finally, the total fusion probability of all categories is normalized to obtain the fusion probability vector as the information fusion result of the two selected sensors.

7. The multi-sensor decision-level information fusion method based on sentence vector and category similarity according to claim 6 is characterized in that: The information fusion results of the two sensors are regarded as the recognition results of the target by a single virtual sensor. The target recognition results of the single virtual sensor are fused with the target recognition results of another sensor using the method of fusing the target recognition results of the two sensors mentioned above. Similarly, the recognition results of the targets of other sensors are fused using the same method, and the target category is determined based on the final fusion result.

8. The multi-sensor decision-level information fusion method based on sentence vector and category similarity according to claim 1 is characterized in that: The target category is determined based on the final fusion result, and the non-maximum suppression strategy is used to determine the target category: Among them C Target is the target category obtained by decision, P Radar_SAR_Infrared_Lidar (C i ) is the final fusion result, C i is the i-th category.

9. A terminal device comprising a processor, a memory, and a computer program stored in the memory; characterized in that: When the processor executes the computer program, it implements the multi-sensor decision-level information fusion method based on sentence vector and category similarity according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program; wherein: When the computer program is executed by a processor, the multi-sensor decision-level information fusion method based on sentence vector and category similarity according to any one of claims 1 to 8 is implemented.

Images