Target classification and uncertainty evaluation method based on semantic association evidence fusion

Abstract

The invention particularly relates to a target classification and uncertainty evaluation method based on semantic association evidence fusion, and the method comprises the following steps: 1, constructing and training a target fusion classification neural network, and calculating a Dirichlet distribution concentration parameter of each modal input data representing an evidence quantity; step 2, associating the generated Dirichlet concentration parameter with the evidence quantity, and calculating single-mode uncertainty; step 3, constructing a semantic association matrix and performing discount correction, exploring potential association and confusion relationships among different categories, and completing multi-source evidence fusion through a Dempster combination rule; step 4, integrating the global uncertainty of the fused evidence and the local uncertainty of the single mode to obtain final uncertainty evaluation; according to the method, the modal information can be effectively fused to obtain high-precision fusion classification, the uncertainty of fusion classification can be quantitatively evaluated, a basis is provided for improving the safety and interpretability of intelligent classification decision, and the method is suitable for the multi-source sensor decision fusion field of automatic driving, medical diagnosis and the like.

Description

Technical Field

[0001] This invention relates to the field of target fusion classification, and more specifically to a target classification and uncertainty assessment method based on semantic association evidence fusion. Background Technology

[0002] In critical fields such as autonomous driving and medical diagnosis, where careful judgment of decision-making results is essential, multimodal fusion has become a core technology for improving the robustness of system perception. Taking autonomous driving as an example, vehicles need to make accurate decisions in complex and dynamic environments. Single-modal perception often has limitations: while visual cameras can provide rich color and texture information, their performance degrades sharply in darkness, strong light, or inclement weather; LiDAR can accurately perceive three-dimensional spatial structures, but it is susceptible to rain and snow and cannot identify color information; millimeter-wave radar has the advantage of all-weather operation, but it lacks in resolution and target classification. Therefore, effectively fusing data from these heterogeneous sensors to achieve complementary advantages is crucial for ensuring driving safety. However, due to the inherent differences in physical principles, data structures, and spatiotemporal resolution among various modal data, direct low-level fusion is extremely challenging. The industry often adopts a decision-level fusion strategy, which involves processing each modal data independently first and then combining their outputs. However, in such high-risk scenarios, simple result synthesis is far from sufficient to meet safety requirements; the final perception result must have extremely high confidence, which places higher demands on the judgment before fusion, thus requiring the introduction of uncertainty assessment.

[0003] To address the aforementioned issues, researchers both domestically and internationally have proposed various uncertainty assessment methods. Early research focused on classic Bayesian neural networks (BNNs), estimating uncertainty by treating network weights as probability distributions. However, their enormous computational overhead and complex variational inference process limited their practical applications. Subsequently, Monte Carlo Dropout, as a more efficient approximate Bayesian method, was proposed. It repeatedly uses Dropout during the inference phase to simulate model uncertainty, greatly promoting the widespread adoption of uncertainty assessment. In recent years, deep ensemble learning has become mainstream. It quantifies uncertainty by training multiple independent models with identical structures but different initializations and statistically analyzing the differences in their predictions. This method typically provides high-quality assessment results. However, these methods generally suffer from a significant drawback: enormous computational and memory resource overhead. For example, deep ensembles require several times the model training and storage costs, posing a severe deployment challenge for applications such as autonomous driving that require real-time decision-making on edge devices or time-sensitive platforms.

[0004] Therefore, there is an urgent need to design a decision fusion method that can assess uncertainty while consuming only a small amount of computational resources. This invention proposes a target classification and uncertainty assessment method based on semantic association evidence fusion. An uncertainty assessment network is constructed for each modality, enabling it to output a set of parameters representing the amount of evidence for each category for each sample to be classified. A single-modality uncertainty is calculated for this modality sample. Then, these evidence representations from various modalities enter the fusion process, where each piece of evidence is corrected through association discounting, and fused using Dempster's combination rule to generate a final fused evidence distribution that aggregates all modalities. Finally, a mixed uncertainty assessment is performed. This invention not only effectively fuses information from various modalities to obtain high-precision fused classification, but also quantitatively assesses the uncertainty of the fused classification, providing a basis for improving the security and interpretability of intelligent classification decisions. Summary of the Invention

[0005] This invention provides a target classification and uncertainty assessment method based on semantic association evidence fusion, which is used to solve the problem of quantitative assessment of uncertainty in multi-source data target fusion identification. It can not only effectively fuse information from various modalities to achieve high-precision classification, but also quantitatively assess the uncertainty of fusion classification.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a target classification and uncertainty assessment method based on semantic association evidence fusion, characterized by including the following steps:

[0007] Step 1: Construct and train the target fusion classification neural network, and calculate the Dirichlet distribution concentration parameter representing the amount of evidence for each modality input data;

[0008] Step 101: Construct a deep neural network model for each single-modal classification to be fused, modify the final output layer of each model, and replace the Softmax function of the last layer with the ReLU non-negative activation function so that the network can directly output the Dirichlet distribution concentration parameter that can represent the amount of evidence.

[0009] Step 102: Train the single-modal network using a variational inference strategy and construct the composite loss function l(x) m ,y), so that the model takes the input data x from the m-th modality. m The algorithm learns and extracts the amount of evidence pointing to the correct category y, and generates a K-dimensional concentration parameter vector for each sample. K represents the total number of categories, and the composite loss function is as follows:

[0010]

[0011] The first term is the expected log-likelihood, y k It is the real label vector, Sm Indicates Dirichlet intensity (i.e., all) The sum of ), ψ(·) represents the digamma function; the second term is the KL divergence regularization, where μ=[μ1,…,μ K Let ] represent a K-dimensional probability vector, where k represents the class index, satisfying And μ k >0, Dir(μ) m |[1,…,1]) denotes the prior Dirichlet distribution, α represents evading punishment based on strong evidence. m Correction, λ t A balancing factor that weighs the accuracy of classification against the reasonableness of distribution;

[0012] Step 2: Correlate the generated Dirichlet concentration parameters with the amount of evidence, and calculate the single-mode uncertainty;

[0013] Step 201: The Dirichlet concentration parameter obtained in step 102 With the amount of evidence Perform association, the mapping relationship is as follows

[0014] Step 202: Iterate through all samples of each modality sequentially, using the formula... Calculate the total amount of evidence, i.e., the Dirichlet strength, using the formula... Calculate the single-mode uncertainty u for each mode in relation to the current sample. m and all single-mode uncertainties {u m Store it;

[0015] Step 3: Construct a semantic association matrix and perform discount correction to explore potential associations and confusion relationships between different categories, and complete the fusion of multi-source evidence through Dempster's combination rule;

[0016] Step 301: Redefine the axiomatic conditions for semantic association: abandon the symmetry requirement and allow associations to be asymmetric;

[0017] Step 302: Design two operators to reduce the computational complexity of semantic association generation. Semantically, the degree to which proposition A is related to B is represented by u. A (B) Then the "Max" operator u A (B) = max a∈A,b∈B {u a (b)}, determined by the maximum correlation between the individual elements contained therein; the “Min_Sum” operator u A (B)=min{∑ b∈B ∑ a∈A u a(b),1}, is determined by the sum of the correlation between the individual elements contained in each element, with an upper limit of 1;

[0018] Step 303: Based on the empirical distribution of each modality, i.e., the confusion matrix, construct a semantic association matrix U describing the asymmetric relationship. Simultaneously, normalize the evidence vector obtained in step 102 to fit the input requirements of the Basic Probability Assignment (BPA) distribution, and use it as initial evidence for subsequent processing, denoted as M. m ;

[0019] Step 304: A new evidence representation model is proposed, combining the initial BPA from Step 303 with the constructed semantic association matrix U, performing association discount correction, calibrating systematic biases between modalities, and obtaining a set of corrected evidence (M). m )′;

[0020] Step 305: According to Dempster's combination rule, analyze all modal-corrected evidence (M... m The fusion is performed using the following formula: Where T represents the conflict coefficient, M final This indicates that the final BPA, which incorporates all modal information, is generated.

[0021] Step 4: Combine the global uncertainty of the integrated evidence with the local uncertainty of the single modality to obtain the final uncertainty assessment;

[0022] Step 401: Retrieve all single-mode uncertainties {u} stored in step 202 m}, treating the reliability of each modality as 1-u m The overall reliability after multimodal fusion should not be lower than the maximum value among all single-modal reliability values. This is equivalent to the overall uncertainty of the fused system not being higher than the minimum value among all single-modal uncertainties. Therefore, the minimum value in this set is taken to determine the single-modal cognitive uncertainty component U. optimistic =min(u 1 ,u 2 ,...,u m );

[0023] Step 402: For the final fused BPA obtained in step 303, calculate its decision fuzziness using Deng entropy as a metric, and normalize it using the theoretical maximum value. fused =E_Deng(bpa_comb) / log_2(N), where N is the number of categories, to obtain the uncertainty component U of the fused data. fused ;

[0024] Step 403: Average and weight the two uncertainty components obtained in steps 401 and 402 to obtain the final mixed uncertainty measure U. final ;

[0025] Step 404: For the final fused BPA, use a likelihood-based distance comparison method to calculate its distance to the definitive evidence representing each category, and select the one that matches M. final The category with the closest distance is used as the final classification result.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] This invention has significant advantages over existing technologies. The proposed method introduces a Dirichlet distribution to deeply quantify the evidence information of each modality and constructs a semantic association matrix to correct the evidence to be processed. Finally, by combining global and local hybrid uncertainty assessment, it can not only effectively fuse information from various modalities to obtain high-precision fusion classification, but also quantitatively assess the uncertainty of the fusion classification. This provides a basis for improving the security and interpretability of intelligent classification decisions and has good versatility and promotion value.

[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0029] Figure 1 Flowchart of a target classification and uncertainty assessment method based on semantic association evidence fusion; Detailed Implementation

[0030] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0035] refer to Figure 1 As shown, this example implementation provides a target classification and uncertainty assessment method based on semantic association evidence fusion, which may specifically include the following steps:

[0036] Step 1: Construct and train the target fusion classification neural network, and calculate the Dirichlet distribution concentration parameter representing the amount of evidence for each modality input data;

[0037] Step 101: Construct a deep neural network model for each single-modal classification to be fused, modify the final output layer of each model, and replace the Softmax function of the last layer with the ReLU non-negative activation function so that the network can directly output the Dirichlet distribution concentration parameter that can represent the amount of evidence.

[0038] Step 102: Train the single-modal network using a variational inference strategy and construct the composite loss function l(x) m ,y), so that the model takes the input data x from the m-th modality. mThe algorithm learns and extracts the amount of evidence pointing to the correct category y, and generates a K-dimensional concentration parameter vector for each sample. K represents the total number of categories, and the composite loss function is as follows:

[0039]

[0040] The first term is the expected log-likelihood, y k It is the real label vector, S m Indicates Dirichlet intensity (i.e., all) The sum of ), ψ(·) represents the digamma function; the second term is the KL divergence regularization, where μ=[μ1,…,μ K Let ] represent a K-dimensional probability vector, where k represents the class index, satisfying And μ k >0, Dir(μ) m |[1,…,1]) denotes the prior Dirichlet distribution, α represents evading punishment based on strong evidence. m Correction, λ t A balancing factor that weighs the accuracy of classification against the reasonableness of distribution;

[0041] Step 2: Correlate the generated Dirichlet concentration parameters with the amount of evidence, and calculate the single-mode uncertainty;

[0042] Step 201: The Dirichlet concentration parameter obtained in step 102 With the amount of evidence Perform association, the mapping relationship is as follows

[0043] Step 202: Iterate through all samples of each modality sequentially, using the formula... Calculate the total amount of evidence, i.e., the Dirichlet strength, using the formula... Calculate the single-mode uncertainty u for each mode in relation to the current sample. m and all single-mode uncertainties {u m Store it;

[0044] Step 3: Construct a semantic association matrix and perform discount correction to explore potential associations and confusion relationships between different categories, and complete the fusion of multi-source evidence through Dempster's combination rule;

[0045] Step 301: Redefine the axiomatic conditions for semantic association: abandon the symmetry requirement and allow associations to be asymmetric;

[0046] Step 302: Design two operators to reduce the computational complexity of semantic association generation. Semantically, the degree to which proposition A is related to B is represented by u. A(B) Then the "Max" operator u A (B) = max a∈A,b∈B {u a (b)}, determined by the maximum correlation between the individual elements contained therein; the “Min_Sum” operator u A (B)=min{∑ b∈B ∑ a∈A u a (b),1}, is determined by the sum of the correlation between the individual elements contained in each element, with an upper limit of 1;

[0047] Step 303: Based on the empirical distribution of each modality, i.e., the confusion matrix, construct a semantic association matrix U describing the asymmetric relationship. Simultaneously, normalize the evidence vector obtained in step 102 to fit the input requirements of the Basic Probability Assignment (BPA) distribution, and use it as initial evidence for subsequent processing, denoted as M. m ;

[0048] Step 304: A new evidence representation model is proposed, combining the initial BPA from Step 303 with the constructed semantic association matrix U, performing association discount correction, calibrating systematic biases between modalities, and obtaining a set of corrected evidence (M). m )′;

[0049] Step 305: According to Dempster's combination rule, analyze all modal-corrected evidence (M... m The fusion is performed using the following formula: Where T represents the conflict coefficient, M final This indicates that the final BPA, which incorporates all modal information, is generated.

[0050] Step 4: Combine the global uncertainty of the integrated evidence with the local uncertainty of the single modality to obtain the final uncertainty assessment;

[0051] Step 401: Retrieve all single-mode uncertainties {u} stored in step 202 m}, treating the reliability of each modality as 1-u m The overall reliability after multimodal fusion should not be lower than the maximum value among all single-modal reliability values. This is equivalent to the overall uncertainty of the fused system not being higher than the minimum value among all single-modal uncertainties. Therefore, the minimum value in this set is taken to determine the single-modal cognitive uncertainty component U. optimistic =min(u 1 ,u 2 ,...,u m );

[0052] Step 402: For the final fused BPA obtained in step 303, calculate its decision fuzziness using Deng entropy as a metric, and normalize it using the theoretical maximum value. fused =E_Deng(bpa_comb) / log_2(N), where N is the number of categories, to obtain the uncertainty component U of the fused data. fused ;

[0053] Step 403: Average and weight the two uncertainty components obtained in steps 401 and 402 to obtain the final mixed uncertainty measure U. final ;

[0054] Step 404: For the final fused BPA, use a likelihood-based distance comparison method to calculate its distance to the definitive evidence representing each category, and select the one that matches M. final The category with the closest distance is used as the final classification result.

[0055] The above description is merely an embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A target classification and uncertainty assessment method based on semantic association evidence fusion, characterized in that, The method includes the following steps: Step 1: Construct and train the target fusion classification neural network, and calculate the Dirichlet distribution concentration parameter representing the amount of evidence for each modality input data; Step 101: Construct a deep neural network model for each single-modal classification to be fused, modify the final output layer of each model, and replace the Softmax function of the last layer with the ReLU non-negative activation function so that the network can directly output the Dirichlet distribution concentration parameter that can represent the amount of evidence. Step 102: Train the single-modal network using a variational inference strategy and construct the composite loss function l(x) m ,y), so that the model takes the input data x from the m-th modality. m The algorithm learns and extracts the amount of evidence pointing to the correct category y, and generates a K-dimensional concentration parameter vector for each sample. K represents the total number of categories, and the composite loss function is as follows: The first term is the expected log-likelihood, y k It is the real label vector, S m Indicates Dirichlet intensity (i.e., all) The sum of ), ψ(·) represents the digamma function; the second term is the KL divergence regularization, where μ=[μ1,…,μ K Let ] represent a K-dimensional probability vector, where k represents the class index, satisfying And μ k >0, Dir(μ) m |[1,…,1]) denotes the prior Dirichlet distribution, α represents evading punishment based on strong evidence. m Correction, λ t A balancing factor that weighs the accuracy of classification against the reasonableness of distribution; Step 2: Correlate the generated Dirichlet concentration parameters with the amount of evidence, and calculate the single-mode uncertainty; Step 201: The Dirichlet concentration parameter obtained in step 102 With the amount of evidence To establish an association, the mapping relationship is as follows: Step 202: Iterate through all samples of each modality sequentially, using the formula... Calculate the total amount of evidence, i.e., the Dirichlet strength, using the formula... Calculate the single-mode uncertainty u for each mode in relation to the current sample. m and all single-mode uncertainties {u m Store it; Step 3: Construct a semantic association matrix and perform discount correction to explore potential associations and confusion relationships between different categories, and complete the fusion of multi-source evidence through Dempster's combination rule; Step 301: Redefine the axiomatic conditions for semantic association: abandon the symmetry requirement and allow associations to be asymmetric; Step 302: Design two operators to reduce the computational complexity of semantic association generation. Semantically, the degree to which proposition A is related to B is represented by u. A (B) Then the "Max" operator u A (B) = max a∈A,b∈B {u a (b)}, determined by the maximum correlation between the individual elements contained in each; the "Min_Sum" operator u A (B)=min{∑ b∈B ∑ a∈A u a (b),1}, is determined by the sum of the correlation between the individual elements contained in each element, with an upper limit of 1; Step 303: Based on the empirical distribution of each modality, i.e., the confusion matrix, construct a semantic association matrix U describing the asymmetric relationship. Simultaneously, normalize the evidence vector obtained in step 102 to fit the input requirements of the Basic Probability Assignment (BPA) distribution, and use it as initial evidence for subsequent processing, denoted as M. m ; Step 304: A new evidence representation model is proposed, combining the initial BPA from Step 303 with the constructed semantic association matrix U, performing association discount correction, calibrating systematic biases between modalities, and obtaining a set of corrected evidence (M). m )′; Step 305: According to Dempster's combination rule, analyze all modal-corrected evidence (M... m The fusion is performed using the following formula: Where T represents the conflict coefficient, M final This indicates that the final BPA, which incorporates all modal information, is generated. Step 4: Combine the global uncertainty of the integrated evidence with the local uncertainty of the single modality to obtain the final uncertainty assessment; Step 401: Retrieve all single-mode uncertainties {u} stored in step 202 m }, treating the reliability of each modality as 1-u m The overall reliability after multimodal fusion should not be lower than the maximum value among all single-modal reliability values. This is equivalent to the overall uncertainty of the fused system not being higher than the minimum value among all single-modal uncertainties. Therefore, the minimum value in this set is taken to determine the single-modal cognitive uncertainty component U. optimistic =min(u 1 ,u 2 ,...,u m ); Step 402: For the final fused BPA obtained in step 303, calculate its decision fuzziness using Deng entropy as a metric, and normalize it using the theoretical maximum value. fused =E_Deng(bpa_comb) / log_2(N), where N is the number of categories, to obtain the uncertainty component U of the fused data. fused ; Step 403: Average and weight the two uncertainty components obtained in steps 401 and 402 to obtain the final mixed uncertainty measure U. final ; Step 404: For the final fused BPA, use a likelihood-based distance comparison method to calculate its distance to the definitive evidence representing each category, and select the one that matches M. final The category with the closest distance is used as the final classification result.

Images