Double-layer evidence fusion learning method, classification evaluation method and device

Abstract

The present disclosure relates to the technical field of machine learning, and specifically to a double-layer evidence fusion learning method and device, and a classification evaluation method and device of an evidence deep learning model. The method includes: obtaining a multimodal data set including training data of multiple sample objects, each sample object corresponding to multiple training data of multiple modalities; using an evidence deep learning model for the multimodal data set, adopting a double-layer evidence fusion strategy for each sample object, and obtaining the classification reliability result and classification uncertainty result of the sample object; constructing a loss function under the Dirichlet distribution according to the classification reliability result and the true category label of each sample object in the multiple sample objects, adjusting the model parameters of the evidence deep learning model, and obtaining an optimized double-layer evidence fusion evidence deep learning model. The present disclosure comprehensively considers the generalization of the model and the ability to estimate uncertainty, and improves the accuracy and confidence of data classification.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and specifically to a double-layer evidence fusion learning method and device for an evidence deep learning model, a classification evaluation method and device for an evidence deep learning model based on double-layer evidence fusion, an electronic device, and a computer-readable medium. Background Art

[0002] In real-world application scenarios, information often exists in multiple modalities to describe the same thing from different perspectives. In recent years, multimodal learning methods have been successful in various applications. These methods usually combine information from different modalities with deep neural networks (DNNs). However, when fusing multimodal information, traditional methods usually assume that the modal information of different samples has the same importance, that is, a fixed weight factor is set for each modality. However, multimodal deep neural network (DNN) learning is greedy, and the model's unbalanced use of each modality will prevent multimodal deep neural network (DNN) learning from using the available modal information of all samples, resulting in poor generalization. In addition, existing multimodal learning methods often ignore the uncertainty of the model, resulting in unreliable prediction results, especially when the information of a certain modality is missing or has large noise (such as sensor anomalies). In critical applications involving human safety (such as medical image classification and autonomous driving), low-reliability predictions will have a significant impact on model deployment.

[0003] Existing uncertainty estimation algorithms usually focus on single-modal data, and Evidential Deep Learning (EDL) has become an important branch of uncertainty estimation. In recent years, some studies have extended this method to multimodal situations, but still do not consider the situation where the information quality of different samples and different modalities is different. The use of a single credibility allocation method leads to limitations in the generalization performance of the model and the ability to estimate uncertainty. New methods need to be explored. Summary of the invention

[0004] In order to solve the problems in the related art, the embodiments of the present disclosure provide a two-layer evidence fusion learning method and device for an evidence deep learning model that comprehensively considers the model generalization and uncertainty estimation ability, a classification evaluation method and device based on the evidence deep learning model based on two-layer evidence fusion, an electronic device and a computer-readable medium.

[0005] In a first aspect, the present disclosure provides a two-layer evidence fusion learning method for an evidence deep learning model, including:

[0006] Acquire a multimodal dataset, the multimodal dataset comprising training data of a plurality of sample objects, wherein each sample object corresponds to a plurality of training data of a plurality of modalities, and the multimodal dataset is a multimodal image set;

[0007] Using an evidence deep learning model for the multimodal data set, obtaining classification confidence results and classification uncertainty results for the multiple sample objects based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification confidence results and classification uncertainty results for the sample object;

[0008] The two-layer evidence fusion strategy includes: for each modality of the sample object, inputting the training data of the modality into the corresponding deep neural network DNN in the evidence deep learning model for learning, setting the output layer activation function of the deep neural network DNN to a linear activation function, and obtaining a corresponding initial deep learning evidence vector; using different credibility distribution methods to redistribute the initial deep learning evidence vector respectively, and obtaining a corresponding plurality of redistributed deep learning evidence vectors; mapping the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters, and obtaining credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and using the Dempster-Shafer evidence fusion rule to fusion the plurality of subjective logic S The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions;

[0009] According to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, a loss function under the Dirichlet distribution is constructed, and the model parameters of the evidence deep learning model are adjusted to obtain an optimized double-layer evidence fusion evidence deep learning model.

[0010] In a second aspect, the present disclosure provides a two-layer evidence fusion learning method for an evidence deep learning model, including:

[0011] Acquire a multimodal data set, the multimodal data set comprising training data of a plurality of sample objects, wherein each sample object corresponds to a plurality of training data of a plurality of modalities, the multimodal data set being a multimodal medical feature data set, the multimodal medical feature data set comprising: a feature tensor obtained by constructing a brain functional connection network by calculating the temporal correlation of each brain region based on a blood oxygen level signal of the brain in a resting state, and a feature tensor obtained by constructing a fiber bundle connection network between brain regions by performing fiber bundle tracing based on the degree of water molecule diffusion;

[0012] Using an evidence deep learning model for the multimodal data set, obtaining classification confidence results and classification uncertainty results for the multiple sample objects based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification confidence results and classification uncertainty results for the sample object;

[0013] The two-layer evidence fusion strategy includes: for each modality of the sample object, inputting the training data of the modality into the corresponding deep neural network DNN in the evidence deep learning model for learning, setting the output layer activation function of the deep neural network DNN to a linear activation function, and obtaining a corresponding initial deep learning evidence vector; using different credibility distribution methods to redistribute the initial deep learning evidence vector respectively, and obtaining a corresponding plurality of redistributed deep learning evidence vectors; mapping the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters, and obtaining credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and using the Dempster-Shafer evidence fusion rule to fusion the plurality of subjective logic S The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions;

[0014] According to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, a loss function under the Dirichlet distribution is constructed, and the model parameters of the evidence deep learning model are adjusted to obtain an optimized double-layer evidence fusion evidence deep learning model.

[0015] In a third aspect, the present disclosure provides a classification evaluation method of an evidence deep learning model based on double-layer evidence fusion, including:

[0016] Acquire multimodal data of an object to be classified, wherein the object to be classified is an image to be identified;

[0017] For each modality in the multimodal data, the data of the modality is input into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors, and the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors are fused using the Dempster-Shafer evidence fusion rule to obtain the credibility results and uncertainty results of the first-layer classification evidence vector under the modality;

[0018] The Dempster-Shafer evidence fusion rule is used to fuse the reliability and uncertainty results of the first-level classification evidence vectors of multiple modalities to obtain the reliability and uncertainty results of the second-level classification evidence vectors.

[0019] A second layer of classification evidence vector is obtained according to the reliability result and uncertainty result of the second layer of classification evidence vector, and the second layer of classification evidence vector is mapped to Dirichlet distribution parameters, and the classification reliability result and classification uncertainty result of the object to be classified are output; wherein, the evidence deep learning model of the double-layer evidence fusion is trained using the method described in the first aspect.

[0020] In a fourth aspect, the present disclosure provides a classification evaluation method of an evidence deep learning model based on double-layer evidence fusion, including:

[0021] Acquire multimodal data of an object to be classified, wherein the object to be classified is medical feature data to be identified, and the medical feature data to be identified includes: a feature tensor of mode 1 obtained by constructing a brain functional connection network based on a time correlation of each brain region calculated based on a blood oxygen level signal of the brain in a resting state, and a feature tensor of mode 2 obtained by constructing a fiber bundle connection network between brain regions by fiber bundle tracing based on the degree of water molecule diffusion;

[0022] For each modality in the multimodal data, the data of the modality is input into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors, and the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors are fused using the Dempster-Shafer evidence fusion rule to obtain the credibility results and uncertainty results of the first-layer classification evidence vector under the modality;

[0023] The Dempster-Shafer evidence fusion rule is used to fuse the reliability and uncertainty results of the first-level classification evidence vectors of multiple modalities to obtain the reliability and uncertainty results of the second-level classification evidence vectors.

[0024] A second layer of classification evidence vector is obtained according to the reliability result and uncertainty result of the second layer of classification evidence vector, and the second layer of classification evidence vector is mapped to Dirichlet distribution parameters, and the classification reliability result and classification uncertainty result of the object to be classified are output; wherein, the evidence deep learning model of the double-layer evidence fusion is trained using the method described in the first aspect.

[0025] In a fifth aspect, the present disclosure provides a dual-layer evidence fusion learning device for an evidence deep learning model, including:

[0026] A first acquisition module acquires a multimodal data set, wherein the multimodal data set includes training data of multiple sample objects, wherein each sample object corresponds to multiple training data of multiple modalities, and the multimodal data set is a multimodal image set;

[0027] A first double-layer evidence fusion module is configured to use an evidence deep learning model for the multimodal data set to obtain classification confidence results and classification uncertainty results of the multiple sample objects based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification confidence result and classification uncertainty result of the sample object;

[0028] The double-layer evidence fusion strategy is specifically as follows: for each modality of the sample object, the training data of the modality is input into the corresponding deep neural network DNN in the evidence deep learning model for learning, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and the plurality of subjective logic S is fused using the Dempster-Shafer evidence fusion rule. The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions;

[0029] The first training module is configured to construct a loss function under the Dirichlet distribution according to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, adjust the model parameters of the evidence deep learning model, and obtain an optimized double-layer evidence fusion evidence deep learning model.

[0030] In a sixth aspect, the present disclosure provides a dual-layer evidence fusion learning device for an evidence deep learning model, including:

[0031] A third acquisition module acquires a multimodal data set, wherein the multimodal data set includes training data of multiple sample objects, wherein each sample object corresponds to multiple training data of multiple modalities, and the multimodal data set is a multimodal medical feature data set, and the multimodal medical feature data set includes: a feature tensor obtained by constructing a brain functional connection network based on a blood oxygen level signal of the brain in a resting state to calculate the time correlation of each brain region, and a feature tensor obtained by constructing a fiber bundle connection network between brain regions by performing fiber bundle tracking based on the degree of water molecule diffusion;

[0032] A second double-layer evidence fusion module is configured to use the evidence deep learning model for the multimodal data set to obtain classification confidence results and classification uncertainty results of the multiple sample objects based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification confidence result and classification uncertainty result of the sample object;

[0033] The double-layer evidence fusion strategy is specifically as follows: for each modality of the sample object, the training data of the modality is input into the corresponding deep neural network DNN in the evidence deep learning model for learning, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and the plurality of subjective logic S is fused using the Dempster-Shafer evidence fusion rule. The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions;

[0034] The second training module is configured to construct a loss function under the Dirichlet distribution according to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, adjust the model parameters of the evidence deep learning model, and obtain an optimized evidence deep learning model for double-layer evidence fusion

[0035] In a seventh aspect, the present disclosure provides a classification and evaluation device for an evidence deep learning model based on double-layer evidence fusion, comprising:

[0036] A second acquisition module is configured to acquire multimodal data of an object to be classified, wherein the object to be classified is an image to be identified;

[0037] The first first-layer fusion module is configured to, for each modality in the multimodal data, input the data of the modality into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, set the output layer activation function of the deep neural network DNN to a linear activation function, and obtain a corresponding initial deep learning evidence vector; use different credibility distribution methods to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; map the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors, and use the Dempster-Shafer evidence fusion rule to fuse the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors to obtain credibility results and uncertainty results of the first-layer classification evidence vector under the modality;

[0038] The second layer fusion module uses the Dempster-Shafer evidence fusion rule to fuse the reliability results and uncertainty results of the first layer classification evidence vectors of the multiple modalities to obtain the reliability results and uncertainty results of the second layer classification evidence vectors;

[0039] The first classification evaluation module is configured to obtain a second-layer classification evidence vector based on the reliability results and uncertainty results of the second-layer classification evidence vector, and map the second-layer classification evidence vector to Dirichlet distribution parameters, and output the classification reliability results and classification uncertainty results of the object to be classified; wherein the evidence deep learning model of the two-layer evidence fusion is trained using the method described in the first aspect.

[0040] In an eighth aspect, the present disclosure provides a classification evaluation device based on a double-layer evidence fusion evidence deep learning model, including:

[0041] a fourth acquisition module, configured to acquire multimodal data of an object to be classified, wherein the object to be classified is medical feature data to be identified, and the medical feature data to be identified includes: a feature tensor of mode 1 obtained by constructing a brain functional connection network by calculating the temporal correlation of each brain region based on a blood oxygen level signal of the brain in a resting state, and a feature tensor of mode 2 obtained by constructing a fiber bundle connection network between brain regions by performing fiber bundle tracking based on the degree of water molecule diffusion;

[0042] The third first-layer fusion module is configured to, for each modality in the multimodal data, input the data of the modality into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, set the output layer activation function of the deep neural network DNN to a linear activation function, and obtain the corresponding initial deep learning evidence vector; use different credibility distribution methods to redistribute the initial deep learning evidence vector to obtain a corresponding plurality of redistributed deep learning evidence vectors; map the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters to obtain credibility results and uncertainty results of the corresponding plurality of subjective logic SL evidence vectors, and use the Dempster-Shafer evidence fusion rule to fuse the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors to obtain the credibility results and uncertainty results of the first-layer classification evidence vector under the modality;

[0043] The fourth second-layer fusion module uses the Dempster-Shafer evidence fusion rule to fuse the reliability results and uncertainty results of the first-layer classification evidence vectors of the multiple modalities to obtain the reliability results and uncertainty results of the second-layer classification evidence vectors;

[0044] The second classification evaluation module is configured to obtain a second-layer classification evidence vector based on the reliability results and uncertainty results of the second-layer classification evidence vector, and map the second-layer classification evidence vector to Dirichlet distribution parameters, and output the classification reliability results and classification uncertainty results of the object to be classified; wherein the evidence deep learning model of the two-layer evidence fusion is trained using the method described in the first aspect.

[0045] In a ninth aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores computer instructions, and wherein the computer instructions are executed by the processor to implement a method as described in any one of the first aspect and the second aspect.

[0046] In a tenth aspect, a computer-readable storage medium is provided in an embodiment of the present disclosure, on which computer instructions are stored. When the computer instructions are executed by a processor, the method described in any one of the first aspect and the second aspect is implemented.

[0047] According to the technical solution provided by the embodiments of the present disclosure, a multimodal data set including training data of multiple sample objects is obtained, and each sample object corresponds to multiple training data of multiple modalities; an evidence deep learning model is used for the multimodal data set, and classification reliability results and classification uncertainty results of the multiple sample objects are obtained based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification reliability results and classification uncertainty results of the sample object; according to the classification reliability results and true category labels of the multiple sample objects, a loss function under the Dirichlet distribution is constructed, and the model parameters of the evidence deep learning model are adjusted to obtain an optimized double-layer evidence fusion evidence deep learning model.

[0048] The present disclosure uses an evidence deep learning model for a multimodal data set including training data of multiple sample objects, and provides a two-layer evidence fusion strategy for each sample object, and each sample object corresponds to multiple training data of multiple modalities. The method uses the evidence deep learning model to achieve estimation of uncertainty; at the same time, a two-layer evidence fusion strategy is adopted, taking into account multiple training data of multiple modalities of the sample object, so that the evidence deep model can utilize training data of more modalities. The two-layer evidence fusion strategy also enables the evidence deep model to give play to the synergy and complementarity of different credibility allocation methods, the model generalization is better, and the estimation of uncertainty is more accurate. Therefore, by comprehensively considering the model generalization and uncertainty estimation capabilities, the evidence deep learning model with two-layer evidence fusion obtained through training in the present disclosure can provide confidence evaluation results for data classification, significantly improving the accuracy of data classification.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0051] Figure 1 A flowchart showing a double-layer evidence fusion learning method of an evidence deep learning model according to an embodiment of the present disclosure is shown;

[0052] Figure 2 A schematic diagram showing a double-layer evidence fusion strategy according to an embodiment of the present disclosure;

[0053] Figure 3 A flowchart showing a classification evaluation method of an evidence deep learning model based on double-layer evidence fusion according to an embodiment of the present disclosure is shown;

[0054] Figure 4A structural block diagram of a double-layer evidence fusion learning device of an evidence deep learning model according to an embodiment of the present disclosure is shown;

[0055] Figure 5 A structural block diagram of a classification and evaluation device based on a double-layer evidence fusion evidence deep learning model according to an embodiment of the present disclosure is shown;

[0056] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0057] Figure 7 A schematic diagram showing some sample objects when a classification evaluation method according to an embodiment of the present disclosure is used to perform classification evaluation on a multimodal digital image set;

[0058] Figure 8 A schematic diagram showing some sample objects of an out-of-distribution image set when a classification evaluation method according to an embodiment of the present disclosure is used to perform classification evaluation on a multimodal digital image set;

[0059] Fig. 9 A schematic diagram showing the cumulative probability distribution of the entropy of the classification evaluation results of the classification evaluation method of the embodiment of the present disclosure and various existing methods for an image set outside the distribution of a multimodal digital image set;

[0060] Fig.10 A schematic diagram showing the density distribution of entropy of classification results of an in-distribution image set and an out-of-distribution image set of a multimodal digital image set according to a classification evaluation method of an embodiment of the present disclosure;

[0061] Fig.11 A schematic diagram showing some sample objects when a classification evaluation method according to an embodiment of the present disclosure is used to classify and evaluate a multimodal medical feature dataset;

[0062] Fig.12 A schematic diagram showing some samples of an out-of-distribution data set when a classification evaluation method according to an embodiment of the present disclosure is used to perform classification evaluation on a multimodal medical feature data set;

[0063] Fig.13 A schematic diagram showing the cumulative probability distribution of the entropy of the classification evaluation results of the classification evaluation method of the embodiment of the present disclosure and various existing methods on the data set outside the distribution of the multimodal medical feature data set;

[0064] Fig.14 A schematic diagram showing the density distribution of entropy of classification evaluation results of the in-distribution data set and the out-of-distribution data set of the multimodal medical feature data set by the classification evaluation method of the embodiment of the present disclosure;

[0065] Fig.15 A flowchart showing the use of EEG data, speech data, and clinical assessment data to construct a multimodal psychiatric disease training set;

[0066] Fig.16 The structure of a computer system suitable for implementing the method according to the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0067] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0068] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0069] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0070] In the present disclosure, if it involves operations of obtaining user information or user data or displaying user information or user data to others, the operations are all authorized and confirmed by the user, or actively selected by the user.

[0071] When fusing multimodal data, it is usually assumed that the modal data of different samples have the same importance, that is, a fixed weight factor is set for each modality. However, multimodal deep neural network (DNN) learning is greedy, and the model uses each modality unbalancedly, which prevents multimodal deep neural network (DNN) learning from using the available modal information of all samples, resulting in poor generalization. In addition, existing multimodal learning methods often ignore the uncertainty of the model, resulting in unreliable prediction results, especially when the information of a certain modality is missing or has large noise (such as sensor anomalies). The present disclosure provides a double-layer evidence fusion learning method for an evidence deep learning model, the method comprising: obtaining a multimodal data set including training data of multiple sample objects, each sample object corresponding to multiple training data of multiple modalities; using the evidence deep learning model for the multimodal data set, and obtaining classification reliability results and classification uncertainty results of the multiple sample objects based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification reliability results and classification uncertainty results of the sample object; constructing a loss function under a Dirichlet distribution according to the classification reliability results and true category labels of the multiple sample objects, adjusting the model parameters of the evidence deep learning model, and obtaining an optimized double-layer evidence fusion evidence deep learning model.

[0072] The present disclosure uses an evidence deep learning model for a multimodal data set including training data of multiple sample objects, and provides a double-layer evidence fusion strategy for each sample object, and each sample object corresponds to multiple training data of multiple modalities. The method adopts the evidence deep learning model to realize the estimation of uncertainty; at the same time, a double-layer evidence fusion strategy is adopted, taking into account multiple training data of multiple modalities of the sample object, so that the evidence deep model can utilize training data of more modalities. The double-layer evidence fusion strategy also enables the evidence deep model to give play to the synergy and complementarity of different credibility allocation methods, the model generalization is better, and the estimation of uncertainty is more accurate. Therefore, by comprehensively considering the model generalization and uncertainty estimation capabilities, the effective fusion of multimodal data can be realized, the generalization is high, and the uncertainty estimation capability is strong, so that the evidence deep learning model with double-layer evidence fusion obtained by training in the present disclosure can provide confidence evaluation results for data classification, which significantly improves the accuracy of data classification.

[0073] Figure 1 A flowchart of a double-layer evidence fusion learning method of an evidence deep learning model according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S101 to S103.

[0074] In step S101 , a multimodal dataset is acquired, where the multimodal dataset includes training data of multiple sample objects, wherein each sample object corresponds to multiple training data of multiple modalities.

[0075] In a specific embodiment, the multimodal data set is a multimodal image set, and the multimodal image set includes training images of multiple sample objects, wherein each sample object corresponds to multiple training images of multiple modalities.

[0076] In a specific embodiment, the multimodal data set is a multimodal medical feature data set, which includes training medical feature data of multiple sample objects, wherein each sample object corresponds to multiple training medical feature data of multiple modalities.

[0077] According to an embodiment of the present disclosure, the multimodal dataset has N sample objects, the N sample objects have C categories and V modalities, and the true label of the nth sample object is The true label of the nth sample object represents the category of the nth sample object, and the representation tensor of the vth mode of the nth sample object is Among them, v∈{1, 2, 3, ...V}, n∈{1, 2, 3, ...N}.

[0078] In step S102, an evidence deep learning model is used for the multimodal data set to obtain classification reliability results and classification uncertainty results of the multiple sample objects based on the training data of the multiple sample objects, including adopting a two-layer evidence fusion strategy for each sample object to obtain the classification reliability results and classification uncertainty results of the sample object.

[0079] In a specific embodiment, an evidence deep learning model is used for a multimodal image set, and image recognition reliability results and image recognition uncertainty results of the multiple sample objects are obtained based on the training images of the multiple sample objects, including using a double-layer evidence fusion strategy for each sample object to obtain the image recognition reliability results and image recognition uncertainty results of the sample object.

[0080] In a specific embodiment, an evidence deep learning model is used for a multimodal medical feature data set, and medical feature classification reliability results and medical feature classification uncertainty results of the multiple sample objects are obtained based on the medical feature training data of the multiple sample objects, including using a two-layer evidence fusion strategy for each sample object to obtain the medical feature classification reliability results and medical feature classification uncertainty results of the sample object.

[0081] The inventors found that the existing uncertainty estimation algorithms usually focus on unimodal data, and can be divided into two categories according to the principle: Bayesian methods and non-Bayesian methods. Among them, non-Bayesian methods include deep integration, EDL and other methods. The BBA (Basic Belief Assignment, credibility distribution) method that combines the EDL method and subjective logic (Subjective Logic, SL), and then uses the Dirichlet model to quantify credibility distribution, has become an important branch of uncertainty estimation. The Deep Evidential Fusion (DEF) model generalizes EDL from unimodal to multimodal situations. Compared with the directly spliced ​​multimodal deep learning network (DNN), the model uses the Dempster-Shafer evidence fusion rule to organically integrate deep learning evidence, giving full play to the synergistic effect of multimodal information. However, the DEF model focuses on the generalization of the model, still assuming that all samples have the same modal information quality, and the uncertainty is not verified. Therefore, regardless of the above situation, the existing models all use a single credibility distribution method to train the model. However, in practice, the information quality of different samples and different modalities may be different, and a single credit assignment method may produce overly fixed training results, resulting in a decrease in the generalization performance of the model.

[0082] Therefore, in view of the generalization and uncertainty estimation ability of the model, the present disclosure proposes a two-layer evidence fusion strategy, which fuses two-layer evidence vectors for multiple training data of multiple modalities of each sample object, so as to comprehensively consider the generalization and uncertainty estimation ability of the model.

[0083] Figure 2 A schematic diagram showing a double-layer evidence fusion strategy according to an embodiment of the present disclosure is shown.

[0084] like Figure 2 As shown, the two-layer evidence fusion strategy includes: for each modality of the sample object, inputting the training data of the modality into the corresponding deep neural network DNN in the evidence deep learning model for learning, setting the output layer activation function of the deep neural network DNN to a linear activation function, and obtaining a corresponding initial deep learning evidence vector; using different credibility distribution methods to redistribute the initial deep learning evidence vector respectively, and obtaining a corresponding plurality of redistributed deep learning evidence vectors; mapping the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters, obtaining credibility results and uncertainty results of the corresponding plurality of subjective logic SL evidence vectors, and using the Dempster-Shafer evidence fusion rule to fuse the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors, and obtaining the credibility results and uncertainty results of the first-layer classification evidence vector under the modality; using the Dempster-Shafer evidence fusion rule to fuse the credibility results and uncertainty results of the first-layer classification evidence vector under the plurality of modalities, and obtaining the credibility results and uncertainty results of the second-layer classification evidence vector; wherein the training data can be a training image or a modal feature tensor manually extracted based on experience.

[0085] According to an embodiment of the present disclosure, a two-layer evidence fusion strategy is adopted for each sample object to obtain the classification reliability result and classification uncertainty result of the sample object, including: obtaining a second-layer classification evidence vector according to the reliability result and uncertainty result of the second-layer classification evidence vector, and mapping the second-layer classification evidence vector to Dirichlet distribution parameters to obtain the classification reliability result and classification uncertainty result of the sample object.

[0086] In a specific embodiment, the two-layer evidence fusion strategy is adopted for each sample object to obtain the classification reliability result and classification uncertainty result of the sample object, including: obtaining the second-layer classification evidence vector according to the reliability result and uncertainty result of the second-layer classification evidence vector, and mapping the second-layer classification evidence vector to Dirichlet distribution parameters to obtain the image recognition reliability result and image recognition uncertainty result of the sample object.

[0087] In a specific embodiment, the two-layer evidence fusion strategy is adopted for each sample object to obtain the classification reliability result and classification uncertainty result of the sample object, including: obtaining the second-layer classification evidence vector according to the reliability result and uncertainty result of the second-layer classification evidence vector, and mapping the second-layer classification evidence vector to Dirichlet distribution parameters to obtain the medical feature classification reliability result and medical feature classification uncertainty result of the sample object.

[0088] According to an embodiment of the present disclosure, the deep neural network DNN may be a ResNet network, a CCNN network, etc., but is not limited thereto.

[0089] According to an embodiment of the present disclosure, for each modality, the training data of the modality is input into the corresponding deep neural network DNN in the evidence deep learning model for learning, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain the corresponding initial deep learning evidence vector, including obtaining the corresponding initial deep learning evidence vector using the following formula:

[0090]

[0091] in, represents the initial deep learning evidence vector under the vth modality of the nth sample object, represents the network parameters of the deep neural network DNN corresponding to the vth mode, f v represents the forward propagation function of the deep neural network DNN corresponding to the vth mode, R C represents a real number set with data dimension C. The training data may be a training image or a modal feature tensor manually extracted based on experience.

[0092] According to an embodiment of the present disclosure, the different credit allocation methods are to use RC to R +C The method of mapping data by any different mapping functions between R C represents a real number set with data dimension C, R +C Represents a set of non-negative real numbers with data dimension C.

[0093] According to an embodiment of the present disclosure, the two-layer evidence fusion strategy combines a variety of different credibility allocation methods to allocate appropriate evidence vectors to each modality of each sample object, thereby providing a classification credibility result for each sample object. Specifically, based on the viewpoint of "different evidences should be complementary" in the Dempster-Shafer evidence fusion rule, the present disclosure provides two examples of complementary credibility allocation methods. Those skilled in the art can understand that the principles and solutions of the present disclosure are also applicable to the use of other complementary credibility allocation methods, as well as the use of more credibility allocation methods.

[0094] The first credibility allocation method uses the softmax function as the mapping function:

[0095] According to an embodiment of the present disclosure, the use of different credibility distribution methods to redistribute the initial deep learning evidence vectors respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors includes using the following formula to redistribute the initial deep learning evidence vector once to obtain a first redistributed deep learning evidence vector among the plurality of redistributed deep learning evidence vectors:

[0096]

[0097] in, express Any element in represents the first redistributed deep learning evidence vector among multiple redistributed deep learning evidence vectors under the vth modality of the nth sample object, express The cth element in represents the total amount of initial deep learning evidence under the vth modality of the nth sample object, express In the cth element, softmax() represents the softmax function, and e() represents the exponential function with e as the base.

[0098] The second credit assignment method uses the ReLU function as the mapping function:

[0099] According to an embodiment of the present disclosure, the use of different credibility distribution methods to respectively redistribute the initial deep learning evidence vectors to obtain a corresponding plurality of redistributed deep learning evidence vectors includes using the following formula to redistribute the initial deep learning evidence vector once to obtain a second redistributed deep learning evidence vector among the plurality of redistributed deep learning evidence vectors:

[0100]

[0101] in, represents the second redistributed deep learning evidence vector under the vth modality of the nth sample object, express The cth element in express In the cth element, ReLU() represents the ReLU function.

[0102] The present invention uses the softmax function to redistribute the initial deep learning evidence vector, so that the subjective logic SL evidence vector is exponentially amplified. The subjective logic SL evidence that clearly points to a certain category will be highlighted, which is conducive to improving the accuracy, but the uncertainty is greatly reduced. The use of the ReLU function continues the methods in the EDL model and the DEF model, retaining the strength of the original subjective logic SL evidence. Therefore, the above two credibility allocation methods are consistent and complementary. Therefore, the fusion of the two can achieve the goal of highlighting the key subjective logic SL evidence while retaining the estimate of the uncertainty of the subjective logic SL evidence.

[0103] According to an embodiment of the present disclosure, mapping the multiple redistributed deep learning evidence vectors into Dirichlet distribution parameters to obtain corresponding credibility results and uncertainty results of multiple subjective logic SL evidence vectors includes:

[0104] The Dirichlet distribution parameters are obtained using the following formula:

[0105]

[0106] in, represents the kth redistributed deep learning evidence vector under the vth modality of the nth sample object, represents the Dirichlet distribution parameter corresponding to the k-th redistributed deep learning evidence vector;

[0107] Construct the Dirichlet distribution as:

[0108]

[0109] Where B() represents the probability density function of Beta distribution. Represents the classification probability vector, p i express Any element in , Dir() represents the probability density function of Dirichlet distribution, represents the Dirichlet distribution parameter of Dir(), T C represents a C-dimensional simplex;

[0110] Then, according to the Dirichlet distribution and Dirichlet distribution parameters, the corresponding reliability results and uncertainty results of multiple subjective logic SL evidence vectors are obtained using the following formula:

[0111]

[0112] in, represents the intensity of the Dirichlet distribution corresponding to the k-th redistributed deep learning evidence vector, represents the kth subjective logic SL evidence vector under the vth modality of the nth sample object, express The reliability results, express The uncertainty result of k∈{1, 2, 3, ...K}, K represents the number of credibility allocation methods adopted.

[0113] According to an embodiment of the present disclosure, the use of the Dempster-Shafer evidence fusion rule to fuse the multiple subjective logic SL evidence vectors includes fusing them using the following formula:

[0114]

[0115] in, Represents the reliability result of the first-level classification evidence vector under V modes of the nth sample object, Represents the uncertainty result of the first-level classification evidence vector under the V mode of the nth sample object;

[0116] It represents the operation based on the Dempster-Shafer evidence fusion rule, and its operation rules are as follows:

[0117]

[0118] Among them, ⊙ represents the multiplication operation of corresponding elements of the vector, and κ represents the normalization coefficient.

[0119] The present invention uses two or more credibility allocation methods to obtain credibility results and uncertainty results of multiple subjective logic SL evidence vectors, and uses the Dempster-Shafer evidence synthesis rule to fuse the credibility results and uncertainty results of multiple subjective logic SL evidence vectors, thereby obtaining the credibility results and uncertainty results of the first-level classification evidence vector under a modality, which can avoid overly biased subjective logic SL evidence allocation.

[0120] According to an embodiment of the present disclosure, the Dempster-Shafer evidence fusion rule is used to fuse the reliability results and uncertainty results of the first-level classification evidence vectors under the multiple modalities, including fusing using the following formula:

[0121]

[0122] in, Represents the confidence result of the second-level classification evidence vector under V modes of the nth sample object, Represents the uncertainty result of the second-level classification evidence vector under V modes of the n-th sample object.

[0123] The present disclosure utilizes the Dempster-Shafer evidence synthesis rule to perform secondary fusion of the reliability results and uncertainty results of the first-layer classification evidence vector, so that the deep evidence learning model can learn the information of all modalities in the multiple samples, so that the present disclosure can comprehensively consider the model generalization and uncertainty estimation capabilities.

[0124] According to an embodiment of the present disclosure, obtaining the second-layer classification evidence vector according to the reliability result and the uncertainty result of the second-layer classification evidence vector includes obtaining the second-layer classification evidence vector using the following formula:

[0125]

[0126] in, Represents the Dirichlet distribution intensity corresponding to the second-level classification evidence vector of the nth sample object, The second-level classification evidence vector representing the nth sample object;

[0127] Mapping the second-layer classification evidence vector to a Dirichlet distribution parameter to obtain the classification confidence result and classification uncertainty result of the sample object includes obtaining the classification confidence result and classification uncertainty result of the sample object using the following formula:

[0128]

[0129] in, Represents the Dirichlet distribution parameter corresponding to the second-level classification evidence of the nth sample object, Indicates the classification confidence result of the nth sample object, u n Represents the classification uncertainty result of the nth sample.

[0130] In step 103, according to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, a loss function under the Dirichlet distribution is constructed, and the model parameters of the evidence deep learning model are adjusted to obtain an optimized double-layer evidence fusion evidence deep learning model.

[0131] According to an embodiment of the present disclosure, the loss function under the Dirichlet distribution is constructed according to the classification reliability results of multiple sample objects and the true category labels of the multiple sample objects, and the model parameters of the evidence deep learning model are adjusted to obtain the optimized evidence deep learning model of the double-layer evidence fusion, including constructing the loss function using the following formula, and obtaining the optimized evidence deep learning model of the double-layer evidence fusion using the loss function:

[0132]

[0133] Among them, MSE i () represents the mean square error loss function of the evidence deep learning model, represents the classification confidence result of the i-th sample object, p ij express Any element of is the true category label of the i-th sample, y ij express Any element of represents expectation, Var() represents variance; Represents the Dirichlet distribution parameter corresponding to the second-level classification evidence vector of the i-th sample object, α ij express Any element of S i express The corresponding Dirichlet distribution intensity, represents the Dirichlet distribution parameter of the i-th sample object after removing the subjective logic SL evidence of the true category, express Any element of KL[] represents the introduction of The KL divergence between the Dirichlet distribution under and the uniform Dirichlet distribution, Γ represents the Gamma function, and ψ represents the digamma function;

[0134]

[0135] Where L represents the optimization target of the evidence deep learning model, λ t represents the annealing coefficient, t represents the number of iterations, and BatchSize represents the batch size.

[0136] Because when outputting the classification results of multiple sample objects, we do not want the subjective logic SL evidence of multiple categories to compete with each other, so we introduce the Dirichlet distribution parameter after removing the subjective logic SL evidence of the true category The KL divergence between the Dirichlet distribution and the uniform Dirichlet distribution under the introduced parameter is added as a regularization term to the training objective. When optimizing the evidence deep learning model, as the number of iterations t increases, the influence of the KL divergence on the overall optimization objective gradually increases, so that the evidence deep learning model can learn a relatively sufficient subjective logic SL at the beginning of training, avoiding the premature convergence of erroneous sample objects to the uniform Dirichlet distribution, thereby improving the ability to identify abnormal data.

[0137] Figure 3 A flow chart of a classification evaluation method based on a double-layer evidence fusion evidence deep learning model according to an embodiment of the present disclosure is shown. Figure 3 As shown, the method includes steps S301 to S304.

[0138] In step S301, multimodal data of an object to be classified is obtained.

[0139] In a specific embodiment, the object to be classified is an image to be identified, medical feature data to be identified, etc., but is not limited thereto.

[0140] In step S302, for each modality in the multimodal data, the data of the modality is input into the trained evidence deep learning model into the deep neural network DNN corresponding to the modal data for learning, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors are fused using the Dempster-Shafer evidence fusion rule to obtain credibility results and uncertainty results of the first layer classification evidence vector under the modality.

[0141] In step S303, the reliability results and uncertainty results of the first-layer classification evidence vectors of the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vectors.

[0142] In step S304, a second-layer classification evidence vector is obtained according to the reliability result and uncertainty result of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to a Dirichlet distribution parameter, and the classification reliability result of the object to be classified is output.

[0143] According to an embodiment of the present disclosure, the evidence deep learning model of the dual-layer evidence fusion is trained using any one of the methods of the dual-layer evidence fusion learning method of the evidence deep learning model.

[0144] Figure 4 The structural block diagram of a dual-layer evidence fusion learning device of an evidence deep learning model according to an embodiment of the present disclosure is shown. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0145] like Figure 4 As shown, the device 400 includes: a first acquisition module 401, a double-layer evidence fusion module 402 and a training module 403;

[0146] A first acquisition module 401 is configured to acquire a multimodal dataset, wherein the multimodal dataset includes training data of multiple sample objects, wherein each sample object corresponds to multiple training data of multiple modalities;

[0147] Specifically, the multimodal data set is a multimodal image set and a multimodal medical feature data set, but is not limited thereto; the multimodal image set includes training images of multiple sample objects, and the multimodal medical feature data set includes medical feature training data of multiple sample objects;

[0148] A two-layer evidence fusion module 402 is configured to use an evidence deep learning model for the multimodal data set to obtain classification reliability results of the multiple sample objects based on the training data of the multiple sample objects, including using a two-layer evidence fusion strategy for each sample object to obtain the classification reliability result of the sample object;

[0149] The training module 403 is configured to construct a loss function under the Dirichlet distribution according to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, adjust the model parameters of the evidence deep learning model, and obtain an optimized double-layer evidence fusion evidence deep learning model.

[0150] Figure 5 The structural block diagram of a classification evaluation device based on a double-layer evidence fusion evidence deep learning model according to an embodiment of the present disclosure is shown. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 5 As shown, the device 500 includes: a second acquisition module 501, a first layer fusion module 502, a second layer fusion module 503 and a classification evaluation module 504;

[0151] A second acquisition module 501 is configured to acquire multimodal data of an object to be classified;

[0152] Specifically, the multimodal dataset is a multimodal image dataset or a multimodal medical feature dataset, but is not limited thereto.

[0153] The first layer fusion module 502 is configured to, for each modality in the multimodal data, input the data of the modality into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, set the output layer activation function of the deep neural network DNN to a linear activation function, and obtain a corresponding initial deep learning evidence vector; use different credibility distribution methods to redistribute the initial deep learning evidence vector to obtain a corresponding plurality of redistributed deep learning evidence vectors; map the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors, and use the Dempster-Shafer evidence fusion rule to fuse the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors to obtain credibility results and uncertainty results of the first layer classification evidence vector under the modality;

[0154] The second layer fusion module 503 is configured to fuse the reliability results and uncertainty results of the first layer classification evidence vectors of the multiple modalities using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second layer classification evidence vectors;

[0155] The classification evaluation module 504 is configured to obtain a second-layer classification evidence vector according to the reliability result and uncertainty result of the second-layer classification evidence vector, map the second-layer classification evidence vector into Dirichlet distribution parameters, and output the classification reliability result and classification uncertainty result of the object to be classified.

[0156] Specifically, when the object to be classified is an image to be recognized, an image recognition reliability result and an image recognition uncertainty result of the object to be classified are output.

[0157] Specifically, when the object to be classified is medical feature data to be classified, a medical feature classification reliability result and a medical feature classification uncertainty result of the object to be classified are output.

[0158] The present disclosure also discloses an electronic device, Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0159] like Figure 6 As shown, the electronic device includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to an embodiment of the present disclosure.

[0160] In the present disclosure, for the convenience of explanation, the classification evaluation method of the embodiment of the present disclosure is applied to a multimodal digital image set and a multimodal medical feature data set as an example for explanation, and an exemplary description for the task of classification of mental and psychological diseases is given, but it is not limited thereto. Those skilled in the art can understand that the principles and schemes of the present disclosure are also applicable to the case of classification evaluation of other multimodal data sets.

[0161] Figure 7 A schematic diagram showing some sample objects when a classification evaluation method according to an embodiment of the present disclosure is used to perform classification evaluation on a multimodal digital image set.

[0162] like Figure 7 As shown, different numbers are different single-mode forms, and different digital image sets are different ways of describing the numbers 0-9, so a multi-modal digital image set is artificially synthesized by pairing. Because the numbers 0-9 are classified, it is a 10-classification problem, C=10.

[0163] In the multimodal digital image set:

[0164] Modality 1 uses the MNIST digital image set of handwritten digits with a data dimension of 28×28×1, and the data dimension is the height, width and number of channels of the MNIST feature map of handwritten digits under modality 1; Modality 2 uses the SVHN digital data set of digital images with a data dimension of 32×32×3, and the data dimension is the height, width and number of channels of the SVHN feature map of digital images under modality 2; the training set of the evidence deep learning model is selected from the corresponding training sets in the MNIST digital data set and the SVHN digital data set, limiting 50 sample objects for each type of digit, and randomly selecting 30 permutations when the two modalities are paired, so that the number of sample objects in the training set obtained is 15,000, and N=15,000 is set.

[0165] Input the training images of the sample objects in the training set of modality 1 into the deep neural network DNN corresponding to the evidence deep learning model, such as the ResNet18 network, but not limited to this; wherein the image size of the input sample object is (28, 28, 1), and the output result of the nth sample object is The output is 10-dimensional; after two complementary credit assignment methods (such as using softma x Function and ReLU function are mapped) to obtain the redistributed deep learning evidence vector and Map it to the Dirichlet distribution parameter and get the subjective logic SL evidence vector and The subjective logic SL evidence vector includes reliability results and uncertainty results.

[0166] Based on the Dempster-Shafer evidence fusion rule, the two subjective logic SL evidence vectors are combined. and Fusion is performed to obtain the first-layer classification evidence vector of modality 1 The first-level classification evidence vector Includes confidence results and uncertainty results

[0167] Input the training images of the sample objects in the training set of modality 2 into the deep neural network DNN corresponding to the evidence deep learning model, such as the ResNet18 network, but not limited to this; wherein the image size of the input sample object is (32, 32, 3), and the output result of the nth sample object is The output is 10-dimensional; after two complementary credit assignment methods (such as using softma x Function and ReLU function are mapped) and the mapping of Dirichlet distribution parameters to obtain subjective logic SL evidence respectively. and The subjective logic SL evidence vector includes reliability results and uncertainty results. and Based on the Dempster-Shafer evidence fusion rule, the two subjective logic SL evidence vectors are combined. and Fusion is performed to obtain the first-layer classification evidence vector of modality 2 The first-level classification evidence vector Includes confidence results and uncertainty results

[0168] Based on the Dempster-Shafer evidence fusion rule, the first-level classification evidence vectors of the two modalities are and Perform secondary fusion to obtain a second-level classification evidence vector, where the second-level classification evidence vector includes a reliability result and an uncertainty result;

[0169] Obtaining a second-layer classification evidence vector according to the reliability result and the uncertainty result of the second-layer classification evidence vector, and mapping the second-layer classification evidence vector to a Dirichlet distribution parameter to obtain a classification reliability result and a classification uncertainty result of the sample object;

[0170] According to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, a loss function under the Dirichlet distribution is constructed, and the model parameters of the evidence deep learning model are adjusted to obtain an optimized double-layer evidence fusion evidence deep learning model.

[0171] According to an embodiment of the present disclosure, during optimization, the annealing coefficient is set t represents the number of iterations, BatchSize represents the batch size, which is set to 64. The set value is an empirical value of deep learning training.

[0172] This paper exploits the complementary information between multiple credibility assignment methods and multimodal data, gives full play to the synergy between subjective logic SL evidence vectors, and obtains a higher accuracy than existing multimodal data classification methods. At the same time, it provides the uncertainty of the evidence fusion results under each modality, thereby providing interpretability.

[0173] The present disclosure designs a test set according to the construction method of the training set. The test set is selected from the corresponding test sets in the MNIST digital recognition image set and the SVHN digital recognition image set. Since the number of samples of each category in the SVHN test set is more than the number of samples in the MNIST test set, the MNIST sample object in the MNIST test set is used as a benchmark, and a matching SVHN sample object is selected for each MNIST sample object. When the two modalities are paired, 10 arrangements are randomly selected, and the sample size of the test set is 100,000, that is, N=100,000.

[0174] Taking the non-evidential model ResNet18 network as the baseline, the EDL model, DEF model and the evidence deep learning model disclosed in this paper were used to perform classification evaluation respectively, with accuracy as the indicator. The results are shown in Table 1.

[0175] Table 1

[0176]

[0177]

[0178] Figure 8 A schematic diagram showing some sample objects of an out-of-distribution image set when a classification evaluation method according to an embodiment of the present disclosure performs classification evaluation on a multimodal digital image set.

[0179] like Figure 8 As shown, in order to further verify the uncertainty estimation ability of the evidence deep learning model disclosed in this disclosure, an out-of-distribution data set of multimodal digital image sets is constructed in the form of multimodal training sets. Specifically, a non-digital letter set is selected for construction, mode 1 comes from the handwritten letter set EMNIST, with a data dimension of 28×28×1, and mode 2 comes from the letter set chars74k of different fonts in real scenes, with a data dimension of 32×32×3. The sample size of the out-of-distribution image set is 44450, that is, N=44450.

[0180] The evidence depth model trained on the corresponding training set of MNIST-SVHN is used to predict the non-numeric letter set, and the entropy of the prediction result is used as the uncertainty indicator. The detection ability of the method disclosed in this disclosure and various existing methods for abnormal sample objects is considered according to the distribution of uncertainty.

[0181] It is known that a trained model should present high uncertainty for out-of-distribution data, that is, the entropy is close to logC, where C is the number of classification categories. In the multimodal digital image set, C=10.

[0182] Fig. 9 A schematic diagram showing the cumulative probability distribution of the entropy of the classification evaluation results of the classification evaluation method of the embodiment of the present disclosure and various existing methods for an out-of-distribution image set of a multimodal digital image set.

[0183] like Fig. 9 As shown, compared with the ResNet18 network model, the EDL model and the DEF model, the cumulative probability distribution of the entropy of the evidence deep learning model disclosed in the present invention is closer to the lower right, indicating that the distribution of digital classification is closer to a uniform distribution.

[0184] Fig.10 A schematic diagram showing the density distribution of entropy of classification results of an in-distribution image set and an out-of-distribution image set of a multimodal digital image set according to the classification evaluation method of an embodiment of the present disclosure is shown.

[0185] like Fig.10 As shown, for out-of-distribution data of a multimodal digital image set, the uncertainty in digital classification is much higher than that for in-distribution data. This shows the rationality of the disclosed deep learning model in estimating uncertainty and can be used to identify abnormal sample objects.

[0186] Fig.11 A schematic diagram showing some sample objects when a classification evaluation method according to an embodiment of the present disclosure is used to perform classification evaluation on a multimodal medical feature dataset.

[0187] like Fig.11As shown, the present disclosure is based on actual clinical data to identify whether the upper limb motor function of stroke patients using brain-computer interface (BCI) assisted robotic hand movement therapy methods has been restored. It is a two-classification problem, so C=2. The human brain is imaged using a nuclear magnetic resonance imaging method, and preprocessing and feature extraction are performed based on the nuclear magnetic resonance image to obtain a multimodal medical feature data set including resting state functional magnetic resonance imaging (rs-fMRI) and structural magnetic resonance imaging (DTI) related features. Among them, the original data of modality 1 is rs-fMRI, which reflects the blood oxygen level signal of the brain in a resting state. Based on this, the temporal correlation of each brain region is calculated to construct a brain functional connection network, and the characteristic tensor of modality 1 is obtained, which shows the functional characteristics of the brain. The original data of modality 2 is a DTI signal, which reflects the degree of water molecule diffusion. Based on this, fiber bundle tracking is performed to construct a fiber bundle connection network between brain regions, and the characteristic tensor of modality 2 is obtained, which shows the structural characteristics of the brain.

[0188] Relevant data of 40 patients were collected. The AAL atlas (one of the most widely used templates in human brain connectome research) was used to define network nodes, with a total of 116 brain regions. Therefore, the feature tensor dimension of modality 1 is 116×116×1, and the feature tensor dimension of modality 2 is 116×116×1.

[0189] The sample object feature tensor in the training set of modality 1 is input into the deep neural network DNN corresponding to the evidence deep learning model, such as the CCNN network, but not limited thereto; wherein the dimension of the feature tensor of the input sample object is (116, 116, 1), the output is 2-dimensional, the number of convolution kernels in the first convolution layer is 64, the number of convolution kernels in the second convolution layer is 128, and the number of neurons in the hidden layer is 96. The sample objects in the training set of modality 2 are input into the deep neural network DNN corresponding to the evidence deep learning model, such as the CCNN network, but not limited thereto; wherein the dimension of the feature tensor of the input sample object is (116, 116, 1), and the output is 2-dimensional.

[0190] Based on the Dempster-Shafer evidence fusion rule, the subjective logic SL evidence vectors obtained by the two credibility assignment methods are fused to obtain the first-level classification evidence vector, and based on the Dempster-Shafer evidence fusion rule, the first-level classification evidence vectors of the two modalities are fused again to obtain the second-level classification evidence vector.

[0191] According to an embodiment of the present disclosure, during optimization, the annealing coefficient is set t represents the number of iterations, and BatchSize represents the batch size, which is set to 4.

[0192] Since the number of sample objects in the multimodal medical feature dataset in the embodiment of the present disclosure is small, a 10-fold cross-validation method is used to train and test the evidence deep learning model, that is, the multimodal medical feature dataset is divided into 10 parts, each containing 4 samples. 9 of them are selected as training sets to train the evidence deep learning model, and the evidence deep learning model is tested on the remaining 1 part, and the above steps are repeated until all sample objects are traversed.

[0193] Using the non-evidential CCNN network as the baseline, we used one of the two confidence assignment methods disclosed in the present invention, and combined the two methods for classification. The average accuracy of the classification results under 10-fold cross validation was used as an indicator. The results are shown in Table 2.

[0194] Table 2

[0195]

[0196] It can be seen that when the two confidence assignment methods are used separately, no one method is significantly better than the other method. When the two confidence assignment methods are combined, the accuracy of the classification results is greatly improved. Therefore, the fusion of the two confidence assignment methods can mine information from different angles under a single modality. This makes the evidence deep learning model have good generalization even when trained on small sample objects.

[0197] To verify the effect of multimodal fusion, since the dimensions of the two modal feature tensors in this embodiment are consistent, the feature tensors of the two modalities are directly spliced ​​to obtain a multimodal feature tensor with a dimension of (116, 116, 2), and the CCNN network model and the EDL model are used as the baseline of the multimodal fusion model. The DEF model and the evidence deep learning model are used for classification, and the average accuracy of the classification under 10-fold cross validation is used as an indicator. The results are shown in Table 3.

[0198] Table 3

[0199] Model Accuracy (%) CCNN 67.5 EDL 72.5 DEF 70.0 Evidence Deep Learning Model for Dual-Layer Evidence Fusion 77.5

[0200] In addition to providing classifications during prediction, the present disclosure can also provide the uncertainty of the classifications. In order to facilitate comparison with other studies, the uncertainty is characterized by the entropy of the classification results. The closer the entropy value is to logC, the higher the uncertainty and the lower the confidence, where C is the number of categories.

[0201] Table 4 lists the true labels, classification results and uncertainty of the classification results of the two sample objects.

[0202] Table 4

[0203] Sample ID True category label Classification results Uncertainty of classification results 10 0 1 0.6023 38 1 1 0.0280

[0204] As can be seen from Table 4, for the 10th sample object, the classification result is inconsistent with the true category label, and the classification result is wrong. At this time, the uncertainty of the recognition result is close to log2, that is, the confidence of the classification result is low. The uncertainty of the classification result of the 38th sample object is close to 0, indicating that the confidence of the classification result is high.

[0205] Compared with the existing multimodal classification models, the present invention proposes to use different confidence assignment methods to mine information from different angles under a single modality, and then perform multimodal fusion, making full use of different confidence assignment methods and complementary information between multimodalities, overcoming the difficulties of multimodal fusion on fewer sample objects, and providing interpretability.

[0206] Fig.12 A schematic diagram showing some samples of an out-of-distribution data set when a classification evaluation method according to an embodiment of the present disclosure is used to perform classification evaluation on a multimodal medical feature data set.

[0207] like Fig.12 As shown, in order to further verify the uncertainty estimation ability of the evidentiary deep learning model, a noise connection network was constructed according to the following rules to form the out-of-distribution data set of the multimodal medical feature data set. The noise generation rule is that the mode 1 functional connection network is superimposed with noise that obeys the beta distribution with parameters 1, 1, and the mode 2 fiber bundle connection network is superimposed with noise that obeys the beta distribution with parameters 2, 2. A total of 100 out-of-distribution data are randomly generated.

[0208] An evidence-based deep learning model trained with a multimodal medical feature dataset of 40 stroke patients who received BCI-assisted treatment was used to classify the noisy connection network. The entropy of the classification results was used as an uncertainty indicator, and the detection capabilities of the disclosed method and various existing methods for abnormal sample objects were evaluated based on the distribution of uncertainty.

[0209] Fig.13 A schematic diagram showing the cumulative probability distribution of the entropy of the classification evaluation results of the classification evaluation method of the embodiment of the present disclosure and various existing methods on a data set outside the distribution of a multimodal medical feature data set is shown.

[0210] like Fig.13 As shown, compared with the CCNN model, the EDL model and the DEF model, the distribution of the entropy of the recognition results obtained using the classification evaluation method disclosed in the present invention is closer to a uniform distribution.

[0211] Fig.14 A schematic diagram showing the density distribution of entropy of classification evaluation results of in-distribution datasets and out-of-distribution datasets of a multimodal medical feature dataset according to the classification evaluation method of an embodiment of the present disclosure is shown.

[0212] like Fig.14As shown, for the out-of-distribution data of the multimodal medical feature dataset, the uncertainty in classification is much higher than that for the in-distribution data. This illustrates the rationality of the classification method of the present disclosure in estimating uncertainty, and can be used to identify abnormal sample objects.

[0213] The inventors have found that there are many types of existing mental and psychological diseases, different diseases may cause similar symptoms, and a single individual may suffer from multiple diseases at the same time. When using machine learning methods to classify mental and psychological diseases, it is usually necessary to refer to the results of multiple examinations or tests. Specifically, multiple modal data of mental and psychological diseases such as electroencephalogram (EEG) data, voice data, and clinical assessment data provide important credentials for the classification task. At the same time, there is currently no gold standard for the judgment of mental and psychological diseases. When the model outputs the classification results, it is also necessary to estimate the uncertainty of the model to explain the confidence of the classification results and provide interpretability.

[0214] Fig.15 A flowchart showing the construction of a multimodal psychiatric disease training set using EEG data, speech data, and clinical assessment data.

[0215] like Fig.15 As shown in the figure, the EEG data is filtered, bad electrodes and bad signal segments are removed, and artifacts are removed by independent principal component analysis to obtain preprocessed EEG data. The preprocessed EEG data can be subjected to power spectrum analysis and functional connectivity analysis to obtain characteristics such as frequency band power, power ratio, lateralization index, coherence, connection pattern and graph theory properties to obtain training data under the EEG modality.

[0216] The EEGLAB toolbox and its plug-ins can be used to preprocess and analyze the EEG data obtained from the scalp electrodes of the subjects. The EEG data are filtered using, for example, an FIR high-pass filter with a cutoff frequency of 0.5 Hz and a low-pass filter with a cutoff frequency of 45 Hz to obtain filtered EEG data. The bad channels and data segments with large noise that need to be removed are determined based on the electrode impedance and manual inspection. Then, independent component analysis (ICA) is performed on the signal, and the artifact components are removed by combining the iclabel tool and visual inspection to obtain the preprocessed EEG data. The preprocessed EEG data is Fourier transformed to obtain the power spectrum, and the corresponding frequency band power, power ratio and lateralization index are obtained for different frequency bands. The Granger causality method and the convergent cross mapping (CCM) method can be used to construct a directed functional network for the signals of different electrodes, and the phase lock value method, the coherence method and the coherence method can also be used to construct an undirected functional network to obtain the coherence and connection pattern of the functional connection. The Brain connectivity Toolbox tool can also be used to calculate graph properties based on the functional network.

[0217] The speech data is subjected to pre-emphasis and de-emphasis, frame windowing and filtering to obtain a pre-processed speech signal. The pre-processed speech signal can be subjected to time domain analysis and frequency domain analysis to obtain corresponding time domain features and frequency domain features, including but not limited to short-time energy, short-time zero-crossing rate, pitch period, spectrogram, formant frequency, and Mel-frequency cepstral coefficients, to obtain training data under speech mode.

[0218] The preprocessing and feature analysis of the speech data obtained during the individual consultation phase of the subject can be performed using the librosa library in Python. First, the speech data is pre-emphasized, and the speech data is framed using a frame length of 25 milliseconds and a step length of 10 milliseconds. A Hamming window is applied to each frame of data, and then the speech data is filtered using, for example, an FIR high-pass filter with a cutoff frequency of 0.5Hz and a low-pass filter with a cutoff frequency of 2000Hz to obtain the preprocessed speech data. According to the definitions of short-time energy, short-time zero-crossing rate, and pitch period, the corresponding features are calculated, and the short-time Fourier transform of the speech data is performed using the librosa.core.stft() function to obtain the amplitude. The amplitude is converted into decibel units using the librosa.amplitude_to_db() function to obtain the spectrogram. The librosa.feature.mfcc() function is used to obtain the Mel cepstral coefficients, and the librosa.feature.chroma_cqt() function is used to obtain the constant Q chromatogram.

[0219] For clinical assessment data, outliers were removed, categorical variables were encoded, and continuous variables were normalized to obtain training data under clinical modality.

[0220] The feature data corresponding to the three modalities constitute the training set for this task. Figure 1 The process is to learn the deep neural network DNN corresponding to the deep learning model of each modality input evidence of each sample object in the collected mental and psychological disease training set, such as CNN, MLP network, but not limited to this. Based on the Dempster-Shafer evidence fusion rule, the subjective logic SL evidence vector obtained by multiple credibility distribution methods is fused to obtain the first layer classification evidence vector, and the first layer classification evidence vectors of the three modalities are secondary fused based on the Dempster-Shafer evidence fusion rule to obtain the second layer classification evidence vector; the second layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification credibility result and classification uncertainty result of each sample object.

[0221] The double-layer evidence fusion strategy proposed in the present invention utilizes different credibility allocation methods to mine single-modal information from different angles, and then performs multimodal fusion, thereby realizing the fusion of different credibility allocation methods and multimodal fusion, and realizing the classification of mental and psychological diseases based on multiple information sources; at the same time, an evidence deep learning model is used to estimate the uncertainty of the model and provide confidence for the classification results.

[0222] Fig.16 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown.

[0223] like Fig.16 As shown, the computer system includes a processing unit, which can perform the various methods in the above-mentioned embodiments according to the program stored in the read-only memory (ROM) or the program loaded from the storage part into the random access memory (RAM). In the RAM, various programs and data required for the operation of the computer system are also stored. The processing unit, ROM and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0224] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN card, a modem, etc. The communication part performs a communication process via a network such as the Internet. The drive is also connected to the I / O interface as needed. Removable media, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed. Among them, the processing unit can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0225] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium.

[0226] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0227] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or programmable hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0228] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A two-layer evidence fusion learning method for an evidence deep learning model, characterized in that: include: Acquire a multimodal dataset, the multimodal dataset comprising training data of a plurality of sample objects, wherein each sample object corresponds to a plurality of training data of a plurality of modalities, and the multimodal dataset is a multimodal image set; Using an evidence deep learning model for the multimodal data set, obtaining classification confidence results and classification uncertainty results for the multiple sample objects based on the training data of the multiple sample objects, including using a double-layer evidence fusion strategy for each sample object to obtain the classification confidence results and classification uncertainty results for the sample object; The two-layer evidence fusion strategy includes: for each modality of the sample object, inputting the training data of the modality into the corresponding deep neural network DNN in the evidence deep learning model for learning, setting the output layer activation function of the deep neural network DNN to a linear activation function, and obtaining a corresponding initial deep learning evidence vector; using different credibility distribution methods to redistribute the initial deep learning evidence vector respectively, and obtaining a corresponding plurality of redistributed deep learning evidence vectors; mapping the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters, and obtaining credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and using the Dempster-Shafer evidence fusion rule to fusion the plurality of subjective logic S The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions; According to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, a loss function under the Dirichlet distribution is constructed, and the model parameters of the evidence deep learning model are adjusted to obtain an optimized double-layer evidence fusion evidence deep learning model.

2. A two-layer evidence fusion learning method for an evidence deep learning model, characterized in that: include: Acquire a multimodal data set, the multimodal data set comprising training data of a plurality of sample objects, wherein each sample object corresponds to a plurality of training data of a plurality of modalities, the multimodal data set being a multimodal medical feature data set, the multimodal medical feature data set comprising: a feature tensor obtained by constructing a brain functional connection network by calculating the temporal correlation of each brain region based on a blood oxygen level signal of the brain in a resting state, and a feature tensor obtained by constructing a fiber bundle connection network between brain regions by performing fiber bundle tracing based on the degree of water molecule diffusion; Using an evidence deep learning model for the multimodal data set, obtaining classification confidence results and classification uncertainty results for the multiple sample objects based on the training data of the multiple sample objects, including using a double-layer evidence fusion strategy for each sample object to obtain the classification confidence results and classification uncertainty results for the sample object; The two-layer evidence fusion strategy includes: for each modality of the sample object, inputting the training data of the modality into the corresponding deep neural network DNN in the evidence deep learning model for learning, setting the output layer activation function of the deep neural network DNN to a linear activation function, and obtaining a corresponding initial deep learning evidence vector; using different credibility distribution methods to redistribute the initial deep learning evidence vector respectively, and obtaining a corresponding plurality of redistributed deep learning evidence vectors; mapping the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters, and obtaining credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and using the Dempster-Shafer evidence fusion rule to fusion the plurality of subjective logic S The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions; According to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, a loss function under the Dirichlet distribution is constructed, and the model parameters of the evidence deep learning model are adjusted to obtain an optimized double-layer evidence fusion evidence deep learning model.

3. The method according to claim 1 or 2, characterized in that: The multimodal dataset has N sample objects, each of which has C categories and V modalities. The true label of the nth sample object is The true label of the nth sample object represents the category of the nth sample object, and the representation tensor of the vth mode of the nth sample object is Among them, v∈{1,2,3,...V}, n∈{1,2,3,...N}.

4. The method according to claim 3, characterized in that For each modality, the training data of the modality is input into the corresponding deep neural network DNN in the evidence deep learning model for learning, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain the corresponding initial deep learning evidence vector, including obtaining the corresponding initial deep learning evidence vector using the following formula: in, represents the initial deep learning evidence vector under the vth modality of the nth sample object, represents the network parameters of the deep neural network DNN corresponding to the vth mode, f v represents the forward propagation function of the deep neural network DNN corresponding to the vth mode, R C Represents a set of real numbers with data dimension C.

5. The method according to claim 4, characterized in that The different credit assignment methods are to use R C to R +C The method of mapping data by any different mapping functions between R +C Represents a set of non-negative real numbers with data dimension C; The method of using different credibility distribution methods to redistribute the initial deep learning evidence vectors respectively to obtain a plurality of corresponding redistributed deep learning evidence vectors includes redistributing the initial deep learning evidence vector once using the following formula to obtain a first redistributed deep learning evidence vector among the plurality of redistributed deep learning evidence vectors: in, express Any element in represents the first redistributed deep learning evidence vector among multiple redistributed deep learning evidence vectors under the vth modality of the nth sample object, express The cth element in represents the total amount of initial deep learning evidence under the vth modality of the nth sample object, express The cth element in , softmax() represents the softmax function, e () Represents an exponential function with base e.

6. The method according to claim 4, characterized in that The different credit assignment methods are to use R C arrive A method for mapping data using any different mapping functions. Represents a set of non-negative real numbers with data dimension C; The method of using different credibility distribution methods to redistribute the initial deep learning evidence vectors respectively to obtain a plurality of corresponding redistributed deep learning evidence vectors includes redistributing the initial deep learning evidence vector once using the following formula to obtain a second redistributed deep learning evidence vector among the plurality of redistributed deep learning evidence vectors: in, represents the second redistributed deep learning evidence vector under the vth modality of the nth sample object, express The cth element in express In the cth element, ReLU() represents the ReLU function.

7. The method according to claim 4, characterized in that The mapping of the multiple redistributed deep learning evidence vectors into Dirichlet distribution parameters to obtain corresponding multiple subjective logic SL evidence vectors' reliability results and uncertainty results includes: The Dirichlet distribution parameters are obtained using the following formula: in, represents the kth redistributed deep learning evidence vector under the vth modality of the nth sample object, represents the Dirichlet distribution parameter corresponding to the k-th redistributed deep learning evidence vector; Construct the Dirichlet distribution as: Where B() represents the probability density function of Beta distribution. Represents the classification probability vector, pi represents Any element in , Dir() represents the probability density function of Dirichlet distribution, represents the Dirichlet distribution parameter of Dir(), T C represents a C-dimensional simplex; Then, according to the Dirichlet distribution and Dirichlet distribution parameters, the corresponding reliability results and uncertainty results of multiple subjective logic SL evidence vectors are obtained using the following formula: in, represents the intensity of the Dirichlet distribution corresponding to the k-th redistributed deep learning evidence vector, represents the kth subjective logic SL evidence vector under the vth modality of the nth sample object, express The reliability results of express The uncertainty result of k∈{1, 2, 3, ...K}, K represents the number of credibility allocation methods adopted.

8. The method according to claim 7, characterized in that The method of fusing the plurality of subjective logic SL evidence vectors by using the Dempster-Shafer evidence fusion rule includes fusing the plurality of subjective logic SL evidence vectors by using the following formula: in, Represents the reliability result of the first-level classification evidence vector under V modes of the nth sample object, Represents the uncertainty result of the first-level classification evidence vector under the V mode of the nth sample object; It represents the operation based on the Dempster-Shafer evidence fusion rule, and its operation rules are as follows: Among them, ⊙ represents the multiplication operation of corresponding elements of the vector, and k represents the normalization coefficient.

9. The method according to claim 8, characterized in that The Dempster-Shafer evidence fusion rule is used to fuse the reliability results and uncertainty results of the first-level classification evidence vectors under the multiple modes, including fusing using the following formula: in, Represents the confidence result of the second-level classification evidence vector under V modes of the nth sample object, Represents the uncertainty result of the second-level classification evidence vector under V modes of the n-th sample object.

10. The method according to claim 9, characterized in that The obtaining of the second-layer classification evidence vector according to the reliability result and the uncertainty result of the second-layer classification evidence vector comprises obtaining the second-layer classification evidence vector using the following formula: in, Represents the Dirichlet distribution intensity corresponding to the second-level classification evidence vector of the nth sample object, The second-level classification evidence vector representing the nth sample object; Mapping the second-layer classification evidence vector to a Dirichlet distribution parameter to obtain the classification confidence result and classification uncertainty result of the sample object includes obtaining the classification confidence result and classification uncertainty result of the sample object using the following formula: in, Represents the Dirichlet distribution parameter corresponding to the second-level classification evidence of the nth sample object, Indicates the classification confidence result of the nth sample object, u n Represents the classification uncertainty result of the nth sample object.

11. The method according to claim 1 or 2, characterized in that: The method constructs a loss function under the Dirichlet distribution according to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, adjusts the model parameters of the evidence deep learning model, and obtains the optimized evidence deep learning model of the double-layer evidence fusion, including constructing the loss function using the following formula, and obtaining the optimized evidence deep learning model of the double-layer evidence fusion using the loss function: Among them, MSE i () represents the mean square error loss function of the evidence deep learning model, represents the classification confidence result of the i-th sample object, p ij express Any element of is the true category label of the i-th sample, y ij express Any element of represents expectation, Var() represents variance; Represents the Dirichlet distribution parameter corresponding to the second-level classification evidence vector of the i-th sample object, α ij express Any element of S i express The corresponding Dirichlet distribution intensity, represents the Dirichlet distribution parameter of the i-th sample object after removing the subjective logic SL evidence of the true category, express Any element of KL[] represents the introduction of The KL divergence between the Dirichlet distribution under and the uniform Dirichlet distribution, Γ represents the Gamma function, and ψ represents the digamma function; Where L represents the optimization target of the evidence deep learning model, λ t represents the annealing coefficient, t represents the number of iterations, and BatchSize represents the batch size.

12. A classification evaluation method based on a double-layer evidence fusion evidence deep learning model, characterized in that: include: Acquire multimodal data of an object to be classified, wherein the object to be classified is an image to be identified; For each modality in the multimodal data, the data of the modality is input into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors, and the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors are fused using the Dempster-Shafer evidence fusion rule to obtain the credibility results and uncertainty results of the first-layer classification evidence vector under the modality; The Dempster-Shafer evidence fusion rule is used to fuse the reliability and uncertainty results of the first-level classification evidence vectors of multiple modalities to obtain the reliability and uncertainty results of the second-level classification evidence vectors. A second layer of classification evidence vector is obtained according to the reliability result and uncertainty result of the second layer of classification evidence vector, and the second layer of classification evidence vector is mapped to Dirichlet distribution parameters, and the classification reliability result and classification uncertainty result of the object to be classified are output; wherein the evidence deep learning model of the double-layer evidence fusion is trained using the method described in any one of claims 1 to 11.

13. A classification evaluation method based on a double-layer evidence fusion evidence deep learning model, characterized in that: include: Acquire multimodal data of an object to be classified, wherein the object to be classified is medical feature data to be identified, and the medical feature data to be identified includes: a feature tensor of mode 1 obtained by constructing a brain functional connection network based on a time correlation of each brain region calculated based on a blood oxygen level signal of the brain in a resting state, and a feature tensor of mode 2 obtained by constructing a fiber bundle connection network between brain regions by fiber bundle tracing based on the degree of water molecule diffusion; For each modality in the multimodal data, the data of the modality is input into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors, and the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors are fused using the Dempster-Shafer evidence fusion rule to obtain the credibility results and uncertainty results of the first-layer classification evidence vector under the modality; The Dempster-Shafer evidence fusion rule is used to fuse the reliability and uncertainty results of the first-level classification evidence vectors of multiple modalities to obtain the reliability and uncertainty results of the second-level classification evidence vectors. A second layer of classification evidence vector is obtained according to the reliability result and uncertainty result of the second layer of classification evidence vector, and the second layer of classification evidence vector is mapped to Dirichlet distribution parameters, and the classification reliability result and classification uncertainty result of the object to be classified are output; wherein the evidence deep learning model of the double-layer evidence fusion is trained using the method described in any one of claims 1 to 11.

14. A double-layer evidence fusion learning device for an evidence deep learning model, characterized in that: include: A first acquisition module acquires a multimodal data set, wherein the multimodal data set includes training data of multiple sample objects, wherein each sample object corresponds to multiple training data of multiple modalities, and the multimodal data set is a multimodal image set; A first double-layer evidence fusion module is configured to use an evidence deep learning model for the multimodal data set to obtain classification confidence results and classification uncertainty results of the multiple sample objects based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification confidence result and classification uncertainty result of the sample object; The double-layer evidence fusion strategy is specifically as follows: for each modality of the sample object, the training data of the modality is input into the corresponding deep neural network DNN in the evidence deep learning model for learning, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and the plurality of subjective logic S is fused using the Dempster-Shafer evidence fusion rule. The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions; The first training module is configured to construct a loss function under the Dirichlet distribution according to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, adjust the model parameters of the evidence deep learning model, and obtain an optimized double-layer evidence fusion evidence deep learning model.

15. A double-layer evidence fusion learning device for an evidence deep learning model, characterized in that: include: A third acquisition module acquires a multimodal data set, wherein the multimodal data set includes training data of multiple sample objects, wherein each sample object corresponds to multiple training data of multiple modalities, and the multimodal data set is a multimodal medical feature data set, and the multimodal medical feature data set includes: a feature tensor obtained by constructing a brain functional connection network based on a blood oxygen level signal of the brain in a resting state to calculate the time correlation of each brain region, and a feature tensor obtained by constructing a fiber bundle connection network between brain regions by performing fiber bundle tracking based on the degree of water molecule diffusion; A second double-layer evidence fusion module is configured to use the evidence deep learning model for the multimodal data set to obtain classification confidence results and classification uncertainty results of the multiple sample objects based on the training data of the multiple sample objects, including adopting a double-layer evidence fusion strategy for each sample object to obtain the classification confidence result and classification uncertainty result of the sample object; The double-layer evidence fusion strategy is specifically as follows: for each modality of the sample object, the training data of the modality is input into the corresponding deep neural network DNN in the evidence deep learning model for learning, and the output layer activation function of the deep neural network DNN is set to a linear activation function to obtain a corresponding initial deep learning evidence vector; different credibility distribution methods are used to redistribute the initial deep learning evidence vector respectively to obtain a corresponding plurality of redistributed deep learning evidence vectors; the plurality of redistributed deep learning evidence vectors are mapped to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors; and the plurality of subjective logic S is fused using the Dempster-Shafer evidence fusion rule. The reliability results and uncertainty results of the L evidence vectors are fused to obtain the reliability results and uncertainty results of the first-layer classification evidence vector under the modality; the reliability results and uncertainty results of the first-layer classification evidence vectors under the multiple modalities are fused using the Dempster-Shafer evidence fusion rule to obtain the reliability results and uncertainty results of the second-layer classification evidence vector; the second-layer classification evidence vector is obtained according to the reliability results and uncertainty results of the second-layer classification evidence vector, and the second-layer classification evidence vector is mapped to the Dirichlet distribution parameter to obtain the classification reliability results and classification uncertainty results of the sample object, and the different reliability allocation methods are methods of mapping data using any different mapping functions; The second training module is configured to construct a loss function under the Dirichlet distribution according to the classification reliability results of the multiple sample objects and the true category labels of the multiple sample objects, adjust the model parameters of the evidence deep learning model, and obtain an optimized double-layer evidence fusion evidence deep learning model.

16. A classification and evaluation device based on a double-layer evidence fusion evidence deep learning model, characterized in that: include: A second acquisition module is configured to acquire multimodal data of an object to be classified, wherein the object to be classified is an image to be identified; The first first-layer fusion module is configured to, for each modality in the multimodal data, input the data of the modality into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, set the output layer activation function of the deep neural network DNN to a linear activation function, and obtain a corresponding initial deep learning evidence vector; use different credibility distribution methods to redistribute the initial deep learning evidence vector to obtain a corresponding plurality of redistributed deep learning evidence vectors; map the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters to obtain credibility results and uncertainty results of a corresponding plurality of subjective logic SL evidence vectors, and use the Dempster-Shafer evidence fusion rule to fuse the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors to obtain credibility results and uncertainty results of the first-layer classification evidence vector under the modality; The second layer fusion module uses the Dempster-Shafer evidence fusion rule to fuse the reliability results and uncertainty results of the first layer classification evidence vectors of the multiple modalities to obtain the reliability results and uncertainty results of the second layer classification evidence vectors; The first classification evaluation module is configured to obtain a second-layer classification evidence vector based on the reliability results and uncertainty results of the second-layer classification evidence vector, and map the second-layer classification evidence vector to Dirichlet distribution parameters, and output the classification reliability results and classification uncertainty results of the object to be classified; wherein the evidence deep learning model of the double-layer evidence fusion is trained using the method described in any one of claims 1 to 11.

17. A classification evaluation device based on a double-layer evidence fusion evidence deep learning model, characterized in that: include: a fourth acquisition module, configured to acquire multimodal data of an object to be classified, wherein the object to be classified is medical feature data to be identified, and the medical feature data to be identified includes: a feature tensor of mode 1 obtained by constructing a brain functional connection network by calculating the temporal correlation of each brain region based on a blood oxygen level signal of the brain in a resting state, and a feature tensor of mode 2 obtained by constructing a fiber bundle connection network between brain regions by performing fiber bundle tracking based on the degree of water molecule diffusion; The third first-layer fusion module is configured to, for each modality in the multimodal data, input the data of the modality into the trained evidence deep learning model to learn the deep neural network DNN corresponding to the modality data, set the output layer activation function of the deep neural network DNN to a linear activation function, and obtain the corresponding initial deep learning evidence vector; use different credibility distribution methods to redistribute the initial deep learning evidence vector to obtain a corresponding plurality of redistributed deep learning evidence vectors; map the plurality of redistributed deep learning evidence vectors to Dirichlet distribution parameters to obtain credibility results and uncertainty results of the corresponding plurality of subjective logic SL evidence vectors, and use the Dempster-Shafer evidence fusion rule to fuse the credibility results and uncertainty results of the plurality of subjective logic SL evidence vectors to obtain the credibility results and uncertainty results of the first-layer classification evidence vector under the modality; The fourth second-layer fusion module uses the Dempster-Shafer evidence fusion rule to fuse the reliability results and uncertainty results of the first-layer classification evidence vectors of the multiple modalities to obtain the reliability results and uncertainty results of the second-layer classification evidence vectors; The second classification evaluation module is configured to obtain a second-layer classification evidence vector based on the reliability results and uncertainty results of the second-layer classification evidence vector, and map the second-layer classification evidence vector to Dirichlet distribution parameters, and output the classification reliability results and classification uncertainty results of the object to be classified; wherein the evidence deep learning model of the double-layer evidence fusion is trained using the method described in any one of claims 1 to 11.

18. An electronic device, characterized in that: The method comprises a memory and a processor; wherein the memory stores computer instructions, and wherein the computer instructions are executed by the processor to implement the method steps described in any one of claims 1 to 13.

19. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method steps of any one of claims 1 to 13 are implemented.

Images