Disease classification method based on memory association multi-information reliable fusion
By introducing memory associations and simplified Hopfield energy to evaluate modal credibility and using generative adversarial networks to achieve cyclic consistency fusion between modals, the existing multimodal fusion methods are solved, and the problem of unreliability and lack of interpretability in the fields of medical diagnosis and other fields is achieved, and higher disease classification accuracy and model interpretability are achieved.
Patent Information
- Application Number
- CN202411847560.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-03-18
AI Technical Summary
The existing multimodal fusion methods have problems of unreliability and lack of interpretability in the fields of medical diagnosis, and it is difficult to effectively utilize the correlation and complementarity between modals.
Using a multi-information reliable fusion method based on memory association, the memory module and reliable fusion module are constructed, and the credibility of the modals is evaluated using the simplified Hopperfield energy, and the cyclic consistency fusion between modals is achieved through a generative adversarial network.
It improves the reliability and interpretability of multimodal models, enhances the model's ability to utilize multimodal information, and significantly improves the accuracy of disease classification.
Smart Images
Figure BDA0005189651060000032 
Figure BDA0005189651060000041 
Figure BDA0005189651060000042
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of classification in computer multimodal fusion, and specifically relates to a disease classification method based on reliable fusion of memory-associated multi-information. Background Art
[0002] In recent years, with the increasing amount of multimodal data, multimodal learning has achieved remarkable success. In real-world scenarios, multimodal learning plays a key role in many fields. By leveraging the complementarity between different modalities, the model can capture more comprehensive information, thereby improving performance (for example, the combination of images and text can provide richer semantic information than a single modality). Although multimodal learning has achieved great success, most multimodal models are still unreliable and lack interpretability due to existing fusion strategies, which makes multimodal models face many challenges in actual deployment, especially in the field of medical diagnosis. Therefore, a reliable multimodal fusion model is crucial for practical applications.
[0003] In multimodal learning, previous multimodal fusion methods mainly focus on feature-level fusion and decision-level fusion. The core idea of feature-level fusion is to use the complementarity between data of different modalities, extract the features of each modality and fuse them in the feature space, so as to provide a richer and more comprehensive representation for subsequent classification, regression or other tasks. Although feature-level fusion has excellent performance, it is difficult to determine the role and importance of each modality during the fusion process. The decision-level fusion method does not directly process the features of each modality, but models and predicts each modality separately, and then integrates the decision results of each modality to make a final judgment. However, since each modality is fused after independent decision-making, the model will ignore the interaction between deep features and cannot fully utilize the correlation and complementarity between modalities. Although these methods are more effective than single-modal methods, they may cause the strong features of a modality to mask other modalities, thereby making the model overly dependent on certain modalities and ignoring the valuable information in other modalities. Therefore, it is necessary to design a novel modality credibility measurement method and feature fusion mechanism between modalities from the representation layer of each modality. Summary of the invention
[0004] In order to solve the above problems, the present invention provides a disease classification method based on reliable fusion of memory-related multi-information, the method comprising the steps of:
[0005] To build a complete disease classification network based on reliable fusion of multiple information based on memory association, it is necessary to construct a memory module and a reliable fusion module respectively to realize the classification network. The memory module is composed of three independent feature encoders, each of which is composed of two layers of linear networks. The potential features are extracted for each modal information, and the memory storage is performed to obtain the respective storage mode information. After obtaining the storage mode of each modal information, the modal information of the input data is evaluated for correlation with the corresponding storage mode, and the obtained correlation coefficient is used as the reliability measure of the modal information. In the reliable fusion module, the reliability modal representation is constructed through a generative adversarial network to construct the cyclic consistency between the modalities. Specifically, the generator is used to generate the fusion information of each modality, and the discriminator is used to judge the generated fusion information. Under the continuous optimization of the optimizer, the generator generates a joint representation information with multiple modal information, and this representation information is used as the final basis for disease classification.
[0006] In the above-mentioned classification process of multi-information fusion, the test data is used as the input information of the model. The potential representation information of the test pattern is obtained through the feature encoder corresponding to the modality. The inner product of the potential representation information and the memory pattern of the modality stored in the training phase is calculated as the credibility of the test pattern. Then, the maximum credibility is multiplied by the potential representation of each modality of the test pattern to obtain the credible modal information. The test pattern combined with the credibility is input into the feature fusion part according to different modalities. The joint representation of the fused multimodal information is obtained by the generators of each modality in the fusion part. This joint representation contains the representation of the fusion of any two modal information. Finally, the joint representations obtained from each modality are spliced to obtain a discriminative final representation, and the classifier classifies the final representation to obtain the final result.
[0007] The distribution of encoders and feature fusion networks and the overall training optimization framework are as follows:
[0008] After inputting the multimodal data features, the encoder first performs feature encoding. The encoder sets different settings for the three modes of the data. The input dimension of each encoder corresponds to the dimension of one modal data, corresponding to D1, D2, and D3 respectively. Then the three modal data X1, X2, and X3 are input into the encoder E. m :X m →Z m After that, we get the potential representations Z1, Z2, and Z3 of the same dimension. In order to verify the feature expression ability of the potential representation, each representation is connected to a linear classifier with the same dimension to obtain the classification result of each modality. The classifier is defined as where f m represents the classifier, m represents the number of modalities, n represents the number of samples, and y represents the true label corresponding to the sample. The probability distribution corresponding to each modal sample obtained by the classifier is expressed as follows:
[0009]
[0010] Among them, p represents the probability distribution, K represents the number of categories, and after obtaining the predicted category corresponding to each modal data, each modal network is trained and optimized respectively. The objective function is expressed as follows:
[0011]
[0012] Among them, L cls Represents the cross entropy loss function for classification;
[0013] After the encoder is optimized, the potential representations of each modality will be fused. In order to mine modal consistency, the fused modal information needs to be optimized by the final classifier, thereby completing the joint optimization process. The formula is as follows:
[0014]
[0015] Among them, Lconf represents the overall classification objective function.
[0016] In order to estimate the confidence of the mode, the Hopfield energy is used to measure the relationship between the storage mode and the potential representation of each mode. The simplified Hopfield energy comes from the modern Hopfield network, which is a classic feedback neural network. The definition formula of the energy function is as follows:
[0017]
[0018] Where LSE represents the log-sum-exp function, β and c represent constants, ξ m represents the mth modal information of the input mode, S represents the storage mode information, and e is equivalent to e j Represents input mode ξ m With the jth storage pattern s j The inner product of i Represents the number of samples in the i-th category, and the simplified Hopfield energy formula is as follows:
[0019]
[0020] Since all features of the same modality of samples of the same category share the same storage mode size N i , so the first term can be treated as a constant, and the simplified Hopfield energy formula is as follows:
[0021]
[0022] The simplified Hopfield energy can quantitatively represent the similarity between two modes. Therefore, the simplified Hopfield energy can be used as a way to measure the modal confidence. The modal confidence potential representation can be expressed as the following formula:
[0023]
[0024] Among them, W m represents the modal assurance criterion obtained by SHE(ξ).
[0025] Leveraging trusted latent representations of each modality Perform cycle consistency constraints. Specifically, establish generators G, F and discriminators D between each two modalities. X ,D Y , given that the two modal data are x and y, the objective function formula of the cycle consistency constraint is defined as follows:
[0026]
[0027] Among them, L cyc represents the cycle consistency objective function. At the same time, the modality generator will be guided by the discriminator. The final adversarial loss function formula is as follows:
[0028]
[0029] Among them, Lgan represents the loss function of the adversarial network, and the final objective function of the model is as follows:
[0030] Lall=Lcls+λ1Lgan+λ2Lcyc
[0031] Among them, λ1 and λ2 represent the adjustable parameters of the balance loss function.
[0032] After all networks are trained to convergence, the test data is input into the feature encoder to obtain the potential representation of the test data. The credibility of the current data is calculated using the storage mode S of each modality, and the credible modal representation is obtained from the credibility. Then it is input into the optimized generator G, F to obtain the information representation of the three fused modalities, and the three modal information is combined, and finally the combined representation is input into the classifier for classification, and the disease classification task is completed through different gene data information. The present invention provides a disease classification method based on reliable fusion of memory-related multi-information, which has the following advantages:
[0033] (1) The method introduces a simplified Hopfield energy and uses the storage mode of the modality at the representation layer to accurately evaluate the confidence of the input modality.
[0034] (2) The method does not simply concatenate the modal features during the fusion stage, but instead captures the relationship and complementarity between the modal features through the cycle consistency of the modalities and generates a comprehensive representation with multiple modal characteristics. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0036] Figure 1 It is a flow chart of a disease classification method based on reliable fusion of memory-related multi-information provided by the present invention;
[0037] Figure 2 It is a schematic diagram of a disease classification method based on reliable fusion of memory-related multi-information, (1) and (2) are respectively the overall structure diagram, the reliable multi-information structure diagram, and the multi-modal fusion module structure diagram of the present invention; DETAILED DESCRIPTION
[0038] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.
[0039] Exemplary Methods
[0040] like Figure 1 The present invention provides a flow chart of a disease classification method based on reliable fusion of memory-related multi-information, and the steps of the method are as follows:
[0041] Step S110: firstly, the data set is standardized.
[0042] Step S120: Input the processed data set into the feature encoder to obtain potential representations of different modalities, and optimize each encoder.
[0043] After inputting the multimodal data features, the encoder first performs feature encoding. The encoder sets different settings for the three modes of the data. The input dimension of each encoder corresponds to the dimension of one modal data, corresponding to D1, D2, and D3 respectively. Then the three modal data X1, X2, and X3 are input into the encoder E. m :X m →Z mAfter that, we get the potential representations Z1, Z2, and Z3 of the same dimension. In order to verify the feature expression ability of the potential representation, each representation is connected to a linear classifier with the same dimension to obtain the classification result of each modality. The classifier is defined as where f m represents the classifier, m represents the number of modalities, n represents the number of samples, and y represents the true label corresponding to the sample. The probability distribution corresponding to each modal sample obtained by the classifier is expressed as follows:
[0044]
[0045] Among them, p represents the probability distribution, K represents the number of categories, and after obtaining the predicted category corresponding to each modal data, each modal network is trained and optimized respectively. The objective function is expressed as follows:
[0046]
[0047] Among them, L cls Represents the cross entropy loss function for classification;
[0048] After the encoder is optimized, the potential representations of each modality will be fused. In order to mine modal consistency, the fused modal information needs to be optimized by the final classifier, thereby completing the joint optimization process. The formula is as follows:
[0049]
[0050] Among them, Lconf represents the total classification objective function;
[0051] Step S130: performing a credible estimation on the optimized modal representation.
[0052] In order to estimate the confidence of the mode, the Hopfield energy is used to measure the relationship between the storage mode and the potential representation of each mode. The simplified Hopfield energy comes from the modern Hopfield network, which is a classic feedback neural network. The definition formula of the energy function is as follows:
[0053]
[0054] Where LSE represents the log-sum-exp function, β and c represent constants, ξ m represents the mth modal information of the input mode, S represents the storage mode information, and e is equivalent to e j Represents input mode ξ m With the jth storage pattern s j The inner product of i Represents the number of samples in the i-th category, and the simplified Hopfield energy formula is as follows:
[0055]
[0056] Since all features of the same modality of samples of the same category share the same storage mode size N i , so the first term can be treated as a constant, and the simplified Hopfield energy formula is as follows:
[0057]
[0058] The simplified Hopfield energy can quantitatively represent the similarity between two modes. Therefore, the simplified Hopfield energy can be used as a way to measure the modal confidence. The modal confidence potential representation can be expressed as the following formula:
[0059]
[0060] Among them, W m represents the modal assurance factor obtained by SHE(ξ);
[0061] Step S140: Perform cyclic consistency constraints on the trusted modal information.
[0062] Leveraging trusted latent representations of each modality Perform cycle consistency constraints. Specifically, establish generators G, F and discriminators D between each two modalities. X ,D Y , given that the two modal data are x and y, the objective function formula of the cycle consistency constraint is defined as follows:
[0063]
[0064] Among them, L cyc represents the cycle consistency objective function. At the same time, the modality generator will be guided by the discriminator. The final adversarial loss function formula is as follows:
[0065]
[0066] Among them, Lgan represents the loss function of the adversarial network;
[0067] Step S150: Calculate the total loss function of the multi-information disease classification network, which can be expressed as:
[0068] Lall=Lcls+λ1Lgan+λ2Lcyc
[0069] Among them, λ1 and λ2 represent the adjustable parameters of the balance loss function;
[0070] Step S160: Use the test data to test the optimized training model parameters. Input the test data into the complete branch of the disease classification network based on the reliable fusion of memory-associated multi-information. The probability distribution output by the final classifier layer of the branch is used to complete the accurate evaluation of the test data.
[0071] Through this implementation, the datasets LGG and ROSMAP are used to perform forward propagation in the trusted representation and multimodal fusion modules, respectively, and back propagation is performed after calculating the total loss function. After the model is trained to convergence, the test data is input into the complete model branch, and the classification result of the disease classification method based on the reliable fusion of memory-related multi-information is judged through the probability distribution output by the last classifier layer of the branch.
[0072] To further illustrate, assuming that the multimodal disease classification dataset is classified according to this embodiment, a classification result with higher accuracy than most methods will be obtained.
[0073] Detailed description of the invention Results
[0074] This implementation uses the publicly available multimodal disease classification datasets LGG and ROSMAP. The details of the datasets are described as follows:
[0075] ROSMAP is used for the diagnosis of Alzheimer's disease and contains 351 samples in 2 categories, and LGG is used for the grading of gliomas and contains 510 samples in 2 categories.
[0076] In order to verify the superiority of this implementation (Ours), this implementation is compared with several existing multimodal disease classification methods, including GRidge, BPLSDA, BSPLSDA and other methods. The accuracy (ACC), F1 score and area under the AUC curve of these methods on the above public datasets will be compared. The specific data comparison is shown in Table 1.
[0077] Table 1 Accuracy of disease dataset (%)
[0078]
[0079] By comparing the data in the above table, we can clearly see that Ours achieves the best performance and significantly improves the accuracy of disease classification. The quantitative results fully demonstrate the superiority of Ours, because Ours can better overcome the problem of modal unreliability. Ours uses simplified Hopfield energy to estimate the credibility of the modality and uses adversarial networks to generate joint representations with multi-modal information, making the network more generalizable and able to mine modal consistency. A large number of experiments show that this method is superior to existing methods.
[0080] This implementation proposes a disease classification method based on reliable fusion of memory-related multi-information for disease classification tasks.
[0081] By training the feature encoder to obtain the storage mode of the modality, and estimating the credibility of the test data through the storage mode, the final disease classification task is completed by constraining the cyclic consistency between the modalities through the subsequent generation network. The credibility assessment uses a simplified Hopfield energy, which effectively solves the unreliability problem of modal information, enhances the interpretability of the model, and makes the learned features more discriminative through consistency fusion. The experimental results on the public datasets LGG and ROSMAP show that this implementation has higher accuracy and better superiority than other methods.
[0082] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.
Claims
1. A disease classification method based on reliable fusion of memory-related multi-information, the method comprising the steps of: To build a complete disease classification network based on memory association and reliable fusion of multiple information, it is necessary to build memory modules and reliable fusion modules to realize the classification network. The memory module is composed of three independent feature encoders, each of which is composed of a two-layer linear network. It extracts potential features from each modal information and obtains the respective storage mode information through memory storage. Then, the modal information of the input data is evaluated for correlation with the corresponding storage mode, and the obtained correlation coefficient is used as the reliability measure of the modal information. In the reliable fusion module, the reliability modal representation is constructed through the generative adversarial network to construct the cyclic consistency between the modalities. Specifically, the generator is used to generate the fusion information of each modality, and the discriminator is used to judge the generated fusion information. Under the continuous optimization of the optimizer, the generator generates the joint representation information with multiple modal information, and this representation information is used as the final basis for disease classification. In the above-mentioned classification process of multi-information fusion, the test data is used as the input information of the model, and the potential representation information of the test pattern is obtained through the feature encoder corresponding to the modality. The inner product of the potential representation information and the memory pattern of the modality stored in the training stage is calculated as the credibility of the test pattern, and then the maximum credibility is multiplied by the potential representation of each modality of the test pattern to obtain the credible modal information, and the test pattern combined with the credibility is input into the feature fusion part according to different modalities, and the joint representation of the fused multi-modal information is obtained by the generators of each modality in the fusion part. This joint representation contains the representation of the fusion of any two modal information. Finally, the joint representations obtained from each modality are spliced to obtain a discriminative final representation, and the final representation is classified by the classifier to obtain the final result; The distribution of encoders and feature fusion networks and the overall training optimization framework are as follows: After inputting the multimodal data features, the encoder first performs feature encoding. The encoder sets different settings for the three modes of the data. The input dimension of each encoder corresponds to the dimension of one modal data, corresponding to D1, D2, and D3 respectively. Then the three modal data X1, X2, and X3 are input into the encoder E. m :X m →Z m After that, we get the potential representations Z1, Z2, and Z3 of the same dimension. In order to verify the feature expression ability of the potential representation, each representation is connected to a linear classifier with the same dimension to obtain the classification result of each modality. The classifier is defined as where f m represents the classifier, m represents the number of modalities, n represents the number of samples, and y represents the true label corresponding to the sample. The probability distribution corresponding to each modal sample obtained by the classifier is expressed as follows: Among them, p represents the probability distribution, K represents the number of categories, and after obtaining the predicted category corresponding to each modal data, each modal network is trained and optimized respectively. The objective function is expressed as follows: Among them, L cls Represents the cross entropy loss function for classification; After the encoder is optimized, the potential representations of each modality will be fused. In order to mine modal consistency, the fused modal information needs to be optimized by the final classifier, thereby completing the joint optimization process. The formula is as follows: Among them, Lconf represents the overall classification objective function.
2. According to the disease classification method based on reliable fusion of memory association multi-information according to claim 1, the reliability calculation process is as follows: In order to estimate the confidence of the mode, the Hopfield energy is used to measure the relationship between the storage mode and the potential representation of each mode. The simplified Hopfield energy comes from the modern Hopfield network, which is a classic feedback neural network. The definition formula of the energy function is as follows: in, LSE represents the log-sum-exp function, β and c represent constants, ξ m represents the mth modal information of the input mode, S represents the storage mode information, and e is equivalent to e j Represents input mode ξ m With the jth storage pattern s j The inner product of i Represents the number of samples in the i-th category, and the simplified Hopfield energy formula is as follows: Since all features of the same modality of samples of the same category share the same storage mode size N i , so the first term can be treated as a constant, and the simplified Hopfield energy formula is as follows: The simplified Hopfield energy can quantitatively represent the similarity between two modes. Therefore, the simplified Hopfield energy can be used as a way to measure the modal confidence. The modal confidence potential representation can be expressed as the following formula: Among them, W m represents the modal assurance criterion obtained by SHE(ξ).
3. The disease classification method based on reliable fusion of memory-related multi-information according to claim 1 is characterized in that Generate a joint representation with multimodal characteristics and fuse it. The process is as follows: Leveraging trusted latent representations of each modality Perform cycle consistency constraints. Specifically, establish generators G, F and discriminators D between each two modalities. X ,D Y , given that the two modal data are x and y, the objective function formula of the cycle consistency constraint is defined as follows: Among them, L cyc represents the cycle consistency objective function, p(g) represents the distribution, and at the same time, the modality generator will be guided by the discriminator. The final adversarial loss function formula is as follows: Among them, Lgan represents the loss function of the adversarial network, and the final objective function of the model is as follows: Lall=Lcls+λ1Lgan+λ2Lcyc Among them, λ1 and λ2 represent the adjustable parameters of the balance loss function.
4. According to the disease classification method based on reliable fusion of memory-related multi-information according to claim 1, the complete multi-information disease classification process is as follows: After all networks are trained to convergence, the test data is input into the feature encoder to obtain the potential representation of the test data. The credibility of the current data is calculated using the storage mode S of each modality, and the credible modal representation is obtained from the credibility. Then it is input into the optimized generator G, F to obtain the information representation of the three fused modalities, and the three modal information is combined. Finally, the combined representation is input into the classifier for classification, and the disease classification task is completed through different genetic data information.