Multi-modal image fusion analysis method and system based on causal characterization

Abstract

The invention discloses a multi-modal image fusion analysis method and system based on causal characterization. The method comprises the steps that an initial feature map is extracted, and a causal decoupling module maps and decouples the initial feature map into a causal factor and a confusion factor; the learnable modal disturbance module introduces an adaptive disturbance mechanism on a confusion factor, keeps a causal factor unchanged, and introduces disturbances of different levels into the model, so that the model is effectively generalized to an unknown domain; and the multi-modal feature fusion module splices the causal factor features of different modals to obtain fused causal factor features, flattens the fused causal factor features, inputs the flattened fused causal factor features to a self-attention layer for feature interaction, classifies the flattened fused causal factor features through a classifier, and outputs the category of the image. According to the method, a causal relationship driving method is adopted, irrelevant confusion factors are eliminated through a causal decoupling module, key causal factors are reserved, interference is reduced, key features are strengthened, meanwhile, an adaptive modal disturbance module is introduced, and the adaptability and generalization ability of the model to different data distributions are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal image fusion analysis method, system, terminal and computer-readable storage medium based on causal representation. Background Art

[0002] With the rapid development of medical imaging technology, the role of medical images in clinical diagnosis has become increasingly prominent. However, due to the differences in imaging equipment and their principles, single-modality medical images are difficult to fully reflect the health status of patients and cannot provide doctors with accurate and comprehensive image information.

[0003] Multimodal medical images can provide more comprehensive information. However, existing technologies are insufficient in integrating multimodal medical images and fail to fully utilize the complementarity between multimodal images. In addition, deep learning technology has shown great potential in the field of medical image processing, but its performance is limited by the consistency of the distribution of training data and test data. In actual medical scenarios, due to differences in equipment parameters, scanning instruments or scanning methods, medical image data under the same imaging modality have differences in feature distribution, making it difficult to reuse deep learning models on new data sets. The interpretation and diagnosis of medical images is a labor-intensive task that requires manual annotation of target areas in medical images, which imposes a heavy burden on the daily work of clinicians.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal image fusion analysis method, system, terminal and computer-readable storage medium based on causal representation, aiming to solve the problem in the prior art that manual annotation of target areas in medical images is required, resulting in low accuracy and efficiency in medical image processing.

[0006] To achieve the above object, the present invention provides a multimodal image fusion analysis method based on causal representation, and the multimodal image fusion analysis method based on causal representation comprises the following steps:

[0007] An encoder equipped with convolutional blocks is used to extract an initial feature map from the multimodal endoscopic image, and a causal decoupling module decouples the initial feature map mapping into causal factors and confusion factors through a multi-layer perceptron;

[0008] The learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged. By introducing different levels of perturbations in the model, the model can be effectively generalized to unknown domains.

[0009] The multimodal feature fusion module concatenates the causal factor features of different modalities to obtain fused causal factor features, flattens the fused causal factor features on the channel and inputs them into the self-attention layer for feature interaction, and outputs the image category after classification by the classifier.

[0010] In addition, to achieve the above-mentioned purpose, the present invention also provides a multimodal image fusion analysis system based on causal representation, wherein the multimodal image fusion analysis system based on causal representation includes:

[0011] A causal decoupling module, for decoupling the initial feature map into causal factors and confusion factors through a multi-layer perceptron after extracting the initial feature map from the multi-modal endoscopic image using an encoder equipped with a convolutional block;

[0012] A learnable modal perturbation module is used to introduce an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged. By introducing different levels of perturbations in the model, the model can be effectively generalized to unknown domains.

[0013] The multimodal feature fusion module is used to splice the causal factor features of different modalities to obtain fused causal factor features, flatten the fused causal factor features on the channel and input them into the self-attention layer for feature interaction, and output the image category after classification by the classifier.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multimodal image fusion analysis program based on causal representation stored in the memory and executable on the processor, wherein the multimodal image fusion analysis program based on causal representation, when executed by the processor, implements the steps of the multimodal image fusion analysis method based on causal representation as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal image fusion analysis program based on causal representation, and when the multimodal image fusion analysis program based on causal representation is executed by a processor, the steps of the multimodal image fusion analysis method based on causal representation as described above are implemented.

[0016] In the present invention, an encoder equipped with a convolutional block is used to extract an initial feature map from a multimodal endoscopic image, and a causal decoupling module decouples the initial feature map mapping into causal factors and confusion factors through a multilayer perceptron; a learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged, and introduces different levels of perturbations in the model to enable the model to be effectively generalized to an unknown domain; a multimodal feature fusion module splices the causal factor features of different modalities to obtain a fused causal factor feature, flattens the fused causal factor feature on the channel and inputs it into the self-attention layer for feature interaction, and outputs the image category after classification by a classifier. The present invention accurately identifies and removes confusion factors that do not play a key role from complex multimodal data, while retaining those causal factors that have a decisive influence on the classification results, to improve the efficiency and accuracy of medical image interpretation and processing, further enhance the model's generalization ability to unknown fields, and applies adaptive perturbations on the confusion factor without changing the causal factor, so that the model can flexibly adapt to different data distributions, effectively alleviating the problem of decreased generalization performance due to domain differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a preferred embodiment of the multimodal image fusion analysis method based on causal representation of the present invention;

[0018] Figure 2 It is a schematic diagram of a multimodal image fusion learning framework based on causal representation in a preferred embodiment of the multimodal image fusion analysis method based on causal representation of the present invention;

[0019] Figure 3 It is a schematic diagram of the processing process of the causal decoupling module in a preferred embodiment of the multimodal image fusion analysis method based on causal representation of the present invention;

[0020] Figure 4 Schematic diagram of the process of the learnable modal disturbance module in a preferred embodiment of the multimodal image fusion analysis method based on causal representation of the present invention;

[0021] Figure 5 It is a schematic diagram of the processing process of the multimodal feature fusion module in a preferred embodiment of the multimodal image fusion analysis method based on causal representation of the present invention;

[0022] Figure 6 It is a schematic diagram of a Frobenius norm calculation process of a difference of a Gram matrix composed of feature maps in a preferred embodiment of the multimodal image fusion analysis method based on causal representation of the present invention;

[0023] Figure 7 It is a structural diagram of a preferred embodiment of a multimodal image fusion analysis system based on causal representation of the present invention;

[0024] Figure 8 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] Medical image classification is a key technology in the field of digital medicine. It involves the use of computer vision and machine learning algorithms to automatically identify and label different categories of pathological features in medical images. With the advancement of medical imaging technology and the surge in the amount of medical data. Traditional medical image classification methods rely on expert experience and manual feature extraction, but these methods have limitations in dealing with complex and changeable medical image data. In recent years, deep learning technology, especially convolutional neural networks (CNNs), has shown great potential in medical image classification. They can automatically learn rich feature representations from images, significantly improving the accuracy and efficiency of classification. Despite this, medical image classification still faces challenges, including the need for high-quality labeled data, limitations in model generalization capabilities, and the complexity of deployment in clinical environments.

[0027] Typical image classification networks, such as VGGNet, GoogLeNet, ResNet, and DenseNet, although they have their own characteristics in structure and details, they share some core basic design principles. For example, these networks all adopt a hierarchical architecture, gradually extracting high-level features of the image through multiple convolutional layers and pooling layers. They support end-to-end learning, directly mapping from the original image to the classification result without manual feature extraction. All of these networks make extensive use of convolutional layers to automatically learn image features, and use pooling layers to reduce the spatial size of features while extracting important information. At the end of the network, a fully connected layer is used for the final classification decision. In addition, they usually use ReLU as the activation function to increase nonlinear expression capabilities, and use regularization techniques such as Dropout and data enhancement to prevent overfitting.

[0028] In the field of multimodal medical image classification, existing technical solutions mainly focus on how to effectively fuse data from different imaging modalities. Common fusion methods include feature layer fusion, decision layer fusion, input layer fusion, hybrid fusion, model layer fusion, attention mechanism fusion, dynamic weight fusion and graph-based fusion. These strategies combine multimodal information at different stages (such as after feature extraction, before classification decision or in the middle layer of the model) to enable the model to understand and utilize data more comprehensively, thereby improving the accuracy and robustness of classification. The selection of a suitable fusion strategy depends on the specific task requirements, data characteristics and computing resources. For example, the proposed CADE model achieves the classification of pulmonary nodule diseases by fusing early and late professional knowledge features. The proposed three-branch attention-guided convolutional neural network improves the accuracy of disease classification in chest X-ray images by fusing global and local branches. The proposed clinical knowledge enhancement model uses the attention mechanism to adaptively fuse cross-domain features to improve the performance of cardiovascular disease diagnosis. The proposed multi-task learning model fuses multiple semantic attribute features of pulmonary nodule CT images to improve the performance of pulmonary nodule classification. The proposed multi-task learning framework improves the performance of benign and malignant risk prediction of pulmonary nodules by integrating multiple attribute label features. The proposed domain transfer convolutional neural network improves the classification performance of X-ray images by integrating local image dictionaries and sparse spatial pyramid features.

[0029] Domain generalization refers to learning a model from one or more different but related domains (i.e., different training datasets) that can generalize well on an unseen test domain. Existing domain generalization methods can be roughly divided into two categories: learning domain-invariant feature representations from multiple source domains and generating more diverse samples through data augmentation. In learning domain-invariant feature representations, domain adversarial training is widely used. Domain Adversarial Neural Networks (DANN) for domain adaptation have been proposed. In DANN, the discriminator is trained to distinguish domains, while the generator is trained to deceive the discriminator to learn domain-invariant feature representations. Adversarial training is used by gradually reducing the domain differences in the manifold space. In addition, some recent methods exploit causality to learn domain-invariant semantic representations for domain generalization. Learning causality and mitigating confounding factors usually requires causal intervention. Considering the cross-domain invariance of the causal relationship between semantic factors and predictions, a causal semantic generation model is proposed to eliminate the domain-specific correlation between semantic factors and variation factors, so that the prediction is only affected by the semantic factors. Some people assume that images of the same object across domains should have the same representation, and use the cross-domain invariance of the causal relationship between objects and feature representations to capture intra-class variations for domain generalization. Some people introduce causal reasoning to extract causal factors that are invariant across domains to learn invariant feature representations. Some people focus on using causal reasoning to discover the intrinsic reasons for domain shift by constructing counterfactual scenarios on the learned causal graph.

[0030] Among the multimodal features extracted from the neural network, there are some features that have no significant effect on the classification accuracy. They do not play a key role in the classification task and are called confusion factors. In contrast, other features have a decisive influence on the classification accuracy. They play a vital role in the classification process. Such features are called causal factors. At present, most multimodal-based classification models mainly focus on the optimization of feature fusion strategies, but they often ignore the confusion factors and causal factors in modal features. These models do not effectively eliminate those confusion factors, resulting in interference between these features in the network, affecting the accuracy of the classification and recognition effects. In order to improve the performance of the model, the present invention focuses on identifying and eliminating those confusion factors that do not contribute to the classification accuracy, while strengthening those causal factors that have a significant impact on the classification results.

[0031] Existing domain generalization models face the challenge of insufficient generalization ability when transferring knowledge from the source domain to the target domain. These models usually fail to fully utilize the source domain data to generalize to the wide and diverse target domain, resulting in unstable performance when facing unknown data distributions.

[0032] In the current field of artificial intelligence, there is relatively little research and application of multimodal domain generalization classification models, which limits the effectiveness and flexibility of machine learning systems in dealing with cross-domain tasks. In order to fill this research gap and improve the performance of models in diverse data environments, this paper proposes a new method to address the challenges of multimodal domain generalization.

[0033] In order to utilize the unique features and common features of different modalities in multimodality and reduce the burden of doctors' daily work, the present invention proposes a multimodal image fusion assisted diagnosis framework based on causal representation. Furthermore, in order to solve the domain offset distribution problem and enable the model (neural network) trained in the source domain to generalize to the target domain with different distribution, the present invention, based on the above framework, intends to expand the distribution of the source domain by perturbing the confusion factor, so that the network learns the causal invariant factors of the final decision among many changing factors, which can effectively generalize the model to the unknown domain (target domain).

[0034] The multimodal image fusion analysis method based on causal representation described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the multimodal image fusion analysis method based on causal representation includes the following steps:

[0035] Step S10: extracting an initial feature map from the multimodal endoscopic image using an encoder equipped with a convolutional block, and the causal decoupling module decouples the initial feature map mapping into causal factors and confusion factors through a multi-layer perceptron.

[0036] Specifically, an encoder equipped with convolutional blocks is used to extract an initial feature map F from a multimodal endoscopic image; in order to extract the causal factor C from the initial feature map F (i.e., feature map), remove the confusion factor U, explore the intrinsic causal mechanism between data and labels, and obtain better generalization ability. Figure 3 As shown in the figure, after the original image X is extracted into the feature map F by the feature extractor E, the entire feature map is decoupled into the causal factor C and the confusion factor U by the multi-layer perceptron MLP; during the training process, the multi-layer perceptron MLP is supervised to decouple the causal factor C and the confusion factor U through three loss functions, as shown in the following formula:

[0037] U,C=MLP[E(X)]; (1)

[0038]

[0039] Among them, X represents the original image, E represents the feature extractor, MLP represents the multi-layer perceptron, C and U represent the causal factor and confusion factor respectively, and N Crepresents the number of classifications, KL represents relative entropy loss, CLS represents classifier, CSE represents cross entropy loss, Y represents true label, i represents each modality in multimodal, G represents Gram operation, N represents Frobenius norm, and Represents three loss functions used to supervise the model to separate causal factors and confounding factors.

[0040] The decoupled causal factor C and confusion factor U are predicted to have their own distribution through the classifier CLS, so that the confusion factor U is as independent of the label as possible. Through the supervision of the cross entropy loss function, its distribution is close to the uniform distribution, constraining it to be independent of the label, so that it does not participate in the prediction of the label; on the contrary, the distribution of the causal factor C is constrained to align with the real distribution, making it a factor that determines the label. In order to make the decoupling more thorough, the distribution of the causal factor C and the confusion factor U is maximized by maximizing the Frobenius norm of the difference between the Gram matrix composed of the feature graphs of the causal factor C and the confusion factor U, that is, the distribution of the causal factor C and the confusion factor U is as far away as possible, so as to achieve the effect of decoupling.

[0041] Step S20: The learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged, and introduces different levels of perturbations into the model so that the model can be effectively generalized to unknown domains.

[0042] Specifically, the challenge of model domain generalization is that the target domain distribution is unknown. One of the most common ways to solve domain generalization is to merge all data from the source domain and learn the model by minimizing the training error of the pooled data. This is called the empirical risk minimization (ERM) principle, which is expressed by the following formula:

[0043]

[0044] Among them, E' represents the expectation, represents the loss function, P src represents the empirical distribution on the source domain (training set), θ represents the parameters that the model needs to learn, x and y represent the image data and the corresponding labels respectively, Indicates that the joint distribution of (x, y) is closest to the empirical distribution P src The parameter θ.

[0045] In this regard, the present invention proposes a new method to solve the domain generalization problem from the perspective of causal decoupling. Therefore, the key to the generalization of the model of the present invention to unknown domains lies in how to create modality-aware causal perturbations. In the causal decoupling module, the causal decoupling of the original modal features has been achieved, but in the case of only a single source domain, the distribution of the confusion features is relatively simple. In order to enhance the richness of model features, such as Figure 4As shown, an adaptive perturbation is added to the confusion factor U that is independent of the label, while not changing the causal factor C that determines the label, that is:

[0046] F′=(1+γξ)U+C,ξ~Z(0,1); (6)

[0047] Among them, F′ represents the feature map after perturbation, γ represents the learnable parameter, ξ obeys the standard normal distribution, and ξ is resampled every time the data flows through the network. Z(0,1) represents a normal distribution with 0 as the mean and 1 as the standard deviation.

[0048] Considering that the basic requirement of domain generalization is to enable the model to generalize well to unknown domains, and further, to make the feature distribution after perturbation as different as possible from the feature distribution before perturbation, a distance metric is needed to measure the difference between the source domain distribution and the virtual target domain distribution. The ideal goal of this metric is to create a virtual target distribution with a large domain difference from the source distribution and retain semantic distinction information. Therefore, the present invention proposes a matrix metric to improve the dissimilarity of the front and back feature maps, that is, by maximizing the Frobenius norm of the difference between the Gram matrix composed of the front and back feature maps, so that the feature map F before perturbation and the feature map F after perturbation are equal. ′ The distribution of is maximized away from, in order to maximize the proximity to the unknown domain.

[0049] The present invention expands the distribution of the source domain by perturbing the confusion factor, so that the network learns the causal invariant factors of the final decision among many changing factors, and can effectively generalize the model to the unknown domain (target domain), thereby improving the accuracy and efficiency of medical image processing.

[0050] Step S30, the multimodal feature fusion module splices the causal factor features of different modalities to obtain fused causal factor features, flattens the fused causal factor features on the channel and inputs them into the self-attention layer for feature interaction, and outputs the category of the image after classification by the classifier.

[0051] Specifically, Figure 5 As shown in the figure, the multimodal feature fusion module concatenates the causal factor features obtained by the causal decoupling module or the causal decoupling module plus the learnable modal perturbation module to obtain fused causal factor features; after flattening the fused causal factor features on the channel, they are sent to the self-attention layer, so that each feature interacts with the causal features in its own modality and the causal features of the other modality (that is, each feature can interact not only with the causal features in its own modality, but also with the causal features of the other modality), using all the information of the two modalities, and finally sent to the classifier for classification to output the category of the image.

[0052] Furthermore, the Gram matrix is ​​used to reflect the correlation between channels of the feature map, and one channel of the feature map is obtained by convolving a convolution kernel with an image (or a feature map of the previous layer), that is, one channel of the feature map corresponds to one convolution kernel. Usually, a convolution layer will have multiple convolution kernels, which are independent of each other. The meanings of features extracted by different convolution kernels are also different. For example, convolution kernel A extracts coarse and fine features, and convolution kernel B extracts color features. The Gram matrix actually reflects the correlation between features with different meanings. At the same time, the Gram matrix also reflects the feature distribution to a certain extent. Therefore, in order to improve the decoupling effect and increase the diversity of feature changes, for any mode i, the Frobenius norm of the difference between the Gram matrices composed of different feature maps is constrained, that is:

[0053]

[0054] Among them, F and F ′ They represent the feature graph before and after the disturbance, respectively. C′ and U′ represent the feature graph composed of causal factors and confusion factors before the disturbance, respectively. C″ and U″ represent the feature graph composed of causal factors and confusion factors after the disturbance, respectively. and It represents the loss function used by the ith mode to maximize the feature map. G represents the Gram operation. The above calculation process is as follows: Figure 6 shown.

[0055] For all modalities, the final loss is:

[0056]

[0057] Where M represents the total number of modes, represents the i-th modality classification cross entropy loss, and represents the hyperparameter used to balance the loss strength of the Gram matrix in the i-th mode, represents the final loss; the final loss Used to supervise and guide the model to generalize to unknown domains.

[0058] Furthermore, the method proposed in the present invention is not only applicable to two modes, but can also be extended to multiple modes. In addition, the scheme of the present invention has been verified to be feasible through theory and experiment, and has greatly improved the classification accuracy, which is 5% higher than the accuracy of other optimal methods.

[0059] The present invention proposes an innovative multimodal image recognition framework for multimodal medical endoscopic image recognition tasks. The framework consists of three core components: a causal decoupling module, a learnable modal perturbation module, and a multimodal feature fusion module. First, an encoder equipped with a convolutional block is used to extract the initial feature map from the multimodal endoscopic image. Then, the causal decoupling module decouples these initial feature maps into causal factors and confusion factors through a multi-layer perceptron, realizing the independent decomposition of different modal image features under causal relationships, so as to facilitate a deeper understanding and classification of images, and significantly improve the screening accuracy of specific recognition tasks. On this basis, in order to extend the model to unknown fields and improve its generalization ability, the learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor, while keeping the causal factor unchanged. By introducing different levels of perturbations in the model, the generalization ability of the model in unknown fields is effectively evaluated and optimized, thereby effectively alleviating the problem of decreased model generalization performance caused by differences in domain data distribution. Finally, the features of the two modalities are spliced ​​together through the multimodal feature fusion module and enter the attention layer for feature interaction. By introducing the strategies of causal decoupling, modal perturbation and feature fusion, the present invention not only improves the accuracy of multimodal endoscopic image recognition, but also enhances the generalization performance of the model when facing unknown fields, providing an efficient and reliable solution for the field of medical image recognition.

[0060] Beneficial effects:

[0061] (1) This invention adopts an innovative method with causality as the core. Through a carefully designed causal decoupling module, it can accurately identify and eliminate those confounding factors that do not play a key role from complex multimodal data, while retaining those causal factors that have a decisive influence on the classification results. This method not only reduces the interference caused by irrelevant information, but also significantly improves the accuracy and execution efficiency of the classification task by strengthening key features.

[0062] (2) The present invention introduces a learnable modal perturbation module and a multimodal feature fusion module, which further enhances the model's generalization ability for unknown domains. Adaptive perturbations are applied to the confusion factor without changing the causal factor, so that the model can flexibly adapt to different data distributions, effectively alleviating the problem of decreased generalization performance due to inter-domain differences. This comprehensive strategy not only improves the accuracy of laryngeal cancer image recognition, but also provides an efficient and reliable solution for the field of medical image recognition, demonstrating the important role of causal analysis in improving model performance.

[0063] Furthermore, the method proposed in the present invention can adapt to the fusion and classification of multimodal data. It can be flexibly applied to multiple modalities to achieve cross-modal feature extraction and integration, and can be combined with technologies such as domain adaptation to cope with more complex application scenarios. Through this scalability, the present invention can improve the generalization ability and adaptability of the model under different data distributions and different task requirements.

[0064] Furthermore, if Figure 7 As shown, based on the above-mentioned multimodal image fusion analysis method based on causal representation, the present invention also provides a multimodal image fusion analysis system based on causal representation, wherein the multimodal image fusion analysis system based on causal representation includes:

[0065] A causal decoupling module 51, configured to decouple the initial feature map into causal factors and confusion factors through a multi-layer perceptron after extracting the initial feature map from the multi-modal endoscopic image using an encoder equipped with a convolutional block;

[0066] A learnable modal perturbation module 52, which is used to introduce an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged, so that the model can be effectively generalized to unknown domains by introducing different levels of perturbations in the model;

[0067] The multimodal feature fusion module 53 is used to splice the causal factor features of different modalities to obtain fused causal factor features, flatten the fused causal factor features on the channel and input them into the self-attention layer for feature interaction, and output the category of the image after classification by the classifier.

[0068] Furthermore, if Figure 8 As shown, based on the above-mentioned multimodal image fusion analysis method and system based on causal representation, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 8 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0069] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a multimodal image fusion analysis program 40 based on causal representation is stored on the memory 20, and the multimodal image fusion analysis program 40 based on causal representation can be executed by the processor 10, thereby realizing the multimodal image fusion analysis method based on causal representation in the present application.

[0070] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multimodal image fusion analysis method based on causal representation.

[0071] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0072] In one embodiment, when the processor 10 executes the multimodal image fusion analysis program 40 based on causal representation in the memory 20 , the steps of the multimodal image fusion analysis method based on causal representation as described above are implemented.

[0073] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal image fusion analysis program based on causal representation, and when the multimodal image fusion analysis program based on causal representation is executed by a processor, the steps of the multimodal image fusion analysis method based on causal representation as described above are implemented.

[0074] In summary, the present invention provides a multimodal image fusion analysis method, system, terminal and storage medium based on causal representation, the method comprising: using an encoder equipped with a convolution block to extract an initial feature map from a multimodal endoscopic image, a causal decoupling module decouples the initial feature map mapping into causal factors and confusion factors through a multilayer perceptron; a learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged, and introduces different levels of perturbations into the model so that the model can be effectively generalized to an unknown domain; a multimodal feature fusion module splices the causal factor features of different modalities to obtain a fused causal factor feature, flattens the fused causal factor feature on the channel and inputs it into a self-attention layer for feature interaction, and outputs the category of the image after classification by a classifier. The present invention accurately identifies and eliminates confounding factors that do not play a key role from complex multimodal data, while retaining causal factors that have a decisive influence on the classification results, thereby improving the efficiency and accuracy of medical image interpretation and processing, further enhancing the generalization ability of the model to unknown fields, and applying adaptive perturbations to the confounding factors without changing the causal factors, so that the model can flexibly adapt to different data distributions, effectively alleviating the problem of decreased generalization performance due to inter-domain differences.

[0075] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0076] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0077] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal image fusion analysis method based on causal representation, characterized in that: The multimodal image fusion analysis method based on causal representation includes: An encoder equipped with convolutional blocks is used to extract an initial feature map from the multimodal endoscopic image, and a causal decoupling module decouples the initial feature map mapping into causal factors and confusion factors through a multi-layer perceptron; The learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged. By introducing different levels of perturbations in the model, the model can be effectively generalized to unknown domains. The multimodal feature fusion module concatenates the causal factor features of different modalities to obtain fused causal factor features, flattens the fused causal factor features on the channel and inputs them into the self-attention layer for feature interaction, and outputs the image category after classification by the classifier.

2. The multimodal image fusion analysis method based on causal representation according to claim 1 is characterized in that: The encoder equipped with convolutional blocks is used to extract an initial feature map from the multimodal endoscopic image, and the causal decoupling module decouples the initial feature map mapping into causal factors and confusion factors through a multi-layer perceptron, specifically including: After the original image X is extracted into the feature map F by the feature extractor E, the whole feature map is decoupled into the causal factor C and the confusion factor U through the multi-layer perceptron MLP; During the training process, the multi-layer perceptron MLP is supervised to decouple the causal factor C and the confusion factor U through three loss functions: U,C=MLP[E(X)]; (1) Among them, X represents the original image, E represents the feature extractor, MLP represents the multi-layer perceptron, C and U represent the causal factor and confusion factor respectively, and N C represents the number of classifications, KL represents relative entropy loss, CLS represents classifier, CSE represents cross entropy loss, Y represents true label, i represents each modality in multimodal, G represents Gram operation, N represents Frobenius norm, and Represents three loss functions used to supervise the model to separate causal factors and confounding factors; The decoupled causal factor C and confusion factor U are each predicted to have a distribution through the classifier CLS. The distribution of the constraint confusion factor U is close to a uniform distribution and does not participate in the prediction of the label. The distribution of the constraint causal factor C is aligned with the true distribution and becomes the factor that determines the label.

3. The multimodal image fusion analysis method based on causal representation according to claim 1 or 2, characterized in that: The encoder equipped with convolutional blocks is used to extract an initial feature map from the multimodal endoscopic image, and the causal decoupling module decouples the initial feature map mapping into causal factors and confusion factors through a multi-layer perceptron, and then further includes: By maximizing the Frobenius norm of the difference between the Gram matrices of the feature maps of the causal factor C and the confusion factor U, the distribution of the causal factor C and the confusion factor U is maximized.

4. The multimodal image fusion analysis method based on causal representation according to claim 1 is characterized in that: The learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged. By introducing different levels of perturbations in the model, the model can be effectively generalized to unknown domains, including: Add an adaptive perturbation to the confusion factor U that is independent of the label, while not changing the causal factor C that determines the label: F′=(1+γξ)U+C,ξ~Z(0,1); (6) Among them, F′ represents the feature map after perturbation, γ represents the learnable parameter, ξ obeys the standard normal distribution, and Z(0,1) represents a normal distribution with 0 as the mean and 1 as the standard deviation; By maximizing the Frobenius norm of the difference between the Gram matrix formed by the previous and next feature maps, the distribution of the feature map F before the perturbation and the feature map F′ after the perturbation are maximized to maximize the proximity to the unknown domain.

5. The multimodal image fusion analysis method based on causal representation according to claim 1 is characterized in that: The multimodal feature fusion module splices the causal factor features of different modalities to obtain fused causal factor features, flattens the fused causal factor features on the channel and inputs them into the self-attention layer for feature interaction, and outputs the category of the image after classification by the classifier, specifically including: The multimodal feature fusion module combines the causal factor features obtained by the causal decoupling module or the causal decoupling module plus the learnable modal perturbation module to obtain the fused causal factor features; After flattening the fused causal factor features on the channel, they are sent to the self-attention layer, so that each feature interacts with the causal features within its own modality and the causal features of the other modality, using all the information of the two modalities, and finally sent to the classifier for classification, outputting the category of the image.

6. The multimodal image fusion analysis method based on causal representation according to claim 2 is characterized in that: The Gram matrix is ​​used to reflect the correlation between feature map channels; For any mode i, the Frobenius norm of the difference of the Gram matrices formed by constraining their respective feature maps is: Among them, F and F′ represent the feature map before and after the disturbance, respectively, C′ and U′ represent the feature map composed of causal factors and confusion factors before the disturbance, respectively, and C″ and U″ represent the feature map composed of causal factors and confusion factors after the disturbance, respectively. and represents the loss function used by the i-th modality to maximize its respective feature map.

7. The multimodal image fusion analysis method based on causal representation according to claim 6 is characterized in that: For all modalities, the final loss is: Where M represents the total number of modes, represents the cross entropy loss of the i-th modality classification, and represents the weight used to balance the loss strength of the Gram matrix in the i-th mode, Indicates final loss; The final loss Used to supervise and guide the model to generalize to unknown domains.

8. A multimodal image fusion analysis system based on causal representation, characterized in that: The multimodal image fusion analysis system based on causal representation includes: A causal decoupling module, for decoupling the initial feature map into causal factors and confusion factors through a multi-layer perceptron after extracting the initial feature map from the multi-modal endoscopic image using an encoder equipped with a convolutional block; A learnable modal perturbation module is used to introduce an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged. By introducing different levels of perturbations in the model, the model can be effectively generalized to unknown domains. The multimodal feature fusion module is used to splice the causal factor features of different modalities to obtain fused causal factor features, flatten the fused causal factor features on the channel and input them into the self-attention layer for feature interaction, and output the image category after classification by the classifier.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a multimodal image fusion analysis program based on causal representation stored in the memory and executable on the processor. When the multimodal image fusion analysis program based on causal representation is executed by the processor, the steps of the multimodal image fusion analysis method based on causal representation are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal image fusion analysis program based on causal representation, and when the multimodal image fusion analysis program based on causal representation is executed by a processor, the steps of the multimodal image fusion analysis method based on causal representation as described in any one of claims 1-7 are implemented.

Images