A multi-modal image fusion analysis method and system based on causal representation

Abstract

The application discloses a kind of multi-modal image fusion analysis method and system based on causal representation, the method includes: extracting initial feature map, initial feature map is mapped decoupling into causal factor and confusion factor by causal decoupling module;Learnable modal disturbance module introduces adaptive disturbance mechanism on confusion factor, keep causal factor unchanged, by introducing different levels of disturbance in model, so that model effectively generalizes to unknown domain;Multi-modal feature fusion module is spliced into fusion causal factor feature by the causal factor feature of different modalities, is flattened after input to self-attention layer and carries out feature interaction, the category of image is output after classification by classifier.This application uses the method driven by causal relationship, removes irrelevant confusion factor by causal decoupling module, retains key causal factor, to reduce interference and strengthen key features, while introducing adaptive modal disturbance module, enhance the adaptability and generalization ability of model to different data distribution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-modal image fusion analysis method and system based on causal representation, a terminal and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of medical imaging technology, medical images play an increasingly important role in clinical diagnosis. However, due to the differences in imaging equipment and its principles, single modality medical images are difficult to fully reflect the health status of patients, and cannot provide accurate and comprehensive image information for doctors.

[0003] Multi-modal medical images can provide more comprehensive information. However, the existing technology has deficiencies in integrating multi-modal medical images, and fails to fully utilize the complementarity between multi-modal images. In addition, deep learning technology has shown great potential in the field of medical image processing, but its performance is limited by the consistency of the distribution of training data and test data. In actual medical scenarios, due to differences in device parameters, scanning instruments or scanning methods, medical image data under the same imaging modality has differences in feature distribution, making it difficult for deep learning models to be reused on new data sets. The interpretation and diagnosis of medical images is a labor-intensive task that requires manual annotation of the target region of the medical image, which has brought a heavy burden to the daily work of clinical doctors.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a multi-modal image fusion analysis method and system based on causal representation, a terminal and a computer readable storage medium, which aims to solve the problem that manual annotation of the target region of the medical image is required in the prior art, resulting in low accuracy and efficiency of medical image processing.

[0006] To achieve the above purpose, the present application provides a multi-modal image fusion analysis method based on causal representation, which comprises the following steps:

[0007] An encoder equipped with a convolution block is used to extract an initial feature map from a multi-modal endoscopic image, and a causal decoupling module maps the initial feature map to causal factors and confusion factors through a multi-layer perception;

[0008] A learnable modality perturbation module introduces an adaptive perturbation mechanism on the confusion factors while keeping the causal factors unchanged, and introduces different levels of perturbations in the model to enable the model to effectively generalize to unknown domains;

[0009] The multi-modal feature fusion module splices the causal factor features of different modes to obtain fused causal factor features, inputs the fused causal factor features after being flattened in channels to a self-attention layer for feature interaction, and outputs the category of the image through a classifier.

[0010] In addition, to achieve the above object, the present application also provides a multi-modal image fusion analysis system based on causal representation, wherein the multi-modal image fusion analysis system based on causal representation comprises:

[0011] A causal decoupling module is configured to map the initial feature map to causal factors and confusion factors through a multi-layer perception after the initial feature map is extracted from the multi-modal endoscopic image using an encoder equipped with a convolution block;

[0012] A learnable modal perturbation module is configured to introduce an adaptive perturbation mechanism on the confusion factors while keeping the causal factors unchanged, and to effectively generalize the model to unknown domains by introducing different levels of perturbations in the model.

[0013] A multi-modal feature fusion module is configured to splice the causal factor features of different modes to obtain fused causal factor features, to input the fused causal factor features after being flattened in channels to a self-attention layer for feature interaction, and to output the category of the image through a classifier.

[0014] In addition, to achieve the above object, the present application also provides a terminal, wherein the terminal comprises a memory, a processor, and a multi-modal image fusion analysis program based on causal representation stored in the memory and executable on the processor, and the multi-modal image fusion analysis program based on causal representation implements the steps of the multi-modal image fusion analysis method based on causal representation when executed by the processor.

[0015] In addition, to achieve the above object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a multi-modal image fusion analysis program based on causal representation, and the multi-modal image fusion analysis program based on causal representation implements the steps of the multi-modal image fusion analysis method based on causal representation when executed by a processor.

[0016] In the present application, an encoder equipped with a convolution block is used to extract an initial feature map from a multi-modal endoscopic image, and a causal decoupling module maps the initial feature map to a causal factor and a confusion factor through a multi-layer perception; a learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged, and introduces different levels of perturbation in the model to enable the model to effectively generalize to unknown domains; a multi-modal feature fusion module splices the causal factor features of different modalities to obtain fused causal factor features, which are flattened in the channel and input to a self-attention layer for feature interaction, and the image category is output after classification by a classifier. The present application accurately identifies and removes the confusion factors that do not play a key role from complex multi-modal data, while retaining the causal factors that have a decisive influence on the classification result, to improve the efficiency and accuracy of medical image interpretation and processing, further enhance the generalization ability of the model to unknown domains, and apply adaptive perturbation on the confusion factor without changing the causal factor, so that the model can flexibly adapt to different data distributions and effectively alleviate the problem of decreased generalization performance caused by domain differences. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of a preferred embodiment of the multi-modal image fusion analysis method based on causal representation of the present application;

[0018] Figure 2 is a schematic diagram of a multi-modal image fusion learning framework based on causal representation in a preferred embodiment of the multi-modal image fusion analysis method based on causal representation of the present application;

[0019] Figure 3 is a schematic diagram of the processing process of the causal decoupling module in a preferred embodiment of the multi-modal image fusion analysis method based on causal representation of the present application;

[0020] Figure 4 is a schematic diagram of the processing process of the learnable modal perturbation module in a preferred embodiment of the multi-modal image fusion analysis method based on causal representation of the present application;

[0021] Figure 5 is a schematic diagram of the processing process of the multi-modal feature fusion module in a preferred embodiment of the multi-modal image fusion analysis method based on causal representation of the present application;

[0022] Figure 6 is a schematic diagram of the calculation process of the Frobenius norm of the difference of the Gram matrix composed of feature maps in a preferred embodiment of the multi-modal image fusion analysis method based on causal representation of the present application;

[0023] Figure 7 is a structural diagram of a preferred embodiment of the multi-modal image fusion analysis system based on causal representation of the present application;

[0024] Figure 8 Structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0026] Medical image classification is a key technology in the field of digital medicine, which involves the use of computer vision and machine learning algorithms to automatically identify and label different pathological features in medical images. With the advancement of medical imaging technology and the explosive growth of medical data, traditional medical image classification methods rely on expert experience and manual feature extraction, but these methods have limitations in dealing with complex and variable medical image data. In recent years, deep learning technology, especially convolutional neural networks (CNN), has shown great potential in medical image classification, as they can automatically learn rich feature representations from images, significantly improving classification accuracy and efficiency. Despite this, medical image classification still faces challenges, including the need for high-quality labeled data, limitations on model generalization, and the complexity of deployment in clinical environments.

[0027] Typical image classification networks, such as VGGNet, GoogLeNet, ResNet, and DenseNet, although each has its own characteristics in structure and details, share some core basic design principles. For example, these networks all adopt a hierarchical architecture, gradually extracting high-level features of images through multiple convolutional and pooling layers. They support end-to-end learning, directly mapping from raw images to classification results without the need for manual feature extraction. All these networks use a large number of convolutional layers to automatically learn image features and use pooling layers to reduce the spatial size of features while extracting important information. At the end of the network, fully connected layers are used for final classification decisions. In addition, they usually use ReLU as the activation function to increase the non-linear expression ability, and use regularization techniques such as Dropout and data augmentation to prevent overfitting.

[0028] In the field of multi-modal medical image classification, existing technical solutions mainly focus on how to effectively fuse data from different imaging modalities. Common fusion methods include feature layer fusion, decision layer fusion, input layer fusion, hybrid fusion, model layer fusion, attention mechanism fusion, dynamic weight fusion, and graph-based fusion. These strategies combine multi-modal information at different stages (such as after feature extraction, before classification decision, or in the middle layers of the model) to enable the model to more comprehensively understand and utilize data, thereby improving the accuracy and robustness of classification. Selecting the appropriate fusion strategy depends on specific task requirements, data characteristics, and computational resources. For example, the CADE model proposed by someone combines early and late professional knowledge features to achieve lung nodule disease classification. The three-branch attention-guided convolutional neural network proposed by someone improves the accuracy of disease classification in chest X-ray images through the fusion of global and local branches. The clinical knowledge enhancement model proposed by someone uses attention mechanisms to adaptively fuse cross-domain features, improving the performance of cardiovascular disease diagnosis. The multi-task learning model proposed by someone combines multiple semantic attribute features of lung nodule CT images to improve lung nodule classification performance. The multi-task learning framework proposed by someone improves the performance of lung nodule benign and malignant risk prediction by fusing multiple attribute label features. The domain transfer convolutional neural network proposed by someone improves the classification performance of X-ray images by fusing local image dictionaries and sparse spatial pyramid features.

[0029] Domain generalization refers to learning a model from one or more different but related domains (i.e., different training datasets) that can generalize well on unseen test domains. Existing domain generalization methods can be broadly categorized into two categories: learning domain-invariant feature representations from multiple source domains and generating more diverse samples through data augmentation. In learning domain-invariant feature representations, domain adversarial training is widely used. Domain Adversarial Neural Networks (DANN) for domain adaptation has been proposed. In DANN, discriminators are trained to distinguish between domains, while generators are trained to deceive discriminators to learn domain-invariant feature representations. Some people use adversarial training by gradually reducing the domain difference in manifold space. In addition, some recent methods utilize causal relationships to learn domain-invariant semantic representations for domain generalization. Learning causal relationships and mitigating confounding factors often require causal interventions. Considering the cross-domain invariance of the causal relationship between semantic factors and predictions, some people propose a causal semantic generation model to eliminate the specific domain correlation between semantic factors and variant factors, so that the prediction is only affected by the semantic factors. Some people assume that the images of the same object across domains should have the same representation, and use the cross-domain invariance of the causal relationship between objects and feature representations to capture intra-class variations for domain generalization. Some people introduce causal reasoning to extract cross-domain invariant causal factors to learn invariant feature representations. Some people focus on using causal reasoning to discover the intrinsic reasons for domain shift by constructing counterfactual scenarios on learned causal graphs.

[0030] In the multi-modal features extracted in neural networks, there are some features that have no significant influence on classification accuracy, and they do not play a key role in the classification task, which are called confounding factors. On the contrary, some other features have a decisive influence on classification accuracy, and they play a crucial role in the classification process, which are called causal factors. At present, most multi-modal classification models focus on optimizing feature fusion strategies, but they often ignore confounding factors and causal factors in modal features. These models do not effectively eliminate confounding factors, resulting in interference of these features in the network, affecting the accuracy of classification and recognition effect. In order to improve the performance of the model, the present invention focuses on identifying and eliminating confounding factors that do not contribute to classification accuracy, while strengthening causal factors that have a significant impact on classification results.

[0031] Existing domain generalization models face the challenge of insufficient generalization ability when transferring knowledge from source domains to target domains. These models often fail to fully utilize source domain data to generalize to a wide and diverse range of target domains, resulting in unstable performance when facing unknown data distributions.

[0032] In the current field of artificial intelligence, the research and application of multi-modal domain generalization classification models are relatively less, which limits the efficiency and flexibility of machine learning systems in processing cross-domain tasks. In order to fill this research gap and improve the performance of the model in a diversified data environment, the present application proposes a new method to deal with the challenge of multi-modal domain generalization.

[0033] In order to utilize the unique features and common features of different modalities in multi-modal and reduce the burden of doctors in their daily work, the present application proposes a multi-modal image fusion assisted diagnosis framework based on causal representation. Further, in order to solve the problem of domain shift distribution, so that the model (neural network) trained in the source domain can be generalized to the target domain with different distributions, the present application expands the distribution of the source domain by perturbing the confusion factor based on the above framework, so that the network learns the causal invariant factor of the final decision among many change factors, which can make the model effectively generalize to the unknown domain (target domain).

[0034] The multi-modal image fusion analysis method based on causal representation according to the preferred embodiment of the present application, as shown in Figure 1 and Figure 2 The multi-modal image fusion analysis method based on causal representation includes the following steps:

[0035] Step S10, using an encoder equipped with a convolution block to extract an initial feature map from a multi-modal endoscopic image, and a causal decoupling module maps the initial feature map to a causal factor and a confusion factor through a multi-layer perceptron.

[0036] Specifically, an initial feature map F is extracted from a multi-modal endoscopic image using an encoder equipped with a convolution block; in order to extract a causal factor C from the initial feature map F (i.e. a feature map), remove the confusion factor U, mine the internal causal mechanism between data and labels, and obtain better generalization ability. As shown in Figure 3 After the original image X is extracted to the feature map F by the feature extractor E, the whole feature map is decoupled into the causal factor C and the confusion factor U through the multi-layer perceptron MLP; during the training process, the multi-layer perceptron MLP decoupling the causal factor C and the confusion factor U is supervised through three loss functions, as shown in the following formula:

[0037] U,C=MLP[E(X)]; (1)

[0038]

[0039] Wherein, X represents the original image, E represents the feature extractor, MLP represents the multi-layer perceptron, C and U represent the causal factor and the confusion factor respectively, N Cdenotes the number of classes, KL denotes the relative entropy loss, CLS denotes the classifier, CSE denotes the cross-entropy loss, Y denotes the true label, i denotes each modality in the multi-modal, G denotes the Gram operation, N denotes the Frobenius norm, and denote three loss functions for the supervised model to separate the causal factor and the confounding factor.

[0040] The causal factor C and the confounding factor U obtained by decoupling are respectively subjected to a predicted distribution by the classifier CLS, the confounding factor U is made as irrelevant to the label as possible, and the distribution thereof is made close to the uniform distribution by the supervision of the cross-entropy loss function, thereby constraining the independence of the label and preventing the confounding factor U from participating in the prediction of the label; on the contrary, the distribution of the causal factor C is constrained to align with the real distribution, so that the causal factor C becomes a factor for determining the label. In order to make the decoupling more thorough, the distributions of the causal factor C and the confounding factor U are maximized to be far away from each other by maximizing the Frobenius norm of the difference of the Gram matrix constituted by the feature maps of the causal factor C and the confounding factor U, so that the distributions of the causal factor C and the confounding factor U are made as far away from each other as possible, thereby achieving the effect of decoupling.

[0041] In step S20, the learnable modality perturbation module introduces an adaptive perturbation mechanism on the confounding factor while keeping the causal factor unchanged, and introduces different levels of perturbation in the model to enable the model to effectively generalize to the unknown domain.

[0042] Specifically, the challenge faced by the model domain generalization is that the target domain distribution is unknown, and the most common method to solve the domain generalization is to combine all data from the source domain and learn the model by minimizing the training error of the pooled data, which is called the empirical risk minimization (ERM) principle, which is expressed by the following formula, that is:

[0043]

[0044] wherein E' represents expectation, denotes the loss function, P src denotes the empirical distribution on the source domain (training set), θ represents the parameters of the model to be learned, x and y respectively represent the image data and the corresponding label, denotes the parameters θ that make the joint distribution of (x, y) closest to the empirical distribution P src .

[0045] The present application proposes a new method in this regard, which considers the problem of domain generalization from the perspective of causal decoupling. The key to enabling the model of the present application to generalize to the unknown domain lies in how to create a modality-aware causal perturbation. In the causal decoupling module, the causal decoupling of the original modality features has been achieved, but in the case of only a single source domain, the distribution of the confounding features is relatively single. In order to enhance the richness of the model features, such as Figure 4As shown, an adaptive perturbation is added on the confusion factor U independent of the label, while the causal factor C determining the label is not changed, that is:

[0046] F'=(1+γξ)U+C, ξ~Z(0,1) (6)

[0047] Wherein, F' represents the perturbed feature map, γ represents a learnable parameter, ξ is subject to a standard normal distribution, and ξ is resampled every time the data flows through the network, and Z(0,1) represents a normal distribution with a mean of 0 and a standard deviation of 1.

[0048] Considering that the basic requirement of domain generalization is to enable the model to generalize well to unknown domains, further, the perturbed feature distribution is as different as possible from the feature distribution before perturbation, a distance measure is needed to measure the difference between the source domain distribution and the virtual target domain distribution, and the ideal goal of this measure is to create a virtual target distribution with a large domain difference from the source distribution and retain semantic distinction information. Therefore, the application proposes a matrix measure to improve the dissimilarity of the before and after feature maps, that is, by maximizing the Frobenius norm of the difference of the Gram matrix composed of the before and after feature maps, the distribution of the feature map F before perturbation and the feature map F ′ after perturbation is maximized to be far away, so as to maximize the proximity to the unknown domain.

[0049] The application expands the distribution of the source domain by perturbing the confusion factor, enables the network to learn the causal invariant factor of the final decision in a large number of change factors, and enables the model to effectively generalize to the unknown domain (target domain), thereby improving the accuracy and efficiency of medical image processing.

[0050] In step S30, the multimodal feature fusion module splices the causal factor features of different modalities to obtain fused causal factor features, flattens the fused causal factor features in the channel, and inputs the flattened fused causal factor features into a self-attention layer for feature interaction. After classification by a classifier, the class of the image is output.

[0051] Specifically, as shown in Figure 5 , the multimodal feature fusion module splices the causal factor features obtained by the causal decoupling module or the causal decoupling module plus the learnable modal perturbation module to obtain fused causal factor features. After flattening the fused causal factor features in the channel, the flattened fused causal factor features are sent to a self-attention layer, so that each feature interacts with the causal features in its own modality and also interacts with the causal features of another modality (i.e. each feature can not only interact with the causal features in its own modality, but also interact with the causal features of another modality). By using all the information of the two modalities, the flattened fused causal factor features are finally sent to a classifier for classification, and the class of the image is output.

[0052] Further, the Gram matrix is used to reflect the correlation between the feature map channels and the channels, and one channel of the feature map is obtained by convolving one convolution kernel with the image (or the feature map of the previous layer), that is, one channel of the feature map corresponds to one convolution kernel. Generally, a convolution layer has multiple convolution kernels, and the convolution kernels are independent of each other, and different convolution kernels extract different feature meanings, for example, convolution kernel A extracts fine and thick features, and convolution kernel B extracts color features. The Gram matrix actually reflects the correlation between features with different meanings, and the Gram matrix also reflects the feature distribution to some extent. Therefore, in order to improve the decoupling effect and increase the diversity of feature changes, for any modality i, the Frobenius norm of the difference of the Gram matrix composed of different feature maps is constrained, that is:

[0053]

[0054] Wherein, F and F ′ respectively represent the feature map before perturbation and the feature map after perturbation, C' and U' respectively represent the feature map composed of the causal factor and the confounding factor before perturbation, C" and U" respectively represent the feature map composed of the causal factor and the confounding factor after perturbation, and represent the loss function of each feature map of the i-th modality, G represents the Gram operation, and the calculation process is as shown in Figure 6 .

[0055] For all modalities, the final loss is:

[0056]

[0057] Wherein, M represents the total number of modalities, represents the classification cross-entropy loss of the i-th modality, and represents the hyperparameter used to balance the loss strength of the Gram matrix in the i-th modality, represents the final loss; the final loss is used to supervise and guide the model to generalize to unknown domains.

[0058] Further, the method proposed by the present application is not only suitable for two modalities, but also can be extended to multiple modalities. In addition, the scheme of the present application is theoretically and experimentally verified to be feasible, and the classification accuracy is greatly improved, which is 5% higher than that of other optimal methods.

[0059] The present application is directed to a multi-modal medical endoscope image recognition task, and an innovative multi-modal image recognition framework is proposed, which is composed of three core components of causal decoupling module, learnable modal perturbation module and multi-modal feature fusion module. First, an encoder equipped with a convolution block is used to extract initial feature maps from multi-modal endoscopic images. Then, the causal decoupling module maps these initial features to causal factors and confounding factors through a multi-layer perceptron, realizing independent decomposition of different modal image features under causal relationship, so as to facilitate deeper understanding and classification of images, and significantly improve the screening accuracy of specific recognition tasks. On this basis, in order to extend the model to unknown fields and improve its generalization ability, the learnable modal perturbation module introduces an adaptive perturbation mechanism on the confounding factors while keeping the causal factors unchanged. By introducing perturbations of different levels in the model, the generalization ability of the model in unknown fields is effectively evaluated and optimized, thereby effectively alleviating the problem of decline in model generalization performance caused by domain data distribution difference. Finally, the multi-modal feature fusion module splices the features of the two modalities together and enters the attention layer for feature interaction. By introducing the strategies of causal decoupling, modal perturbation and feature fusion, the present application not only improves the accuracy of multi-modal endoscopic image recognition, but also enhances the generalization performance of the model when facing unknown fields, providing an efficient and reliable solution for the medical image recognition field.

[0060] Advantages:

[0061] (1) The present application adopts an innovative method with causal relationship as the core. Through the carefully designed causal decoupling module, it can accurately identify and eliminate those confounding factors that do not play a key role from complex multi-modal data, while retaining those causal factors that have a decisive influence on the classification result. This method not only reduces the interference caused by irrelevant information, but also significantly improves the accuracy and execution efficiency of the classification task by strengthening the key features.

[0062] (2) The present application introduces the learnable modal perturbation module and the multi-modal feature fusion module, which further enhances the generalization ability of the model to unknown fields. By applying adaptive perturbation on the confounding factors without changing the causal factors, the model can adapt to different data distributions flexibly, effectively alleviating the problem of decline in generalization performance caused by domain difference. This comprehensive strategy not only improves the accuracy of tasks such as throat cancer image recognition, but also provides an efficient and reliable solution for the medical image recognition field, demonstrating the important role of causal analysis in improving model performance.

[0063] Further, the method proposed by the present application can adapt to the fusion and classification of multi-modal data, which can be flexibly applied to various modalities, realize cross-modal feature extraction and integration, and combine with domain adaptation and other technologies to cope with more complex application scenarios. Through this expansibility, the present application can improve the generalization ability and adaptability of the model under different data distributions and different task requirements.

[0064] Further, as shown in Figure 7 Based on the above multi-modal image fusion analysis method based on causal representation, the present application also correspondingly provides a multi-modal image fusion analysis system based on causal representation, wherein the multi-modal image fusion analysis system based on causal representation comprises:

[0065] A causal decoupling module 51 is configured to map the initial feature map extracted from the multi-modal endoscopic image using an encoder equipped with a convolution block to a causal factor and a confusion factor through a multi-layer perception after the initial feature map is extracted from the multi-modal endoscopic image using the encoder equipped with the convolution block.

[0066] A learnable modal perturbation module 52 is configured to introduce an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged, and effectively generalize the model to unknown domains by introducing different levels of perturbation in the model.

[0067] A multi-modal feature fusion module 53 is configured to splice the causal factor features of different modalities to obtain fused causal factor features, flatten the fused causal factor features in the channel, input the flattened fused causal factor features to a self-attention layer for feature interaction, and output the class of the image through a classifier.

[0068] Further, as shown in Figure 8 Based on the above multi-modal image fusion analysis method and system based on causal representation, the present application also correspondingly provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 8 Only part of the components of the terminal are shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0069] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a multi-modal image fusion analysis program based on causal representation 40, which can be executed by the processor 10 to implement the multi-modal image fusion analysis method based on causal representation in the present application.

[0070] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as to execute the multi-modal image fusion analysis method based on causal representation, etc.

[0071] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the terminal and to display a visualized user interface. The components 10-30 of the terminal communicate with each other through a system bus.

[0072] In an embodiment, the steps of the multi-modal image fusion analysis method based on causal representation as described above are implemented when the processor 10 executes the multi-modal image fusion analysis program based on causal representation 40 in the memory 20.

[0073] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a multi-modal image fusion analysis program based on causal representation, which implements the steps of the multi-modal image fusion analysis method based on causal representation as described above when executed by a processor.

[0074] In summary, the application provides a multi-modal image fusion analysis method, system, terminal and storage medium based on causal representation, the method comprising: using an encoder equipped with a convolution block to extract an initial feature map from a multi-modal endoscopic image, and a causal decoupling module mapping and decoupling the initial feature map into causal factors and confusion factors through a multi-layer perception; a learnable modal disturbance module introducing an adaptive disturbance mechanism on the confusion factors while keeping the causal factors unchanged, introducing different levels of disturbance in the model to enable the model to effectively generalize to unknown domains; a multi-modal feature fusion module splicing the causal factor features of different modalities to obtain fused causal factor features, flattening the fused causal factor features in the channel and inputting them into a self-attention layer for feature interaction, and outputting the category of the image through a classifier. The application accurately identifies and removes confusion factors that do not play a key role from complex multi-modal data, while retaining causal factors that have a decisive influence on the classification result, improving the efficiency and accuracy of medical image interpretation and processing, further enhancing the generalization ability of the model to unknown domains, applying adaptive disturbance on the confusion factors without changing the causal factors, so that the model can adapt to different data distributions flexibly, effectively alleviating the problem of decreased generalization performance caused by domain differences.

[0075] It should be noted that in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or terminals including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles or terminals. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or terminal including the element.

[0076] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0077] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all such improvements and changes shall fall within the scope of protection of the claims of the application.

Claims

1. A multimodal image fusion analysis method based on causal representation, characterized in that: The multimodal image fusion analysis method based on causal representation includes: An encoder equipped with convolutional blocks is used to extract an initial feature map from the multimodal endoscopic image, and a causal decoupling module decouples the initial feature map into causal factors and confusion factors through a multi-layer perceptron. The learnable modal perturbation module introduces an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged. By introducing different levels of perturbation in the model, the model can be effectively generalized to unknown domains, including: The confounding factor is independent of the label Add an adaptive perturbation to the original image without changing the causal factor that determines the label : （1）； in, represents the feature map after perturbation, represents the learnable parameters, It obeys the standard normal distribution. represents a normal distribution with a mean of 0 and a standard deviation of 1; By maximizing the Frobenius norm of the difference between the Gram matrix of the feature maps before and after, the feature map before perturbation is And the perturbed feature map The distribution of is maximized away from, in order to maximize the proximity to the unknown domain; The multimodal feature fusion module splices the causal factor features of different modalities to obtain fused causal factor features, flattens the fused causal factor features on the channel and inputs them into the self-attention layer for feature interaction, and outputs the image category after classification by the classifier.

2. The multimodal image fusion analysis method based on causal representation according to claim 1 is characterized in that: The encoder equipped with a convolutional block is used to extract an initial feature map from the multimodal endoscopic image, and the causal decoupling module decouples the initial feature map into causal factors and confusion factors through a multi-layer perceptron, specifically including: Original image After feature extractor Extract feature maps Then, through the multi-layer perceptron Decouple the entire feature map from causal factors and confounding factors ; During the training process, the multilayer perceptron is supervised by three loss functions MLP Decoupling causal factors and confounding factors : （2）； （3）； （4）； （5）； in, represents the original image, represents the feature extractor, represents a multilayer perceptron, and denote causal factors and confounding factors respectively, Indicates the number of categories, represents the relative entropy loss, represents a classifier, represents the cross entropy loss, represents the true label, Indicates that for each mode in the multimodal, represents Gram operation, represents the Frobenius norm, 、 and Represents three loss functions used to supervise the model to separate causal factors and confounding factors; Decoupled causal factors and confounding factors Through the classifier CLS Each obtains a predictive distribution and constrains the confusion factor The distribution is close to uniform distribution, does not participate in the prediction of labels, and constrains causal factors The distribution of is aligned with the true distribution and becomes the factor that determines the label.

3. The multimodal image fusion analysis method based on causal representation according to claim 1 or 2, characterized in that: The encoder equipped with a convolutional block extracts an initial feature map from the multimodal endoscopic image, and the causal decoupling module decouples the initial feature map into causal factors and confusion factors through a multi-layer perceptron, and then further includes: By maximizing the causal factor and confounding factors The Frobenius norm of the difference between the Gram matrices of the feature maps makes the causal factor and confounding factors The distribution of maximization is far away.

4. The multimodal image fusion analysis method based on causal representation according to claim 1 is characterized in that: The multimodal feature fusion module splices the causal factor features of different modalities to obtain a fused causal factor feature, flattens the fused causal factor feature on the channel and inputs it into the self-attention layer for feature interaction, and outputs the image category after classification by the classifier, specifically including: The multimodal feature fusion module combines the causal factor features obtained by the causal decoupling module or the causal decoupling module plus the learnable modal perturbation module to obtain the fused causal factor features. After flattening the fused causal factor features on the channel, they are sent to the self-attention layer, so that each feature interacts with the causal features within its own modality and the causal features of the other modality at the same time, utilizing all the information of the two modalities, and finally sent to the classifier for classification, outputting the category of the image.

5. The multimodal image fusion analysis method based on causal representation according to claim 2 is characterized in that: The Gram matrix is ​​used to reflect the correlation between feature map channels; For any mode , by constraining the Frobenius norm of the difference between the Gram matrices of the respective feature maps: （6）； （7）； （8）； in, and Represent the feature map before and after perturbation, and Respectively represent the feature maps composed of causal factors and confusion factors before disturbance, and Respectively represent the feature maps composed of causal factors and confusion factors after disturbance, 、 and Indicates the Each modality maximizes the loss function used by its own feature map.

6. The multimodal image fusion analysis method based on causal representation according to claim 5 is characterized in that: For all modalities, the final loss is: （9）； in, represents the total number of modes, Indicates the Cross entropy loss for multi-modal classification, 、 and Indicates the The weight used to balance the loss strength of the Gram matrix in each mode, Indicates final loss; The final loss Used to supervise and guide the model to generalize to unknown domains.

7. A multimodal image fusion analysis system based on causal representation, characterized in that: The multimodal image fusion analysis system based on causal representation is used to implement the multimodal image fusion analysis method based on causal representation according to any one of claims 1 to 6, and the multimodal image fusion analysis system based on causal representation includes: a causal decoupling module for extracting an initial feature map from a multimodal endoscopic image using an encoder equipped with a convolutional block, and then decoupling the initial feature map mapping into causal factors and confusion factors through a multilayer perceptron; A learnable modal perturbation module is used to introduce an adaptive perturbation mechanism on the confusion factor while keeping the causal factor unchanged. By introducing different levels of perturbations in the model, the model can be effectively generalized to unknown domains. The multimodal feature fusion module is used to splice the causal factor features of different modalities to obtain fused causal factor features, flatten the fused causal factor features on the channel and input them into the self-attention layer for feature interaction, and output the image category after classification by the classifier.

8. A terminal, characterized in that: The terminal includes: a memory, a processor, and a multimodal image fusion analysis program based on causal representation stored in the memory and executable on the processor. When the multimodal image fusion analysis program based on causal representation is executed by the processor, the steps of the multimodal image fusion analysis method based on causal representation are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a multimodal image fusion analysis program based on causal representation. When the multimodal image fusion analysis program based on causal representation is executed by a processor, the steps of the multimodal image fusion analysis method based on causal representation are implemented as described in any one of claims 1 to 6.

Images