Intelligent analysis system based on multi-modal medical data

Abstract

The invention relates to the technical field of medical health, in particular to an intelligent analysis system based on multi-modal medical data, which comprises a multi-modal feature extraction module used for extracting local and global features from medical images and text reports; the cross-modal alignment module is used for aligning potential representations of image and text modals through a variational auto-encoder and a decoupling characterization method; and the cross-modal interaction and fusion module is used for carrying out interaction and fusion on the modal features through a multi-head attention mechanism and a semantic guidance mechanism to generate shared multi-modal representation. According to the method, complementary information of medical images and report texts can be effectively integrated, meanwhile, the integrity of information in modals is ensured, meanwhile, through design and optimization of a plurality of innovative modules, the framework shows excellent performance in cross-modals of medical report analysis, and the method is suitable for popularization and application. An efficient and robust solution is provided for complex medical image and text report analysis.

Description

Technical Field

[0001] The present invention relates to the field of medical health technology, and in particular to an intelligent analysis system based on multimodal medical data. Background Art

[0002] At present, medical reports are an important application of artificial intelligence in the medical field. Currently, there are mainly the following types of related technologies: 1. Medical report analysis methods based on a single modality. They are mainly divided into two categories: natural language processing methods based on medical texts and deep learning methods based on medical images.

[0003] Methods based on medical text data aim to extract structured information from medical texts (such as electronic medical records, doctor's orders, etc.) and automatically generate summary reports. The main technical means include named entity recognition, that is, identifying key entity terms such as diseases, drugs, surgeries, anatomical parts, etc. from medical texts, which is the basic step of medical text analysis; including medical entity linking, that is, linking the identified entity terms to the medical knowledge base to disambiguate their accurate semantics; including medical relationship extraction, that is, extracting semantic relationships between entities from medical texts, such as the relationship between diseases and symptoms, drugs and dosages, etc.; including medical text summarization, that is, automatically summarizing lengthy medical texts to generate concise diagnosis and treatment reports.

[0004] The automatic report generation method based on medical images mainly analyzes medical image data such as X-rays, CT, and MRI to automatically generate diagnostic reports. The main technical routes include the following directions: Methods based on deep learning models such as CNN: cross-modality medical image classification, segmentation, and detection are used as a pre-step for report generation; Methods based on image-language generation models: Using the CNN+RNN encoder-decoder architecture, a diagnosis report is directly generated from medical images in an end-to-end manner. This type of method draws on the idea of ​​image description generation (Image Description Generation); Transformer-based method: Use a pure Transformer architecture to model the mapping relationship from image to report, where the encoder is responsible for extracting image features and the decoder is responsible for generating diagnostic report text.

[0005] The single-modality report analysis method has limitations such as one-sided semantic analysis and insufficient knowledge utilization. It requires the integration of multimodal information and more medical knowledge to obtain comprehensive and accurate analysis capabilities.

[0006] 2. Simple multimodal fusion methods. Currently, there are mainly the following multimodal fusion methods: Feature splicing: Medical image features and text features are simply spliced ​​together as input for downstream cross-modal functions (such as disease classification and report generation).

[0007] Feature superposition: Add or multiply the feature vectors of different modes element by element to obtain the fused feature representation.

[0008] Attention fusion: Use the attention mechanism to assign weights to features of different modalities to achieve adaptive feature fusion.

[0009] Although these simple multimodal fusion methods achieve information-level integration, they fail to fully utilize the uniqueness and cross-modal requirements of medical data.

[0010] This patent proposes a medical report analysis method based on multimodal medical data, aiming to capture the deep semantic associations between modalities while enhancing the focus on local key features, providing an efficient and robust solution for cross-modal analysis of complex medical images and text reports. Summary of the invention

[0011] The purpose of the present invention is to provide an intelligent analysis system based on multimodal medical data to solve the limitations of the existing single-modal report analysis method proposed in the above background technology, such as one-sided semantic analysis and insufficient knowledge. It is necessary to integrate multimodal information and incorporate more medical knowledge to obtain comprehensive and accurate analysis capabilities. Although the existing simple multimodal fusion method has achieved information-level integration, it has failed to fully utilize the uniqueness of medical data and cross-modal requirements.

[0012] To achieve the above object, the present invention provides an intelligent analysis system based on multimodal medical data, comprising: A multimodal feature extraction module that extracts local and global features from medical images and text reports to generate high-dimensional representations; A cross-modal alignment module that aligns the latent representations of image and text modalities through a variational autoencoder and a disentangled representation method; As well as a cross-modal interaction and fusion module, it is used to interact and fuse modal features through a multi-head attention mechanism and a semantic guidance mechanism to generate a shared multimodal representation.

[0013] Specifically, the multimodal feature extraction module at least includes: Image feature extractor, which uses deep convolutional neural network to extract local and global features of medical images; The text feature extractor uses a pre-trained language model to extract local and global features of text reports.

[0014] Specifically, the image feature extractor enhances local features through a self-attention mechanism and aggregates them into global features.

[0015] Specifically, the text feature extractor uses the BERT model to process each word in the text report, generates contextual semantic representation, and aggregates it into global text features through a self-attention mechanism.

[0016] Specifically, The image feature extractor enhances local features through the self-attention mechanism, specifically: by calculating the weighted average of local features, the global feature vector is obtained : , in the above formula, represents the total number of local regions, H f and W f is the spatial resolution of the feature map, V i is the feature vector at the i-th position in the feature map, and d is the feature dimension; Global Features It represents the semantic information of the whole image, and together with the local feature V, it constitutes the output of the image feature extraction module, which is recorded as: .

[0017] Specifically, Aggregate into global text features through self-attention mechanism: Aggregate word-level features using self-attention mechanism to obtain global representation of text : , in the above formula, t j Represents the contextual semantic representation of the jth word in the text, N W is the total number of words reported; Global Features It represents the overall semantic information of the text report and is recorded as the output of the text feature extraction module together with the word-level feature T: .

[0018] Specifically, the cross-modal alignment module learns the latent representation of images and texts through a variational autoencoder, aligns the relevant parts of the image and text modalities, and decouples the irrelevant parts.

[0019] Specifically, the cross-modal interaction and fusion module interacts with the aligned features of the image and text through a multi-head attention mechanism, generates enhanced image and text features, and fuses them into a final multimodal representation.

[0020] Specifically, the semantic guidance mechanism generates semantically consistent features by introducing a semantic prototype matrix and using an attention mechanism to enhance image and text features.

[0021] Specifically, the cross-modal alignment module learns the potential representation of images and texts through a variational autoencoder, specifically: Given an image modality input and text modal input , use independent variational autoencoders for the two modalities to learn their potential representations Z m , , v and t represent image and text respectively; Variational Autoencoders include Conditional Encoders and decoder ,in is the input of another mode, and are the parameters of the encoder and decoder respectively. To achieve conditional information bottleneck and decoupled representation learning, the potential representation Z m Decomposed into cross-modally relevant parts Part not related to cross-modality ,in, , where the cross-modal related part Represents semantic information shared across modalities and irrelevant parts across modalities It captures modality-specific information and avoids the interference of irrelevant cross-modal information on cross-modal learning.

[0022] Specifically, The disentanglement of latent representations enables cross-modal alignment through three loss functions: reconstruction loss , KL divergence loss and decoupling loss ; Reconstruction loss To ensure that the decoder can m Reconstruct the original modal input X m , ensuring the fidelity of the latent representation to the intra-modality information; KL divergence loss Used to regularize the latent space and make its distribution close to the prior distribution ; Decoupling loss Ensure the clarity of feature decomposition by reducing the correlation between cross-modal related parts and cross-modal irrelevant parts; The final optimization goal combines the above three loss functions into an alignment loss: ,By optimizing the above loss function, the cross-modal related part provides a generalized cross-modal representation, which enables the image modality and text modality to be aligned in a shared semantic space.

[0023] Specifically, the alignment features of images and texts are interacted through a multi-head attention mechanism, as follows: Image modality features , with text modality features As keys and values ​​for multi-head attention calculation, the updated image features are: ,in, represents the multi-head attention mechanism, Q, K, V are query, key and value matrices respectively; Text modality features , based on image modality features As the key and value for multi-head attention calculation, the updated text features are: , after the above interaction process, the updated image features are obtained and text features , which respectively embed semantic information from another modality while preserving the intra-modality features; Finally, the features after image and text alignment and Fusion with the features in the original modality to form the final features: .

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. In this intelligent analysis system based on multimodal medical data, the multimodal feature extraction module extracts local and global features from medical images and text reports respectively to generate high-dimensional representations, laying a good foundation for subsequent processing. At the same time, the self-attention mechanism is used to enhance the expression ability of local features and better capture the dependencies between regions.

[0026] 2. In this intelligent analysis system based on multimodal medical data, the conditional information bottleneck method based on variational autoencoders and decoupled representations is used to effectively learn compact cross-modal representations and promote semantic alignment between modalities. At the same time, by decomposing the latent representation and combining it with a carefully designed loss function, the model is guided to learn cross-modal features with more discriminative and generalizable capabilities.

[0027] 3. In this intelligent analysis system based on multimodal medical data, a cross-modal multi-head attention mechanism is introduced through the modal interaction and fusion module, so that the image modal features and text modal features complement each other semantically, enhance the expression of complementary information, and at the same time fuse the modal features into a shared multimodal representation, providing richer and more consistent semantic information for downstream cross-modality.

[0028] 4. In this intelligent analysis system based on multimodal medical data, the semantic enhancement module captures the shared semantic information in cross-modality through a learnable semantic prototype matrix, and embeds it into the fused features through the attention mechanism, further improving the semantic expression ability of cross-modal features.

[0029] 5. In this intelligent analysis system based on multimodal medical data, the CLIP model is introduced through the IAMMD-CLIP joint training framework for feature enhancement to improve the expressiveness and generalization of features. At the same time, the model's adaptability to medical data is enhanced by prompting fine-tuning and introducing an adapter layer.

[0030] 6. In this intelligent analysis system based on multimodal medical data, phased training is adopted through a joint optimization strategy to ensure the stability of model training and the optimality of the final performance. At the same time, by balancing the weights of multimodal alignment loss, contrast loss, classification loss and generation task loss, the robustness and generalization ability of the model in medical tasks are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the intelligent analysis system based on multimodal medical data of the present invention; Figure 2 A schematic diagram of the working process of the intelligent analysis system based on multimodal medical data of the present invention; Figure 3 Schematic diagram of the principles of the three key loss functions of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Cross-modality medical image analysis involves the processing of multimodal data, where the input includes data from different modalities: medical images and corresponding text reports. A medical image can be represented as a three-dimensional tensor ,in Represents the height, width and number of channels of the image respectively. The report text is usually composed of multiple word sequences, Indicates that is the total number of words in the report. The goal is to design an end-to-end multimodal framework that can efficiently extract and fuse these modal features, thereby achieving the goal of cross-modal analysis and diagnostic reasoning of medical reports.

[0034] The core challenge of this problem lies in cross-modal semantic alignment and information interaction. Medical images and report texts differ significantly in data structure and semantic expression. Medical images provide rich spatial information and image patterns, while text reports convey semantic information and diagnostic logic in a serialized manner. Therefore, achieving efficient alignment and fusion between modalities is the key to the success of this cross-modal approach.

[0035] To solve the above problems, Figure 1-Figure 3 As shown, the present invention provides an intelligent analysis system based on multimodal medical data. The system includes three main modules: a multimodal feature extraction module, a cross-modal alignment module, and a cross-modal interaction and fusion module. First, the multimodal feature extraction module extracts local and global features from medical images and text reports, respectively, and generates high-dimensional representations for subsequent processing. Then, the cross-modal alignment module extracts compressed redundant features through a variational autoencoder module while retaining complementary information between modalities. Finally, the cross-modal interaction and fusion module and the semantic guidance mechanism realize the deep fusion of features between modalities.

[0036] The system is designed to capture deep semantic associations between modalities while enhancing attention to local key features, providing an efficient and robust solution for cross-modal analysis of complex medical images and text reports.

[0037] 1. Multimodal feature extraction.

[0038] In order to achieve multimodal fusion of medical images and text reports, it is first necessary to extract features from the data of the two modalities separately to obtain local and global representations that can fully express the modal characteristics. The system uses independent image feature extractors and text feature extractors to process image and text data respectively. These features will then be used as input to the information bottleneck and cross-modal alignment module based on variational autoencoders.

[0039] 1.1 Image feature extraction.

[0040] Image feature extraction of medical images aims to capture the spatial information of local regions and the global image representation. Given an input medical image , firstly, a deep convolutional neural network (CNN) is used to process it. Specifically, the image is processed layer by layer through several convolutional layers and pooling layers to generate a low-resolution feature map ,in and is the spatial resolution of the feature map, is the number of channels. Each position of the feature map represents the characteristics of the local area of ​​the image, so the feature map contains rich local information.

[0041] In order to further extract local features, the feature map Flattened into a set of local eigenvectors: .

[0042] in, represents the total number of local areas, It is the first The feature vector of each position, d is the feature dimension. The spatial information of the receptive field corresponding to the position is encoded, reflecting the image pattern of the local area.

[0043] In order to further improve the expressiveness of local features, the present invention introduces a self-attention mechanism to enhance local features. By calculating the similarity between local features, the attention mechanism can dynamically adjust the weight of each local feature to better capture the dependencies between regions. The enhanced local features are still expressed as , but includes global context information.

[0044] In addition, in order to extract the global features of the image, the self-attention mechanism is applied to the local features for global aggregation. Specifically, the global feature vector is obtained by calculating the weighted average of the local features. : .

[0045] Global Features Represents the overall semantic information of the image and local features The output of the image feature extraction module is recorded as: .

[0046] 1.2 Text feature extraction.

[0047] The feature extraction of text reports is completed through a pre-trained language model (BERT), which can capture the contextual dependencies and rich semantic information of the text. Given a text report ,in Indicates First, map each word to an embedding vector space to generate an initial embedding representation. ,in .

[0048] Then, the embedding representation After being processed by the multi-layer Transformer encoder of the BERT model, word-level feature representation is generated: .

[0049] in, Indicates the first The contextual semantic representation of a word.

[0050] In order to extract the global features of the text, the present invention also uses the self-attention mechanism to aggregate the word-level features to obtain the global representation of the text : .

[0051] Global Features Represents the overall semantic information of the text report, and word-level features The output of the text feature extraction module is recorded as: .

[0052] 2. Cross-modal alignment with conditional information bottleneck based on variational autoencoder and disentangled representation.

[0053] In order to effectively learn compact cross-modal representations and promote semantic alignment between modalities, this paper proposes a conditional information bottleneck method based on variational autoencoder (VAE) and decoupled representation. The core of this method is to decompose the potential representation of the modality, combined with a well-designed loss function, to guide the model to learn cross-modal features with more discriminative and generalizable capabilities.

[0054] Given an image modality input and text modal input , using independent variational autoencoders for the two modalities to learn their potential representations ( , v and t represent images and text respectively). Variational autoencoders include conditional encoders and decoder ,in is the input of another mode, and are the parameters of the encoder and decoder respectively. To achieve conditional information bottleneck and decoupled representation learning, the potential representation Decomposed into cross-modally relevant parts Part not related to cross-modality : .

[0055] Among them, the cross-modal related part Represents semantic information shared across modalities and irrelevant parts across modalities It captures modality-specific information and avoids the interference of irrelevant cross-modal information on cross-modal learning.

[0056] like Figure 2 As shown in Figure 2, in order to achieve the decoupling of latent representations and ensure cross-modal alignment, three key loss functions are designed: reconstruction loss , KL divergence loss and decoupling loss .

[0057] 2.1 Reconstruction Loss.

[0058] Reconstruction loss To ensure that the decoder can Reconstruct the original modal input , ensuring the fidelity of the potential representation to the information within the modality. It is defined as:

[0059] in, is a conditional encoder that depends on the input of the current modality and the other modality; is the decoder, responsible for reconstructing the original input from the latent representation, represents the expectation operation, represents the Kullback-Leibler divergence.

[0060] 2.2KL divergence loss.

[0061] KL divergence loss Used to regularize the latent space and make its distribution close to the prior distribution , in order to improve the generalization ability of the model. It is defined as:

[0062] in, is the prior distribution, usually assumed to be a standard normal distribution, is the identity matrix.

[0063] 2.3 Decoupling loss.

[0064] Decoupling loss By reducing the cross-modal correlation Part not related to cross-modality The correlation between them ensures the clarity of the feature decomposition. It is defined as:

[0065] in, is the covariance matrix of the cross-modal related part and the cross-modal irrelevant part, is the Frobenius norm. Minimizing this loss can reduce the cross-correlation between the two parts, thereby achieving effective feature decoupling.

[0066] 2.4 Cross-modal alignment.

[0067] The final optimization goal combines the above three loss functions into an alignment loss:

[0068] By optimizing the above loss function, the cross-modal related parts It provides a generalized cross-modal representation, which enables the image modality and text modality to be aligned in a shared semantic space. The modality-specific information is retained, which prevents the alignment process from losing important characteristics of the original modality. It can effectively integrate the complementary information between modalities while ensuring the integrity of the information within the modality.

[0069] 3. Modal interaction and fusion.

[0070] After completing the conditional information bottleneck cross-modal alignment based on variational autoencoders, the next step is to enhance the complementary information expression of image modality and text modality through an explicit modal interaction mechanism, and at the same time fuse the modal features into a shared multimodal representation for downstream cross-modality.

[0071] The goal of modal interaction is to introduce a cross-modal multi-head attention mechanism to make the image modality features and text modality features They complement each other semantically. Specifically, the interaction of modal features is achieved through the following process: Features of image modality (cross-modality related features) , based on text modality features (cross-modality related features) As keys and values ​​for multi-head attention calculation, the updated image features are:

[0072] in, represents the multi-head attention mechanism, , , are the query, key, and value matrices respectively.

[0073] Similarly, for text modality features , based on image modality features As the key and value for multi-head attention calculation, the updated text features are: .

[0074] After the above interaction process, the updated image features are obtained and text features , which respectively embed semantic information from another modality while preserving the intra-modality characteristics.

[0075] Finally, the features after image and text alignment and Fusion with the features in the original modality to form the final features: .

[0076] For the convenience of remembering Represents the image features after alignment. Similarly, represents the aligned text features. M represents the total number of paired images and reports.

[0077] 4. Semantic enhancement.

[0078] Based on modal interaction and fusion, the system further introduces a semantic guidance mechanism to enhance the semantic expression ability of cross-modal features through a learnable semantic prototype matrix P. The semantic prototype matrix captures the shared semantic information in cross-modalities and embeds it into the fused features through an attention mechanism.

[0079] 4.1 Semantic Attention Computation.

[0080] Features after image alignment Features aligned with text , respectively calculate the attention distribution between them and the semantic prototype. The semantic attention distribution of image features is:

[0081] The semantic attention distribution of text features is:

[0082] in, and Represents the attention distribution between the fused features of image and text modalities and the semantic prototype.

[0083] 4.2 Semantic feature enhancement.

[0084] The semantic prototype is weighted by using semantic attention distribution, and then embedded into the original features through a multi-layer perceptron (MLP) to generate enhanced image features. And the enhanced text features :

[0085] .

[0086] Final Enhancement Features and It contains both intra-modal and cross-modal semantic information and embeds the guidance information of the shared semantic space, providing a richer and more consistent semantic expression for subsequent cross-modal analysis. Finally, the enhanced feature set of all images and texts after semantic guidance is recorded as and .

[0087] 4.3 Semantic consistency loss.

[0088] Semantic consistency loss It is used to further enhance the semantic expression ability of the alignment features by minimizing:

[0089] in is the Frobenius norm. It ensures that the aligned image and text features are distributed consistently in the semantic space.

[0090] 5. Joint optimization and training.

[0091] The joint optimization process of this system aims to comprehensively improve the alignment quality, semantic consistency and cross-modal adaptability of multimodal features through the design of multi-objective loss functions. Based on the aligned and fused features, IAMMD is trained through three types of loss functions: alignment loss ( ), semantic consistency loss ( ). These loss functions guide model training from three perspectives: alignment quality, consistency of feature distribution, and complementation of semantic information.

[0092] 5.1 Alignment Loss.

[0093] Combine the three losses in step 3 to form a comprehensive alignment loss: .

[0094] 5.2 Semantic consistency loss.

[0095]

[0096] in is the Frobenius norm.

[0097] 5.3 Total loss function.

[0098] The final joint optimization objective integrates the above three loss functions into:

[0099] in is the weight coefficient of each loss, which is used to balance the contribution between alignment loss and semantic consistency loss. The system can improve the distribution consistency and semantic expression ability of cross-modal features while ensuring the alignment quality between modalities, and provide accurate and robust feature representation for downstream cross-modal medical semantic analysis.

[0100] 6.IAMMD-CLIP joint training framework.

[0101] 6.1 CLIP feature enhancement.

[0102] After obtaining the features of the system that integrates multimodal information, in order to further improve the expressiveness and generalization of the features, this paper introduces the CLIP model [model proposed by openai] for feature enhancement. Given the image features generated by IAMMD and text features , mapped to the feature space of CLIP through the feature adaptation layer: .

[0103] in, are learnable mapping parameters.

[0104] 6.2 Prompt Tuning

[0105] In order to further optimize the feature space representation of CLIP, this paper introduces prompt tuning in image and text features. Prompt tuning guides the model to adapt to new tasks by adding a set of learnable prompt vectors. Given an image prompt vector and text hint vector ,in and Represents the length of the image and text prompt respectively, is the embedding dimension.

[0106] The hint fine-tuning of image features is expressed as follows: .

[0107] in, Represents concatenation in the sequence length dimension.

[0108] Modified image features CLIP image encoder Get image embedding: .

[0109] The hint fine-tuning expression of text features is as follows: .

[0110] Modified text features CLIP text encoder Get text embedding: .

[0111] During hint fine-tuning, only the hint vector is updated and The parameters of the CLIP encoder and The parameters remain frozen.

[0112] 6.3 Adapter Layers (AdapterLayers).

[0113] In order to enhance the adaptability of tasks, the present invention further introduces adapter layers. The adapter layer integrates task-specific learning capabilities into the feature encoder of the CLIP model by inserting a small bottleneck network module. Each adapter layer consists of the following parts: Dimensionality reduction matrix ; Using nonlinear activation function ); Dimensional matrix .

[0114] The adapter function is defined as follows: .

[0115] in, Represents the activation function.

[0116] The adapter layer is integrated into each layer of the Transformer module in the CLIP encoder, and its output is expressed as: .

[0117] During training, only the parameters of the adapter layer are optimized and , the original weights of the CLIP encoder remain frozen. Image and text The embedding after prompt fine-tuning and adapter enhancement is denoted as and .

[0118] 6.4 Fusion feature generation.

[0119] Image embeddings enhanced with hints and adapters and text embedding is integrated into the cross-modal attention mechanism to generate the final fused features: .

[0120] in, Represents feature concatenation.

[0121] 6.5 Downstream Tasks.

[0122] 6.5.1 Diagnostic classification.

[0123] Based on fusion features Build a hierarchical classification head to predict the diagnosis class: .

[0124] 6.5.2 Report generation.

[0125] Using fusion features As input, a medical report is generated through the Transformer decoder: .

[0126] 6.6 Joint optimization strategy.

[0127] 6.6.1 Training in stages.

[0128] In order to ensure the stability of model training and the optimality of final performance, the present invention adopts a phased training strategy. The entire training process is divided into the following three stages, each of which is optimized for the training requirements of different modules.

[0129] ①IAMMD pre-training stage.

[0130] The goal of the first stage is to optimize the multimodal alignment capability of the system to ensure that image and text features can be effectively fused in the same semantic space. In this stage, all parameters of the CLIP model are frozen, and only the learnable mapping parameters and fusion modules in the system are trained. By optimizing the multimodal alignment loss, the image features and text features of IAMMD are kept highly consistent in the embedding space.

[0131] ②CLIP feature enhancement stage.

[0132] The goal of the second stage is to further enhance the image and text features using the CLIP model. In this stage, the parameters of the system are fixed, and only the parameters such as the prompt vector and adapter layer of the CLIP model are optimized to improve CLIP's adaptability to medical data. By optimizing the contrast loss of CLIP and the cross-modal loss of downstream tasks, the image and text features of CLIP can more accurately express cross-modal semantic relationships in medical scenarios.

[0133] ③End-to-end fine-tuning stage.

[0134] The goal of the third phase is to jointly optimize the system and the CLIP model to ensure that the image and text features of the overall model can be efficiently integrated. In this phase, all parameters in the system and the CLIP model are unfrozen and all modules are jointly trained. By balancing the weights of multimodal alignment loss, contrast loss, classification loss, and generation task loss, the model achieves robustness and generalization in medical tasks.

[0135] 6.6.2 Cross-modal specific losses.

[0136] ①Classification cross-modal loss: .

[0137] in is the cross entropy loss, used for coarse classification across modalities; is the binary cross entropy loss for fine-grained classification across modalities.

[0138] Coarse-grained classification refers to the division of data into large or main categories, with rough distinctions between categories. For example, in medical image classification, coarse-grained cross-modal classification may be based on the type of disease, without involving specific subtypes or specific parts of the disease.

[0139] Representation of fine-grained classification: refers to a finer division of data, usually used for more specific and detailed cross-modal. For example, in medical images, fine-grained cross-modal classification may not only classify "lung cancer", but may also further distinguish "non-small cell lung cancer" from "small cell lung cancer", etc.

[0140] ②Report generation cross-modal loss: .

[0141] in, is the maximum likelihood estimation loss used to report cross-modal generation.

[0142] ③Contrast loss of CLIP model: .

[0143] in is the temperature coefficient that controls the strength of the contrastive loss. By minimizing the CLIP contrastive loss, the alignment of images and text in a shared semantic space is enhanced.

[0144] is the similarity score of the positive sample pair (matching image-text pair), Perform an exponential transformation on the similarity, K is the batch size, including 1 positive sample and K-1 negative samples, Represents the features of the kth text sample.

[0145] 6.6.3 Final optimization goal.

[0146] In the end-to-end fine-tuning stage, the final optimization goal is: .

[0147] is the total loss of IAMMD, focusing on the alignment and optimization of images and texts.

[0148] It is the contrastive loss of CLIP, which enhances the cross-modal alignment of images and texts.

[0149] is the downstream cross-modal loss (classification or generation cross-modality corresponds to and ), ensuring the accuracy of the model in cross-modality medical image interpretation.

[0150] is the weight coefficient for balancing the loss terms and is a manually set hyperparameter.

[0151] In summary, the intelligent analysis system based on multimodal medical data provided by the present invention can effectively integrate the complementary information of medical images and report texts while ensuring the integrity of the information within the modality. At the same time, through the design and optimization of multiple innovative modules, the framework shows excellent performance in cross-modal medical report analysis, providing an efficient and robust solution for the analysis of complex medical images and text reports.

[0152] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An intelligent analysis system based on multimodal medical data, characterized in that: include: A multimodal feature extraction module that extracts local and global features from medical images and text reports to generate high-dimensional representations; A cross-modal alignment module that aligns the latent representations of image and text modalities through a variational autoencoder and a disentangled representation method; As well as a cross-modal interaction and fusion module, it is used to interact and fuse modal features through a multi-head attention mechanism and a semantic guidance mechanism to generate a shared multimodal representation.

2. The intelligent analysis system based on multimodal medical data according to claim 1, characterized in that: The multimodal feature extraction module at least includes: Image feature extractor, which uses deep convolutional neural network to extract local and global features of medical images; The text feature extractor uses a pre-trained language model to extract local and global features of text reports.

3. The intelligent analysis system based on multimodal medical data according to claim 2, characterized in that: The image feature extractor enhances local features through a self-attention mechanism and aggregates them into global features.

4. The intelligent analysis system based on multimodal medical data according to claim 2, characterized in that: The text feature extractor uses the BERT model to process each word in the text report, generates contextual semantic representations, and aggregates them into global text features through a self-attention mechanism.

5. The intelligent analysis system based on multimodal medical data according to claim 3, characterized in that: The image feature extractor enhances local features through the self-attention mechanism, specifically: by calculating the weighted average of local features, the global feature vector is obtained : , in the above formula, represents the total number of local regions, H f and W f is the spatial resolution of the feature map, V i is the feature vector at the i-th position in the feature map, and d is the feature dimension; Global Features It represents the semantic information of the whole image, and together with the local feature V, it constitutes the output of the image feature extraction module, which is recorded as: 。 6. The intelligent analysis system based on multimodal medical data according to claim 4, characterized in that: Aggregate into global text features through self-attention mechanism: Aggregate word-level features using self-attention mechanism to obtain global representation of text : , in the above formula, t j Represents the contextual semantic representation of the jth word in the text, N W is the total number of words reported; Global Features It represents the overall semantic information of the text report and is recorded as the output of the text feature extraction module together with the word-level feature T: 。 7. The intelligent analysis system based on multimodal medical data according to claim 1, characterized in that: The cross-modal alignment module learns the latent representations of images and texts through a variational autoencoder, aligns the relevant parts of the image and text modalities, and decouples the irrelevant parts.

8. The intelligent analysis system based on multimodal medical data according to claim 1, characterized in that: The cross-modal interaction and fusion module interacts with the aligned features of the image and text through a multi-head attention mechanism, generates enhanced image and text features, and fuses them into the final multimodal representation.

9. The intelligent analysis system based on multimodal medical data according to claim 1, characterized in that: The semantic guidance mechanism introduces a semantic prototype matrix and uses an attention mechanism to enhance image and text features to generate semantically consistent features.

10. The intelligent analysis system based on multimodal medical data according to claim 7, characterized in that: The cross-modal alignment module learns the potential representation of images and texts through a variational autoencoder, specifically: Given an image modality input and text modal input , use independent variational autoencoders for the two modalities to learn their potential representations Z m , , v and t represent image and text respectively; Variational Autoencoders include Conditional Encoders and decoder ,in is the input of another mode, and are the parameters of the encoder and decoder respectively. To achieve conditional information bottleneck and decoupled representation learning, the potential representation Z m Decomposed into cross-modally relevant parts Part not related to cross-modality ,in, , where the cross-modal related part Represents semantic information shared across modalities and irrelevant parts across modalities It captures modality-specific information and avoids the interference of irrelevant cross-modal information on cross-modal learning.

11. The intelligent analysis system based on multimodal medical data according to claim 10, characterized in that: The disentanglement of latent representations enables cross-modal alignment through three loss functions: reconstruction loss , KL divergence loss and decoupling loss ; Reconstruction loss To ensure that the decoder can m Reconstruct the original modal input X m , ensuring the fidelity of the latent representation to the intra-modality information; KL divergence loss Used to regularize the latent space and make its distribution close to the prior distribution ; Decoupling loss Ensure the clarity of feature decomposition by reducing the correlation between cross-modal related parts and cross-modal irrelevant parts; The final optimization goal combines the above three loss functions into an alignment loss: ,By optimizing the above loss function, the cross-modal related part provides a generalized cross-modal representation, which enables the image modality and text modality to be aligned in a shared semantic space.

12. The intelligent analysis system based on multimodal medical data according to claim 11, characterized in that: The alignment features of images and texts are interacted through a multi-head attention mechanism, specifically: Image modality features , with text modality features As keys and values ​​for multi-head attention calculation, the updated image features are: ,in, represents the multi-head attention mechanism, Q, K, V are query, key and value matrices respectively; Text modality features , based on image modality features As the key and value for multi-head attention calculation, the updated text features are: , after the above interaction process, the updated image features are obtained and text features , which respectively embed semantic information from another modality while preserving the intra-modality features; Finally, the features after image and text alignment and Fusion with the features in the original modality to form the final features: .

Images