A multi-modal medical data representation learning method and system

Abstract

The application discloses a kind of multimodal medical data representation learning method and system, it is related to medical research technical field, including the following steps: inputting multimodal medical data to representation learning model;Dynamic input modeling module encodes corresponding medical data based on the complexity of different modal medical data, obtains multiple token sequences corresponding to each mode;Dynamic window module allocates and weights fusion of multiple token sequences of each mode at different scales, obtains multiple weighted sequences;Multi-head attention network captures and fuses the semantic association information of multiple weighted sequences, obtains corresponding fusion vector, and feedforward network maps fusion vector, obtains corresponding modal feature representation;Cross-modal contrast knowledge distillation module aligns the features of modal feature representation of different modes, obtains the modal feature representation after alignment.The application significantly improves the precision and robustness of multimodal medical data representation.

Description

Technical Field

[0001] This invention relates to the field of medical research technology, and in particular to a multimodal medical data representation learning method and system. Background Technology

[0002] Hepatocellular carcinoma (HCC) is the fourth most common malignant tumor in my country and the second leading cause of cancer-related death. The risk factors and mechanisms of HCC are complex and diverse. Despite advancements in various diagnostic and treatment methods, patient prognosis remains poor, with a 5-year overall survival rate of only 12.1%. With the rapid development of technologies such as medical imaging, gene sequencing, and pathological analysis, the medical field has accumulated a vast amount of multimodal data. Analyzing multimodal medical data on HCC patients is crucial for a deeper understanding of the pathogenesis and for implementing precise diagnosis and treatment. In recent years, thanks to the rapid improvement in computing power and the establishment of large-scale medical databases, artificial intelligence technologies, represented by deep learning, have achieved many breakthroughs in the medical field.

[0003] Different modalities of clinical data related to liver cancer can aid in its diagnosis, including tumor imaging, histopathological images, and genetic data. These modalities are highly complementary; for example, tumor imaging and histopathological images focus on the macroscopic and microscopic characteristics of the tumor, respectively, while genetic data, case reports, and laboratory data provide more granular information related to the disease from a more refined perspective. While these data contain a wealth of information, their complex data types and diverse formats make it difficult for existing diagnostic methods to fully utilize them for comprehensive analysis. Multimodal representation learning technology can extract comprehensive information from different data sources and has broad application prospects.

[0004] Existing representation learning methods, such as the Transformer model, use fixed-length token sequences to represent input data, which cannot adapt to the significant dimensionality heterogeneity in medical data. At the same time, when extracting modal feature representations, they only learn shallow, keyword-based associations and fail to fully explore deeper relationships between modalities, thus reducing the accuracy of the extracted modal feature representations. Summary of the Invention

[0005] Based on the shortcomings of the existing technologies, this invention provides a multimodal medical data representation learning method and system, which solves the problems of existing representation learning methods such as the Transformer model, which use fixed-length token sequences to represent input data and cannot adapt to the significant dimensionality heterogeneity in medical data; at the same time, when extracting modal feature representations, they only learn shallow, keyword-based associations and fail to fully explore deeper relationships between modalities, thus reducing the accuracy of the extracted modal feature representations.

[0006] The present invention adopts the following technical solution: In a first aspect, the present invention provides a multimodal medical data representation learning method, comprising the following steps: Collect multimodal medical data; Multimodal medical data is input into a representation learning model to obtain aligned modal feature representations; the representation learning model includes a dynamic input modeling module, a dynamic window module, a multi-head attention network, a feedforward network, and a cross-modal comparative knowledge distillation module; The dynamic input modeling module encodes the corresponding medical data based on the complexity of different modalities, obtaining multiple token sequences for each modality. The dynamic window module assigns and weights these multiple token sequences at different scales, resulting in multiple weighted sequences. The multi-head attention network captures and fuses the semantic association information of these multiple weighted sequences, obtaining a corresponding fusion vector. The feedforward network maps the fusion vector to obtain the corresponding modal feature representation. The cross-modal comparative knowledge distillation module aligns the modal feature representations of different modalities, obtaining aligned modal feature representations.

[0007] Preferably, the dynamic input modeling module uses a fixed number of tokens in a pool, and activates different numbers of tokens through a gating mechanism for input data of different resolutions.

[0008] Preferably, the dynamic window module allocates and weights multiple token sequences of each modality at different scales, including the following steps: Multiple tokens in the token sequence of each modality are allocated using sliding windows of different scales; The allocated tokens are weighted and merged according to dynamic weights to obtain multiple weighted sequences.

[0009] Preferably, the cross-modal comparative knowledge distillation module performs feature alignment on the modal feature representations of different modalities to obtain aligned modal feature representations, specifically including the following steps: The teacher model is constructed, which is a 3-layer multilayer perceptron (MLP). The input is the modal feature representation of all modalities, and the output is the aligned modal feature representation. The student model is constructed using a 3-layer multilayer perceptron (MLP). The input is the modal feature representations of all other modalities when one modal feature is randomly missing, and the output is the aligned modal feature representations. For each sample, samples of the same category as the sample are defined as positive samples, and samples of different categories are defined as negative samples; Alignment is achieved using a contrastive learning loss function.

[0010] Preferably, the multimodal data includes image data, pathological data, genetic data, and diagnostic reports.

[0011] Preferably, the modal features of image data are represented as spatial local features, the modal features of pathological data are represented as graph structures, and the modal features of gene data and diagnostic reports are represented as contextual semantic features.

[0012] Secondly, the present invention provides a multimodal medical data representation learning system, comprising: The acquisition module is used to acquire multimodal medical data; The input module is used to input multimodal medical data into the representation learning model to obtain aligned modal feature representations; the representation learning model includes a dynamic input modeling module, a dynamic window module, a multi-head attention network, a feedforward network, and a cross-modal comparative knowledge distillation module; The processing module is used to dynamically input the modeling module to encode the corresponding medical data based on the complexity of different modal medical data, obtaining multiple token sequences for each modality; the dynamic window module allocates and weights the multiple token sequences of each modality at different scales to obtain multiple weighted sequences; the multi-head attention network captures and fuses the semantic association information of the multiple weighted sequences to obtain the corresponding fusion vector; the feedforward network maps the fusion vector to obtain the corresponding modal feature representation; the cross-modal comparison knowledge distillation module aligns the modal feature representations of different modalities to obtain the aligned modal feature representation.

[0013] Compared with the prior art, the above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects: This invention constructs a representation learning model, including a dynamic input modeling module, a dynamic window module, a multi-head attention network, a feedforward network, and a cross-modal comparative knowledge distillation module. The dynamic input modeling module adaptively adjusts the number of tokens in different modalities, addressing the insufficient representation capability caused by the dimensional heterogeneity of medical data. The dynamic window module distributes and weights multiple token sequences from each modality at different scales, resulting in multiple weighted sequences. This invention weights multiple tokens to varying degrees, fully exploring deeper relationships within each modality. Then, the multi-head attention network and feedforward network capture and fuse semantic association information from the multiple weighted sequences, obtaining corresponding fusion vectors. Mapping these fusion vectors yields the corresponding modal feature representations, achieving collaborative modeling of local details and global semantics, thus improving the richness and accuracy of feature representation. Finally, the cross-modal comparative knowledge distillation module aligns the modal feature representations of different modalities, effectively eliminating distribution differences between modalities. Overall, this invention significantly improves the accuracy and robustness of multimodal medical data representation. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a multimodal medical data representation learning method according to the present invention; Figure 2 This is a schematic diagram of the dynamic window module of the present invention; Figure 3 This is a schematic diagram of the multi-branch modal expert unit of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 To address the aforementioned problems, embodiments of the present invention provide a multimodal medical data representation learning method, referring to... Figures 1-3 This method specifically includes the following steps: S1: Collect multimodal medical data.

[0018] This embodiment uses liver cancer as an example for illustration. First, multimodal data characterizing liver cancer patients are collected. The multimodal data includes imaging data, pathological data, genetic data, and diagnostic reports.

[0019] S2: Multimodal medical data is input into the representation learning model to obtain aligned modal feature representations. This model is based on a deep Transformer architecture and uses multi-branch Transformer units to efficiently encode multimodal medical data. The core architecture of the model includes a dynamic input modeling module, a dynamic window module, multi-branch modality expert units, and a cross-modal comparative knowledge distillation module. The multi-branch modality expert units include a multi-head attention network and a feedforward network.

[0020] S3: The dynamic input modeling module encodes the corresponding medical data based on the complexity of different modal medical data.

[0021] Traditional Transformer models use fixed-length token sequences to represent input data, which cannot adapt to the significant dimensionality heterogeneity in medical data. This model proposes a dynamic input modeling module that adaptively adjusts the token allocation strategy based on sample complexity. For high-dimensional complex data (such as high-resolution pathological images), the module automatically generates more tokens to achieve fine-grained feature characterization; for low-dimensional data (such as structured clinical indicators), the number of tokens is reduced to improve computational efficiency.

[0022] In this embodiment, the dynamic input modeling module uses a fixed number of tokens. For input data of different resolutions, different tokens are activated through a gating mechanism to obtain a dynamic number of tokens. Ultimately, for high-dimensional complex data (such as high-resolution pathological images), the module automatically generates more tokens to achieve fine-grained feature characterization; for low-dimensional data (such as structured clinical indicators), the number of tokens is reduced to improve computational efficiency.

[0023] S4: The dynamic window module distributes and weights multiple token sequences of each modality at different scales to obtain multiple weighted sequences.

[0024] This module achieves feature fusion through a multi-scale window mechanism: the module designs windows of different scales to fuse token information, and the outputs of each scale window branch are fused through learnable dynamic weights. The weight values ​​are determined by the inherent characteristics of the input data, thereby realizing adaptive multi-scale information integration and obtaining multiple weighted sequences. Figure 2 The numbers in the table represent the assigned weights.

[0025] S5: The multi-head attention network captures and fuses semantic association information from multiple weighted sequences to obtain the corresponding fusion vector. The feedforward network maps the fusion vector to obtain the corresponding modal feature representation.

[0026] Multiple weighted sequences are semantically associated and fused in different subspaces to obtain corresponding fusion vectors. The fusion vectors are then mapped to obtain the corresponding modal feature representations.

[0027] The model employs a multi-branch structure combining a shared self-attention mechanism with a modality-specific feedforward network (modality-specific expert model). Each Transformer unit contains a shared multi-head self-attention module and a feedforward network branch. The self-attention module is responsible for capturing long-distance dependencies within a modality, such as global anatomical structural associations in medical images or long-range regulatory patterns in gene sequences. Each feedforward network branch acts as a modality-specific "expert model," learning unique feature representations for different data types (e.g., images, pathology, genes, clinical texts). This design preserves the ability to interact with general features across modalities while enabling refined modeling of heterogeneous data through modality expert networks. Each modality expert network adopts a differentiated hierarchical structure: the image branch focuses on the progressive aggregation of spatial local features, the gene branch emphasizes relational reasoning of graph structures, and the clinical text branch focuses on deep parsing of contextual semantics.

[0028] Different sized receptive windows are assigned to different attention heads in the Transformer, allowing the model to capture both local details and global context simultaneously. The multi-head attention network is an existing network consisting of linear layer mappings and dot-product-based attention computation. Multiple weighted sequences are first processed through three different linear transformation layers to obtain Query, Key, and Value, respectively. These transformed vectors are then divided into several "heads," each with its own independent Query, Key, and Value matrices. For each head, a Scaled Dot-Product Attention operation is performed. Finally, the outputs of all heads are concatenated and then fused through a linear layer to obtain the final fused vector.

[0029] Modality-specific expert models are existing networks that include linear layers with self-attention mechanisms, responsible for learning data from different modalities.

[0030] S6: The cross-modal comparative knowledge distillation module aligns the modal feature representations of different modalities to obtain aligned modal feature representations. This includes the following steps:

[0031] S61: Construct a teacher model, which is a 3-layer multilayer perceptron (MLP). The input is the modal feature representation of all modalities, and the output is the aligned modal feature representation.

[0032] S62: Construct a student model, which is also a 3-layer multilayer perceptron (MLP). The input is the modal feature representation of all other modalities when one modal feature is randomly missing, and the output is the aligned modal feature representation.

[0033] S63: Define positive and negative samples: For each sample, samples of the same class and similar to the sample are defined as positive samples, and samples of different classes are defined as negative samples.

[0034] S64: Alignment using contrastive learning loss function: bringing positive samples closer together and negative samples further apart, aligning the feature relationships and structural information between different modalities.

[0035] To achieve intermodal knowledge complementarity, the model introduces a category-guided cross-modal contrastive learning mechanism. For each training sample, the system treats cross-modal samples of the same category as positive samples and samples of different categories as negative samples. By using a contrastive loss function, the model narrows the distance between positive samples in the feature space while widening the distribution of negative samples. This process achieves a triple optimization objective:

[0036] Feature alignment: Eliminate distribution differences between modes and establish a unified representation space.

[0037] Structure Preservation: Maintain the semantic relationship topology in the original data (such as the similarity hierarchy between disease subtypes).

[0038] Knowledge transfer: Facilitating the transfer of knowledge from high-information modalities (such as genomics) to low-information modalities (such as clinical scales).

[0039] The knowledge distillation process employs a bidirectional supervision strategy, constraining intermodal consistency at the feature level while guiding semantic alignment through diagnostic labels. Furthermore, the model incorporates an intermodal attention gating mechanism to dynamically adjust the intensity of knowledge transfer, avoiding information redundancy or excessive smoothing.

[0040] The proposed model is based on the deep network Transformer. Each unit consists of a shared self-attention module and a feedforward network pool corresponding to different modalities of data. Each feedforward network branch acts as an "expert model (several linear layers with self-attention mechanisms)" responsible for selectively learning different modalities of data and capturing long-range dependencies within each modality through the self-attention module, thereby mining disease markers for each modality. Unlike the traditional approach of using fixed-length token sequences to represent all images in a dataset, the proposed model proposes a dynamic input modeling module that adaptively uses the most appropriate number of tokens for each sample. For complex high-dimensional data, this module generates more tokens to fully represent its features; for low-dimensional data, it reduces the required token data, thus lowering the model's computational complexity. The module designs windows of different scales to fuse token information. It obtains multi-scale information by assigning windows of different scales to different multi-head self-attention groups within the Transformer and achieves dynamic information fusion by assigning weights to the multi-scale window branches. Furthermore, the proposed model transmits complementary information between features of different modalities through cross-modal contrastive knowledge distillation. Contrastive learning achieves feature alignment and knowledge distillation between different modalities by bringing positive samples closer and pushing negative samples further away. By introducing class label guidance in the cross-modal contrastive knowledge distillation module, for each sample, samples of the same class and similar to that sample are defined as positive samples, and samples of different classes are defined as negative samples. By bringing positive samples closer and pushing negative samples further away, feature relationships and structural information between different modalities can be aligned, thereby achieving better joint diagnostic performance based on the aligned features.

[0041] Example 2 Based on the same concept, the present invention also provides a multimodal medical data representation learning system, including an acquisition module, an input module and a processing module.

[0042] The acquisition module is used to collect multimodal medical data.

[0043] The input module is used to input multimodal medical data into the representation learning model to obtain aligned modal feature representations. The representation learning model includes a dynamic input modeling module, a dynamic window module, a multi-head attention network, a feedforward network, and a cross-modal comparative knowledge distillation module.

[0044] The processing module encodes the corresponding medical data based on the complexity of different modal medical data, obtaining multiple token sequences for each modality. The dynamic windowing module distributes and weights the multiple token sequences of each modality at different scales, resulting in multiple weighted sequences. The multi-head attention network captures and fuses the semantic association information of the multiple weighted sequences, obtaining a corresponding fusion vector. The feedforward network maps the fusion vector to obtain the corresponding modal feature representation. The cross-modal comparative knowledge distillation module aligns the modal feature representations of different modalities, obtaining aligned modal feature representations.

[0045] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0046] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multimodal medical data representation learning method, characterized in that, Includes the following steps: Collect multimodal medical data; Multimodal medical data is input into a representation learning model to obtain aligned modal feature representations; The representation learning model includes a dynamic input modeling module, a dynamic window module, a multi-head attention network, a feedforward network, and a cross-modal contrastive knowledge distillation module; The dynamic input modeling module encodes the corresponding medical data based on the complexity of different modal medical data, resulting in multiple token sequences for each modality; The dynamic window module distributes and weights multiple token sequences of each modality at different scales to obtain multiple weighted sequences; the multi-head attention network captures and fuses the semantic association information of the multiple weighted sequences to obtain the corresponding fusion vector; the feedforward network maps the fusion vector to obtain the corresponding modality feature representation. The cross-modal comparative knowledge distillation module performs feature alignment on the modal feature representations of different modalities to obtain aligned modal feature representations.

2. The multimodal medical data representation learning method as described in claim 1, characterized in that, The dynamic input modeling module uses a fixed number of tokens. For input data with different resolutions, a gating mechanism is used to activate different numbers of tokens.

3. The multimodal medical data representation learning method as described in claim 1, characterized in that, The dynamic window module allocates and weights multiple token sequences for each modality at different scales, including the following steps: Multiple tokens in the token sequence of each modality are allocated using sliding windows of different scales; The allocated tokens are weighted and merged according to dynamic weights to obtain multiple weighted sequences.

4. The multimodal medical data representation learning method as described in claim 1, characterized in that, The cross-modal comparative knowledge distillation module aligns the modal feature representations of different modalities to obtain aligned modal feature representations. This process includes the following steps: The teacher model is constructed, which is a 3-layer multilayer perceptron (MLP). The input is the modal feature representation of all modalities, and the output is the aligned modal feature representation. The student model is constructed using a 3-layer multilayer perceptron (MLP). The input is the modal feature representations of all other modalities when one modal feature is randomly missing, and the output is the aligned modal feature representations. For each sample, samples of the same category as the sample are defined as positive samples, and samples of different categories are defined as negative samples; Alignment is achieved using a contrastive learning loss function.

5. The multimodal medical data representation learning method as described in claim 1, characterized in that, The multimodal data includes image data, pathological data, genetic data, and diagnostic reports.

6. The multimodal medical data representation learning method as described in claim 5, characterized in that, Modal features of imaging data are represented as spatial local features, modal features of pathological data are represented as graph structures, and modal features of genetic data and diagnostic reports are represented as contextual semantic features.

7. A multimodal medical data representation learning system, characterized in that, include: The acquisition module is used to acquire multimodal medical data; The input module is used to input multimodal medical data into the representation learning model to obtain aligned modal feature representations; The representation learning model includes a dynamic input modeling module, a dynamic window module, a multi-head attention network, a feedforward network, and a cross-modal contrastive knowledge distillation module; The processing module is used to dynamically input the modeling module to encode the corresponding medical data based on the complexity of different modal medical data, and obtain multiple token sequences corresponding to each modality. The dynamic window module distributes and weights multiple token sequences of each modality at different scales to obtain multiple weighted sequences; the multi-head attention network captures and fuses the semantic association information of the multiple weighted sequences to obtain the corresponding fusion vector; the feedforward network maps the fusion vector to obtain the corresponding modality feature representation. The cross-modal comparative knowledge distillation module performs feature alignment on the modal feature representations of different modalities to obtain aligned modal feature representations.

Images