A medical multi-modal hidden space alignment fusion method and system based on VQ-GCN

Abstract

The application discloses a medical multi-modal hidden space alignment fusion method and system based on VQ-GCN, and belongs to the technical field of medical multi-modal fusion.The application solves the problem that a traditional fusion method is difficult to maintain semantic consistency in the same feature space.The method comprises the following steps: S1, constructing a feature extractor for each modality data respectively;S2, inputting each modality data into the corresponding feature extractor to obtain modality features of each modality data;S3, performing hidden space alignment on the modality features of each modality data to obtain final semantic coding features of each modality and final common semantic features of all modalities; and S4, performing hierarchical cross-modal attention feature fusion on the final semantic coding features of each modality and the final common semantic features of all modalities to obtain a medical multi-modal feature fusion result.The method can be applied to medical multi-modal hidden space alignment fusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical multi-modal fusion, and particularly relates to a medical multi-modal hidden space alignment fusion method and system based on VQ-GCN (vector quantization-graph convolutional neural network). BACKGROUND

[0002] With the continuous development of medical imaging technology, more and more clinical diagnosis and treatment rely on medical data of multiple modalities, such as image data, pathological data and examination reports. These data provide more comprehensive patient information for clinicians, thereby helping to improve the accuracy of diagnosis and the effectiveness of treatment plans. However, medical data of different modalities usually have different resolutions, scales and feature expression methods, resulting in the complexity of information fusion and comprehensive analysis. At present, the technology of multi-modal data fusion mainly includes early fusion, late fusion and joint alignment methods. Early fusion directly combines data of different modalities at the feature level, while late fusion fuses at the decision level after independent processing. Although these methods improve the utilization efficiency of data to some extent, due to the significant semantic difference and distribution inconsistency between modal data, traditional fusion methods are difficult to maintain semantic consistency in the same feature space, resulting in information loss and diagnosis accuracy decline in the fusion process. Therefore, how to ensure that multi-modal data maintains semantic consistency in the same feature space is still a challenge. SUMMARY

[0003] The purpose of the application is to solve the problem that traditional fusion methods are difficult to maintain semantic consistency in the same feature space, and a medical multi-modal hidden space alignment fusion method and system based on VQ-GCN are proposed.

[0004] The technical scheme adopted by the application to solve the above technical problems is:

[0005] Based on one aspect of the application, a medical multi-modal hidden space alignment fusion method based on VQ-GCN, the method specifically includes the following steps:

[0006] Step S1, the i-th modal data is denoted as d i , i = 1, 2, …, n, n is the number of modalities, a feature extractor is constructed for each modal data, and the feature extractor corresponding to the i-th modal data is denoted as b i ;

[0007] Step S2, input the modal data d i into the corresponding feature extractor b i , and obtain the extracted modal feature f i ;

[0008] Step S3: perform latent space alignment on the features of each modality to obtain the semantic encoding features of each modality and the common semantic features of all modalities;

[0009] Step S4: Perform hierarchical cross-modal attention feature fusion on the semantic encoding features of each modality and the common semantic features of all modalities to obtain the medical multimodal feature fusion result.

[0010] According to another aspect of the present invention, a medical multimodal latent space alignment and fusion system based on VQ-GCN is provided. The system includes a feature extraction module, a feature alignment module, and a feature fusion module, wherein:

[0011] The feature extraction module is used to extract the modal features of each modal data;

[0012] The working process of the feature extraction module is:

[0013] The i-th modal data is recorded as d i , i=1,2,…,n, construct a feature extractor for each modal data, and record the feature extractor corresponding to the i-th modal data as b i , the modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ;

[0014] Where n = 3, d1 represents imaging data, d2 represents pathological data, and d3 represents examination report;

[0015] The feature alignment module is used to obtain the semantic coding features of each modality and the common semantic features of all modalities;

[0016] The working process of the feature alignment module is:

[0017] Step S31: Randomly initialize a codebook CB. r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB;

[0018] Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ;

[0019] Step S33: For the semantic coding feature m i Any row vector m in ij , calculate the row vector m respectively ijThe similarity score of each row vector in the codebook CB is obtained, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ;

[0020] Semantic encoding feature m i The similarity score similarity(m ij ,cb k )for:

[0021]

[0022] Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, “×” represents the multiplication of numbers, and “·” represents the multiplication of vectors;

[0023] Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m i After replacing each row vector in the , we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ;

[0024] Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively;

[0025] Step S34: concatenate the modal features f1, f2, and f3 in the channel dimension, use the concatenation result as the input of the multimodal shared semantic encoder C, and output the multimodal shared semantic encoding feature P through the multimodal shared semantic encoder C;

[0026] The specific process of step S34 is as follows:

[0027] The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a cross-attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image and text encoding results are then passed through the cross-attention module to extract shared information between different modalities. Finally, the information extraction results are passed through the MLP to obtain the shared semantic coding feature P.

[0028] Step S35: The multimodal shared semantic coding feature P and the semantic coding features vqm1 and vqm after vector quantization of each modality are converted into 23, vqm3 as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, vqm′3 of each modality and the final common semantic coding feature P′ of all modalities;

[0029] The feature fusion module is used to perform hierarchical cross-modal attention feature fusion on the semantic coding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result;

[0030] The working process of the feature fusion module is as follows:

[0031] Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is i The constructed feature normalization layer is recorded as norm i ;

[0032] Step S42: The semantic encoding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ;

[0033] Step S43: Normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ;

[0034] Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ;

[0035] Step S45: input h1, h2 and h3 into the adaptive attention fusion network, and output the multimodal feature fusion result through the adaptive attention fusion network;

[0036] The adaptive attention fusion network includes MLP and Cross-Attention modules;

[0037] Concatenate h1, h2, and h3 in the channel dimension and input the concatenation result into the MLP to obtain the multimodal adaptive weight α;

[0038] Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module;

[0039] The weight alpha is multiplied by the feature Gamma to obtain the final medical multi-modal feature fusion result.

[0040] The beneficial effects of the present application are:

[0041] The present application unifies multi-modal medical data into the same hidden space by using the vector quantization mode, realizes information interaction by using the graph convolutional neural network, realizes multi-modal data alignment by maximizing-minimizing mutual information, and finally realizes cross-modal interaction and multi-modal data fusion based on Attention, realizes the alignment and fusion of multi-modal medical data in the same hidden space, and makes different modal medical data maintain semantic consistency in the same feature space. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of a medical multi-modal hidden space alignment and fusion method based on VQ-GCN of the present application;

[0043] Figure 2 is a flowchart of the cross-modal feature alignment part based on VQ-GCN. DETAILED DESCRIPTION

[0044] Specific implementation one: combined with Figure 1 This embodiment is described. The medical multi-modal hidden space alignment and fusion method based on VQ-GCN described in this embodiment, the method specifically includes the following steps:

[0045] Step S1, the i-th modal data is denoted as d i , i=1, 2, …, n, n is the number of modalities, a feature extractor is constructed for each modal data, and the feature extractor corresponding to the i-th modal data is denoted as b i ;

[0046] Step S2, input the modal data d i into the corresponding feature extractor b i , and obtain the extracted modal feature f i ;

[0047] Step S3, align the features of each modality in the hidden space, respectively obtain the semantic encoding features of each modality and the common semantic features of all modalities;

[0048] Step S4, perform hierarchical cross-modal attention feature fusion on the semantic encoding features of each modality and the common semantic features of all modalities to obtain the medical multi-modal feature fusion result.

[0049] The medical multi-modal feature fusion result obtained by the present application can be used for downstream tasks.

[0050] Specific embodiment 2: This embodiment differs from specific embodiment 1 in that the modality number n is 3, d1 represents image data, d2 represents pathological data, and d3 represents the examination report;

[0051] The feature extractor corresponding to d1 is DenseNet, the feature extractor corresponding to d2 is CLAM, and the feature extractor corresponding to d3 is Transformer.

[0052] Other steps and parameters are the same as those in the first embodiment.

[0053] Specific embodiment 3: This embodiment differs from specific embodiment 1 or 2 in that the specific process of step S3 is as follows:

[0054] Step S31: Randomly initialize a codebook CB. r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB;

[0055] Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ;

[0056] Step S33: For the semantic coding feature m i Any row vector m in ij , calculate the row vector m respectively ij The similarity score of each row vector in the codebook CB is obtained, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ;

[0057] Semantic encoding feature m i The similarity score similarity(m ij ,cb k )for:

[0058]

[0059] Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, “×” represents the multiplication of numbers, and “·” represents the multiplication of vectors;

[0060] Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m iAfter replacing each row vector in the , we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ;

[0061] Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively; the semantic information of different modalities is mapped in the same latent space;

[0062] Step S34: concatenate the modal features f1, f2, and f3 along the channel dimension, and use the concatenation result as the input of the multimodal shared semantic encoder C. The multimodal shared semantic encoder C outputs the multimodal shared semantic encoding feature P.

[0063] The specific process of step S34 is as follows:

[0064] The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a cross-attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image and text encoding results are then passed through the cross-attention module to extract shared information between different modalities. Finally, the information extraction results are passed through the MLP to obtain the shared semantic coding feature P.

[0065] The image encoding module and text encoding module are both the image encoder and text encoder of the CLIP model;

[0066] Step S35: Use the multimodal shared semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, and vqm′3 of each modality and the final shared semantic coding feature P′ of all modalities.

[0067] Other steps and parameters are the same as those in the first or second embodiment.

[0068] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the graph convolutional neural network maximizes the loss function of the mutual information between the multimodal common semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality. for:

[0069]

[0070] Among them, similarity(P,vqm i) is the multimodal shared semantic encoding feature P and the semantic encoding feature vqm i Similarity score of

[0071] The graph convolutional neural network minimizes the loss function of the mutual information between the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality for:

[0072]

[0073] Among them, similarity(vqm i ,vqm i’ ) is vqm i With vqm i’ Similarity score.

[0074] The other steps and parameters are the same as those in the first to third embodiments.

[0075] Specific implementation method five: Combination Figure 2 This embodiment is different from any one of the first to fourth embodiments in that the semantic encoder s1 is a ResNet network, the semantic encoder s2 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM.

[0076] And the semantic encoder s i The training process is:

[0077] The semantic coding feature vqm after vector quantization i Input the decoder sd corresponding to the i-th modality i , the reconstruction loss is calculated based on the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality;

[0078] The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

[0079] The other steps and parameters are the same as those in the first to fourth embodiments.

[0080] In this embodiment, the i-th mode corresponds to the decoder sd i The structure is that the i-th modality corresponds to the semantic encoder s i inversion of the structure.

[0081] Specific embodiment 6: This embodiment differs from any one of specific embodiments 1 to 5 in that the specific process of step S4 is as follows:

[0082] Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is iThe constructed feature normalization layer is recorded as norm i ;

[0083] The feature normalization layer in this invention is BatchNorm;

[0084] Step S42: The semantic encoding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ;

[0085] Step S43: Normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ;

[0086] It should be noted that the cross-modal interaction module in the present invention adopts Cross-Attention;

[0087] Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ;

[0088] Step S45: input h1, h2 and h3 into the adaptive attention fusion network (Fusion), and output the multimodal feature fusion result through the adaptive attention fusion network.

[0089] The other steps and parameters are the same as those in the first to fifth embodiments.

[0090] Specific embodiment 7: This embodiment differs from any one of specific embodiments 1 to 6 in that the adaptive attention fusion network includes an MLP (multi-layer perceptron) and a Cross-Attention module;

[0091] Concatenate h1, h2, and h3 in the channel dimension and input the concatenation result into the MLP to obtain the multimodal adaptive weight α;

[0092] Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module;

[0093] Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

[0094] The other steps and parameters are the same as those in the first to sixth embodiments.

[0095] Specific implementation eight: a medical multi-modal hidden space alignment fusion system based on VQ-GCN is described in this embodiment. The system comprises a feature extraction module, a feature alignment module and a feature fusion module, wherein:

[0096] The feature extraction module is used to extract modal features of each modal data;

[0097] The working process of the feature extraction module is as follows:

[0098] Let the i-th modal data be d i , i = 1, 2, …, n, construct a feature extractor for each modal data respectively, and let the feature extractor corresponding to the i-th modal data be b i , input the modal data d i into the corresponding feature extractor b i , and obtain the extracted modal features f i ;

[0099] Wherein, n = 3, d1 represents image data, d2 represents pathological data, and d3 represents examination report;

[0100] The feature alignment module is used to obtain semantic encoding features of each modal and common semantic features of all modalities;

[0101] The working process of the feature alignment module is as follows:

[0102] Step S31, randomly initialize a codebook CB, r is the number of row vectors in the codebook CB, and q is the length of each row vector in the codebook CB;

[0103] Step S32, input the modal features f i into the corresponding semantic encoder s i , and obtain the semantic encoding features m i of the i-th modal;

[0104] Step S33, for any one row vector m i in the semantic encoding features m ij , calculate the similarity score of the row vector m ij with each row vector in the codebook CB, and obtain the corresponding row vector in the codebook CB with the highest similarity score, and then replace the row vector m i in the semantic encoding features m ij with the obtained row vector;

[0105] The similarity score of the j-th row vector in the semantic encoding features m i and the k-th row vector in the codebook CB is similarity(m ij , cbk ) is:

[0106]

[0107] wherein, cb k is the kth column vector in the codebook CB, ||·|| represents the 2-norm, "x" represents the multiplication of numbers, and "·" represents the multiplication of vectors;

[0108] After traversing each column vector in the semantic coding feature m i , each column vector in the semantic coding feature m i is replaced respectively, the vector-quantized semantic coding feature vqm i corresponding to the ith modality is obtained.

[0109] Similarly, the vector-quantized semantic coding feature corresponding to each modality is obtained respectively; and the semantic information of different modalities is mapped in the same latent space.

[0110] In step S34, the modality features f1, f2 and f3 are concatenated in the channel dimension, and the connection result is taken as the input of the multi-modal shared semantic encoder C, and the multi-modal shared semantic coding feature P is output through the multi-modal shared semantic encoder C.

[0111] The specific process of the step S34 is as follows:

[0112] The multi-modal shared semantic encoder C includes an image coding module, a text coding module, a Cross-Attention module and an MLP; the modality feature of the image data and the modality feature of the pathological data are coded through the image coding module, the modality feature of the examination report is coded through the text coding module, then the image coding result and the text coding result are subjected to common information extraction between different modalities through the Cross-Attention module, and finally the information extraction result is subjected to the MLP to obtain the shared semantic coding feature P.

[0113] The image coding module and the text coding module are both image encoders and text encoders of the CLIP model.

[0114] In step S35, the multi-modal shared semantic coding feature P and the vector-quantized semantic coding features vqm1, vqm2 and vqm i are taken as nodes to construct a graph convolutional neural network, semantic information transmission is performed in the graph convolutional neural network, and the final semantic coding features vqm'1, vqm'2 and vqm'3 of each modality and the final shared semantic coding feature P' of all modalities are obtained.

[0115] The feature fusion module is used for hierarchical cross-modal attention feature fusion on the semantic encoding features of each modality and the common semantic features of all modalities, to obtain a medical multi-modal feature fusion result.

[0116] The working process of the feature fusion module is as follows:

[0117] In step S41, a feature normalization layer is constructed for the semantic encoding feature of each modality, wherein the semantic encoding feature vqm' of the i-th modality is input into the feature normalization layer norm i The constructed feature normalization layer is denoted as norm i ;

[0118] The feature normalization layer in the present application is BatchNorm.

[0119] In step S42, the semantic encoding feature vqm' of the i-th modality is input into the corresponding feature normalization layer norm i , to obtain the feature normalization result nvqm i of the semantic encoding feature vqm' of the i-th modality. i i ;

[0120] In step S43, a cross-modal interaction module cma o is constructed for each feature normalization result nvqm i .

[0121] It should be noted that the cross-modal interaction module in the present application adopts Cross-Attention.

[0122] In step S44, the multi-modal final common semantic encoding feature P' and the nvqm i are input into the cma i to perform cross-modal interaction, to obtain a cross-modal interaction feature h i .

[0123] In step S45, the h1, h2 and h3 are input into an adaptive attention fusion network (Fusion), and the multi-modal feature fusion result is output through the adaptive attention fusion network.

[0124] The adaptive attention fusion network includes an MLP (Multi-Layer Perceptron) and a Cross-Attention module.

[0125] The h1, h2 and h3 are connected in the channel dimension, and the connection result is input into the MLP, to obtain a multi-modal adaptive weight a.

[0126] The h1, h2 and h3 are input into the Cross-Attention module, and the attention fusion feature G is output through the Cross-Attention module.​

[0127] The weight alpha is multiplied by the feature Gamma to obtain the final medical multi-modal feature fusion result.

[0128] Specific implementation nine: the difference between this embodiment and specific implementation eight is that the feature extractor corresponding to the image data d1 is DenseNet, the feature extractor corresponding to the pathological data d2 is CLAM, and the feature extractor corresponding to the examination report d3 is Transformer.

[0129] The other steps and parameters are the same as those in specific implementation eight.

[0130] Specific implementation ten: the difference between this embodiment and specific implementation eight or nine is that the semantic encoder s1 is a ResNet network, the semantic encoder s3 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM.

[0131] The training process of the semantic encoder s i is as follows:

[0132] The vector-quantized semantic encoding feature vqm i is input into the decoder sd i corresponding to the i-th modality, and the reconstruction loss is calculated according to the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality.

[0133] The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

[0134] The other steps and parameters are the same as those in specific implementation eight or nine.

[0135] The above examples of the present application are only to illustrate the calculation model and calculation process of the present application, and are not limited to the embodiments of the present application. For those skilled in the art, other different forms of changes or variations can be made on the basis of the above description, and it is impossible to enumerate all the embodiments here. Any obvious changes or variations derived from the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A medical multimodal latent space alignment and fusion method based on VQ-GCN, characterized by: The method specifically comprises the following steps: Step S1: record the i-th modal data as d i , i=1,2,…,n, n is the number of modes, a feature extractor is constructed for each mode data, and the feature extractor corresponding to the i-th mode data is recorded as b i ; The modality number n is set to 3, d1 represents imaging data, d2 represents pathological data, and f3 represents examination report; Step S2: Modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ; Step S3: perform latent space alignment on the features of each modality to obtain the semantic encoding features of each modality and the common semantic features of all modalities; The specific process of step S3 is: Step S31: Randomly initialize a codebook r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB; Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ; Step S33: For the semantic coding feature m i Any row vector m in ij , calculate the row vector m respectively ij The similarity score of each row vector in the codebook CB is obtained, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ; Semantic encoding feature m i The similarity score similarity(m ij ,cb k )for: Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, "×" represents the multiplication of numbers, and "·" represents the multiplication of vectors; Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m i After replacing each row vector in the , we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ; Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively; Step S34: concatenate the modal features f1, f2, and f3 along the channel dimension, use the concatenation result as the input of the multimodal shared semantic encoder C, and output the multimodal shared semantic encoding feature P through the multimodal shared semantic encoder C; The specific process of step S34 is as follows: The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a cross-attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image and text encoding results are then passed through the cross-attention module to extract shared information between different modalities. Finally, the information extraction results are passed through the MLP to obtain the shared semantic coding feature P. Step S35: Use the multimodal shared semantic coding feature P and the vector quantized semantic coding features vqm1, vqm2, and vqm3 of each modality as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, and vqm′3 of each modality and the final shared semantic coding feature P′ of all modalities; Step S4: performing hierarchical cross-modal attention feature fusion on the semantic encoding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result; The specific process of step S4 is as follows: Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is i The constructed feature normalization layer is recorded as norm i ; Step S42: The semantic coding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of mvqm i ; Step S43: Normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ; Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ; Step S45: input h1, h2 and h3 into the adaptive attention fusion network, and output the multimodal feature fusion result through the adaptive attention fusion network.

2. A medical multimodal latent space alignment and fusion method based on VQ-GCN according to claim 1, characterized in that: The feature extractor corresponding to the image data d1 is DenseNet, the feature extractor corresponding to the pathological data d2 is CLAM, and the feature extractor corresponding to the examination report d3 is Transformer.

3. The medical multimodal latent space alignment and fusion method based on VQ-GCN according to claim 2 is characterized in that: The graph convolutional neural network maximizes the loss function of the mutual information between the multimodal common semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality. for: Among them, similarity(P,vqm i ) is the multimodal shared semantic encoding feature P and the semantic encoding feature vqm i Similarity score of The graph convolutional neural network minimizes the loss function of the mutual information between the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality for: Among them, similarity(vqm i ,vqm i’ ) is vqm i With vqm i’ Similarity score.

4. The medical multimodal latent space alignment and fusion method based on VQ-GCN according to claim 3 is characterized in that: The semantic encoder s1 is a ResNet network, the semantic encoder s2 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM; And the semantic encoder s i The training process is: The semantic coding feature vqm after vector quantization i Input the decoder sd corresponding to the i-th modality i , the reconstruction loss is calculated based on the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality; The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

5. The medical multimodal latent space alignment and fusion method based on VQ-GCN according to claim 4 is characterized in that: The adaptive attention fusion network includes MLP and Cross-Attention modules; Concatenate h1, h2, and h3 in the channel dimension and input the concatenation result into the MLP to obtain the multimodal adaptive weight α; Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module; Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

6. A medical multimodal latent space alignment and fusion system based on VQ-GCN, characterized by: The system includes a feature extraction module, a feature alignment module and a feature fusion module, wherein: The feature extraction module is used to extract the modal features of each modal data; The working process of the feature extraction module is: The i-th modal data is recorded as d i , i=1,2,…,n, construct a feature extractor for each modal data, and record the feature extractor corresponding to the i-th modal data as b i , the modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ; Where n = 3, d1 represents imaging data, d2 represents pathological data, and d3 represents examination report; The feature alignment module is used to obtain the semantic coding features of each modality and the common semantic features of all modalities; The working process of the feature alignment module is: Step S31: Randomly initialize a codebook r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB; Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ; Step S33: For the semantic coding feature m i Any row vector m in ij , calculate the row vector m respectively ij The similarity score of each row vector in the codebook CB is obtained, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ; Semantic encoding feature m i The similarity score similarity(m ij ,cb k )for: Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, "×" represents the multiplication of numbers, and "·" represents the multiplication of vectors; Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m i After replacing each row vector in the , we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ; Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively; Step S34: concatenate the modal features f1, f2, and f3 along the channel dimension, use the concatenation result as the input of the multimodal shared semantic encoder C, and output the multimodal shared semantic encoding feature P through the multimodal shared semantic encoder C; The specific process of step S34 is as follows: The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a cross-attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image and text encoding results are then passed through the cross-attention module to extract shared information between different modalities. Finally, the information extraction results are passed through the MLP to obtain the shared semantic coding feature P. Step S35: Use the multimodal shared semantic coding feature P and the vector quantized semantic coding features vqm1, vqm2, and vqm3 of each modality as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, and vqm′3 of each modality and the final shared semantic coding feature P′ of all modalities; The feature fusion module is used to perform hierarchical cross-modal attention feature fusion on the semantic coding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result; The working process of the feature fusion module is as follows: Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is i The constructed feature normalization layer is recorded as norm i ; Step S42: The semantic coding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ; Step S43: Normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ; Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ; Step S45: input h1, h2 and h3 into the adaptive attention fusion network, and output the multimodal feature fusion result through the adaptive attention fusion network; The adaptive attention fusion network includes MLP and Cross-Attention modules; Concatenate h1, h2, and h3 in the channel dimension and input the concatenation result into the MLP to obtain the multimodal adaptive weight α; Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module; Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

7. The medical multimodal latent space alignment and fusion system based on VQ-GCN according to claim 6, characterized in that: The feature extractor corresponding to the image data d1 is DenseNet, the feature extractor corresponding to the pathological data d2 is CLAM, and the feature extractor corresponding to the examination report d3 is Transformer.

8. The medical multimodal latent space alignment and fusion system based on VQ-GCN according to claim 7 is characterized in that: The semantic encoder s1 is a ResNet network, the semantic encoder s2 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM; And the semantic encoder s i The training process is: The semantic coding feature vqm after vector quantization i Input the decoder sd corresponding to the i-th modality i , the reconstruction loss is calculated based on the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality; The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

Images