Medical multi-mode hidden space alignment fusion method and system based on VQ-GCN

Abstract

The invention discloses a medical multi-mode hidden space alignment fusion method and system based on VQ-GCN, and belongs to the technical field of medical multi-mode fusion. The method solves the problem that a traditional fusion method is difficult to maintain semantic consistency in the same feature space. The method specifically comprises the following steps: S1, respectively constructing a feature extractor for each modal data; s2, inputting each piece of modal data into a corresponding feature extractor to obtain a modal feature of each piece of modal data; s3, carrying out hidden space alignment on the modal features of the modal data, and respectively obtaining final semantic coding features of each modal and final common semantic features of all modals; and S4, carrying out hierarchical cross-modal attention feature fusion on the final semantic coding features of each modal and the final common semantic features of all modals to obtain a medical multi-modal feature fusion result. The method can be applied to medical multi-mode hidden space alignment fusion.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical multimodal fusion, and specifically relates to a medical multimodal latent space alignment fusion method and system based on VQ-GCN (vector quantization-graph convolutional neural network). Background Art

[0002] With the continuous development of medical imaging technology, more and more clinical diagnosis and treatment rely on medical data of multiple modalities, such as imaging data, pathological data, and examination reports. These data provide clinicians with more comprehensive patient information, which helps to improve the accuracy of diagnosis and the effectiveness of treatment plans. However, medical data of different modalities usually have different resolutions, scales, and feature expressions, which leads to the complexity of information fusion and comprehensive analysis. At present, the technologies of multimodal data fusion mainly include early fusion, late fusion, and joint alignment. Early fusion directly merges data of different modalities at the feature level, while late fusion performs decision-level fusion after independent processing. Although these methods have improved the utilization efficiency of data to a certain extent, due to the significant semantic differences and distribution inconsistency between modal data, traditional fusion methods are difficult to maintain semantic consistency in the same feature space, resulting in information loss and decreased diagnostic accuracy during the fusion process. Therefore, how to ensure that multimodal data maintains semantic consistency in the same feature space is still a challenge. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that traditional fusion methods are difficult to maintain semantic consistency in the same feature space, and propose a medical multimodal latent space alignment fusion method and system based on VQ-GCN.

[0004] The technical solution adopted by the present invention to solve the above technical problems is:

[0005] According to one aspect of the present invention, a medical multimodal latent space alignment fusion method based on VQ-GCN is provided, and the method specifically comprises the following steps:

[0006] Step S1: record the i-th modal data as d i , i = 1, 2, ..., n, n is the number of modes, a feature extractor is constructed for each mode data, and the feature extractor corresponding to the i-th mode data is recorded as b i ;

[0007] Step S2: Modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ;

[0008] Step S3: perform latent space alignment on the features of each modality to obtain the semantic coding features of each modality and the common semantic features of all modalities;

[0009] Step S4: perform hierarchical cross-modal attention feature fusion on the semantic encoding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result.

[0010] According to another aspect of the present invention, a medical multimodal latent space alignment and fusion system based on VQ-GCN, the system includes a feature extraction module, a feature alignment module and a feature fusion module, wherein:

[0011] The feature extraction module is used to extract the modal features of each modal data;

[0012] The working process of the feature extraction module is:

[0013] The i-th modal data is recorded as d i , i = 1, 2, ..., n, construct a feature extractor for each modal data, and record the feature extractor corresponding to the i-th modal data as b i , the modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ;

[0014] Where n = 3, d1 represents imaging data, d2 represents pathological data, and d3 represents examination report;

[0015] The feature alignment module is used to obtain the semantic coding features of each modality and the common semantic features of all modalities;

[0016] The working process of the feature alignment module is:

[0017] Step S31: randomly initialize a codebook CB. r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB;

[0018] Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ;

[0019] Step S33: for the semantic coding feature m i Any row vector m in ij , respectively calculate the row vector m ijThe similarity score of each row vector in the codebook CB is calculated, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ;

[0020] Semantic encoding feature m i The similarity score similarity(m ij ,cb k )for:

[0021]

[0022] Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, “×” represents the multiplication of numbers, and “·” represents the multiplication of vectors;

[0023] Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m i After replacing each row vector in, we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ;

[0024] Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively;

[0025] Step S34, the modal features f1, f2 and f3 are connected in the channel dimension, and the connection result is used as the input of the multimodal common semantic encoder C, and the multimodal common semantic encoder C outputs the multimodal common semantic coding feature P;

[0026] The specific process of step S34 is as follows:

[0027] The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a Cross-Attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image encoding results and the text encoding results are then used to extract shared information between different modalities through the Cross-Attention module. Finally, the information extraction results are used to obtain shared semantic coding features P through the MLP.

[0028] Step S35: The multi-modal common semantic coding feature P and the semantic coding features vqm1 and vqm 23, vqm3 as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, vqm′3 of each modality and the final common semantic coding feature P′ of all modalities;

[0029] The feature fusion module is used to perform hierarchical cross-modal attention feature fusion on the semantic coding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result;

[0030] The working process of the feature fusion module is as follows:

[0031] Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is i The constructed feature normalization layer is denoted as norm i ;

[0032] Step S42: The semantic coding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ;

[0033] Step S43: normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ;

[0034] Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ;

[0035] Step S45, input h1, h2 and h3 into the adaptive attention fusion network, and output the multimodal feature fusion result through the adaptive attention fusion network;

[0036] The adaptive attention fusion network includes MLP and Cross-Attention modules;

[0037] Connect h1, h2 and h3 in the channel dimension and input the connection result into MLP to obtain the multimodal adaptive weight α;

[0038] Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module;

[0039] Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

[0040] The beneficial effects of the present invention are:

[0041] The present invention uses vector quantization to unify multimodal medical data into the same latent space, then adopts graph convolutional neural network to realize information interaction, realizes multimodal data alignment by maximizing-minimizing mutual information, and finally realizes cross-modal interaction and multimodal data fusion based on Attention, realizing the alignment and fusion of multimodal medical data in the same latent space, so that different modal medical data can maintain semantic consistency in the same feature space. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a medical multimodal latent space alignment and fusion method based on VQ-GCN of the present invention;

[0043] Figure 2 This is a flowchart of the cross-modal feature alignment part based on VQ-GCN. DETAILED DESCRIPTION

[0044] Specific implementation method 1: Combination Figure 1 The present embodiment describes a medical multimodal latent space alignment fusion method based on VQ-GCN, and the method specifically comprises the following steps:

[0045] Step S1: record the i-th modal data as d i , i = 1, 2, ..., n, n is the number of modes, a feature extractor is constructed for each mode data, and the feature extractor corresponding to the i-th mode data is recorded as b i ;

[0046] Step S2: Modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ;

[0047] Step S3: perform latent space alignment on the features of each modality to obtain the semantic coding features of each modality and the common semantic features of all modalities;

[0048] Step S4: perform hierarchical cross-modal attention feature fusion on the semantic encoding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result.

[0049] The medical multimodal feature fusion results obtained by the present invention can be used for downstream tasks.

[0050] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the modality number n is 3, d1 represents image data, d2 represents pathological data, and d3 represents the examination report;

[0051] The feature extractor corresponding to d1 is DenseNet, the feature extractor corresponding to d2 is CLAM, and the feature extractor corresponding to d3 is Transformer.

[0052] The other steps and parameters are the same as those in the first embodiment.

[0053] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the specific process of step S3 is as follows:

[0054] Step S31: randomly initialize a codebook CB. r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB;

[0055] Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ;

[0056] Step S33: for the semantic coding feature m i Any row vector m in ij , respectively calculate the row vector m ij The similarity score of each row vector in the codebook CB is calculated, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ;

[0057] Semantic encoding feature m i The similarity score similarity(m ij ,cb k )for:

[0058]

[0059] Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, “×” represents the multiplication of numbers, and “·” represents the multiplication of vectors;

[0060] Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m iAfter replacing each row vector in, we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ;

[0061] Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively; the mapping of semantic information of different modalities in the same latent space is realized;

[0062] Step S34, concatenate the modal features f1, f2 and f3 in the channel dimension, use the concatenation result as the input of the multimodal common semantic encoder C, and output the multimodal common semantic coding feature P through the multimodal common semantic encoder C;

[0063] The specific process of step S34 is as follows:

[0064] The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a Cross-Attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image encoding results and the text encoding results are then used to extract shared information between different modalities through the Cross-Attention module. Finally, the information extraction results are used to obtain shared semantic coding features P through the MLP.

[0065] The image encoding module and the text encoding module are both the image encoder and text encoder of the CLIP model;

[0066] Step S35: Use the multimodal common semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, vqm′3 of each modality and the final common semantic coding feature P′ of all modalities.

[0067] The other steps and parameters are the same as those in the first or second embodiment.

[0068] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the graph convolutional neural network maximizes the loss function of the mutual information between the multimodal common semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality. for:

[0069]

[0070] Among them, similarity(P,vqm i) is the multimodal shared semantic encoding feature P and the semantic encoding feature vqm i Similarity score of

[0071] The graph convolutional neural network minimizes the loss function of the mutual information between the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality. for:

[0072]

[0073] Among them, similarity(vqm i ,vqm i’ ) is vqm i With vqm i’ Similarity score.

[0074] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0075] Specific implementation method five: Combination Figure 2 This embodiment is described. The difference between this embodiment and any one of the specific embodiments 1 to 4 is that the semantic encoder s1 is a ResNet network, the semantic encoder s2 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM;

[0076] And the semantic encoder s i The training process is:

[0077] The vector quantized semantic coding feature vqm i Input the decoder sd corresponding to the i-th mode i , the reconstruction loss is calculated based on the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality;

[0078] The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

[0079] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0080] In this implementation, the i-th mode corresponds to the decoder sd i The structure is that the i-th modality corresponds to the semantic encoder s i inversion of the structure.

[0081] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the specific process of step S4 is as follows:

[0082] Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is iThe constructed feature normalization layer is denoted as norm i ;

[0083] The feature normalization layer in the present invention is BatchNorm;

[0084] Step S42: The semantic coding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ;

[0085] Step S43: normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ;

[0086] It should be noted that the cross-modal interaction module in the present invention adopts Cross-Attention;

[0087] Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ;

[0088] Step S45, input h1, h2 and h3 into the adaptive attention fusion network (Fusion), and output the multimodal feature fusion result through the adaptive attention fusion network.

[0089] The other steps and parameters are the same as those in Specific Implementations 1 to 5.

[0090] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the adaptive attention fusion network includes an MLP (multi-layer perceptron) and a Cross-Attention module;

[0091] Connect h1, h2 and h3 in the channel dimension and input the connection result into MLP to obtain the multimodal adaptive weight α;

[0092] Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module;

[0093] Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

[0094] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0095] Specific implementation eight: This implementation is a medical multimodal latent space alignment and fusion system based on VQ-GCN, the system includes a feature extraction module, a feature alignment module and a feature fusion module, wherein:

[0096] The feature extraction module is used to extract the modal features of each modal data;

[0097] The working process of the feature extraction module is:

[0098] The i-th modal data is recorded as d i , i = 1, 2, ..., n, construct a feature extractor for each modal data, and record the feature extractor corresponding to the i-th modal data as b i , the modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ;

[0099] Where n = 3, d1 represents imaging data, d2 represents pathological data, and d3 represents examination report;

[0100] The feature alignment module is used to obtain the semantic coding features of each modality and the common semantic features of all modalities;

[0101] The working process of the feature alignment module is:

[0102] Step S31: randomly initialize a codebook CB. r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB;

[0103] Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ;

[0104] Step S33: for the semantic coding feature m i Any row vector m in ij , respectively calculate the row vector m ij The similarity score of each row vector in the codebook CB is calculated, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ;

[0105] Semantic encoding feature m i The similarity score similarity(m ij ,cbk )for:

[0106]

[0107] Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, “×” represents the multiplication of numbers, and “·” represents the multiplication of vectors;

[0108] Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m i After replacing each row vector in, we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ;

[0109] Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively; the mapping of semantic information of different modalities in the same latent space is realized;

[0110] Step S34, concatenate the modal features f1, f2 and f3 in the channel dimension, use the concatenation result as the input of the multimodal common semantic encoder C, and output the multimodal common semantic coding feature P through the multimodal common semantic encoder C;

[0111] The specific process of step S34 is as follows:

[0112] The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a Cross-Attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image encoding results and the text encoding results are then used to extract shared information between different modalities through the Cross-Attention module. Finally, the information extraction results are used to obtain shared semantic coding features P through the MLP.

[0113] The image encoding module and the text encoding module are both the image encoder and text encoder of the CLIP model;

[0114] Step S35: The multi-modal common semantic coding feature P and the semantic coding features vqm1, vqm2, vqm i As nodes, we construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, vqm′3 of each modality and the final common semantic coding feature P′ of all modalities;

[0115] The feature fusion module is used to perform hierarchical cross-modal attention feature fusion on the semantic coding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result;

[0116] The working process of the feature fusion module is as follows:

[0117] Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is i The constructed feature normalization layer is denoted as norm i ;

[0118] The feature normalization layer in the present invention is BatchNorm;

[0119] Step S42: The semantic coding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ;

[0120] Step S43: normalize the result nvqm for each feature o Construct cross-modal interaction modules cma separately i ;

[0121] It should be noted that the cross-modal interaction module in the present invention adopts Cross-Attention;

[0122] Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ;

[0123] Step S45, input h1, h2 and h3 into an adaptive attention fusion network (Fusion), and output a multimodal feature fusion result through the adaptive attention fusion network;

[0124] The adaptive attention fusion network includes MLP (multi-layer perceptron) and Cross-Attention modules;

[0125] Connect h1, h2 and h3 in the channel dimension and input the connection result into MLP to obtain the multimodal adaptive weight α;

[0126] Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module;

[0127] Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

[0128] Specific implementation method nine: This implementation method is different from specific implementation method eight in that the feature extractor corresponding to the image data d1 is DenseNet, the feature extractor corresponding to the pathological data d2 is CLAM, and the feature extractor corresponding to the examination report d3 is Transformer.

[0129] The other steps and parameters are the same as those in the eighth embodiment.

[0130] Specific embodiment ten: This embodiment is different from specific embodiment eight or nine in that the semantic encoder s1 is a ResNet network, the semantic encoder s3 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM;

[0131] And the semantic encoder s i The training process is:

[0132] The vector quantized semantic coding feature vqm i Input the decoder sd corresponding to the i-th mode i , the reconstruction loss is calculated based on the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality;

[0133] The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

[0134] The other steps and parameters are the same as those in the eighth or ninth embodiment.

[0135] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A medical multimodal latent space alignment fusion method based on VQ-GCN, characterized in that: The method specifically comprises the following steps: Step S1: record the i-th modal data as d i , i = 1, 2, ..., n, n is the number of modes, a feature extractor is constructed for each mode data, and the feature extractor corresponding to the i-th mode data is recorded as b i ; Step S2: Modal data d i Input the corresponding feature extractor b i , obtain the extracted modal features f i ; Step S3: perform latent space alignment on the features of each modality to obtain the semantic coding features of each modality and the common semantic features of all modalities; Step S4: perform hierarchical cross-modal attention feature fusion on the semantic encoding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result.

2. According to claim 1, a medical multimodal latent space alignment fusion method based on VQ-GCN is characterized in that: The modality number n is 3, d1 represents image data, d2 represents pathological data, and d3 represents examination report; The feature extractor corresponding to d1 is DenseNet, the feature extractor corresponding to d2 is CLAM, and the feature extractor corresponding to d3 is Transformer.

3. According to claim 2, a medical multimodal latent space alignment fusion method based on VQ-GCN is characterized in that: The specific process of step S3 is: Step S31: Randomly initialize a codebook r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB; Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ; Step S33: for the semantic coding feature m i Any row vector m in ij , respectively calculate the row vector m ij The similarity score of each row vector in the codebook CB is calculated, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ; Semantic encoding feature m i The similarity score similarity(m ij , cb k )for: Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, "×" represents the multiplication of numbers, and "·" represents the multiplication of vectors; Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m i After replacing each row vector in, we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ; Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively; Step S34, the modal features f1, f2 and f3 are connected in the channel dimension, and the connection result is used as the input of the multimodal common semantic encoder C, and the multimodal common semantic encoder C outputs the multimodal common semantic coding feature P; The specific process of step S34 is as follows: The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a Cross-Attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image encoding results and the text encoding results are then used to extract shared information between different modalities through the Cross-Attention module. Finally, the information extraction results are used to obtain shared semantic coding features P through the MLP. Step S35: Use the multimodal common semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, vqm′3 of each modality and the final common semantic coding feature P′ of all modalities.

4. According to claim 3, a medical multimodal latent space alignment fusion method based on VQ-GCN is characterized in that: The graph convolutional neural network maximizes the loss function of the mutual information between the multimodal common semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality. for: Among them, similarity(P, vqm i ) is the multimodal shared semantic encoding feature P and the semantic encoding feature vqm i Similarity score of The graph convolutional neural network minimizes the loss function of the mutual information between the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality. for: Among them, similarity(vqm i ,vqm i ') is vqm i With vqm i ' similarity score.

5. According to claim 4, a medical multimodal latent space alignment fusion method based on VQ-GCN is characterized in that: The semantic encoder s1 is a ResNet network, the semantic encoder s2 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM; And the semantic encoder s i The training process is: The vector quantized semantic coding feature vqm i Input the decoder sd corresponding to the i-th mode i , the reconstruction loss is calculated based on the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality; The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

6. According to claim 5, a medical multimodal latent space alignment fusion method based on VQ-GCN is characterized in that: The specific process of step S4 is as follows: Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is i The constructed feature normalization layer is denoted as norm i ; Step S42: The semantic coding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ; Step S43: normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ; Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ; Step S45, input h1, h2 and h3 into the b adaptive attention fusion network, and output the multimodal feature fusion result through the b adaptive attention fusion network.

7. According to claim 6, a medical multimodal latent space alignment fusion method based on VQ-GCN is characterized in that: The adaptive attention fusion network includes MLP and Cross-Attention modules; Concatenate h1, h2, and h3 in the channel dimension and input the concatenation result into the MLP to obtain the multimodal b adaptation weight α; Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module; Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

8. A medical multimodal latent space alignment fusion system based on VQ-GCN, characterized in that: The system includes a feature extraction module, a feature alignment module and a feature fusion module, wherein: The feature extraction module is used to extract the modal features of each modal data; The working process of the feature extraction module is: The i-th modal data is recorded as d i , i = 1, 2, ..., n, construct a feature extractor for each modal data, and record the feature extractor corresponding to the i-th modal data as b i , the modal data d i Input the corresponding feature extractor b i , and obtain the extracted modal features f i ; Where n = 3, d1 represents imaging data, d2 represents pathological data, and d3 represents examination report; The feature alignment module is used to obtain the semantic coding features of each modality and the common semantic features of all modalities; The working process of the feature alignment module is: Step S31: Randomly initialize a codebook r is the number of row vectors in the codebook CB, q is the length of each row vector in the codebook CB; Step S32: Modal feature f i Input the corresponding semantic encoder s i , get the semantic encoding feature m of the i-th modality i ; Step S33: for the semantic coding feature m i Any row vector m in ij , respectively calculate the row vector m ij The similarity score of each row vector in the codebook CB is calculated, and the row vector corresponding to the highest similarity score in the codebook CB is obtained, and then the obtained row vector is used to replace the semantic coding feature m i The row vector m in ij ; Semantic encoding feature m i The similarity score similarity(m ij , cb k )for: Among them, cb k is the k-th row vector in the codebook CB, ||·|| represents the 2-norm, "×" represents the multiplication of numbers, and "·" represents the multiplication of vectors; Traverse the semantic encoding feature m i After each row vector in, the semantic encoding feature m i After replacing each row vector in, we get the vector quantized semantic coding feature vqm corresponding to the i-th mode. i ; Similarly, the vector quantized semantic coding features corresponding to each modality are obtained respectively; Step S34, the modal features f1, f2 and f3 are connected in the channel dimension, and the connection result is used as the input of the multimodal common semantic encoder C, and the multimodal common semantic encoder C outputs the multimodal common semantic coding feature P; The specific process of step S34 is as follows: The multimodal shared semantic encoder C includes an image encoding module, a text encoding module, a Cross-Attention module, and an MLP. The modal features of the imaging data and the modal features of the pathological data are encoded by the image encoding module, and the modal features of the examination report are encoded by the text encoding module. The image encoding results and the text encoding results are then used to extract shared information between different modalities through the Cross-Attention module. Finally, the information extraction results are used to obtain shared semantic coding features P through the MLP. Step S35: Use the multimodal common semantic coding feature P and the semantic coding features vqm1, vqm2, and vqm3 after vector quantization of each modality as nodes to construct a graph convolutional neural network, transfer semantic information within the graph convolutional neural network, and obtain the final semantic coding features vqm′1, vqm′2, and vqm′3 of each modality and the final common semantic coding feature P′ of all modalities; The feature fusion module is used to perform hierarchical cross-modal attention feature fusion on the semantic coding features of each modality and the common semantic features of all modalities to obtain a medical multimodal feature fusion result; The working process of the feature fusion module is as follows: Step S41: construct a feature normalization layer for the semantic coding features of each modality, wherein the semantic coding feature vqm′ of the i-th modality is i The constructed feature normalization layer is denoted as norm i ; Step S42: The semantic coding feature vqm′ of the i-th modality i Input the corresponding feature normalization layer norm i , obtain the semantic encoding feature vqm′ i The feature normalization result of nvqm i ; Step S43: normalize the result nvqm for each feature i Construct cross-modal interaction modules cma separately i ; Step S44: Combine the multimodal final shared semantic coding feature P′ with nvqm i Enter cma i Perform cross-modal interaction and obtain cross-modal interaction features h i ; Step S45, input h1, h2 and h3 into the b adaptive attention fusion network, and output the multimodal feature fusion result through the b adaptive attention fusion network; The b-adaptive attention fusion network includes MLP and Cross-Attention modules; Concatenate h1, h2, and h3 in the channel dimension and input the concatenation result into the MLP to obtain the multimodal b adaptation weight α; Input h1, h2 and h3 into the Cross-Attention module, and output the attention fusion feature Γ through the Cross-Attention module; Multiply the weight α by the feature Γ to obtain the final medical multimodal feature fusion result.

9. The medical multimodal latent space alignment fusion system based on VQ-GCN according to claim 8, characterized in that: The feature extractor corresponding to the image data d1 is DenseNet, the feature extractor corresponding to the pathological data d2 is CLAM, and the feature extractor corresponding to the examination report d3 is Transformer.

10. The medical multimodal latent space alignment fusion system based on VQ-GCN according to claim 9, characterized in that: The semantic encoder s1 is a ResNet network, the semantic encoder s2 is a Transformer, and the semantic encoder s3 is a CNN-iBLSTM; And the semantic encoder s i The training process is: The vector quantized semantic coding feature vqm i Input the decoder sd corresponding to the i-th mode i , the reconstruction loss is calculated based on the output of the decoder corresponding to each modality and the input of the semantic encoder corresponding to each modality; The training is stopped until the reconstruction loss converges, and the trained semantic encoders are obtained.

Images