A multi-granularity medical text information guided 3D multi-modal fusion method

Abstract

The present application relates to the technical field of data processing, in particular to a 3D multi-modal fusion method guided by multi-granularity medical text information, comprising: extracting key description information in 3D images and image reports as input data, encoding to obtain a vector corresponding to the input data; performing image feature extraction and aggregation based on KD at a fine-grained level to obtain slice-level features; multiplying the slice importance score obtained based on coarse-grained FT with the slice-level features to obtain final overall 3D image features; fusing the overall 3D image features, residual data stream features and complete text vectors to obtain multi-modal features for classification; and inputting the multi-modal features into a classification head to obtain model prediction output. The present application makes up for the lack of text utilization in 3D medical multi-modal methods, solves the problems of difficulty in accurately identifying fine-grained features and difficulty in effectively solving coarse-grained redundancy in 3D medical images, and is not limited to specific diseases in method design.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a 3D multimodal fusion method guided by multi-granularity medical text information. Background Art

[0002] With the development of technology, deep learning methods have been widely used in medical image analysis and have achieved good results in many image tasks. However, a considerable portion of clinical images are 3D images obtained through magnetic resonance imaging (MRI) technology, which are usually composed of multiple 2D slices. They contain small-scale lesion information that is difficult to identify by the naked eye at the regional level of the slices. At the same time, there is obvious redundancy between coarse-grained slices, making the features of these 3D images particularly difficult to extract. In fact, medical text information paired with medical images, such as imaging reports, often contains high-value, multi-granular semantic information. It can not only provide multi-level guidance for medical image analysis and help extract and filter complex image features, but also participate in the multimodal fusion process to improve the performance of traditional image-based methods.

[0003] In the past, medical text information was often used for automated, noisy data annotation to mitigate the high cost of medical data annotation. In recent years, with the development of multimodal deep learning technologies and the public release of several 2D medical multimodal datasets, many methods have combined clinical images and text information for multimodal learning, exploring potential connections between different modalities and achieving significant performance improvements compared to image-only models. However, these methods also have obvious defects: first, most models are designed only for 2D images and text and are not suitable for 3D clinical images and paired text; second, even if there are a few 3D multimodal methods, most of them lack the utilization of text information paired with images; most importantly, most existing multimodal methods are too simple in their use of medical text. Text information is usually divided into words and sentences as a whole or simply divided into words and sentences and then directly input into the model to participate in multimodal interaction and fusion. In fact, there are great differences in the granularity of descriptions between different descriptive statements in medical text. For example, the description of some key lesions is usually very critical for identifying specific lesion areas on the slice, while some overall descriptions only describe visible changes at the coarse-grained slice level. Current methods lack the distinction and in-depth utilization of descriptive information of different granularities in medical text. Summary of the Invention

[0004] In response to some or all of the above-mentioned deficiencies in the prior art, the technical problems to be solved by the present invention are the current lack of multimodal methods for utilizing 3D images and medical text in the medical field, the difficulty in extracting 3D medical image features, and the inadequate utilization of high-value medical text information by existing models. To this end, the present invention provides a 3D multimodal fusion method guided by multi-granular medical text information, which is applied to the 3D multimodal fusion framework MedMM guided by multi-granular medical text information. The 3D multimodal fusion framework MedMM includes a KD automatic extraction module, an encoding module, a KD-based regional attention interaction module, and a slice aggregation module based on complete text information. The method comprises:

[0005] Step 1: The KD automatic extraction module extracts input data in a rule-based manner, where KD stands for key descriptive information;

[0006] Step 2: The encoding module encodes the input data to obtain the image vector corresponding to the input data KD text vector and the complete text vector

[0007] Step 3: The KD-based regional attention interaction module is used to calculate the image vector KD text vector and the complete text vector After projection operation, the images are mapped to a common multimodal space, and then cross-modal interaction and fusion are performed. Fine-grained image feature extraction and aggregation are performed based on the KD to obtain slice-level features.

[0008] Step 4: The slice aggregation module based on complete text information uses the complete text as coarse-grained FT supervision information to alleviate the redundancy between slices and guide the further aggregation of coarse-grained levels. The slice importance score obtained based on the coarse-grained FT is multiplied by the slice level feature to obtain the final overall 3D image feature.

[0009] Step 5: Fusion of the overall 3D image features, residual data stream features, and complete text vector Multimodal features for classification are obtained, and the multimodal features are sent to a classification head to obtain a model prediction output.

[0010] Preferably, the KD is lesion-related information, including lesion size, lesion signal and lesion location.

[0011] In one embodiment of the present invention, the input data includes a 3D image X composed of multiple 2D slices. I 、KD Text X KD and the full text X T ,in represents the i-th slice, N s Represents the total number of slices.

[0012] In one embodiment of the present invention, step 2 specifically includes:

[0013] The X I Input image encoder E I Encode to obtain the corresponding image vector The X KD and X T Input text encoder E respectively T The input data KD text vector is obtained by encoding and the complete text vector

[0014]

[0015] in Represents: the image vector of the first slice, Indicates: Nth s Image vector of slices:

[0016] In one embodiment of the present invention, step 3 specifically includes:

[0017] Regarding the and After projection operation, it is mapped to a common multimodal space for cross-modal interaction and fusion;

[0018] Using the KD as a query vector, the KD is made to interact with the regional features of each slice across modalities to obtain an attention matrix of the KD and the attention degree of all regions on each slice;

[0019] Based on the attention matrix, the slice area is aggregated to obtain the slice level feature

[0020] In one embodiment of the present invention, step 4 specifically includes:

[0021] The regional features output by the image encoder After a default average pooling operation AVGPOOL, the slice features are obtained

[0022]

[0023] The and Mapping into multimodal space:

[0024]

[0025] Where W O and W G are the mapping matrices, for: The vector in the multimodal space, G mm for: A vector in a multimodal space.

[0026] The cosine similarity method is used to calculate the correlation score between the mapped complete text information and the slice information:

[0027]

[0028] Score i It is called the slice importance score, which represents the relevance of each slice to the full text.

[0029] The slice-level features obtained based on fine-grained aggregation are multiplied by the slice importance scores obtained based on coarse-grained FT to obtain the final overall 3D image feature O:

[0030]

[0031] Preferably, the residual data stream is calculated by the slice importance score and the relatively comprehensive slice features obtained after pooling:

[0032]

[0033] Preferably, the multimodal features used for classification in step 5 are:

[0034]

[0035] By adopting the above technical solution, the present invention has the following beneficial effects:

[0036] (1) The present invention designs a 3D multimodal fusion method guided by multi-granularity medical text information, which can accept 3D medical images and text input, and makes up for the lack of text utilization in existing 3D medical multimodal methods.

[0037] (2) The present invention designs a method that can efficiently extract 3D image features, extract key features on the image, and solve the problems that fine-grained features in 3D medical images are difficult to accurately identify and coarse-grained redundancy is difficult to effectively solve.

[0038] (3) The present invention fully recognizes and utilizes the different granularity information in the text in terms of method and framework design, guides the extraction and aggregation of different granularity features on the image based on the different granularity semantic information of the text, and at the same time explores the cross-modal mapping relationship between images and texts at different levels.

[0039] (4) The method design of the present invention is not limited to specific diseases, and it can be used across diseases and data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of a 3D multimodal fusion method according to an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the 3D multimodal fusion framework MedMM according to an embodiment of the present invention;

[0043] Figure 3 This is a visualization diagram of the 3D multimodal fusion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0046] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0047] The present invention will be further explained below with reference to specific embodiments.

[0048] like Figure 1 As shown, the 3D multimodal fusion method guided by multi-granularity medical text information in the embodiment of the present invention is applied to the 3D multimodal fusion framework MedMM guided by multi-granularity medical text information, characterized in that the 3D multimodal fusion framework MedMM includes a KD automatic extraction module, an encoding module, a KD-based region attention interaction module (KDRA) and a slice aggregation module (FT-based slice aggregation, FTSA) based on complete text information. The method includes:

[0049] Step 1: The KD automatic extraction module extracts input data in a rule-based manner, where KD stands for key descriptive information;

[0050] Step 2: The encoding module encodes the input data to obtain the image vector corresponding to the input data KD text vector and the complete text vector

[0051] Step 3: The KD-based regional attention interaction module projects the image vector and text vector into a common multimodal space, performs cross-modal interaction and fusion, and extracts and aggregates fine-grained image features based on the KD to obtain slice-level features.

[0052] Step 4: The slice aggregation module based on complete text information uses the complete text as coarse-grained FT supervision information to alleviate the redundancy between slices and guide the further aggregation of coarse-grained levels. The slice importance score obtained based on the coarse-grained FT is multiplied by the slice level feature to obtain the final overall 3D image feature.

[0053] Step 5: Fusion of the overall 3D image features, residual data stream features, and complete text vector Multimodal features for classification are obtained, and the multimodal features are sent to a classification head to obtain a model prediction output.

[0054] The present invention designs a multimodal framework that can accept 3D images and imaging reports as input. Secondly, to address the difficulties in extracting 3D medical image features and the inadequate utilization of high-value medical text information by existing models, the present invention first automatically extracts key descriptions (KD) from imaging reports as fine-grained supervision information. Based on KD, the accurate extraction of key image features at the regional level is guided. Then, the full text of the imaging report (FT) is used as coarse-grained supervision information. Its comprehensive semantic information guides the aggregation of different slices at the coarse-grained level, assigning different importance to different slices and minimizing redundancy between slices. In addition, the framework design and application of the present invention are not limited to specific diseases and datasets, and no pre-training is required. For clinical scenarios, regardless of the size of the dataset, as long as there are 3D images and textual information paired with the images, the present invention can be directly applied, showing cross-disease versatility and flexibility. Finally, the present invention also provides multi-granular interpretability results, namely, the region-level attention weight matrix is ​​used to reflect the fine-grained mapping relationship between KD and regions, and the slice-level importance score is used to reflect the coarse-grained mapping relationship between FT and slices.

[0055] In one embodiment of the present invention, KD is first automatically extracted. Referring to the text automatic extraction method in the prior art, the present invention pre-designs a rule-based method to extract the KD. Some examples of rules are given in Table 1. The KD is lesion-related information, which includes descriptions such as lesion size, lesion signal, and lesion location, which are usually substantially different from other descriptions in the report text. Therefore, on different data sets and tasks, as long as similar rules are designed with a small amount of doctor guidance, automatic extraction can be achieved simply and quickly.

[0056] Table 1

[0057]

[0058] The 3D multimodal fusion method guided by multi-granularity medical text information provided by the embodiment of the present invention can accept 3D medical images and text input, making up for the lack of text utilization in existing 3D medical multimodal methods.

[0059] In another embodiment of the present invention, the encoding module collects 3D images and corresponding image reports for each patient. After necessary preprocessing and KD automatic extraction, the input data of each patient includes a 3D image X composed of multiple 2D slices. I 、KD Text X KD and the full text X T ,in represents the i-th slice, N sRepresents the total number of slices.

[0060] The X I Input image encoder E I Encode to obtain the corresponding image vector The X KD and X T Input text encoder E respectively T The input data KD text vector is obtained by encoding and the complete text vector

[0061]

[0062]

[0063] Among them, Represents: the image vector of the first slice, Indicates: Nth S The image vector of each slice, E I 2D slices are encoded using 2DResNet18 pre-trained on ImageNet. T We use RoBERTa pre-trained on the Chinese corpus according to the whole word masking strategy (wwm) as the text encoder. Here we use the same text encoder E T KD and FT are encoded separately.

[0064] In a preferred embodiment of the present invention, step 3 specifically includes:

[0065] For the image vector KD text vector and the complete text vector After projection operation, it is mapped to a common multimodal space for cross-modal interaction and fusion;

[0066]

[0067] Where W Q , W K and W V Next, considering that KD is a description of fine-grained lesion information on a slice, and these lesions usually only occupy a small area of ​​the entire slice, the present invention uses the KD as a query vector based on the attention mechanism, so that the query vector interacts with the regional features of each slice across modalities, and obtains the attention matrix A of the KD and the attention degree of all regions on each slice. i :

[0068]

[0069] Then, the slice regions are aggregated based on the attention matrix to obtain slice-level features

[0070]

[0071] Therefore, the image feature extraction and aggregation of fine-grained level (slice area) is completed based on KD, and the slice level feature is obtained.

[0072] In addition to KD, the report text actually contains a lot of important information, such as some changes unrelated to the lesion, some descriptions of normal manifestations, etc. Although this information is unrelated to the lesion, it can only be observed on certain slices and not on other slices. The present invention takes advantage of this feature and uses the complete report as coarse-grained supervision information to alleviate the redundancy between slices and guide the further aggregation of coarse-grained levels (slices). Therefore, in another preferred embodiment of the present invention, step 4 specifically includes:

[0073] Slice features aggregated by the slice aggregation module based on complete text information It is only strongly correlated with the KD in the complete text and has nothing to do with other descriptions. Therefore, this embodiment converts the regional features output by the image encoder into After a default average pooling operation AVGPOOL, the slice features are obtained

[0074]

[0075] Will With the Mapping into multimodal space:

[0076]

[0077] Where W O and W G are the mapping matrices, for: The vector in the multimodal space, G mm for: A vector in a multimodal space.

[0078] The cosine similarity method is used to calculate the correlation score between the mapped complete information and the slice information:

[0079]

[0080] Score iIt is called the slice importance score, which represents the correlation between each slice and the complete text. The cosine similarity is a measurement method for comparing the similarity between two vectors, and is usually used to calculate the similarity of text, image, audio and other data. It is often used in fields such as text mining, information retrieval and recommendation systems. Given two vectors A and B, their cosine similarity cos(A, B) can be calculated by dividing their inner product by the product of their moduli. The embodiment of the present invention assigns higher scores to slices that are closely related to the report and lower scores to irrelevant slices through model learning, thereby maximizing feature quality and reducing redundant information at the slice level.

[0081] The slice-level features obtained based on fine-grained KD aggregation are multiplied by the slice importance scores obtained based on coarse-grained FT to obtain the final overall 3D image feature O:

[0082]

[0083] The present invention can fully identify and utilize information of different granularities in text, guide the extraction and aggregation of features of different granularities on images based on the semantic information of different granularities in text, and simultaneously explore the cross-modal mapping relationship between images and texts at different levels.

[0084] In another preferred embodiment of the present invention, in order to avoid information loss in the bottom-up multi-level aggregation process, the present invention additionally adds a residual data stream of the image in the final multimodal feature fusion part, where the residual information is calculated by the slice importance score and the relatively comprehensive slice features obtained after pooling:

[0085]

[0086] Finally, the overall 3D image features, residual features and complete text features are integrated to obtain the multimodal features for classification:

[0087]

[0088] Feeding multimodal features into the classification head yields the model prediction output.

[0089] In order to verify the effectiveness of the medical 3D multimodal fusion model designed by the present invention under the guidance of multi-granularity text information, the present invention was verified on tumor classification tasks in two different parts (spinal cord and brain), and additional ablation experiments were performed to prove the effectiveness of each module.

[0090] Specifically, the embodiment of the present invention collected two data sets from Beijing Tiantan Hospital affiliated to Capital Medical University: a spinal cord tumor classification data set and a brain tumor classification data set. The spinal cord tumor classification data set contains a total of 486 samples and two categories (patients are divided into spinal cord ependymoma / spinal cord astrocytoma according to the gold standard of pathological examination), including 256 cases of spinal cord ependymoma and 230 cases of spinal cord astrocytoma; the brain tumor classification data set includes a total of 2231 samples and three categories (patients are divided into glioblastoma, metastasis or primary central nervous system lymphoma according to the gold standard of pathological examination), including 880 cases of glioblastoma, 416 cases of metastasis and 935 cases of primary central nervous system lymphoma. Each sample in each data set contains multi-slice 3D images and Chinese imaging reports paired with the images as model input.

[0091] In order to fully verify the effectiveness of the method designed in the embodiment of the present invention, a comparison of multiple methods was added to the experiment, which are mainly divided into three categories: pure image method, pure text method and multimodal method.

[0092] Specifically, image-only methods include models that encode 3D images in different ways:

[0093] 1) MID: (middle, representing the method of selecting the middle slice for encoding) selects the middle slice with the highest probability of lesion occurrence as the 2D encoding model representing the 3D image feature. The 2D encoder uses the 2D ResNet18 image model and is divided into MID (Random.) (random initialization, representing the MID method with random initialization parameters) and MID (ImageNet.) (pre-training parameters, representing the MID method with ImageNet pre-training parameters) according to whether the parameters are randomly initialized or loaded.

[0094] 2) AVG: (average, representing the method of averaging all slice features after encoding) A 2.5D model that uses 2D ResNet18 to perform 2D encoding on all slice features and then performs a simple averaging operation.

[0095] 3) GA: (gated attention, representing a method of using a gated attention mechanism to aggregate slice features), a 2.5D model that aggregates slice features encoded using 2D ResNet18.

[0096] 4) 3DRN18: (3D ResNet 18, representing the encoding method using the 3D ResNet18 image model) This model uses 3D ResNet18 to directly encode the entire 3D image. The plain text method includes the following models:

[0097] 5) FTB: (Full Text using BERT, representing a method for encoding full text using the BERT series model) uses the basic bert-base-chinese model pre-trained on Chinese corpus to encode full text.

[0098] 6) FTR: (Full Text using ROBERTa, representing the method of encoding full text using the ROBERTa series of models) uses the roberta-wwm-chinese model pre-trained on Chinese corpus according to the whole word masking strategy (wwm) to encode full text.

[0099] 7) KDR: (Key Description using ROBERTa, representing a method for encoding key descriptions using the ROBERTa series of models) uses a model that is pre-trained on Chinese corpus using the roberta-wwm-chinese strategy using whole word masking (wwm) to encode only KD.

[0100] Due to the current lack of models that can accept 3D images and text as input, this paper directly merges the features encoded by the pure image and pure text models in the verification phase to construct an additional multimodal method without cross-modal interaction:

[0101] 8) 3DRN18+FTR: A classification model that directly combines the image features encoded by 3D ResNet18 and the text features encoded by roberta-wwm-chinese on the complete text.

[0102] 9) GA+FTR: A classification model that directly combines the image features encoded by GA and the text features encoded by roberta-wwm-chinese on the complete text.

[0103] 10) IRENE: According to the method in

[11] , the model takes 3D images and reports as input and performs classification after cross-modal interaction within the Transformer.

[0104] The performance of each model on the two classification tasks is shown in Tables 2 and 3, where precision (P), recall (R), F1 value, accuracy (ACC) and area under the ROC curve (AUC) are used as indicators to evaluate model performance. The experimental results are given as mean ± standard deviation, and each result is the result of five repeated experiments.

[0105] Table 2: Performance comparison of models on the spinal cord tumor binary classification task

[0106]

[0107]

[0108] Table 3: Performance comparison of models on the brain tumor three-classification task

[0109]

[0110] As shown in Tables 2 and 3, MedMM achieved the best results across all evaluation metrics for two classification tasks involving different organs. Its performance significantly improved over image-only, text-only, and multimodal approaches, fully demonstrating the effectiveness and versatility of the multi-granular text-based medical 3D multimodal fusion framework designed in this embodiment of the present invention. Furthermore, since imaging reports are highly condensed from expert observations and typically contain precise semantic information, the table also demonstrates the superior performance of the text-only approach. Meanwhile, while KD includes key lesion information, it is often limited to lesion information and too brief. Therefore, KDR's performance at the overall disease classification level is significantly inferior to that of FTR, which relies on complete text. For the two non-interactive multimodal models, 3DRN18+FTR and GA+FTR, the addition of text information significantly improved over image-only approaches, demonstrating that the inclusion of text information can provide valuable supplementary modal information for model-based automatic diagnosis. However, the results still lagged behind those of MedMM, further demonstrating the necessity and efficiency of cross-modal interaction in the proposed framework. Comparison with IRENE, the only 3D image-text model in the medical field, further demonstrates the advantages of our multi-granular cross-modal interaction module. Since IRENE only focuses on regional-level interactions and its use of text is very crude and direct (all words are fed into the model for indiscriminate interaction with slice features), it lacks both a rational design at the slice level and full utilization of the multi-granularity of text information. The framework proposed in this invention fully utilizes imaging reports to guide the extraction, filtering, and aggregation of 3D image features at multiple levels, both fine and coarse. This is both efficient and consistent with clinical facts. The results also show that MedMM is clearly superior to IRENE in both tasks. In addition, IRENE's large Transformer structure and its reliance on the pre-training process make it less suitable than the framework proposed in this invention for 3D multimodal dataset scenarios in clinical settings with limited sample size and no pre-training conditions. MedMM's design is not limited to specific datasets and has universality and generalization across diseases. This is also verified by the consistently excellent experimental results on both the brain and spinal cord.

[0111] In addition, to verify the effectiveness of each module, an ablation experiment was designed on the spinal cord tumor binary classification task. The results are shown in Table 4.

[0112] Table 4: Ablation experiment of the model on the spinal cord tumor binary classification task

[0113]

[0114]

[0115] In Table 4, AVG is the baseline for comparison. It uses the default averaging operation in the image encoder to obtain features at both the region and slice levels. The comparison between KDRA+GAVG and AVG proves that the design of the KDRA module is more beneficial than simply averaging all region features to obtain slice features. LAVG+FTSA has further improved over both KDRA+GAVG and AVG, reflecting that the FTSA module at the slice level that measures the importance of slices based on the complete text is more reasonable than simply averaging all slice features. After integrating the two modules, KDRA+FTSA has a significant improvement over the other comparison baselines, demonstrating the advantages of the two granular cross-modal interaction modules designed in this invention. Finally, after adding additional text information and residual data streams, MedMM achieved the best classification results, highlighting the advantages of multi-granularity and multi-modal fusion.

[0116] Finally, a visualization example is given and compared with the expert annotation results, such as Figure 3 As shown in the figure, the solid line frame is the lesion area marked by clinical experts, and the dotted line frame is the lesion slice marked by clinical experts. Figure 3 It can be seen that at the fine-grained regional level, the regional attention matrix given by the model accurately focuses on the lesion and the surrounding area, and the attention matrix is ​​very concentrated, which is almost consistent with the expert annotation results; at the coarse-grained slice level, the slice score given by the model is completely consistent with the lesion slice annotated by the expert, that is, the important slices paid attention to by the model are also the slices that the experts think are important, which is consistent with clinical facts. It concisely, directly and powerfully proves that the framework proposed in this invention can accurately focus on key features at multiple levels of regions and slices, and is reliable.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D multimodal fusion method guided by multi-granular medical text information, applied to the 3D multimodal fusion framework MedMM guided by multi-granular medical text information, characterized by: The 3D multimodal fusion framework MedMM includes a KD automatic extraction module, an encoding module, a KD-based regional attention interaction module, and a slice aggregation module based on complete text information. The method includes: Step 1: The KD automatic extraction module extracts input data in a rule-based manner, where KD stands for key descriptive information; Step 2: The encoding module encodes the input data to obtain the 3D image vector corresponding to the input data , KD text vector and the complete text vector ; Step 3: The KD-based regional attention interaction module is used to calculate the image vector , KD text vector and the complete text vector Perform projection operations to map to a common multimodal space, then perform cross-modal interaction and fusion, and perform fine-grained image feature extraction and aggregation based on the KD to obtain slice-level features; Step 4: The slice aggregation module based on complete text information uses the complete text as coarse-grained FT supervision information to alleviate the redundancy between slices and guide the further aggregation of coarse-grained levels. The slice importance score obtained based on the coarse-grained FT is multiplied by the slice level feature to obtain the final overall 3D image feature. Step 5: Fusion of the overall 3D image features, residual data stream features, and complete text vector Obtain multimodal features for classification, and feed the multimodal features into the classification head to obtain the model prediction output. The residual data stream features are calculated by the slice importance scores and the relatively comprehensive slice features obtained after pooling. The slice feature is obtained by performing an average pooling operation on the image vector. get.

2. The 3D multimodal fusion method guided by multi-granularity medical text information according to claim 1, characterized in that: The KD is lesion-related information, including lesion size, lesion signal, and lesion location.

3. The 3D multimodal fusion method guided by multi-granularity medical text information according to claim 1, characterized in that: The input data includes a 3D image composed of multiple 2D slices , KD text and the full text ,in , represents the i-th slice, Represents the total number of slices.

4. The 3D multimodal fusion method guided by multi-granularity medical text information as claimed in claim 3, characterized in that: The step 2 specifically includes: The Input image encoder Encode to obtain the corresponding image vector , the and Input text encoder separately The input data KD text vector is obtained by encoding and the complete text vector : ， ， in Represents: the image vector of the first slice, Indicates: The image vector of the slices.

5. The 3D multimodal fusion method guided by multi-granularity medical text information according to claim 4, characterized in that: The step 3 specifically includes: Regarding the 、 and After projection operation, it is mapped to a common multimodal space for cross-modal interaction and fusion; Using the KD as a query vector, the KD is made to interact with the image vector of each slice across modalities to obtain an attention matrix of the KD and the attention degree of all regions on each slice; Based on the attention matrix, the slice area is aggregated to obtain the slice level feature .

6. The 3D multimodal fusion method guided by multi-granularity medical text information according to claim 5, characterized in that: The step 4 specifically includes: The image vector output by the image encoder After a default average pooling operation Get slice features : ， The and Mapping into multimodal space: ， ， in and are the mapping matrices, for: A vector in multimodal space, for: A vector in multimodal space, The cosine similarity method is used to calculate the correlation score between the mapped complete text information and the slice information: ， in It is called the slice importance score, which represents the relevance between each slice and the complete text. The slice-level features obtained based on fine-grained aggregation are multiplied by the slice importance scores obtained based on coarse-grained FT to obtain the final overall 3D image features. : .。 7. The 3D multimodal fusion method guided by multi-granularity medical text information according to claim 6, characterized in that: The residual data stream is calculated by the slice importance score and the relatively comprehensive slice features obtained after pooling: 。 8. The 3D multimodal fusion method guided by multi-granularity medical text information according to claim 7, characterized in that: The multimodal features used for classification in step 5 are: 。

Images