Multi-modal multi-scale transformation fusion method and system based on exchange

Abstract

The invention discloses an exchange-based multi-modal multi-scale transformation fusion method and system, the method comprises the steps of obtaining original image data and original text data and inputting the original image data and the original text data into an MMTF model to generate a fusion result, and the MMTF model comprises a multi-modal encoder module, a decoder module, a channel-based information exchange module and a multi-scale fusion module. The multi-mode encoder module comprises a text encoder and a double-branch image decoder; the decoder module decodes the embedded part generated by the encoder; the channel-based information exchange module performs information exchange on embedding of different modes on different channels; and the multi-scale fusion module is used for fusing the cls token from one branch and the patch token from the other branch according to the image features and the text features on the different branches. According to the method, reliable decision support can be provided in various medical environments, the diagnosis accuracy is improved, and the workload of clinicians is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and in particular relates to a multi-modal multi-scale transformation fusion method and system based on exchange. Background Art

[0002] In recent years, data-driven artificial intelligence (AI) has developed rapidly and profoundly changed the field of multimodal machine learning. Multimodal fusion is a core challenge in this field, which aims to unify heterogeneous data into a single representation that effectively reduces the dimensionality while retaining the semantic characteristics of each modality. In the field of healthcare, patient data often includes imaging (such as radiological scans) and text records (such as electronic health records or diagnostic reports). Text data, in particular, contains detailed descriptions of medical conditions observed by radiologists, resulting in complex and less transparent cross-modal relationships, making data integration more complicated.

[0003] To address this challenge, research has increasingly adopted deep multimodal fusion techniques. Deep multimodal fusion techniques are mainly divided into aggregation-based and alignment-based methods. Aggregation-based methods represent each modality through sub-networks and combine these representations using various operators. However, this approach sacrifices the ability to make independent predictions for each modality, resulting in the loss of some intra-modal information. In contrast, alignment-based methods use alignment losses to maintain the consistency of multimodal features while retaining the independent outputs of multiple sub-networks and weighting the final predictions. However, alignment-based fusion simply adds a regularization term to the original unimodal optimization objective without achieving true cross-modal fusion or facilitating the necessary inter-modal information exchange. In addition, some studies have explored channel-based modality exchange methods to balance intra-modal processing and inter-modal fusion. For example, by using batch normalization (BN) scaling factors to measure channel importance, channels with factors close to zero are replaced with the average channels of another modality. However, this channel exchange technique often ignores the heterogeneity between image and text data, making it difficult to effectively represent cross-modal data in a unified low-dimensional space. Furthermore, since the pathological region usually occupies only a small portion of a medical image, single-scale feature representation faces limitations in capturing local lesion details.

[0004] As a solution, integrating multi-scale structures into the model has been shown to effectively improve feature learning capabilities. The multi-scale network structure samples the input data at different granularities, which may result in different fine-grained and coarse-grained features in the region of interest (ROI). Fine-grained features retain more detailed input data, while coarse-grained features capture overall trends. Most disease detection tasks consider multi-scale factors when classifying images.

[0005] Integrating features from various data scales can enhance the convergence of deep neural networks during artificial intelligence training for medical applications. Fine-grained features capture detailed information, while coarse-grained features reveal overall trends, both of which are critical for optimizing model performance. This recognition has led to increasing interest in combining multimodal and multiscale learning strategies. However, existing multiscale methods in medical artificial intelligence mainly focus on combining image modalities, with limited exploration of multiscale complementary learning between images and text reports. Summary of the invention

[0006] In view of this, an object of the present invention is to provide a multi-modal multi-scale transformation fusion method and system based on exchange.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A multi-modal multi-scale transformation fusion method based on exchange, comprising:

[0009] Acquire multimodal data, namely, original image data and original text data corresponding to the original image data;

[0010] The original image data and the original text data are input into a multimodal multiscale transformation fusion model to generate a fusion result, wherein the multimodal multiscale transformation fusion model includes: a multimodal encoder module, a decoder module, a channel-based information exchange module and a multiscale fusion module,

[0011] The multimodal encoder module consists of a text encoder and a dual-branch image decoder.

[0012] The text encoder is used to encode the input text data and generate text embedding;

[0013] The dual-branch image encoder includes a small-branch image encoder and a large-branch image encoder, which are used to extract a fine-grained image embedding and a coarse-grained image embedding of input image data respectively;

[0014] The decoder module contains two decoders, which decode the embeddings generated by the text encoder and the dual-branch image encoder respectively, so as to generate images from text and text from images, so as to regularize the embeddings generated by the text encoder and the dual-branch image decoder.

[0015] The channel-based information exchange module is used to exchange information between embeddings of different modalities on different channels according to the attention scores of different channels, thereby generating image features and text features on different branches;

[0016] A multi-scale fusion module is used to fuse the cls tokens from one branch and the patch tokens from another branch based on the image features and text features on different branches.

[0017] The present invention also provides a multi-modal multi-scale transformation fusion system based on exchange, comprising:

[0018] A multimodal data acquisition engine, used to acquire multimodal data, namely, image data and text data corresponding to the image data;

[0019] Model training engine, used to train multi-modal and multi-scale transformation fusion models.

[0020] The fusion generation engine is used to generate fusion results based on the acquired multimodal data using the trained multimodal multi-scale transformation fusion model.

[0021] The multimodal multiscale transformation fusion model includes: a multimodal encoder module, a decoder module, a channel-based information exchange module and a multiscale fusion module.

[0022] The multimodal encoder module consists of a text encoder and a dual-branch image decoder.

[0023] The text encoder is used to encode the input text data and generate text embedding;

[0024] The dual-branch image encoder includes a small-branch image encoder and a large-branch image encoder, which are used to extract a fine-grained image embedding and a coarse-grained image embedding of input image data respectively;

[0025] The decoder module contains two decoders, which decode the embeddings generated by the text encoder and the dual-branch image encoder respectively, so as to generate images from text and text from images, so as to regularize the embeddings generated by the text encoder and the dual-branch image decoder.

[0026] The channel-based information exchange module is used to exchange information between embeddings of different modalities on different channels according to the attention scores of different channels, thereby generating image features and text features on different branches;

[0027] A multi-scale fusion module is used to fuse the cls tokens from one branch and the patch tokens from another branch based on the image features and text features on different branches.

[0028] Furthermore, the information exchange of embeddings of different modalities includes the information exchange of coarse-grained image embeddings and text embeddings, and the information exchange of fine-grained image embeddings and text embeddings. The channel-based information exchange module also uses two hyperparameters η and μ to control the starting layer and the ending layer of the multimodal information exchange, respectively.

[0029] Furthermore, the channel-based information exchange module includes a multi-head self-attention unit and a multimodal information exchange unit. The text embedding, fine-grained image embedding and coarse-grained image embedding plus their corresponding cls tokens are fed to the multi-head self-attention unit to obtain the attention scores of each token in the text embedding, fine-grained image embedding and coarse-grained image embedding. The calculated attention scores are sent to the multimodal information exchange unit through a residual connected feedforward network. The multimodal information exchange unit adopts multiple Transformer encoders with shared parameters to exchange multimodal information of different scales in a complementary manner.

[0030] Furthermore, the multimodal information exchange unit exchanges multimodal information of different scales in a complementary manner, including: for information exchange at any layer, for any channel, if the attention score of the current channel is lower than a set threshold, the embedding vector of the channel will be replaced by the average embedding of a preset percentage of tokens in another modality, otherwise, its embedding vector remains unchanged, thereby obtaining an updated embedding matrix.

[0031] Furthermore, the channel-based information exchange module also includes FFN and CB units. After the current layer of information exchange is completed, FFN generates input embeddings for the next layer of information exchange based on the updated embedding matrix, and CB is used to broadcast context to each token.

[0032] Furthermore, the cls tokens from one branch and the patch tokens from another branch are merged, including:

[0033] The image cls token from the large branch and the text patch token from the small branch are concatenated and cross-attention is performed with the image cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the large branch to obtain the fused image feature of the large branch, i.e., the coarse-grained fused image feature.

[0034] The image cls token from the small branch and the text patch token from the large branch are concatenated and cross-attention is performed with the image cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the small branch to obtain the fused image feature of the small branch, i.e., the fine-grained fused image feature.

[0035] The text cls token from the large branch and the image patch token from the small branch are concatenated and cross-attention is performed with the text cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the large branch to obtain the fused text feature of the large branch, i.e., the coarse-grained fused text feature.

[0036] The text cls token from the small branch and the image patch token from the large branch are concatenated and cross-attention is performed with the text cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the small branch to obtain the fused text feature of the small branch, that is, the fine-grained fused text feature.

[0037] The beneficial effects of the present invention are:

[0038] The present invention captures the correlation between patient imaging and medical reports through image and report generation tasks, jointly normalizes multimodal embeddings, and aligns them into a shared latent space to achieve cross-modal interaction; the present invention constructs a multimodal multi-scale transform fusion model (i.e., MMTF model), introduces a multi-path Transformer, and effectively combines features from different modalities and scales through a cross-attention mechanism, thereby aggregating cross-modal correlations at multiple scales and achieving semantic knowledge exchange between images and text reports of different scales; through extensive experiments on four public lung disease datasets, it is shown that the proposed MMTF model achieves state-of-the-art classification performance.

[0039] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0041] Figure 1 It is a schematic diagram of the MMTF model;

[0042] Figure 2 It is a schematic diagram of channel-based multimodal information exchange;

[0043] Figure 3 It is a schematic diagram of large-branch multi-scale fusion;

[0044] Figure 4 is the comparison result of the MMTF model with the baseline models with different proportions of labeled data in the target domain;

[0045] Figure 5 This is a comparison chart between the MMTF model and the SOTA medical image text pre-training model in NIH ChestX-ray and VinBigData chest X-ray anomaly detection;

[0046] Figure 6 This is a comparison chart between the MMTF model and SOTA multimodal lung diagnosis on NIH ChestX-ray, VinBigData ChestX X-ray anomaly detection and Shenzhen tuberculosis datasets;

[0047] Figure 7 is a visualization of randomly selected samples;

[0048] Figure 8 This is the t-SNE visual clustering result diagram of the Shenzhen tuberculosis and pneumonia image data collection dataset caused by the new coronavirus. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0050] The present invention proposes a multi-modal multi-scale transformation fusion system based on exchange, comprising:

[0051] A multimodal data acquisition engine, used to acquire multimodal data, namely, image data and text data corresponding to the image data;

[0052] Model training engine, used to train multi-modal and multi-scale transformation fusion models;

[0053] The fusion generation engine is used to generate fusion results based on the acquired multimodal data using the trained multimodal multi-scale transformation fusion model.

[0054] The present invention also provides a multi-modal multi-scale transformation fusion method based on exchange, the method comprising:

[0055] Acquire multimodal data, namely, original image data and original text data corresponding to the original image data;

[0056] The original image data and the original text data are input into the multimodal multi-scale transformation fusion model to generate a fusion result.

[0057] Figure 1 The following is a schematic diagram of a multi-modal multi-scale transform fusion model (MMTF model) using lung disease as an example. Figure 1 As shown, the multi-modal multi-scale transformation fusion model mainly includes four functional modules, namely: a multi-modal encoder module (i.e. Figure 1 The components marked in ①), the decoder module (i.e. Figure 1 ② and ③), channel-based information exchange module (i.e. Figure 1 The component marked in (4) and the multi-scale fusion module (i.e. Figure 1 Components marked in ⑤).

[0058] The multimodal encoder module consists of a text encoder and a dual-branch image decoder.

[0059] The text encoder is used to encode the input text data (e.g., medical reports, etc.) and generate text embeddings.

[0060] A dual-branch image encoder is designed to obtain multi-scale features of an image (e.g., a medical radiograph), which may include a small-branch image encoder and a large-branch image encoder, which are used to extract fine-grained image embedding and coarse-grained image embedding of the input image data, respectively.

[0061] Given a pair of inputs [x v ,x t ], where x v stands for radiograph, x t is the corresponding report. The text encoder is used to extract the semantic information of the report and obtain the text embedding T t , and obtain a wider embedding of the image (i.e., coarse-grained image embedding I L ) and smaller embeddings (i.e., fine-grained image embeddings I S ). These encoded features are input into subsequent components to learn multimodal and multi-scale representations. Finally, the learned multimodal fusion feature representation is used for multi-label classification tasks.

[0062] In some embodiments, a dual-branch image encoder is used to extract features from radiographs to obtain multi-scale image features. For example, a Resnet34 architecture can be used to extract coarse-grained features. That is, coarse-grained image embedding, while the Resnet152 architecture can be used to extract fine-grained features That is, coarse-grained image embedding. Where M and N are the number of tags and feature dimensions at each scale, respectively. Represents a real number set, C represents the number of channels of the image, and is expressed as follows:

[0063] I L =Renet34(x v ), (Formula 1)

[0064] I S =Resnet152(x v ), (Formula 2)

[0065] Reports usually consist of long paragraphs and require reasoning across multiple sentences. Therefore, the present invention uses a language model based on a self-attention mechanism to learn long-range semantic dependencies in reports. Unlike using a dual-branch image encoder to obtain multi-scale image features, there is no additional processing on the original report. The cumbersome and complex processing of local text features not only ignores obscure words, but may also confuse synonyms in the medical field. In some embodiments, the MMTF model can adopt the BioClinicalBERT model as a text encoder to obtain clinically aware text embedding, which is expressed as:

[0066] T t = BertEncoder(x t ), (Formula 3)

[0067] The BertEncoder in Formula 3 is the BioClinicalBERT model.

[0068] In the medical field, due to the modality variability between radiographs and reports, the encoded multimodal features are usually located in different vector spaces. Therefore, for channel-based multimodal information exchange, these embeddings need to be pulled into the same space first. In order to capture the correlation between reports and radiographs, the present invention designs two generation tasks (mutual generation between reports and radiographs). As shown in components ② and ③, the generation task regularizes the encoder embedding through two decoders.

[0069] like Figure 1 As shown, the decoder module includes two decoders, which respectively decode the embeddings generated by the text encoder and the dual-branch image encoder, thereby generating two generation tasks, namely, generating images from texts and generating texts from images, to regularize the embeddings generated by the text encoder and the dual-branch image decoder.

[0070] In component ②, the received text embedding is processed by MLP and then decoded by the image decoder to generate an image; in component ③, the received large branch image embedding (i.e., coarse-grained image embedding) is processed by MLP and then decoded by the text decoder to generate text corresponding to different scales (or called "branches").

[0071] For the generation task of generating text from images (i.e., the generation task shown in component ③), the NIC model can be used as a decoder to generate the corresponding report based on the input radiograph. Similarly, the image generation model PixelCNN can be used as a decoder to extract the embedding from the text encoder and generate the corresponding radiograph (as shown in component ②).

[0072] By leveraging these two generation tasks, the relationship between images and texts can be strengthened and mined. By comparing the generated radiographs and reports with the original input radiographs and reports, two generation losses are generated, namely, and in, is the generation loss for generating text from images, is the generation loss of generating images based on text. These two generation tasks are auxiliary tasks of the prediction task. Figure 1 middle is the main task loss for multi-label classification (e.g., disease classification).

[0073] The channel-based information exchange module is used to exchange information between embeddings of different modalities on different channels according to the attention scores of different channels, thereby generating image features and text features on different scales (i.e., branches).

[0074] The information exchange between embeddings of different modalities includes exchanging information between coarse-grained image embeddings and text embeddings, and exchanging information between fine-grained image embeddings and text embeddings.

[0075] like Figure 2 As shown in Figure 2, the channel-based information exchange module includes a multi-head self-attention unit and a multimodal information exchange unit (i.e. Figure 1 The multimodal information exchange unit adopts multiple transformer encoders (i.e., Transformer encoders, or transformer encoders) with shared parameters, which are used to learn to embed text and visual modalities at multiple scales. The Transformer encoder learns global context information from the input vector in its shallow layer and performs channel-based information exchange between modalities, that is, multimodal information of different scales is exchanged in a complementary manner. Since the downstream task is to perform multi-label classification, cls tags (or "tokens") for global feature aggregation are added to the beginning of the input sequence (i.e., text embedding and image embedding). The text embedding, fine-grained image embedding, and coarse-grained image embedding, plus their respective corresponding cls tokens, are fed to the multi-head self-attention unit to learn global context information and obtain the attention scores of each token (i.e., patch token) in the text embedding, fine-grained image embedding, and coarse-grained image embedding.

[0076] In the self-attention mechanism, the input is linearly mapped to obtain three matrices: query matrix Q, key matrix K, and value matrix V. The output calculation is expressed as:

[0077]

[0078] Among them, d k is a scaling factor, and Softmax represents a normalization function.

[0079] The channel-based information exchange module sets its shallow layer to a regular Transformer encoder layer, followed by multiple exchange layers for information exchange. When the multimodal multi-scale fusion is completed, the exchange process ends. To this end, the channel-based information exchange module also uses two hyperparameters η and μ to control the start and end layers of the multimodal information exchange, respectively. When the multimodal embedding enters the η layer, the information exchange starts and stops at the μ layer. In some embodiments, there are 6 exchange layers, so 6 information exchanges are performed (such as Figure 1 In component ②, the “transformer based on information exchange” has a 6-layer structure) After the channel-based multimodal information exchange is completed, the exchanged embedding results are fed to the subsequent modules for multimodal and multi-scale fusion.

[0080] For information exchange at any layer (i.e., any layer of the exchange layer), for any channel, if the attention score of the current channel is lower than the set threshold, the embedding vector of the channel will be replaced by the average embedding of a preset percentage of tokens in the other modality, otherwise, the embedding vector of the channel remains unchanged, thereby obtaining an updated embedding matrix.

[0081] Among them, the coarse-grained image embedding of one channel in the multimodal information exchange process can be expressed as:

[0082]

[0083] Among them, θ is the preset threshold, γ is the replacement rate of information exchange, and They are the intermediate embeddings of the coarse-grained image generated by the multi-head self-attention unit and the text corresponding to the coarse-grained image; express The attention score of the i-th row in , j represents the intermediate embedding of the text The row of n represents a channel The total number of tokens in .

[0084] The information exchange process between fine-grained images and text is consistent with the information exchange between coarse-grained images and text, and will not be repeated here.

[0085] The text embedding on one channel in the multimodal information exchange process can be expressed as:

[0086]

[0087] in, represents the corresponding coarse-grained text embedding generated by the information exchange between the coarse-grained image embedding and the text, and n represents the information on a channel The total number of tokens in .

[0088] The corresponding fine-grained text embedding generated by the information exchange between fine-grained image embedding and text can be expressed as The specific implementation method is the same as that of formula 6. Replace the intermediate embeddings of fine-grained and coarse-grained images generated by multi-head self-attention units That's it.

[0089] The channel-based information exchange module also includes FFN (Feed-Forward Network) and CB (Context Broadcasting) units. After the information exchange of the current layer is completed, FFN generates the input embedding of the next layer of information exchange based on the updated embedding matrix, and CB is used to broadcast the context to each token.

[0090] Figure 2 The detailed structure of FNN and CB is also enlarged, which can be referred to Figure 2 , I will not go into details here.

[0091] As mentioned above, radiographic multimodal data contains a wide range of pathological information and different focus points. In order to provide sufficient feature extraction in each modality, the present invention also designs a multimodal multiscale fusion module (referred to as "multiscale fusion module"). After processing by the channel-based multimodal information exchange module, coarse-grained Coarse-grained Fine-grained and fine-grained The multi-modal features of the image are then sent to the multi-scale fusion module for further representation learning.

[0092] The multimodal multi-scale fusion module can be used to fuse the cls tokens from one branch and the patch tokens from another branch according to the image features and text features on different branches. Figure 1As shown, the multi-modal multi-scale fusion module may include a multi-scale fusion transformer (or transformer), which can fuse image features and text features on different branches to generate coarse-grained fused image features, fine-grained fused image features, coarse-grained fused text features, and coarse-grained fused image features. Each fusion will generate the above four fusion features. The multi-scale fusion transformer has a total of four layers, that is, four fusion processes are performed.

[0093] Figure 3 It is a schematic diagram of multi-scale fusion of large branches (i.e. large scale or coarse granularity). Figure 3 As shown, the cls token of the large branch acts as a query token and interacts with the patch token of the small branch through a cross-attention mechanism. The smaller branch follows the same process but exchanges the patch token from the other branch.

[0094] The model proposed in the present invention can use the information exchange between the image cls token from the large branch and the text patch token from the small branch, and then return to the corresponding branch (i.e., the large branch), and at the same time, perform the same operation between the image cls token of the small branch and the text patch token of the large branch; it can also use the information exchange between the text cls token from the large branch and the image patch token from the small branch, and then return to the corresponding branch (i.e., the large branch), and at the same time, perform the same operation between the text cls token of the small branch and the image patch token of the large branch.

[0095] Since the image cls tag at each scale has learned the abstract information from all the patch tags in its branch, interacting with the text patch tags on other branches helps learn information at different scales. After being fused with the text tags from other branches, the image cls tag interacts with its patch tags again in the subsequent transformer encoder. This operation helps to efficiently transfer the information learned from other branches to its patch tokens, thereby enriching the multi-scale representation of the image patch tokens.

[0096] The image cls token from the large branch and the text patch token from the small branch are concatenated and cross-attention is performed with the image cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the large branch to obtain the fused image feature of the large branch, i.e., the coarse-grained fused image feature I′ L .

[0097] The image cls token from the small branch and the text patch token from the large branch are concatenated and cross-attention is performed with the image cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the small branch to obtain the fused image feature of the small branch, i.e., the fine-grained fused image feature I′ S ;

[0098] The text cls token from the large branch and the image patch token from the small branch are concatenated and cross-attention is performed with the text cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the large branch to obtain the fused text feature of the large branch, i.e., the coarse-grained fused text feature.

[0099] The text cls token from the small branch and the image patch token from the large branch are concatenated and cross-attention is performed with the text cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the small branch to obtain the fused text feature of the small branch, that is, the fine-grained fused text feature.

[0100] The fused image features of different scales generated in the above multi-scale interaction process can be expressed by formulas 7 and 8 respectively:

[0101]

[0102] Among them, the cls subscript indicates the cls token of the image or text feature generated after the information exchange is completed; the coarse granularity is indicated by the superscript L, and the fine granularity is indicated by the superscript S; the patch subscript indicates the corresponding patch token; is the concatenation operation, and CA is the cross attention, as shown in Formula 9:

[0103]

[0104] Among them, W q ,W k ,W v are learnable matrix parameters.

[0105] It should be noted that the above formulas 7 and 8 only represent the process of generating fused image features of different scales in the multi-scale interaction process; since the process of generating fused text features of different scales in the multi-scale interaction process is similar to the formula expression, it is necessary to replace I used to represent the image and T used to represent the text in formulas 7 and 8. t Just make the exchange.

[0106] After multi-scale information fusion, the output embeddings of the multi-scale feature fusion module are finally connected to a fully connected network to obtain the final fused embedding matrix and subsequent multi-label classification prediction.

[0107] In summary, the overall loss of the overall optimization objective function during MMTF training is It can be expressed as:

[0108]

[0109] Among them, α and β are hyperparameters used to measure the importance of the three tasks. (In the following experiments, α and β are both set to 0.0001).

[0110] In order to evaluate the effectiveness and reliability of the MMTF model, a large-scale chest X-ray image and radiology report dataset MIMIC-CXR-JPG was used for pre-training, and then fine-tuned on four public datasets. The experimental results show that the method of the present invention surpasses the previous state-of-the-art network model in classification performance. In the follow-up, an in-depth overview of the datasets, evaluation metrics, and experimental settings will be provided. Then, the experimental results of all datasets are shown, including a comprehensive performance analysis of the model of the present invention based on a series of ablation studies conducted on the VinBigData dataset.

[0111] The present invention uses the MIMIC-CXR-JPG dataset for pre-training. MIMIC-CXR-JPG is a publicly accessible dataset of chest X-rays with accompanying X-ray reports. MIMIC-CXR-JPG contains more than 370,000 radiographs, equivalent to 227,835 radiological studies performed on 65,279 patients. Each radiological study is accompanied by a chest X-ray image, a free text radiology report, and abnormality / disease labels obtained through human-assisted intervention. Among them, the report is a radiologist's summary of their findings, including examinations, inductions, impressions, findings, techniques, and comparisons. In fact, the report only retains two main parts: findings and impressions. Findings are a description of the essential parts of the radiograph, while impressions summarize the most directly relevant findings.

[0112] To evaluate the model, fine-tuning and ablation experiments were also performed on the model. The training set, validation set, and test set were all divided using the official split, with the split percentages being 70%, 10%, and 20%, respectively. A detailed description of the fine-tuning dataset and implementation is provided below.

[0113] NIHChestX-ray14: NIHChestX-ray14 has 112,120 disease-labeled chest X-rays from 30,805 patients. The dataset includes 14 chest abnormalities: atelectasis, cardiomegaly, effusion, infiltration, mass, nodule, pneumonia, pneumothorax, consolidation, edema, emphysema, fibrosis, pleural thickening, and hernia. In the radiographs, all labels are obtained by mining the relevant reports using natural language processing tools.

[0114] VinBigData: VinBigData provides 14 chest abnormalities, namely aortic enlargement, atelectasis, pneumothorax, pulmonary opacity, pleural thickening, interstitial lung disease, pulmonary fibrosis, calcification, pleural effusion, consolidation, cardiomegaly, other lesions, nodules, masses, and infiltration. The training set contains 15,000 individually labeled chest X-rays, and the test set contains 3,000 chest X-rays.

[0115] ShenzhenTuberculosis: Shenzhen Tuberculosis was collected by Guangzhou Medical College in collaboration with Shenzhen Third People's Hospital. Chest X-rays were mainly from outpatient clinics, including 662 frontal chest X-rays. Among them, 326 were normal subjects and the remaining 336 were cases with tuberculosis manifestations.

[0116] COVID-19Image DataCollection: The COVID-19 image dataset contains more than 900 cases of pneumonia chest X-rays. Experiments were conducted to distinguish between COVID-19 and non-COVID-19 cases, and between viral and bacterial pneumonia.

[0117] The MIMIC-CXR-JPG dataset was pre-trained. During the entire pre-training process, the batch size was set to 64 and the epoch was set to 5. The optimizer used AdamW and its initial learning rate was set to 5×10 -5 A linear learning rate warm-up strategy is used to accelerate model convergence. Training is performed on a single 4090 GPU using the MIMIC-CXR-JPG dataset in a PyTorch environment.

[0118] During the fine-tuning phase, the batch size was set to 32 and the learning rate was set to 10. The AdamW optimizer was used and the learning rate was set to 1×10 -5 , the decay strategy is consistent with the pre-training process. No additional data augmentation is applied except cropping the images to pixels of uniform size. The area under the ROC curve (AUROC) is used to evaluate the model performance to ensure the consistency of the metrics in the experiments. This setting allows a direct transition from pre-training to fine-tuning, emphasizing the stability and simplicity of parameter processing.

[0119] According to the mode used in the pre-training stage, the state-of-the-art models are generally divided into two categories: medical image pre-training methods and medical image text pre-training methods. Therefore, MMTF is compared and validated with the existing state-of-the-art models on four fine-tuning datasets.

[0120] The medical image pre-training methods involved in the comparison include:

[0121] ModelGenesis: is a self-supervised visual representation learning method that can perform multiple types of generative restoration based on input radiographs;

[0122] Comparedto Learn: is a self-supervised visual representation learning method that builds homogeneous and heterogeneous data pairs by mixing images and features.

[0123] ImageNet Pre-training: is a representative pre-training method for supervised learning on large-scale natural image datasets.

[0124] In addition, for better comparison, supervised ConvNet and Transformer-based comparative experiments are also used.

[0125] The medical image text pre-training methods involved in the comparison include:

[0126] MuSE: Performs multimodal fusion via channel exchange and excels in entity recognition and sentiment analysis tasks.

[0127] ConVIRT: Jointly training visual and textual encoders with paired medical images and reports via bidirectional contrastive learning.

[0128] GLoRIA: Multimodal feature representation for medical images via global and local contrast learning.

[0129] MedKLIP: Use report filters to extract medical entities and use more complex modality fusion to aggregate features.

[0130] The embodiments of the present invention are compared with other SOTA image pre-training and image-text pre-training models under the same deployment environment, and ablation experiments are performed accordingly. All image pre-training and image-text pre-training baselines are first pre-trained on MIMIC-CXR-JPG. In the fine-tuning stage, the transferability of the model is evaluated by fine-tuning the model using 1%, 10%, and 100% parts of four mature datasets (including NIHChestX-ray, VinBigData chest X-ray abnormality detection, Shenzhen tuberculosis, and pneumonia image datasets caused by the new coronavirus).

[0131] In Table 1, classification results for image pre-training methods on four fine-tuning datasets are examined, covering both limited and full supervision. These results are consistent with the broader medical image analysis literature. Evaluation results for specific disease types are detailed in the Appendix. Overall, this analysis highlights that MMTF consistently outperforms baseline methods on datasets with varying labeling ratios, especially when annotated data is limited.

[0132] Table 1 Classification results of image pre-training methods on four fine-tuning datasets

[0133]

[0134]

[0135] In Table 1, NIH, VBD, and SZ represent NIH ChestX-ray, VinBigData ChestX X-ray anomaly detection, and Shenzhen tuberculosis datasets. C-T1 (or CT1) and C-T2 (or CT2) represent two tasks in the collection of pneumonia image data caused by the new coronavirus: distinguishing between pneumonia cases caused by the new coronavirus and pneumonia cases not caused by the new coronavirus (C-T1) and distinguishing between viral pneumonia and bacterial pneumonia cases (C-T2). The evaluation indicator is AUROC (i.e., Area Under the Receiver Operating Characteristic), where the best results are shown in bold.

[0136] Specifically, in the fine-tuning stage, MMTF shows superior performance by achieving the highest AUROC compared to traditional pre-trained models, such as Figure 4 As shown. Figure 4 NIH ChestX-ray (a) and VinbBigData chest X-ray abnormality detection (b). AUROC indicates the macro average of all diseases. It should be noted that the percentage of labels used in the training data. Figure 4 As shown, on the NIH ChestX-ray dataset, MMTF achieves an average improvement of 6% over pre-trained models such as Model Genesis and ImageNet-based models with limited supervision. This result is particularly important in the context of medical image analysis, as it is challenging to obtain a large amount of expert annotations. In addition, on the VinBigData dataset, MMTF achieves an AUROC of 90.9% with only 10% labeled data, surpassing models trained on fully annotated datasets. These findings illustrate the powerful representation capabilities of MMTF, especially when dealing with small datasets.

[0137] Further analysis shows that MMTF performs well in tasks involving tuberculosis detection, outperforming the C2L model by 2% in AUROC. Even in simpler tasks, such as the classification of pneumonia caused by the new coronavirus, MMTF outperforms the Transformer model by 1.4%, largely due to its efficient channel-based multimodal information exchange module. The model's most notable advantage is in distinguishing between viral and bacterial infections, with an AUROC at least 4.8% higher than the baseline method. These results highlight MMTF's ability to transfer learned multimodal representations across tasks, making it particularly valuable in clinical settings where labeled data is often scarce.

[0138] Table 2 shows the image-text pre-training classification results on four fine-tuning datasets, where the image-text pre-training baseline is based on the same backbone as MMTF. AUROC is the evaluation metric. The best results are shown in bold. Table 2 demonstrates the consistent advantage of MMTF over the baseline models.

[0139] Table 2 Comparison of the present invention with the image-text pre-training baseline

[0140]

[0141]

[0142] Figure 5 This is a comparison between the MMTF model proposed in this paper and the SOTA medical image text pre-training model in NIH ChestX-ray and VinBigData chest X-ray abnormality detection. The AUROC scores of the multi-label classification task on the NIH ChestX-ray dataset are shown in (a), and the AUROCs scores of the multi-label classification task on the VinBigData dataset are shown in (b). The optimal result is used as the upper limit of each category in the radar chart.

[0143] like Figure 5 As shown in Figure 2, MMTF outperforms existing image-text pre-trained models on almost all chest diseases. This performance is particularly remarkable under constrained supervision, such as when using only 1% of labeled data on the NIH Chest X-ray dataset, MMTF significantly outperforms MuSE by at least 15%. This large advantage can be attributed to MMTF's multi-scale representation learning, which enhances its generalization ability across tasks and datasets, especially when labeled data is scarce.

[0144] It should be noted that both MMTF and MuSE adopt channel-based multimodal information exchange. Although MuSE is effective in entity recognition and sentiment analysis tasks, it is not optimized for complex medical image analysis, where capturing fine-grained hierarchical features is critical. MMTF expands the potential of channel-based fusion by combining multi-scale learning, making it perform better in medical tasks such as identifying consolidation, lung opacity, and pneumothorax. For example, MMTF achieves AUROC scores of 97.3%, 95.7%, and 97.9% in these cases, respectively, surpassing other models such as MedKLIP and ConVIRT.

[0145] In addition, MMTF showed even greater advantages on the VinBigData dataset, performing well in distinguishing viral from bacterial pneumonia (9% improvement over baseline) and diagnosing pneumonia caused by the new coronavirus (nearly 3% improvement over MedKLIP). These findings are not only practically significant, but also statistically supported. The p-value analysis emphasizes that the performance improvement of MMTF is extremely unlikely to be due to chance, with a p-value as low as 3.25×10 -4 , using a dataset with 10% labeled data, the robustness of the model under different supervision levels is demonstrated. Even when limited to 1% of labeled data, the gains of MMTF remain statistically significant, confirming that its multimodal fusion and multiscale learning approach provides concrete improvements in medical image analysis.

[0146] In the embodiment, the proposed MMTF model is also compared with other state-of-the-art multimodal lung disease diagnosis methods, including MultiFusionNet, Late Fusion, and CheXMed. All three methods involve inputting multimodal data, including electronic medical records and chest X-rays. It is worth noting that, for the sake of fairness, in subsequent experiments, all these models were first pre-trained on the MIMIC-CXR dataset and then fine-tuned on downstream datasets.

[0147] Figure 6 The comparison results of the proposed MMTF and SOTA multimodal lung diagnosis on NIH ChestX-ray, VinBigData ChestXX-ray anomaly detection and Shenzhen tuberculosis datasets.

[0148] like Figure 6As shown in the figure, the MMTF model consistently outperforms other competing methods in terms of generalization and diagnostic accuracy, especially when dealing with scarce labeled data. For example, MultiFusionNet uses a multi-layer fusion technique to merge features from different convolutional layers, improving classification by leveraging different feature maps from deeper layers. However, MMTF achieves an accuracy of 76.4% on the NIH dataset with only 1% labeled data, significantly higher than MultiFusionNet's 68.9%, indicating that MMTF has stronger generalization capabilities under limited supervision. Similarly, Late Fusion integrates multimodal data such as clinical records and chest X-rays, which has some advantages in multimodal settings, but performs poorly in more complex tasks. On the SZ and C-T2 datasets, MMTF achieves accuracies of 97.8% and 92.9%, respectively, which are better than Late Fusion's 92.8% and 70.2%, highlighting MMTF's ability to capture hierarchical features that are critical for accurate medical image analysis. CheXMed combines clinical records with chest X-ray data and performs well in specific areas such as age-based pneumonia detection, but still has shortcomings overall. On the NIH dataset with 1% labeled data, CheXMed scored only 66.8%, while MMTF scored 76.4%. MMTF's multi-scale fusion mechanism enables it to capture finer details of lung pathologies such as pneumothorax and lung opacity, resulting in more accurate diagnoses on multiple datasets. Overall, MMTF's advanced multimodal fusion, multi-scale aggregation, and Transformer-based feature extraction enable it to more effectively handle the complexity of medical data, with higher robustness and diagnostic accuracy than similar products.

[0149] MMTF provides reliable decision support for clinicians by accurately identifying and localizing lung lesions. In addition to diagnostic accuracy, interpretability plays a vital role in AI-assisted diagnosis. To this end, an interpretability analysis was performed using the NIH ChestX-ray and VinBigData chest X-ray datasets to evaluate the ability of MMTF to detect lesion areas.

[0150] To make the decision process of the model more transparent, the cross-attention maps of each transformation layer of the multi-scale fusion module are averaged and visualized.

[0151] Figure 7 is the visualization result of randomly selected samples. Figure 7 Eight randomly sampled images from the NIH ChestX-ray dataset (ad) and VinBigData (eh) are shown. On the left side of the eight randomly sampled images, the original image and the attention generated by MMTF Figure 1In each original image, a red box highlights the lesion area annotated by the radiologist. The attention map uses a color spectrum from red to blue overlaid on the original image, where red indicates high attention areas and blue indicates low attention areas.

[0152] pass Figure 7 It can be seen that MMTF effectively integrates information from different regions, which is closely related to the pathological location determined by the radiologist. It is worth noting that the multi-scale fusion mechanism allows MMTF to detect lung lesions at various scales, ensuring accuracy even for smaller lesions (e.g., the first image in the NIH Chest X-ray, Figure 7 a).

[0153] In addition, to evaluate the transfer learning ability of the model on smaller datasets, t-distributed stochastic neighbor embedding (t-SNE) was applied on the Shenzhen tuberculosis dataset and visualized. Figure 8 Figures ac show the t-SNE visual clustering results of the Shenzhen tuberculosis and pneumonia caused by the new coronavirus image data collection dataset. Figure 8 The clustering results in show that MMTF can accurately capture the local and global structure of the data. Whether it is the relatively simple task of tuberculosis classification or the more challenging task of distinguishing pneumonia caused by the new coronavirus from other types of pneumonia, MMTF shows strong discrimination ability. This highlights the robustness of the model, especially in identifying viral and bacterial pneumonia cases - a key challenge in real-world clinical settings.

[0154] Therefore, through the above experiments, we can see that MMTF not only performs well in diagnostic accuracy, but also improves interpretability and provides practical clinical value by providing more transparent and reliable diagnostic insights. This feature enhances its practicality for clinicians and increases trust in AI-driven medical solutions.

[0155] A thorough ablation experiment was conducted on MMTF by removing or replacing individual modules. Table 3 shows the quantitative results of the ablation experiment. In Table 3, Task1 represents the generation task of generating images from text; Task2 represents the generation task of generating text from images; L-encoder represents the large-branch image encoder; S-encoder represents the small-branch image encoder; Channel-based module represents the channel-based information exchange module. The check mark in Table 3 indicates that the component, unit or module exists, and the unchecked one indicates that the component, unit or module is removed. The results show that the proposed module can effectively solve the problem of poor multimodal representation before.

[0156] Table 3 Ablation study of MMTF by removing or replacing individual modules

[0157]

[0158] First, the effect of replacing the two generation tasks (rows 0 and 1 in Table 3) in order of examination on the prediction performance. When only image generation (i.e., Task 1) or text generation (i.e., Task 2) is retained in the model of the present invention, the overall performance of MMTF on the Vinbigdata dataset decreases by 1.28% and 1.09% respectively (compared to row 6). It is worth noting that removing the generation task causes the overall model performance to decrease by 2.38%. This indicates that the generation tasks used in the model reduce and improve the loss of fine-grained semantics due to modality fusion.

[0159] In addition to the generation task within the model, the effect of the multi-scale feature fusion module is also evaluated. First, the multi-scale fusion module is removed to study the impact of the single-branch image encoder on the model performance. The results show that in the single-scale case (rows 3 and 4 in Table 3), the model prediction performance is significantly reduced (6.77% and 7.31%) compared with the dual-branch model (row 6 in Table 3). This result means that single-scale multimodal fusion cannot effectively represent the relationship between lesions and text descriptions in different image regions. Next, the channel-based information exchange module is removed to verify the effectiveness of channel-based multimodal information exchange. The results show that multimodal information fusion directly between multiple scales after encoding is 2.84% lower than the original model AUROC. This shows that channel-based inter-modal information exchange is necessary before performing multi-scale information fusion.

[0160] The proposed MMTF model has great potential for clinical applications, especially in resource-limited settings where annotated data is scarce. By leveraging existing free text radiology reports, MMTF reduces the reliance on extensive manual labeling and facilitates a more efficient diagnostic process. This is particularly important in real-world clinical settings, where the availability of fully annotated datasets is often limited.

[0161] In addition, the strong performance of MMTF in various datasets (such as NIH ChestX-ray, VinBigData, and Shenzhen Tuberculosis) highlights its adaptability to different clinical tasks, from common diseases such as tuberculosis to more complex diseases such as pneumonia and pneumonia caused by the new coronavirus. This shows that the model can provide reliable decision support in various medical settings, improve diagnostic accuracy, and reduce the workload of clinicians.

[0162] In addition, MMTF integrates multi-scale feature fusion and auxiliary generation tasks, allowing clinicians to better visualize and understand the model's decision-making process, thereby improving interpretability. This feature is well aligned with the growing demand for transparent AI systems in the healthcare field.

[0163] Finally, the model's ability to achieve high performance with limited supervision and its ability to capture fine-grained semantic details suggest that it can be applied to other medical imaging tasks, including segmentation and reconstruction, making it a valuable tool for a wide range of medical applications.

[0164] A novel multimodal multiscale transform fusion (MMTF) method proposed in this paper can be used to analyze medical images and corresponding reports. MMTF integrates multimodal, multi-stage, and multi-level fine-grained features to address the limitations of cross-modal fusion of heterogeneous medical data. Detailed fusion is achieved by exchanging token embeddings between image and text modalities, while the cross-attention-based multiscale fusion module enables effective information exchange between branches. In addition, two generation tasks capture the correlation between text and image, narrowing the gap in cross-modal attention representation. Experimental results show that MMTF achieves state-of-the-art performance on several benchmark lung disease datasets, and interpretability analysis further confirms the model's ability to accurately capture lung abnormalities. Good performance is obtained on four public datasets compared to existing methods, supporting the effectiveness of the model in diagnosing lung abnormalities.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-modal multi-scale transformation fusion method based on exchange, characterized in that: include: Acquire multimodal data, namely, original image data and original text data corresponding to the original image data; The original image data and the original text data are input into a multimodal multiscale transformation fusion model to generate a fusion result, wherein the multimodal multiscale transformation fusion model includes: a multimodal encoder module, a decoder module, a channel-based information exchange module and a multiscale fusion module, The multimodal encoder module consists of a text encoder and a dual-branch image decoder. The text encoder is used to encode the input text data and generate text embedding; The dual-branch image encoder includes a small-branch image encoder and a large-branch image encoder, which are used to extract a fine-grained image embedding and a coarse-grained image embedding of input image data respectively; The decoder module contains two decoders, which decode the embeddings generated by the text encoder and the dual-branch image encoder respectively, so as to generate images from text and text from images, so as to regularize the embeddings generated by the text encoder and the dual-branch image decoder. The channel-based information exchange module is used to exchange information between embeddings of different modalities on different channels according to the attention scores of different channels, thereby generating image features and text features on different branches; A multi-scale fusion module is used to fuse the cls tokens from one branch and the patch tokens from another branch based on the image features and text features on different branches.

2. The multi-modal multi-scale transformation fusion method based on exchange according to claim 1, characterized in that: The information exchange of embeddings of different modalities includes the information exchange between coarse-grained image embedding and text embedding, and the information exchange between fine-grained image embedding and text embedding. The channel-based information exchange module also uses two hyperparameters η and μ to control the starting layer and the ending layer of the multimodal information exchange, respectively.

3. The multi-modal multi-scale transformation fusion method based on exchange according to claim 1, characterized in that: The channel-based information exchange module includes a multi-head self-attention unit and a multimodal information exchange unit. The text embedding, fine-grained image embedding and coarse-grained image embedding plus their corresponding cls tokens are fed to the multi-head self-attention unit to obtain the attention scores of each token in the text embedding, fine-grained image embedding and coarse-grained image embedding. The calculated attention scores are sent to the multimodal information exchange unit through a residual connected feedforward network. The multimodal information exchange unit uses multiple Transformer encoders with shared parameters to exchange multimodal information of different scales in a complementary manner.

4. The multi-modal multi-scale transformation fusion method based on exchange according to claim 3 is characterized in that: The multimodal information exchange unit exchanges multimodal information of different scales in a complementary manner, including: for information exchange at any layer, for any channel, if the attention score of the current channel is lower than the set threshold, the embedding vector of the channel will be replaced by the average embedding of a preset percentage of tokens in another modality, otherwise, the embedding vector of the channel remains unchanged, thereby obtaining an updated embedding matrix.

5. The multi-modal multi-scale transformation fusion method based on exchange according to claim 4 is characterized in that: The channel-based information exchange module also includes FFN and CB units. After the information exchange of the current layer is completed, FFN generates the input embedding of the next layer of information exchange according to the updated embedding matrix, and CB is used to broadcast the context to each token.

6. The multi-modal multi-scale transformation fusion method based on exchange according to claim 4, characterized in that: Merge cls tokens from one branch with patch tokens from another branch, specifically: The image cls token from the large branch and the text patch token from the small branch are concatenated and cross-attention is performed with the image cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the large branch to obtain the fused image feature of the large branch, i.e., the coarse-grained fused image feature. The image cls token from the small branch and the text patch token from the large branch are concatenated and cross-attention is performed with the image cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the small branch to obtain the fused image feature of the small branch, i.e., the fine-grained fused image feature. The text cls token from the large branch and the image patch token from the small branch are concatenated and cross-attention is performed with the text cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the large branch to obtain the fused text feature of the large branch, i.e., the coarse-grained fused text feature. The text cls token from the small branch and the image patch token from the large branch are concatenated and cross-attention is performed with the text cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the small branch to obtain the fused text feature of the small branch, that is, the fine-grained fused text feature.

7. A multi-modal multi-scale transformation fusion system based on exchange, characterized in that: include: A multimodal data acquisition engine, used to acquire multimodal data, namely, image data and text data corresponding to the image data; Model training engine, used to train multi-modal and multi-scale transformation fusion models. The fusion generation engine is used to generate fusion results based on the acquired multimodal data using the trained multimodal multi-scale transformation fusion model. The multimodal multiscale transformation fusion model includes: a multimodal encoder module, a decoder module, a channel-based information exchange module and a multiscale fusion module. The multimodal encoder module consists of a text encoder and a dual-branch image decoder. The text encoder is used to encode the input text data and generate text embedding; The dual-branch image encoder includes a small-branch image encoder and a large-branch image encoder, which are used to extract a fine-grained image embedding and a coarse-grained image embedding of input image data respectively; The decoder module contains two decoders, which decode the embeddings generated by the text encoder and the dual-branch image encoder respectively, so as to generate images from text and text from images, so as to regularize the embeddings generated by the text encoder and the dual-branch image decoder. The channel-based information exchange module is used to exchange information between embeddings of different modalities on different channels according to the attention scores of different channels, thereby generating image features and text features on different branches; A multi-scale fusion module is used to fuse the cls tokens from one branch and the patch tokens from another branch based on the image features and text features on different branches.

8. The multi-modal multi-scale transformation fusion system based on exchange according to claim 7, characterized in that: The channel-based information exchange module includes a multi-head self-attention unit and a multimodal information exchange unit. The text embedding, fine-grained image embedding and coarse-grained image embedding plus their corresponding cls tokens are fed to the multi-head self-attention unit to obtain the attention scores of each token in the text embedding, fine-grained image embedding and coarse-grained image embedding. The calculated attention scores are sent to the multimodal information exchange unit through a residual connected feedforward network. The multimodal information exchange unit uses multiple Transformer encoders with shared parameters to exchange multimodal information of different scales in a complementary manner.

9. The multi-modal multi-scale transformation fusion system based on exchange according to claim 8, characterized in that: The multimodal information exchange unit exchanges multimodal information of different scales in a complementary manner, including: for information exchange at any layer, for any channel, if the attention score of the current channel is lower than the set threshold, the embedding vector of the channel will be replaced by the average embedding of a preset percentage of tokens in another modality, otherwise, the embedding vector of the channel remains unchanged.

10. The multi-modal multi-scale transformation fusion system based on exchange according to claim 9, characterized in that: The multi-scale fusion module performs the following operations: The image cls token from the large branch and the text patch token from the small branch are concatenated and cross-attention is performed with the image cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the large branch to obtain the fused image feature of the large branch, i.e., the coarse-grained fused image feature. The image cls token from the small branch and the text patch token from the large branch are concatenated and cross-attention is performed with the image cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the image cls token of the small branch to obtain the fused image feature of the small branch, i.e., the fine-grained fused image feature. The text cls token from the large branch and the image patch token from the small branch are concatenated and cross-attention is performed with the text cls token of the large branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the large branch to obtain the fused text feature of the large branch, i.e., the coarse-grained fused text feature. The text cls token from the small branch and the image patch token from the large branch are concatenated and cross-attention is performed with the text cls token of the small branch. Then, the patch token in the result of the cross-attention is concatenated with the text cls token of the small branch to obtain the fused text feature of the small branch, that is, the fine-grained fused text feature.

Images