Mutual learning method and system between multi-view medical images and text reports

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically discloses a mutual learning method and system between multi-view medical images and text reports. By constructing a mutual learning model and dividing the learning process of the mutual learning model into an image reconstruction task, a report reconstruction task, and a multi-view alignment task, the image reconstruction task obtains the feature representation of the multi-view medical image, and the report reconstruction task obtains the feature representation of the text report and cross-modally fuses it with the image feature representation to obtain a reconstructed text report. The integration of multimodal reconstruction tasks enables the model to learn richer and more detailed feature representations, thereby improving its robustness to lost or damaged data. In addition, the exchange-based multimodal fusion method in cross-modal text reconstruction aims to fully integrate visual features and enrich the semantic representation of domain-specific knowledge. Through pre-training and joint training, the model performance is optimized so that more accurate and comprehensive diagnostic results can be obtained after applying the mutual learning model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for mutual learning between multi-view medical images and text reports. Background Art

[0002] With the rapid development of deep learning technology, artificial intelligence (AI) is playing a vital role in assisting medical diagnosis and monitoring medical conditions. AI-assisted technologies have demonstrated their effectiveness in routine clinical tasks, including disease diagnosis and quantification of medical variables. However, training a single model from scratch for a specific application can be computationally intensive and require extensive manual annotation, fundamentally limiting the advancement of medical applications. Therefore, there is an urgent need to develop foundational medical models that can handle a wide range of clinical applications.

[0003] The persistent lack of large-scale annotated datasets remains a significant challenge in medicine. A widely adopted solution to reduce the annotation burden for downstream tasks involves leveraging pre-trained models. The application of the Vision-Language-Based Model in a clinical setting avoids the reliance on traditional annotated data by leveraging medical reports as auxiliary signals. This approach enables the model to learn robust and transferable feature representations applicable to a variety of tasks and domains. By deploying task-specific heads and leveraging the model's high-quality feature representations, the need for extensive manual annotation is significantly reduced.

[0004] ConVIRT is one of the earliest methods for medical visual language pre-training, which pioneered the use of radiology text reports to guide visual representation learning through cross-modal bidirectional contrastive learning. This approach enables the model to generalize to previously unseen samples and skillfully distinguish between matched and mismatched image-text pairs. However, ConVIRT only focuses on global feature comparison and ignores the key role of local fine-grained features. To address this gap, GLoRIA introduces a fine-grained weighting mechanism that calculates the similarity between individual words and specific image patches to generate word-level responses. Building on this concept, MGCA further improves GLoRIA by introducing multi-granularity alignment to learn the correspondence between fine-grained visual and textual tags.

[0005] Reconstruction-based self-supervised learning provides a complementary approach to cross-modal alignment methods such as ConVIRT, GLoRIA, and MGCA. Instead of focusing on the direct comparison between visual and textual features, it enhances feature extraction by reconstructing the masked portion of the input data to capture global and local semantics. Models such as MAE and M3AE adopt this strategy by jointly reconstructing images and text and learning robust representations. However, M3AE does not fully exploit the relationship between the two modalities. To address this issue, MRM integrates visual information through global average pooling (GAP) during the report reconstruction process to optimize semantic feature representation. Despite this, these models still have difficulty in fully integrating visual and textual information, limiting their overall effectiveness.

[0006] While advances in mask reconstruction strategies for visual language models represent progress, existing methods often overlook the importance of multi-view medical images, which are frequently generated in clinical practice and possess key inter-view relationships. These relationships hold great potential for enriching visual representation learning. Although recent studies have attempted to incorporate multi-view images into joint training, simply concatenating them and feeding them into the model has proven insufficient to capture lesion semantics or personalized features. This simplistic approach often leads to information redundancy, highlighting the need for more sophisticated techniques to effectively leverage multi-view data in medical AI. Summary of the Invention

[0007] The present invention provides a mutual learning method and system between multi-view medical images and text reports, and solves the technical problem that the current visual language basic model cannot effectively apply to multi-view image data due to the complex cross-modal relationship between multi-view images and corresponding text reports.

[0008] To solve the above technical problems, the present invention provides a mutual learning method between multi-view medical images and text reports, comprising the steps of:

[0009] S1. Build a mutual learning model;

[0010] The mutual learning model is used to implement image reconstruction tasks, report reconstruction tasks, and multi-view alignment tasks. The image reconstruction task is to extract visual features from multi-view medical images taken from multiple angles to obtain a reconstructed medical image; the report reconstruction task is to extract report features from the text report corresponding to the multi-view medical image, and integrate the report features and the visual features to generate a cross-modal reconstructed text report; the multi-view alignment task is to align the multi-view medical images and the text report in the feature space;

[0011] S2. Pre-training the image reconstruction task, the report reconstruction task, and the multi-view alignment task respectively;

[0012] S3. Jointly train the pre-trained image reconstruction task, the report reconstruction task, and the multi-view alignment task.

[0013] Furthermore, the image reconstruction task includes a visual encoding stage and an image reconstruction stage; the visual encoding stage is to use an image mask to mask the input view images, and then use an image encoder to extract the features of each masked image and fuse them to obtain a fused image; the image reconstruction stage is to use a visual decoder to reconstruct the fused image obtained in the visual encoding stage to obtain a reconstructed image.

[0014] Furthermore, the use of an image mask to mask the input view images specifically includes an image input process, an image mask process, and an image encoding process. The image input process is to perform image segmentation on the input multi-view image to obtain corresponding image blocks to form a patch sequence; the image mask process is to use an image mask to randomly select i blocks in the patch sequence of each view image to mask and obtain a masked image; the image encoding process is to input the masked image corresponding to each view image into a shared image encoder for feature extraction to obtain corresponding masked image features.

[0015] Furthermore, the method of extracting features of each masked image using an image encoder and fusing them specifically includes a semantic evaluation process, a weight calculation process and an image fusion process; the semantic evaluation process is to multiply the mask matrix corresponding to each view image with the learnable matrix in the image encoding process to obtain a corresponding semantic evaluation score matrix; the weight calculation process is to calculate the weight of the uncovered part of the masked image of each view image based on the semantic evaluation score matrix; the image fusion process is to multiply the weight corresponding to each view image with the uncovered image features in the masked image features, and then add the multiplication results of all view images to obtain a fused image.

[0016] Furthermore, the report reconstruction task includes a report input process, a report masking process, a report encoding process, a multimodal fusion process, and a report decoding process; the report input process is to convert the original text report into an original word block sequence through a word segmenter; the report masking process is to use a report mask to randomly select n word blocks from the original word block sequence to mask, and obtain an original word block masked sequence and an original word block uncovered sequence; the report encoding process is to obtain a text embedding by extracting features from the original word block uncovered sequence using a report encoder; the multimodal fusion process is to integrate the text embedding with the fused image to obtain a new text embedding.

[0017] Furthermore, the multimodal fusion process designs an exchange-based image-text fusion module; the image-text fusion module uses two encoders with shared parameters to respectively calculate the attention scores of the fused image and text embeddings, and then replaces the text information at the corresponding position with the image information with the lowest attention score in the exchange layer to obtain a new text embedding.

[0018] Furthermore, the multi-view alignment task includes a multi-view encoding stage, a multi-text encoding stage, and a multi-view alignment stage; the multi-view encoding stage uses a shared image encoder to encode the cover image corresponding to each view image into a feature space to obtain the corresponding cover image features of each view;

[0019] The multi-text encoding stage includes enhancing the original text report to obtain an enhanced text report, and then masking the enhanced text report to obtain an enhanced masked text report; then the original masked text report and the enhanced masked text report are fed into a shared report encoder to generate corresponding original masked text report features and enhanced masked text report features;

[0020] The multi-view alignment stage is used to pair multiple view mask image features with multiple mask text report features one by one and then perform feature alignment through training.

[0021] Furthermore, during the pre-training process of the image reconstruction task, the mean square error loss function is used to calculate the loss between the reconstructed i image blocks and the corresponding true value blocks in the reconstructed image. During the pre-training process of the report reconstruction task, a conditional probability function is used to calculate the loss between the image encoder, report decoder and image-text fusion module. During pre-training for the multi-view alignment task, the loss between each pair of view mask image features and mask text report features is calculated.

[0022] Furthermore, the loss function used for the joint training of the pre-trained image reconstruction task, the report reconstruction task and the multi-view alignment task is equal λ RA To balance the weight coefficients of various losses, are the losses of the image reconstruction task, the report reconstruction task, and the multi-view alignment task respectively; It is equal to the sum of the alignment losses between multiple pairs of view images and text reports with weight coefficients.

[0023] The present invention also provides a mutual learning system between multi-view medical images and text reports, the key of which is that the system is provided with an intelligent agent, which is used to implement the mutual learning method between multi-view medical images and text reports.

[0024] The present invention provides a mutual learning method and system between multi-view medical images and text reports. By constructing a mutual learning model and dividing the learning process of the mutual learning model into an image reconstruction task, a report reconstruction task, and a multi-view alignment task, the image reconstruction task obtains the feature representation of the multi-view medical image, and the report reconstruction task obtains the feature representation of the text report and cross-modally fuses it with the image feature representation to obtain a reconstructed text report. The integration of multimodal reconstruction tasks enables the model to learn richer and more detailed feature representations, thereby improving its robustness to lost or damaged data. In addition, the exchange-based multimodal fusion method in cross-modal text reconstruction aims to fully integrate visual features and enrich the semantic representation of domain-specific knowledge. This method enhances the model's ability to explore lesion information. Through pre-training and joint training, the model performance is optimized so that more accurate and comprehensive diagnostic results can be obtained after applying the mutual learning model. Experimental results demonstrate the effectiveness and excellence of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of a method and system for mutual learning between multi-view medical images and text reports provided by an embodiment of the present invention;

[0026] Figure 2 Schematic diagram of semantic evaluation, weight calculation and image fusion provided by an embodiment of the present invention;

[0027] Figure 3 is a structural diagram of an image-text fusion module provided by an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of the principle of multi-view cross-modal alignment provided by an embodiment of the present invention;

[0029] Figure 5 is the hyperparameter λ IL and λ RA Diagram of the structure for performing grid search. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings. The embodiments are provided for illustrative purposes only and are not to be construed as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of protection of the present invention. Many changes may be made to the present invention without departing from the spirit and scope of the present invention.

[0031] The embodiment of the present invention provides a mutual learning method between multi-view medical images and text reports, such as Figure 1 As shown in the schematic diagram, the specific steps include:

[0032] S1. Build a mutual learning model;

[0033] The mutual learning model is used to implement image reconstruction tasks, report reconstruction tasks, and multi-view alignment tasks. The image reconstruction task is to extract visual features from multi-view medical images taken from multiple angles to obtain a reconstructed medical image; the report reconstruction task is to extract report features from the text report corresponding to the multi-view medical image and integrate the report features and visual features to generate a cross-modal reconstructed text report; the multi-view alignment task is to align the multi-view medical images and text reports in the feature space;

[0034] S2, pre-training the image reconstruction task, report reconstruction task and multi-view alignment task respectively;

[0035] S3. Jointly train the pre-trained image reconstruction task, report reconstruction task, and multi-view alignment task.

[0036] The mutual learning method between multi-view medical images and text reports provided by an embodiment of the present invention constructs a mutual learning model and divides the learning process of the mutual learning model into an image reconstruction task, a report reconstruction task, and a multi-view alignment task. The image reconstruction task obtains the feature representation of the multi-view medical image, and the report reconstruction task obtains the feature representation of the text report and cross-modally fuses it with the image feature representation to obtain a reconstructed text report. The integration of multimodal reconstruction tasks enables the model to learn richer and more detailed feature representations, thereby improving its robustness to lost or damaged data. In addition, the exchange-based multimodal fusion method in cross-modal text reconstruction aims to fully integrate visual features and enrich the semantic representation of domain-specific knowledge. This method enhances the model's ability to explore lesion information. Through pre-training and joint training, the model performance is optimized so that more accurate and comprehensive diagnostic results can be obtained after applying the mutual learning model. Each step is described in detail below.

[0037] In step S1, Figure 1As shown, the mutual learning model is built around three pre-training tasks: image reconstruction (MIM), report reconstruction (MLM), and multi-view alignment (MVA). The model uses frontal and side views of medical images (X-rays) as input for the image reconstruction task, further extracting features of the X-ray lesion region from different angles. The image reconstruction task employs the MAE method, which masks image patches across views using image masks to reduce visual information redundancy through extensive masking. The report reconstruction task uses WordPiece as a word segmenter and introduces an exchange-based cross-modal embedding fusion (CEF) module to integrate visual and report features to generate cross-modal representations. These cross-modal representations are enriched with visual information and then fed into the report decoder for report reconstruction. This report reconstruction method improves the masking rate of text reports to enhance cross-modal fusion and interaction with visual data. The multi-view alignment task focuses on aligning images and text in the feature space to explore semantic consistency between different views, thereby achieving zero-shot functionality.

[0038] (1) Image reconstruction task (MIM)

[0039] like Figure 1 As shown in Figure 2, the image reconstruction task (MIM) includes the visual encoding stage and the image reconstruction stage. In the visual encoding stage, each input view image is masked with an image mask, and then the features of each masked image are extracted and fused using an image encoder to obtain a fused image.

[0040] Specifically, the visual encoding stage includes image input, image masking, image encoding, semantic evaluation, weight calculation, and image fusion. The image input stage segments the input multi-view image to obtain corresponding image blocks, forming a patch sequence. The image masking stage uses an image mask to randomly select i blocks from the patch sequence of each view image and mask them to obtain a masked image. The image encoding stage inputs the masked image corresponding to each view image into a shared image encoder for feature extraction, obtaining the corresponding masked image features. The semantic evaluation stage multiplies the mask matrix corresponding to each view image by the learnable matrix used in the image encoding stage to obtain the corresponding semantic evaluation score matrix. The weight calculation stage calculates the weights of the uncovered portions of the masked image for each view image based on the semantic evaluation score matrix. The image fusion stage multiplies the weights corresponding to each view image by the uncovered image features in the masked image features, and then sums the multiplication results for all view images to obtain the fused image.

[0041] During the image input process of the visual encoding stage, the input image of a single perspective is represented as C, H, and W represent the number of channels, height, and width of the input image, respectively. This example takes the input image including the front view and the side view as an example. Through the reshape operation, the front view and the side view are divided into non-overlapping blocks of size P×P, and N p =(H×W) / P 2 blocks, thus representing the front view and side view as corresponding patch sequences, each of which consists of N p P 2 ×C patches (i.e., image blocks).

[0042] In the image masking process of the visual encoding stage, in order to cover more image areas and minimize the redundancy between multiple views, an image mask with a masking rate of 0.75 is used to randomly select i blocks from the patch sequence of the front view for masking. Similarly, an image mask with a masking rate of 0.75 is used to randomly select i blocks from the patch sequence of the side view for masking. Considering the front view and the side view as a whole, the overall masking rate is 50%. Therefore, the entire front view can be divided into the front uncovered image sequence and the front mask image sequence The entire side view can be divided into a sequence of side unmasked images and side mask image sequences

[0043] In the image encoding process of the visual encoding stage, MAE is used as the image encoder E in order to extract meaningful feature representations from the input multi-view images. I In this process, the front unmasked image sequence and the front mask image sequence Input into the shared image encoder to obtain the corresponding feature representation and At the same time, the side unmasked image sequence and side mask image sequences Input to the shared image encoder E I In the above example, we obtain the corresponding feature representation and It should be noted that the image encoder E I The self-attention mechanism is applied to image features, enabling them to simulate complex spatial relationships and capture long-distance dependencies. Then, after encoding the image, position embedding is added to the obtained encoded features to provide the original position information for the encoded features to obtain an image feature sequence.

[0044] Side views play a crucial auxiliary role in understanding the frontal view. By providing complementary perspectives, side views can reveal spatial and structural details that may not be visible in the frontal view. But how can the differences between views be exploited to achieve high-level semantic consistency in representation learning while avoiding feature redundancy? The answer to this question depends on whether the multi-view sampled patches can complement each other. If the sampled patches accurately cover the lesion area from different angles, it can ensure that the multi-view patches are highly correlated, thereby promoting complementary differentiation. To achieve this process, as Figure 2 As shown, the present invention designs a semantic evaluation process, a weight calculation process and an image fusion process.

[0045] During the semantic evaluation process in the visual encoding stage, the sampled images are scored according to the mask matrix, and then these scores are used to adjust the weights of the patches in each view. By doing so, regions with high correlation between views are given higher weights and are integrated into the final multi-view fusion. Given the mask matrices M for the front view and the side view f and M l (Notice Figure 2 For illustration purposes only, the image mask shown is Figure 1 is inconsistent. In the actual processing process, for the same set of multi-view images, the mask matrix is ​​the matrix corresponding to the image mask used in the masking process). In the mask matrix, the masked area of ​​the image is assigned a value of 0, while the uncovered area is assigned a value of 1. Then, the learnable matrix P is used respectively. f and P l to capture the semantic distribution of the front view and the side view. Finally, the mask matrix of the same view is multiplied by the learnable matrix to obtain the semantic evaluation score of each image. Therefore, the semantic evaluation scores of a set of front and side views can be expressed as and (B represents the batch size). The learnable matrix P f and P l It is a core component of deep learning models and is usually represented as the parameters that the model needs to optimize during training. These matrices are obtained through model initialization and training and represent the weights required for the model to learn specific tasks from the data.

[0046] During the weight calculation process of the visual encoding stage, the weights W of the uncovered areas in the mask images of the front view and the side view are calculated according to the semantic evaluation score matrix F and W L :

[0047]

[0048]

[0049] in, Indicates the importance of a single (i-th) front view, represents the weighted importance of the kth frontal view in a batch, exp() represents the exponential function, Indicates the importance of a single (i-th) side image, represents the weighted importance of the k-th side view in a batch.

[0050] During the image fusion process at the visual encoding stage, the weights W of the uncovered areas in the mask images of the front view and the side view are calculated. F and W L As the positive unmasked image feature I fU and side unmasked image features I lU The weight factor is used to fuse the features and obtain the fused view features:

[0051] I Fusion =exp(W F )×I fU +exp(W L )×I lU (3)

[0052] Fused view features (D represents the feature dimension.) This fusion function effectively combines key details from the two viewpoints, enhancing the model’s ability to capture a more comprehensive representation of the anatomical structure.

[0053] The image reconstruction stage is based on the visual decoder D I The fused image I obtained in the visual encoding stage Fusion Reconstruct and obtain the reconstructed image D I (I Fusion ), the focus is to restore the masked area. Here only the front view is reconstructed.

[0054] In step S2, during the pre-training process of the image reconstruction task (MIM), the reconstruction quality is evaluated using the mean square error (MSE) loss function, which calculates the reconstructed image D I (I Fusion ) reconstructed i image blocks (masked areas) and the corresponding i ground-truth blocks The loss between To ensure accurate restoration of the original image content, the image reconstruction loss function is expressed as follows:

[0055]

[0056] (2) Reporting Reconstruction Mission (MLM)

[0057] The report reconstruction task includes the report input process, report mask process, report encoding process, multimodal fusion process, and report decoding process.

[0058] The report input process is to convert the original text report into a sequence of raw word blocks through a word segmenter (WordPiece is used as the default word segmenter) Where K represents the dimension of each word chunk and M represents the number of word chunks. Figure 1 Taking English words as an example, there is no restriction on language category in actual application.

[0059] The report mask process is to use the report mask (with an overall mask rate of 50%) to the original word block sequence Randomly select n word blocks for masking to obtain the original word block masking sequence and the original word block unmasked sequence

[0060] A higher masking rate encourages the report encoder to better learn the image content by trying to reconstruct most of the masked text. At the same time, a higher masking rate enhances the model's ability to capture key information. Compared to the typical 15% masking rate, this example sends the word block sequence to the report encoder E. t Previously, a 50% masking rate was applied to it. The masked chunks are marked with "[MASK]" to maintain the same dimensionality as the chunk sequence before masking.

[0061] In the report encoding process, the original word block is unmasked by using the report encoder Perform feature extraction to obtain text embedding Among them, W l1 represents the learnable parameters, E pos1 is the position embedding of the entire report, and D represents the dimension of the feature.

[0062] The multimodal fusion process combines text embeddings with fused images to generate new text embeddings. To complement textual information with visual data for report reconstruction, a swap-based image-text fusion module is designed. Rather than directly concatenating visual and textual embeddings, this image-text fusion module first learns global contextual information about the input vector in the shallow layers of the image-text fusion module's encoder. The learned representations are then integrated across modalities to enhance overall feature alignment.

[0063] For convenience, use V e =I Fusion and T e =E tRespectively represent the image embedding and text embedding input in the multimodal fusion process. The image-text fusion module (cross-modal fusion module, MCF) introduces CrossTransformer based on the Transformer architecture. CrossTransformer uses two encoders with shared parameters (multi-head attention modules) to learn to fuse image and text embeddings respectively, thereby enhancing text content with visual information. The structure of the image-text fusion module is as follows: Figure 3 As shown in the figure, the “CLS” tag is appended to the front of the generated image embedding and text embedding. e and T e As the input of the two encoders respectively. To ensure that the global context information between the two input vectors is learned before the swap, the shallow layer of CrossTransformer is a regular Transformer encoder layer (multi-head attention) followed by a swap layer. In the swap layer, the token with the lowest attention score is selected from the text embedding and its embedding vector is replaced with the average score embedding of the image. The specific swap process is as follows: After adding the cls token, the embedding is generated and These embeddings are then processed and passed to the CrossTransformer, which consists of two submodules: the self-attention submodule and the exchange submodule. The self-attention submodule captures V e and T e Importance distribution of each token in , generating intermediate embeddings for images and text and In the exchange submodule, a certain θ proportion of tokens with the lowest attention scores are selected from the text embedding, and the corresponding text information is replaced with image information. Subsequently, a specified proportion of tokens with the lowest attention scores are selected from the text embedding, and the corresponding text information is replaced with image information. Therefore, the updating process of the text embedding can be expressed as:

[0064]

[0065] represents the average embedding of image features, Represents the embedding of text features, and by adding the two, new text features are obtained.

[0066] During the exchange process, residual connections are adopted to minimize the information loss during the replacement process. Once the information exchange is completed, the updated text embedding matrix is ​​passed through a feed-forward neural network (FFN) with layer normalization.

[0067] like Figure 3As shown in the figure, residual connections and regularization layers are placed after the multi-head attention and feedforward neural networks. Residual connections are a type of skip connection structure in deep neural networks. By directly adding the input and output, they alleviate the vanishing gradient problem in deep networks and accelerate the training process. Regularization is an important strategy in machine learning to prevent model overfitting. Its goal is to improve model generalization by constraining model complexity.

[0068] Finally, the report decoding process is performed. In this process, a report decoder is used to reconstruct the text embedding matrix passed by the feedforward network (FFN) to obtain a reconstructed text report. This decoder uses a simple Transformer architecture for report reconstruction.

[0069] In step S2, the model is trained using the loss function of mask language modeling for the report reconstruction task:

[0070]

[0071] Here, P() represents the conditional probability function, and Θ F Represent the weight parameters of the image encoder, report decoder and image-text fusion module respectively, T m represents the mth word block of the output of the report decoder, represents T that matches the i-th mask module m , represents the M encoded image blocks that match the M word blocks.

[0072] (3) Multi-view alignment task (MVA)

[0073] When developing a medical foundation model, multi-view contrastive learning is employed to study the semantic consistency between different views and their corresponding reports, thereby improving model performance. This approach not only addresses challenges associated with imprecise multi-view alignment but also improves the model's ability to manage complex medical imaging tasks.

[0074] Figure 4 The schematic diagram of the multi-view cross-modal alignment principle. The multi-view alignment task includes the multi-view encoding stage, the multi-text encoding stage, and the multi-view alignment stage.

[0075] The multi-view encoding stage uses a shared image encoder (with Figure 1 The image encoder structure is the same as that of the multi-view encoding stage. The image encoder in the multi-view encoding stage emphasizes the multi-view cross-modal alignment process) to encode the cover image corresponding to each view image into the feature space to obtain the corresponding cover image features of each view. For example Figure 4As shown, in the multi-view encoding stage, the image encoder is used to encode the cover images corresponding to the front view and the side view into the feature space to obtain the front cover image feature and side mask image features

[0076] However, when processing reports paired with images, due to the high-dimensional semantic characteristics of text reports and the high-dimensional visual characteristics of image data, if a single text report is aligned with images from two different views, a common representation needs to be found in this high-dimensional space, which may lead to information loss or noise increase in the alignment process. In the multi-text encoding stage, in order to ensure accurate alignment between multiple views and reports, the original text report is enhanced to obtain an enhanced text report (the number of enhanced text reports is equal to the number of multiple views minus 1). The enhancement operation is to randomly replace the characters in the text report with adjacent characters on the keyboard at the character level. This simulation of common errors in clinical scenarios is expected to enhance the model's adaptability to noisy text data, thereby improving robustness and generalization capabilities. The same process of masking the original text report through a text mask is then used to mask the enhanced text report to obtain an enhanced masked text report. In Figure 4 Based on the example shown, there is only one enhanced mask text report, and then the original mask text report and the enhanced mask text report are fed into the report encoder to generate the corresponding original mask text report features. and enhanced masked text reporting features

[0077] The multi-view alignment stage is used to pair the multi-view mask image features and the multi-view mask text report features one by one and then perform feature alignment through training. In the multi-view alignment stage, before performing multi-view cross-modal alignment, another problem must be solved: how to determine the relevance of different views to the corresponding reports. Since clinical reports are based on lesion details in multi-view images, assigning higher weights to image patches containing important pathological information is crucial for determining the relevance between image and report pairs. Therefore, the semantic importance score W obtained from the masking process is F and W L Can be used to adjust cross-modal contrast loss:

[0078]

[0079]

[0080] here, It is v l The mean along the first dimension, It is v l Mean along the first dimension. represents the pooled output of t, Indicates t a The pooled output of . <·> represents the dot product similarity computation. τ1 and τ2 are scaling factors that control the temperature, N is the dimension of the batch, and the subscripts i and j denote the i-th and j-th groups of multi-view images in the batch. An effective multi-view alignment scheme assigns greater weight to highly correlated paired samples during training, ensuring that the network captures and coordinates cross-modal relationships.

[0081] Finally, in step S3, a multi-task learning method is used to integrate and jointly train the loss functions of the pre-training tasks. The overall objective function Consists of four loss terms: image reconstruction loss Reporting reconstruction losses and a cross-modal contrast loss between frontal and side views and It can be expressed as:

[0082]

[0083] λ IL ,λ FL ,λ RA To balance the weight coefficients of various losses, They are the losses for image reconstruction task, report reconstruction task and multi-view alignment task respectively; It is equal to the sum of the alignment losses between multiple pairs of view images and text reports (i.e., multiple pairs of view mask image features and mask text report features) with weight coefficients. Taking two pairs of view mask image features and mask text report features as an example, λ FL is the weight coefficient for balancing the cross-modal contrast loss between frontal and side views.

[0084] Based on the above method, an embodiment of the present invention further provides a mutual learning system between multi-view medical images and text reports. The system is provided with an intelligent agent, which is used to implement the above mutual learning method between multi-view medical images and text reports.

[0085] In this example, the mutual learning model was pre-trained on the MIMIC-CXR dataset. The MIMIC-CXR dataset consists of over 370,000 chest X-rays, organized into cases based on the patient. Each case includes multiple X-rays taken for the same patient from different angles or at different times. Each case is accompanied by a complete radiology report, which consists of two main sections: findings, which describe the important aspects of the X-ray in detail, and conclusions, which summarize the findings. Radiology reports are created by radiologists as part of routine clinical practice and, unlike structured labels, provide rich information without requiring additional manual effort. Reports with fewer than three labels were removed from the dataset. After screening, the final number of patient cases used for pre-training was 217,000.

[0086] The CheXpert dataset provides a benchmark for multi-label chest X-ray classification. Five pathology findings were selected to evaluate model performance. Since the CheXpert website does not provide a test set, 5,000 images were selected from the official training set to create a validation set. The final training / validation / test split consists of 218,414 / 5,000 / 234 images.

[0087] The RSNA Pneumonia (RSNA) dataset defines a binary classification task where each radiograph is classified as either pneumonia or normal. The official split is used to partition the dataset into 25,184 / 1,500 / 3,000 images for training, validation, and testing, respectively.

[0088] The National Institutes of Health (NIH) chest X-ray dataset focuses on classifying 14 chest abnormalities, including atelectasis, cardiomegaly, consolidation, edema, and pneumothorax. The dataset contains over 100,000 X-rays from approximately 32,000 patients, divided into a training set (70%), a validation set (10%), and a test set (20%).

[0089] The SIIM-ACR Pneumothorax Segmentation (SIIM) dataset is designed to develop segmentation models for identifying pneumothoraces in chest X-rays. It includes over 120,000 frontal chest X-rays with accurate manual segmentation labels. The data is divided into a training set (70%), a validation set (15%), and a test set (15%).

[0090] MS-CXR provides bounding box and sentence pair annotations describing clinical findings visible in chest X-rays. Each sentence corresponds to a specific pathology, and there may be multiple bounding boxes associated with a single finding. These annotations were collected from a subset of MIMIC-CXR images and include labels for eight different pathologies. A total of 1,162 annotations were collected from 881 cases, and the overlap between the annotated bounding boxes and pre-trained vision-language association results was verified on the entire dataset.

[0091] This example fine-tunes the proposed MMCL model for the medical image classification task using labeled data of different scales from the CheXpert, RSNA pneumonia, and NIH chest X-ray datasets. The classification AUC scores are reported and compared with several state-of-the-art pre-training methods, including MedKLIP, REFERS, MRM, GLoRIA, MGCA, C2L, and TransVW. MedKLIP uses report filters to extract medical entities and integrates a sophisticated modality fusion module for feature aggregation. REFERS introduces a cross-supervision method with multi-view fusion to enhance the representation learning of each X-ray and its associated report. MRM utilizes mask modeling from the image and language domains to evaluate the performance of pre-trained models on medical datasets. GLoRIA and MGCA are cross-modal visual representation learning methods that utilize multi-scale image-text contrastive learning. In addition, image-based self-supervised methods such as C2L and TransVW are also included for comparison.

[0092] As shown in Table 1, MMCL demonstrates significant performance improvements over the state-of-the-art (SOTA) Med-VLP method across three datasets and at different labeling ratios. Notably, when the labeling ratio is low (1% and 10%), MMCL outperforms MedKLIP and REFERS by 2%, 2.5%, and 4.8%, 2.8%, respectively, on the NIH dataset, and by 4.5%, 2.4%, 4.8%, and 1.2%, respectively, on the RSNA pneumonia dataset. This demonstrates that the pre-trained base model of this embodiment transfers well even when fine-tuned using limited labeled data. Furthermore, when fine-tuned on the full set of labeled data, MMCL continues to outperform MRM and GLoRIA by 1.5% and 2.1% on the CheXpert dataset. MRM focuses on the reconstruction task, while GLoRIA uses the alignment task to compare feature similarities and differences between samples. These results further emphasize the effectiveness of the mask alignment pre-training method proposed in this embodiment.

[0093] Table 1

[0094]

[0095] In order to rigorously evaluate the contribution of each component in the proposed MMCL model, the four key components image reconstruction loss l MIM , report reconstruction losses MLM , multi-view alignment module MVA and cross-modal fusion module CEF are combined in different ways to analyze the effects of key components, where only the image reconstruction loss l is used. MIM The experimental results are shown in Table 2. The evaluation metrics for the NIH dataset are the AUC score, the Dice score for the SIIM dataset, and the mIoU score for the MS-CXR dataset. The AUC (Area Under Curve) is the area under the receiver operating characteristic (ROC) curve and is used to evaluate the performance of binary classification models. The Dice score is a metric used to measure the similarity between two sample sets and is commonly used in image segmentation tasks. The mIoU (Mean Intersection over Union) is a commonly used evaluation metric in image segmentation tasks, used to measure the degree of consistency between the model's predictions and the true labels.

[0096] Comparing Table 2 with Table 1, we can see that the baseline performance of the three datasets has dropped sharply, which shows that other components play an important role. The baseline model lacks text-related training and loss design and cannot perform zero-shot reasoning, which further proves the importance of cross-modal alignment pre-training in improving the overall model performance. MLM After adding the multi-view alignment module MVA, significant improvements are achieved, with improvements of 4%, 4.7%, and 0.231, respectively. In addition, the reconstruction loss l MLM The inclusion of the cross-modal fusion module (CEF) significantly improved classification and segmentation tasks, increasing AUC from 79.8 to 83.2 and Dice from 80.3 to 84.6. When all four key components were incorporated simultaneously, the best results were achieved on all three datasets, with improvements of 5.7 on the NIH dataset, 9.6 on SIIM, and 0.272 on MS-CXR compared to the baseline model.

[0097] Table 2

[0098]

[0099] In addition, the hyperparameter λ for balancing the cross-modal contrast loss between frontal and side views is tested on three datasets: NIH, SIIM, and MS-CXR. FL The test results are shown in Table 3. When λ FL When λ is set to 0.5, it achieves the best performance on all three datasets. FLSetting it to 0 is equivalent to single-view contrastive learning, which leads to significantly worse results. Exceeding 1 leads to a sharp drop in performance on all datasets, which may be due to the over-emphasis on side views that interferes with frontal view reconstruction. It is worth noting that in λ FL = 1.5, the performance is even lower than that of single view alignment. Based on these results, FL Take 0.5.

[0100] Table 3

[0101]

[0102] To further explore the relationship between mask reconstruction and multi-view alignment tasks, we perform a scalar multi-view alignment on the NIH dataset. IL and λ RA A grid search was performed with values ​​ranging from 0 to 1, and the results were as follows Figure 5 As shown, where λ IL is the vertical axis, λ RA is the horizontal axis. IL = 0.3 and λ RA = 0.2 achieves the best performance. However, when λ IL and λ RA Beyond 0.5, a significant performance drop is observed, which indicates that MIM still plays a more important role in the pre-training stage.

[0103] The experimental results demonstrate the effectiveness and superiority of the present invention.

[0104] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A mutual learning method between multi-view medical images and text reports, characterized by: Including steps: S1. Build a mutual learning model; The mutual learning model is used to implement image reconstruction tasks, report reconstruction tasks, and multi-view alignment tasks. The image reconstruction task is to extract visual features from multi-view medical images taken from multiple angles to obtain a reconstructed medical image; the report reconstruction task is to extract report features from the text report corresponding to the multi-view medical image, and integrate the report features and the visual features to generate a cross-modal reconstructed text report; the multi-view alignment task is to align the multi-view medical images and the text report in the feature space; The image reconstruction task includes a visual encoding stage and an image reconstruction stage. The visual encoding stage is to mask each input view image with an image mask, extract features of each masked image with an image encoder, and fuse them to obtain a fused image. The image reconstruction stage is to reconstruct the fused image obtained in the visual encoding stage with a visual decoder to obtain a reconstructed image. The report reconstruction task includes a report input process, a report mask process, a report encoding process, a multimodal fusion process, and a report decoding process; the report input process is to convert the original text report into an original word block sequence through a word segmenter; the report mask process is to use a report mask to randomly select the original word block sequence. n The word blocks are masked to obtain the original word block masked sequence and the original word block uncovered sequence; the report encoding process is to extract features of the original word block uncovered sequence by using a report encoder to obtain a text embedding; the multimodal fusion process is to integrate the text embedding with the fused image to obtain a new text embedding; The multi-view alignment task includes a multi-view encoding stage, a multi-text encoding stage, and a multi-view alignment stage; the multi-view encoding stage uses a shared image encoder to encode the cover image corresponding to each view image into a feature space to obtain the corresponding cover image features of each view; The multi-text encoding stage includes enhancing the original text report to obtain an enhanced text report, and then masking the enhanced text report to obtain an enhanced masked text report; then the original masked text report and the enhanced masked text report are fed into a shared report encoder to generate corresponding original masked text report features and enhanced masked text report features; The multi-view alignment stage is used to pair the multiple view mask image features and the multiple mask text report features one by one and then perform feature alignment through training; S2. Pre-training the image reconstruction task, the report reconstruction task, and the multi-view alignment task respectively; In the pre-training process of the image reconstruction task, the mean square error loss function is used to calculate the reconstructed image i The loss between the image patch and the corresponding ground truth patch During the pre-training process of the report reconstruction task, the conditional probability function is used to calculate the loss between the image encoder, report decoder and image-text fusion module. ; During the pre-training process of the multi-view alignment task, calculating the loss between each pair of view mask image features and mask text report features; S3, jointly training the pre-trained image reconstruction task, the report reconstruction task, and the multi-view alignment task; The loss function used for joint training of the pre-trained image reconstruction task, the report reconstruction task and the multi-view alignment task equal To balance the weight coefficients of various losses, are the losses of the image reconstruction task, the report reconstruction task, and the multi-view alignment task respectively; It is equal to the sum of the alignment losses between multiple pairs of view images and text reports with weight coefficients.

2. The mutual learning method between multi-view medical images and text reports according to claim 1, characterized in that: The process of masking each input view image with an image mask specifically includes an image input process, an image mask process, and an image encoding process. The image input process is to perform image segmentation on the input multi-view image to obtain corresponding image blocks to form a patch sequence. The image mask process is to randomly select a patch sequence from the patch sequence of each view image with an image mask. i The blocks are masked to obtain a masked image; the image encoding process is to input the masked image corresponding to each view image into a shared image encoder for feature extraction to obtain corresponding masked image features.

3. The mutual learning method between multi-view medical images and text reports according to claim 2, characterized in that: The extraction of features of each mask image using an image encoder and the fusion thereof specifically include a semantic evaluation process, a weight calculation process, and an image fusion process; the semantic evaluation process is to multiply the mask matrix corresponding to each view image by the learnable matrix in the image encoding process to obtain a corresponding semantic evaluation score matrix; The weight calculation process is to calculate the weight of the uncovered portion in the cover image of each view image according to the semantic evaluation score matrix; The image fusion process is to multiply the weight corresponding to each view image and the uncovered image features in the masked image features, and then add the multiplication results of all view images to obtain a fused image.

4. The mutual learning method between multi-view medical images and text reports according to claim 1, characterized in that: The multimodal fusion process designs an exchange-based image-text fusion module; the image-text fusion module uses two encoders with shared parameters to respectively calculate the attention scores of the fused image and text embeddings, and then replaces the text information at the corresponding position with the image information with the lowest attention score in the exchange layer to obtain a new text embedding.

5. A mutual learning system between multi-view medical images and text reports, characterized by: The system is provided with an intelligent agent, which is used to implement the mutual learning method between multi-view medical images and text reports as described in any one of claims 1 to 4.

Images