Ultrasound image pre-training method based on vision-language multimodal contrastive learning

Abstract

The present invention discloses an ultrasound image pre-training method based on visual-language multimodal contrastive learning, realizes feature alignment through unimodal local contrastive learning and multimodal local contrastive learning, and creatively proposes a self-supervised pre-training method, which uses medical ultrasound image data and diagnostic report text data to learn unimodal and multimodal feature representations of input images and report texts, and uses unimodal contrast loss and multimodal contrast loss to align image and text features. The present invention takes into account the characteristics of ultrasound images and diagnostic reports, and adopts local contrastive learning to fine-grainedly align image and text features. The present invention performs pre-training with masked language modeling and image-text matching as pre-training goals.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to an ultrasound image pre-training method based on visual-language multimodal contrast learning. Background Art

[0002] Medical ultrasound imaging technology, as a non-invasive and radiation-free examination method, plays a core role in the field of clinical diagnosis. This technology can instantly present the morphological structure and functional activities of human internal organs, so it has been widely used in many medical fields such as cardiovascular disease diagnosis, tumor screening, and obstetrics and gynecology examinations. However, traditional medical ultrasound imaging technology faces some challenges, such as the quality of ultrasound images is greatly affected by the operator's skills, the resolution of ultrasound images is limited, and the interpretation of ultrasound images is highly subjective. In recent years, with the rapid development of artificial intelligence technology, especially the breakthrough progress of deep learning technology in image recognition and processing, it has brought revolutionary changes to the field of medical ultrasound imaging. Deep learning, as a method of simulating the human brain to analyze and learn data, can learn the features of ultrasound images through a large amount of pre-training data, and can be used to automatically identify, segment and classify medical ultrasound images.

[0003] Nowadays, with the development of deep learning, downstream tasks such as automatic recognition, segmentation and classification of medical ultrasound images have made great progress. However, in the field of deep learning, most studies are first pre-trained on natural images and then migrated to the field of medical ultrasound images through transfer learning, contrastive learning, etc. Due to the domain differences between natural images and medical ultrasound images, this simple migration effect is often unsatisfactory. With the development of basic models, this problem is expected to be solved.

[0004] The research of Medical Visual Language Pretraining is a rapidly developing field that aims to improve the performance of medical image analysis and text understanding by using deep learning and large-scale pretrained models. The model is pretrained on large-scale, unlabeled medical image and text datasets using self-supervised pretraining methods, and valuable information is obtained from the image and text data. This information can be generalized to various downstream tasks and significantly improve the efficiency of model development. Compared with pre-training with natural images and then migrating to the field of medical image processing through transfer learning, contrastive learning, etc., pre-training with medical images and then fine-tuning on downstream tasks can significantly reduce domain differences. With the continuous development of deep learning technology, ultrasound image visual language pretraining models are expected to become an important tool in the field of ultrasound image analysis. However, ultrasound image data has the characteristics of high noise and unstable image quality, which makes it difficult for traditional image processing methods to effectively process. Therefore, it is particularly important to develop an ultrasound image visual language pretraining model that can handle these challenges. The ultrasound image visual language pretraining model adopts advanced deep learning techniques, including convolutional neural networks (CNNs), Transformer, etc. These technologies can effectively extract the spatiotemporal features of ultrasound image data and text data, and utilize the powerful feature extraction and generalization capabilities of deep learning models to improve the performance of the models when processing complex data.

[0005] The ultrasound image visual language pre-training model faces a series of challenges and difficulties in the design and implementation process, which need to be solved through innovative technologies and methods. Design of pre-training method: Single-modality ultrasound image pre-training models can only be used for limited downstream tasks, and the fine-tuning effect on downstream tasks is often not as good as that of multi-modal pre-training models. Design of loss function: Single-modality local contrast learning and multi-modality local contrast learning require the design of appropriate loss functions to achieve self-supervised pre-training. The design of loss function is related to the overall performance of the pre-training model. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide an ultrasound image pre-training method based on visual-language multimodal contrastive learning. The present invention uses medical ultrasound image data and diagnostic report data to learn the unimodal and multimodal feature representations of input images and report texts, and utilizes unimodal contrast loss and multimodal contrast loss to align image and text features.

[0007] Technical solution: The present invention provides an ultrasound image pre-training method based on visual-language multimodal contrast learning, characterized by comprising the following steps:

[0008] Step S1: Obtain a pair of ultrasound images and a diagnosis report.

[0009] For an ultrasound image, the original ultrasound image is first masked according to the corresponding ratio to obtain a masked ultrasound image, and then the original ultrasound image is input into the image encoder, and the masked ultrasound image is input into the masked image encoder, thereby respectively obtaining the local image feature embedding and intermediate image feature embedding F I ;

[0010] For the diagnosis report, the original diagnosis report is first masked according to the corresponding ratio to obtain the masked text, and then the original diagnosis report text is input into the text encoder, and the masked text is input into the masked text encoder to obtain the local text feature embedding and intermediate text feature embedding F T ;

[0011] Step S2: embed the intermediate image features into F I Send it to the mask image decoder for feature reconstruction to get the reconstructed local image feature embedding Embed the intermediate text features into F T Send it to the masked text decoder for feature reconstruction to get the reconstructed local text feature embedding

[0012] Step S3: Embedding using local image features And the local image feature embedding after reconstruction To calculate the image modality local contrast loss function so as to align the unimodal local features;

[0013] Embedding using local text features And the local text feature embedding after reconstruction To calculate the text modality local contrast loss function so as to align the unimodal local features;

[0014] Embedding using local image features and local text feature embedding To calculate the multimodal local contrast loss function, so as to achieve the alignment of image features and text features;

[0015] Step S4: embedding local image features Embedding with local text features Input them into the multimodal encoder for processing and obtain the encoded feature embedding F M ;

[0016] Step S5: Initialize the image encoder, text encoder, and multimodal encoder using the weights from the pre-training phase; fine-tune a 6-layer Transformer-based answer decoder to generate answers. The answer decoder receives multimodal embeddings and outputs text tokens. The [CLS] token is used as the initial input token for the decoder, and the [SEP] token is appended to indicate the end of the generated sequence. The downstream VQA model is fine-tuned using the masked language model (MLM) loss, using the true answer as the target.

[0017] Furthermore, in step S1, the image encoder and the mask image encoder have the same structure, and both use a 12-layer visual Transformer (e.g., ViT) to extract visual features from the ultrasound image; during pre-training, the ultrasound image is divided into image blocks of size 16×16, which are sent to the image encoder for processing, and 25% of the image blocks are randomly masked, and the remaining unmasked image blocks are encoded by the mask image encoder into an embedded sequence;

[0018] The text encoder and the masked text encoder have the same structure, both using a 6-layer Transformer network and initialized by the first 6 layers of a pre-trained BERT network. During pre-training, the diagnosis report text is converted into a series of tokens using a WordPiece tagger, which are input into the BERT-based text encoder on the one hand, and 25% of the tokens are randomly masked on the other hand, and the remaining unmasked tokens are encoded into an embedded sequence by the masked text encoder;

[0019] In addition, a special token [CLS] is appended to the beginning of the image and text sequences.

[0020] Furthermore, the mask image decoder and the mask text decoder in step S2 are both based on the Transformer architecture; the mask image decoder uses a 12-layer visual Transformer (ViT) to embed the intermediate features of the ultrasound image into F I The masked text decoder uses a 6-layer Transformer to embed the intermediate features of the report text into F T Perform feature reconstruction.

[0021] Furthermore, the multimodal encoder in step S4 is based on the Transformer architecture, and the last 6 layers in the original BERT model are used as the multimodal encoder. A cross-attention mechanism is added to each layer to integrate visual features and language features to promote multimodal interactive learning.

[0022] Furthermore, during the pre-training process, self-supervision is performed through unimodal local contrast loss function and multimodal local contrast loss function. The unimodal local contrast loss function includes the image modality local contrast loss function and the text modality local contrast loss function. Here, the semantic difference between positive and negative samples in the unimodal (image and text) and multimodal domains can be captured simultaneously through self-supervised pre-training. The unimodal local contrast loss function (Unimodal Local Contrastive Loss) can distinguish examples of one modality in the latent space, such as images or texts, so that similar examples are close; the multimodal local contrast loss function (Multimodal Local Contrastive Loss) learns the alignment between the two modalities by maximizing the similarity between the image and its corresponding text, while separating from the negative examples; in addition, during the self-supervised pre-training process, different perspectives or representations of the same input sample need to be generated for the image and text respectively, and these perspectives or representations can be used as positive samples for unimodal contrast learning; the specific method is:

[0023] Denote the image and text embeddings from the unimodal image encoder and text encoder as v cls and t cls , and by transforming g v and g t Further processing is performed to normalize and map the image and text embeddings to low-dimensional representations. The embeddings are then inserted into a lookup table and only the most recent 65535 pairs of image-text embeddings are stored for contrastive learning; here, v cls express or t cls express or

[0024] Use y i2i (V), y t2t (T), y i2t (V) and y t2i (T) represents the ground-truth one-hot similarity, where the probability of a negative pair is 0 and the probability of a positive pair is 1;

[0025] The unimodal contrast loss and multimodal contrast loss can be defined as the cross entropy H as follows:

[0026]

[0027] Among them, s represents the cosine similarity function, s(V,V i ) = g v (v cls ) Tg v (v cls ) i ,s(T,T i ) = g t (t cls ) T g t (t cls ) i ,s(V,T i ) = g v (v cls ) T g t (t cls ) i ,s(T,V i ) = g t (t cls ) T g v (v cls ) i , τ is a learnable temperature parameter; L ulcl represents the unimodal local contrast loss function, L mlcl represents the multimodal local contrast loss function, E (V,T)D represents the expectation on the data distribution D; y i2i (V), y t2t (T), y i2t (V), y t2i (T) represent the true labels, i.e., the one-hot similarity between image to image, text to text, image to text, and text to image; s(V,V i )、s(T,T i )、s(V,T i )、s(T,V i ) represents the cosine similarity function; g v , g t represents the conversion function for normalizing and mapping images and texts respectively; v cls ,t cls denote the image and text embedding g obtained from the image encoder and text encoder respectively. v (v cls ) i , g t (t cls ) i Represents the result of image and text embedding after passing through the transformation function.

[0028] Beneficial effects: The present invention distinguishes examples of one modality in the latent space through single-modality (including image or text) local contrast learning to make similar examples close; at the same time, multi-modal local contrast learning is used to maximize the similarity between the image and its corresponding text to learn the alignment between the two modalities and separate from the negative examples. The present invention also takes into account the characteristics of ultrasound images and diagnostic reports, and adopts local contrast learning to fine-grainedly align image and text features. Compared with the prior art, the present invention has the following advantages:

[0029] (1) The present invention achieves the goal of bringing one modality example in the latent space closer through single-modality local contrast learning.

[0030] (2) The present invention maximizes the similarity between the ultrasound image and the corresponding report text through multimodal local contrast learning, thereby achieving alignment between the ultrasound image and the report.

[0031] (3) The present invention learns the joint feature representation of ultrasound images and corresponding report texts through a multimodal encoder, and performs pre-training with masked language modeling and image-text matching as pre-training objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0033] Figure 2 Schematic diagram of the model framework in the pre-training stage of an embodiment of the present invention.

[0034] Figure 3 Schematic diagram of the model framework in the fine-tuning stage of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0036] First, existing medical image or text model pre-training methods usually only focus on a single modality, ignoring the complementarity between medical images and report text data. Joint pre-training of medical images and text can achieve better results than single modality pre-training. Secondly, there are also some methods that use medical images and text data for multimodal pre-training, but these methods often simply adopt a paradigm similar to CLIP, that is, image and text comparison pre-training. Because medical images are different from natural images, medical images and text reports often cannot be simply aligned as a whole, and there is also a certain local correlation between non-sample pairs.

[0037] To solve the above problems, the ultrasound image pre-training method based on visual-language multimodal contrast learning of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0038] Step S1, obtain a pair of ultrasound images and diagnostic reports. For the ultrasound image, first mask the original ultrasound image according to the corresponding ratio (first divide it into 16×16 image blocks, and then mask 25% of the image blocks) to obtain a masked ultrasound image, and then input the original ultrasound image (divided into 16×16 image blocks) into the image encoder, and input the masked ultrasound image into the masked image encoder, and then obtain the local image feature embedding and intermediate image feature embedding F I ; For the diagnosis report, the original diagnosis report text is first masked according to the corresponding proportion (first converted into a series of tokens using the WordPiece tokenizer, and then 25% of the tokens are masked) to obtain the masked text, and then the original diagnosis report text (converted into a series of tokens using the WordPiece tokenizer, where the token [CLS] is attached to the beginning of the image and text sequence) is input into the text encoder, and the masked text is input into the masked text encoder to obtain the local text feature embedding and intermediate text feature embedding F T ;

[0039] Step S2: embed the intermediate image features into F I Send it to the mask image decoder for feature reconstruction to get the reconstructed local image feature embedding Embed the intermediate text features into F T Send it to the masked text decoder for feature reconstruction to get the reconstructed local text feature embedding

[0040] Step S3: Embedding using local image features And the local image feature embedding after reconstruction To calculate the local contrast loss function of the image modality; using local text feature embedding And the local text feature embedding after reconstruction To calculate the text modality local contrast loss function; using local image feature embedding and local text feature embedding To calculate the multimodal local contrast loss function;

[0041] Step S4: embedding local image features Embedding with local text features Input them into the multimodal encoder for processing and obtain the encoded feature embedding F M ;

[0042] Step S5: Use the weights of the pre-training stage to initialize the image encoder, text encoder, and multimodal encoder; to generate the answer, use a 6-layer Transformer-based answer decoder for fine-tuning, such as Figure 3 As shown, the answer decoder receives multimodal embeddings and outputs text tokens, the [CLS] token is used as the initial input token of the decoder, and the [SEP] token is appended to indicate the end of the generated sequence; the downstream VQA model is fine-tuned with the masked language model (MLM) loss, using the true answer as the target.

[0043] like Figure 2 As shown, the network model of this embodiment involves the following modules: image encoder, text encoder, mask image encoder, mask text encoder, mask image decoder, mask text decoder and multimodal encoder. The image encoder encodes the ultrasound image into feature embedding, the mask image encoder and the mask image decoder provide different representations for the ultrasound image, and the text processing is similar. Specifically, the image encoder and the mask image encoder have the same structure, both using a 12-layer visual Transformer to extract visual features from ultrasound images; during pre-training, the ultrasound image is divided into image blocks of size 16×16, which are fed into the image encoder for processing, while 25% of the image blocks are randomly masked, and the remaining unmasked image blocks are encoded into embedded sequences by the mask image encoder; the text encoder and the mask text encoder have the same structure, both using a 6-layer Transformer network, and initialized by the first 6 layers of the pre-trained BERT network. During pre-training, the diagnostic report text is converted into a series of tokens using the WordPiece tagger, which are fed into the BERT-based text encoder, while 25% of the tokens are randomly masked, and the remaining unmasked tokens are encoded into embedded sequences by the mask text encoder; in addition, a special token [CLS] is appended to the beginning of the image and text sequences.

[0044] The mask image decoder and mask text decoder of this embodiment are both based on the Transformer architecture; the mask image decoder uses a 12-layer visual Transformer (ViT) to embed the intermediate features of the ultrasound image into F I The masked text decoder uses a 6-layer Transformer to embed the intermediate features of the report text into F T Perform feature reconstruction.

[0045] The multimodal encoder of this embodiment is based on the Transformer architecture, and uses the last 6 layers in the original BERT model as the multimodal encoder. A cross-attention mechanism is added to each layer to integrate visual features and language features.

[0046] During the pre-training process, self-supervision is performed through the unimodal local contrast loss function and the multimodal local contrast loss function. The unimodal local contrast loss function includes the image modality local contrast loss function and the text modality local contrast loss function. The specific method is:

[0047] Denote the image and text embeddings from the unimodal image encoder and text encoder as v cls and t cls , and by transforming g v and g t Further processing is performed to normalize and map the image and text embeddings to low-dimensional representations; the embeddings are then inserted into a lookup table and only the most recent 65535 pairs of image-text embeddings are stored for contrastive learning; y i2i (V), y t2t (T), y i2t (V) and y t2i (T) represents the true one-hot similarity, where the probability of a negative sample pair is 0 and the probability of a positive sample pair is 1;

[0048] The unimodal contrast loss and multimodal contrast loss are defined as the cross entropy H as follows:

[0049]

[0050]

[0051] Among them, s represents the cosine similarity function, s(V,V i ) = g v (v cls ) T g v (v cls ) i ,s(T,T i ) = g t (t cls ) T g t (t cls ) i ,s(V,T i ) = g v (v cls ) T g t (t cls ) i ,s(T,V i ) = g t (t cls ) T g v (v cls ) i, τ is a learnable temperature parameter; L ulcl represents the unimodal local contrast loss function, L mlcl represents the multimodal local contrast loss function, E (V,t)D represents the expectation on the data distribution D; y i2i (V), y t2t (T), y i2t (V), y t2i (T) represent the true labels, i.e., the one-hot similarity between image to image, text to text, image to text, and text to image; s(V,V i )、s(T,T i )、s(V,T i )、s(T,V i ) represents the cosine similarity function; g v , g t represents the conversion function for normalizing and mapping images and texts respectively; v cls ,t cls denote the image and text embedding g obtained from the image encoder and text encoder respectively. v (v cls ) i , g t (t cls ) i Represents the result of image and text embedding after passing through the transformation function.

Claims

1. A method for pre-training ultrasound images based on visual-linguistic multimodal contrastive learning, characterized in that: The following steps are involved: Step S1: Obtain a pair of ultrasound images and a diagnosis report. For an ultrasound image, the original ultrasound image is first masked according to the corresponding ratio to obtain a masked ultrasound image, and then the original ultrasound image is input into the image encoder, and the masked ultrasound image is input into the masked image encoder, thereby respectively obtaining the local image feature embedding and intermediate image feature embedding F I ; For the diagnosis report, the original diagnosis report is first masked according to the corresponding ratio to obtain the masked text, and then the original diagnosis report text is input into the text encoder, and the masked text is input into the masked text encoder to obtain the local text feature embedding and intermediate text feature embedding F T ; Step S2: embed the intermediate image features into F I Send it to the mask image decoder for feature reconstruction to get the reconstructed local image feature embedding Embed the intermediate text features into F T Send it to the masked text decoder for feature reconstruction to get the reconstructed local text feature embedding Step S3: Embedding using local image features And the local image feature embedding after reconstruction To calculate the local contrast loss function of the image modality; using local text feature embedding And the local text feature embedding after reconstruction To calculate the text modality local contrast loss function; using local image feature embedding and local text feature embedding To calculate the multimodal local contrast loss function; Step S4: embedding local image features Embedding with local text features Input them into the multimodal encoder for processing and obtain the encoded feature embedding F M ; Step S5: Initialize the image encoder, text encoder, and multimodal encoder using the weights from the pre-training phase; fine-tune a 6-layer Transformer-based answer decoder to generate the answer. The answer decoder receives the multimodal embedding and outputs a text token. The [CLS] token is used as the initial input token of the decoder, and the [SEP] token is appended to indicate the end of the generated sequence. The downstream VQA model is fine-tuned with a masked language model loss, using the true answers as targets.

2. The ultrasound image pre-training method based on visual-language multimodal contrastive learning according to claim 1, characterized in that: The image encoder and the mask image encoder in step S1 have the same structure, and both use a 12-layer visual Transformer to extract visual features from the ultrasound image; during pre-training, the ultrasound image is divided into image blocks of size 16×16, which are sent to the image encoder for processing on the one hand, and 25% of the image blocks are randomly masked on the other hand, and the remaining unmasked image blocks are encoded into an embedded sequence by the mask image encoder; the text encoder and the mask text encoder have the same structure, both use a 6-layer Transformer network, and are initialized by the first 6 layers of the pre-trained BERT network. During pre-training, the diagnosis report text is converted into a series of tokens using a WordPiece tagger, which are input into a BERT-based text encoder on the one hand, and 25% of the tokens are randomly masked on the other hand, and the remaining unmasked tokens are encoded into an embedded sequence by the mask text encoder; in addition, a special token [CLS] is appended to the beginning of the image and text sequences.

3. The ultrasound image pre-training method based on visual-language multimodal contrastive learning according to claim 1, characterized in that: In step S2, the mask image decoder and the mask text decoder are both based on the Transformer architecture; the mask image decoder uses a 12-layer visual Transformer to embed the intermediate features of the ultrasound image into F I The masked text decoder uses a 6-layer Transformer to embed the intermediate features of the report text into F T Perform feature reconstruction.

4. The ultrasound image pre-training method based on visual-language multimodal contrastive learning according to claim 1, characterized in that: The multimodal encoder in step S4 is based on the Transformer architecture, and uses the last 6 layers in the original BERT model as the multimodal encoder. A cross-attention mechanism is added to each layer to integrate visual features and language features.

5. The ultrasound image pre-training method based on visual-language multimodal contrastive learning according to claim 1, characterized in that: During the pre-training process, self-supervision is performed through the unimodal local contrast loss function and the multimodal local contrast loss function. The unimodal local contrast loss function includes the image modality local contrast loss function and the text modality local contrast loss function. The specific method is: Denote the image and text embeddings from the unimodal image encoder and text encoder as v cls and t cls , and by transforming g v and g t Further processing is performed to normalize and map the image and text embeddings to low-dimensional representations; the embeddings are then inserted into a lookup table and only the most recent 65535 pairs of image-text embeddings are stored for contrastive learning; here, v cls express or t cls express or Use y i2i (V), y t2t (T), y i2t (V) and y t2i (T) represents the true one-hot similarity, where the probability of a negative sample pair is 0 and the probability of a positive sample pair is 1; The unimodal contrast loss and multimodal contrast loss are defined as the cross entropy H as follows: Among them, s represents the cosine similarity function, s(V,V i ) = g v (v cls ) T g v (v cls ) i ,s(T,T i ) = g t (t cls ) T g t (t cls ) i ,s(V,T i ) = g v (v cls ) T g t (t cls ) i ,s(T,V i ) = g t (t cls ) T g v (v cls ) i , τ is a learnable temperature parameter; L ulcl represents the unimodal local contrast loss function, L mlcl represents the multimodal local contrast loss function, E (V,T)D represents the expectation on the data distribution D; y i2i (V), y t2t (T), y i2t (V), y t2i (T) represent the true labels, i.e., the one-hot similarity between image to image, text to text, image to text, and text to image; s(V,V i )、s(T,T i )、s(V,T i )、s(T,V i ) represents the cosine similarity function; g v , g t represents the conversion function for normalizing and mapping images and texts respectively; v cls ,t cls denote the image and text embedding g obtained from the image encoder and text encoder respectively. v (v cls ) i , g t (t cls ) i Represents the result of image and text embedding after passing through the transformation function.

Images