Ultrasonic multi-mode pre-training method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, medicine and pre-training, and discloses an ultrasonic multi-modal pre-training method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining an original image, a mask image, an original report, a mask report and an ultrasonic multi-modal pre-training architecture; determining contrast learning loss, mean square error loss and cross entropy loss based on the original image, the mask image, the original report, the mask report and an ultrasonic multi-mode pre-training architecture; parameter updating is carried out on the basis of contrast learning loss, mean square error loss, cross entropy loss and the ultrasonic multi-mode pre-training architecture, a trained ultrasonic multi-mode pre-training architecture is determined, and an image encoder in the trained ultrasonic multi-mode pre-training architecture is used for downstream tasks.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, medicine and pre-training technology, and in particular to an ultrasonic multi-modal pre-training method, device, computer equipment and storage medium. Background Art

[0002] In recent years, multimodal pre-training has attracted widespread attention in industry and academia. This technology uses the complementary relationship between images and text to achieve performance comparable to or even better than unimodal pre-training. By making full use of the complementarity between images and text, multimodal pre-training can also reduce the dependence on unimodal data. In particular, the medical field is a feasible application scenario for multimodal pre-training. Medical images and their reports are natural image-text data pairs that do not require additional annotation by doctors. Some recent work has also demonstrated the great potential of medical multimodal pre-training. However, the current main work focuses on radiology images and reports, and there is relatively little research in the field of ultrasound.

[0003] Existing pre-training techniques usually use labeled images to pre-train models. However, in the medical field, due to the need for privacy and expertise, there is a lack of large-scale labeled medical image datasets, making it difficult to tap the potential of pre-training in the medical field. In the related research on medical multimodal pre-training, the main research is focused on the X-ray field, and the research in the ultrasound field is relatively small. Due to the huge field differences, the model pre-trained on X-ray cannot be directly transferred to the ultrasound field. In addition, ultrasound images and reports have many characteristics. For example, one report usually corresponds to multiple ultrasound images, and ultrasound reports have proprietary templates and professional vocabulary. Many X-ray pre-training technologies cannot be directly transferred. Summary of the invention

[0004] Based on this, it is necessary to address the technical problem that the existing technology has poor pre-training effect on the model of ultrasound images and reports, and propose an ultrasound multimodal pre-training method, device, computer equipment and storage medium.

[0005] In a first aspect, an ultrasound multimodal pre-training method is provided, the method comprising:

[0006] Acquire an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image;

[0007] Encoding the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0008] Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report and the global representation and local representation of the mask report;

[0009] Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0010] Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss;

[0011] Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross entropy loss based on the original report and the prediction results;

[0012] Based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, parameters are updated to determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0013] In a second aspect, an ultrasonic multimodal pre-training device is provided, the device comprising:

[0014] An acquisition module, used to acquire an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder;

[0015] A first encoding module, used for encoding the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, and a global representation and a local representation of the mask image;

[0016] A second encoding module, used for encoding the original report and the mask report respectively based on the text encoder to obtain a global representation and a local representation of the original report, and a global representation and a local representation of the mask report;

[0017] A contrastive learning loss calculation module is used to perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0018] A mean square error loss calculation module is used to encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image, and perform mean square error loss calculation based on the reconstructed image and the original image to obtain a mean square error loss;

[0019] A cross entropy loss calculation module is used to encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross entropy loss based on the original report and the prediction results;

[0020] A parameter updating module is used to update parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, and determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0021] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned ultrasound multimodal pre-training method when executing the computer program.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned ultrasound multimodal pre-training method are implemented.

[0023] The ultrasound multimodal pre-training method proposed in the present invention obtains an original image, a mask image, an original report, a mask report and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder and a convolutional neural network decoder. The original image is an ultrasound image, and then the original image and the mask image are encoded based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image, and the original report and the mask report are encoded based on the text encoder to obtain a global representation and a local representation of the original report, as well as a global representation and a local representation of the mask report, and then comparative learning is performed based on the global representation of the original image and the global representation of the original report. , calculate the contrastive learning loss, then encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image, and calculate the mean square error loss based on the reconstructed image and the original image to obtain the mean square error loss, then encode based on the local representation of the mask report, the local representation of the original image and the text cross-modal encoder to obtain the prediction result of the masked vocabulary in the mask report, and calculate the cross entropy loss based on the original report and the prediction result, and finally update the parameters based on the contrastive learning loss, the mean square error loss, the cross entropy loss, and the ultrasound multimodal pre-training architecture to determine the trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks. The present invention can use medical images and their reports to pre-train the model, significantly improving its performance on downstream visual recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] in:

[0026] Figure 1 is an application environment diagram of an ultrasound multi-modal pre-training method in an embodiment;

[0027] Figure 2 is a flowchart of an ultrasound multi-modality pre-training method in one embodiment;

[0028] Figure 3 is a schematic diagram of the structure of an ultrasound multi-modal pre-training architecture of an ultrasound multi-modal pre-training method in one embodiment;

[0029] Figure 4 A schematic diagram of generating an original image of an ultrasound multimodal pre-training method in one embodiment;

[0030] Figure 5 A result diagram showing a comparison between an ultrasound multimodal pre-training method and other methods in an embodiment;

[0031] Figure 6 A comparative visualization of an ultrasound multimodal pre-training method and other methods in one embodiment;

[0032] Figure 7 A visualization of segmentation and detection of an ultrasound multimodal pre-training method in one embodiment;

[0033] Figure 8 A classification visualization of an ultrasound multi-modality pre-training method in one embodiment;

[0034] Fig. 9 A schematic diagram of word weight analysis of an ultrasound multimodal pre-training method in one embodiment;

[0035] Fig.10 A schematic diagram showing a comparison of multi-image processing strategies of an ultrasound multi-modal pre-training method in one embodiment;

[0036] Fig.11 An image mask analysis diagram of an ultrasound multimodal pre-training method in one embodiment;

[0037] Fig.12 A comparison diagram of image mask strategies of an ultrasound multimodal pre-training method in one embodiment;

[0038] Fig.13 is a structural block diagram of an ultrasonic multi-modal pre-training device in one embodiment;

[0039] Fig.14 is a structural block diagram of a computer device in one embodiment;

[0040] Fig.15 It is a structural block diagram of a computer device in another embodiment. DETAILED DESCRIPTION

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0042] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The ultrasonic multimodal pre-training method provided in the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client 110 communicates with the server 120 through a network. The server 120 can obtain the original image, the mask image, the original report, the mask report and the ultrasound multimodal pre-training architecture through the client 110, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder and a convolutional neural network decoder. The original image is an ultrasound image. Then the server 120 encodes the original image and the mask image based on the image encoder to obtain the global representation and local representation of the original image, as well as the global representation and local representation of the mask image, and encodes the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report, as well as the global representation and local representation of the mask report. Then the server 120 performs comparative learning based on the global representation of the original image and the global representation of the original report to calculate. To contrast learning loss, then the server 120 encodes based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image, and calculates the mean square error loss based on the reconstructed image and the original image to obtain the mean square error loss, then the server 120 encodes based on the local representation of the mask report, the local representation of the original image and the text cross-modal encoder to obtain the prediction result of the masked vocabulary in the mask report, and calculates the cross entropy loss based on the original report and the prediction result, and finally the server 120 updates the parameters based on contrast learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine the trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks. The present invention can pre-train the model using medical images and their reports, significantly improving its performance on downstream visual recognition tasks. Among them, the client 110 can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server 120 can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0045] See also Figure 2 As shown, Figure 2 A schematic flow chart of an ultrasound multimodal pre-training method provided in an embodiment of the present invention includes the following steps:

[0046] Step S101: acquiring an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder;

[0047] Among them, the original image is an ultrasound image, the mask image is an image based on the mask operation performed on the original image, the original report may be an ultrasound image report corresponding to the original image, and the mask report may refer to a report obtained by performing a mask operation on the content in the original report.

[0048] In this embodiment, refer to Figure 3 The proposed ultrasound multimodal pre-training architecture consists of five modules: image encoder f v , text encoder f t , image cross-modal encoder cross v , text cross-modal encoder cross t and convolutional neural network decoder d.

[0049] In one embodiment, the image encoder adopts a ResNet or ViT model, the text encoder adopts a BERT model, the image cross-modal encoder adopts a BERT model, the text cross-modal encoder adopts a BERT model, and the convolutional neural network decoder adopts an upsampling convolutional neural network.

[0050] Step S102: Encode the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, and a global representation and a local representation of the mask image;

[0051] Step S103: Encode the original report and the mask report respectively based on the text encoder to obtain a global representation and a local representation of the original report, and a global representation and a local representation of the mask report;

[0052] As an example, first use the image encoder f v and the text encoder f t Encode the original image and mask image, original report and mask report respectively:

[0053] v g,i ,v l,i =f v (I i )

[0054] t g,i ,t l,i =f t (R i )

[0055] v′ g,i ,v′ l,i =f v (I′ i )

[0056] t′ g,i ,t′ l,i =f t (R′i )

[0057] Among them, i i ,R i Represent the original image and original report of the i-th sample in the mini-batch, I′ i ,R′ i Respectively represent the mask image and mask report of the i-th sample in the mini-batch, v g,i ,v l,i Characterize the global and local representations of the original image of the i-th sample, t g,i ,t l,i The global and local representations of the original report of the i-th sample, v′ g,i ,v′ l,i represents the global and local representations of the i-th sample mask image, t′ g,i ,t′ l,i represents the global and local representations of the i-th sample mask report.

[0058] Step S104: performing contrastive learning based on the global representation of the original image and the global representation of the original report, and calculating the contrastive learning loss;

[0059] It should be noted that contrastive learning maps images and texts into one representation space by shortening the distance between matching image-text pairs in the representation space and increasing the distance between mismatching image-text pairs in the representation space. As a self-supervised learning method, contrastive learning can significantly improve the performance of pre-trained models in zero-shot and few-shot classification tasks.

[0060] In this embodiment, the global representation of the original image and the global representation of the original report, i.e., v g,i and t g,i Specifically, for a small batch of samples of size N, the cosine similarity between each image and all reports and the cosine similarity between each report and all images are first calculated, and the calculated cosine similarity is used to calculate the InfoNCE loss, as shown below:

[0061]

[0062] Among them, τ is the temperature coefficient, s(*,*) is the similarity equation, and the total loss function of contrastive learning is as follows:

[0063]

[0064] Among them, λ1 is the contrastive learning loss weight.

[0065] Step S105: Encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain, and calculate the mean square error loss based on the reconstructed image and the original image to obtain the mean square error loss;

[0066] It should be noted that mask image modeling can enhance the pre-trained model's ability to understand fine-grained features. The mask image can be constructed by exchanging image block information. Specifically, the original image is first divided into image blocks of size 16*16, and then 25% of the image blocks are randomly selected and exchanged with any image block in the image to obtain the mask image.

[0067] Specifically, the local representation of the mask image and the local representation of the original report are input into the image cross-modal encoder to obtain a first feature output by the image cross-modal encoder; the first feature is input into the convolutional neural network decoder to obtain a reconstructed image output by the convolutional neural network decoder.

[0068] As an example, using the local representation v′ of the mask image l,i and the original reported local representation t′ l,i To reconstruct the image. Specifically, v′ l,i , t′ l,i As an image cross-modal encoder v The input and t′ l,i Perform cross attention, cross v The output of will be used as the input of the image decoder d, restored to the original image size, and the masked image is predicted, that is, the reconstructed image. The loss is defined as the mean square error between the reconstructed image and the original image:

[0069]

[0070] Among them, MSE represents the mean squared error loss, and λ2 is the loss weight for cross-modal mask image modeling.

[0071] Step S106: Encode based on the local representation of the mask report, the local representation of the original image and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross entropy loss based on the original report and the prediction results;

[0072] It should be noted that the ultrasound report contains a large number of professional terms and fixed collocations, which is a special language. Therefore, a cross-modal masked language modeling task is introduced to enhance the model's ability to understand ultrasound reports. Unlike English text, which is composed of words, Chinese text is composed of characters. If masking is performed at the character level, the fixed collocation between characters will reduce the difficulty of predicting masked text, especially in medical reports, which usually have obvious structured templates. Therefore, in order to improve the performance of masked language modeling, the present invention performs masking at the word level to obtain a masked report. Specifically, the report is first segmented using the jieba word segmenter, and then 450 commonly used words most relevant to the report content are manually selected to form a vocabulary. For the input report, firstly, the word segmentation is performed according to the vocabulary, and then 25% of the words are randomly masked to obtain a masked report. 15% of the masked words are replaced with special words [MASK], 5% remain unchanged, and 5% are replaced with random words.

[0073] In this embodiment, the local representation of the mask report and the local representation of the original image are first input into the text cross-modal encoder for encoding to obtain a second feature, and then input into a linear layer for prediction based on the second feature to obtain the prediction result.

[0074] In this embodiment, the mask report is first encoded into a global representation t′ g,i , local representation t′ l,o , where the local representation t′ of the mask report l,i As a text cross-modal encoder t The input and the local representation v of the original image l,i Perform cross attention, cross t The output of is passed through a linear layer to predict the masked vocabulary. The prediction effect is characterized by the cross entropy loss:

[0075] Update text cross-modal encoder, text encoder, image encoder.

[0076] Among them, CE is the cross entropy loss and λ3 is the loss weight for cross-modal masked language modeling.

[0077] Step S107: performing parameter updates based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, and determining a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0078] In this embodiment, the parameters of the text encoder and the image encoder are updated based on the contrastive learning loss, and then the parameters of the convolutional neural network decoder, the image cross-modal encoder, the text encoder, and the image encoder are updated based on the mean square error loss, and then the parameters of the text cross-modal encoder, the text encoder, and the image encoder are updated based on the cross entropy loss.

[0079] Finally, it is determined whether the ultrasound multimodal pre-training architecture after the updated parameters meets the training end condition. If so, the ultrasound multimodal pre-training architecture that meets the training end condition is used as the trained ultrasound multimodal pre-training architecture.

[0080] After the model pre-training is completed, the trained model weights can be directly used as the initialization weights of the downstream tasks for downstream migration and fine-tuning. Taking the three downstream tasks of classification, segmentation and target detection as examples, the migration method is as follows: Classification: Use the linear probe technology to linearly map the output features of the pre-trained model to the category dimension for classification; Segmentation: Use the pre-trained weights as the pre-trained weights of segmentation models such as U-Net and SETR for segmentation; Target detection: Use the pre-trained weights as the pre-trained weights of the Yolo series template detection model for target detection; After initializing expert models such as U-Net and Yolo with the pre-trained model, you can use a small amount of data to fine-tune the new model to achieve better results than retraining a new model.

[0081] In one embodiment, the masked report is obtained by masking the original report based on the accuracy of the vocabulary. Medical reports usually have a clear template, that is, doctors use a unified format when writing reports, and only need to fill in the vocabulary related to the disease in a specific position. Therefore, these filled-in words are the most important language information. A dynamic keyword mining technology is proposed, which aims to increase the probability of these keywords being masked in masked language modeling through self-supervision, so that the model spends more time learning these key words. In the dynamic keyword mining technology proposed by the present invention, a weight is maintained for each word in the vocabulary to characterize the prediction accuracy of the word. If a word is often predicted correctly, it means that it may be a template word. This kind of vocabulary prediction difficulty is relatively low, and it can be predicted directly based on the context without relying on image information, so the weight of the word is reduced. If a word is often predicted incorrectly, it means that its prediction may need to combine more image information, and the model has higher requirements, so its corresponding weight is increased. When masking, it will be masked according to the weight table, which means that simple words will be less masked and complex keywords will be more often masked. The model will have more time to learn keywords. The change process of the weight can be formulated as:

[0082]

[0083] Among them, ub and lb are the upper and lower bounds of the weight change, w is the vocabulary weight, and m is a hyperparameter that controls the magnitude of the change.

[0084] In one embodiment, reference Figure 4 , a medical report usually corresponds to multiple medical images. These images correspond to the information in the report. Inputting more pictures can facilitate the model to align modal information, which will be beneficial for contrastive learning and modal reconstruction. Considering the needs of pre-training and downstream tasks, for a case, first build a 1000*1500 canvas, and then randomly divide the canvas into 6 blocks. Next, read the first 6 pictures of the case from the dataset, randomly crop and enhance these 6 pictures and put them into the canvas. If the case has less than 6 pictures, it will be replaced by a completely black image. Finally, transform the canvas into a 256*256 original image.

[0085] It should be noted that the present invention is compared with other similar methods in ultrasound image classification, segmentation and target detection tasks. Other methods have shortcomings in these three tasks, while the present invention has balanced performance in these three methods and achieves relatively excellent performance. In visualization, the present invention can locate the lesion area more accurately, and the ROC curve also achieves the best effect. Figure 5 , the result diagram of comparing the ultrasound multimodal pre-training method proposed in the present invention with similar methods. Figure 6 , which is a comparative visualization of the ultrasound multimodal pre-training method proposed in this invention and similar methods. Figure 7 , which is the segmentation and detection visualization of the ultrasound multimodal pre-training method proposed in this invention. Figure 8 , which is the classification visualization of the ultrasound multimodal pre-training method proposed in this invention.

[0086] refer to Fig. 9 , Classification visualization of the ultrasound multimodal pre-training method proposed in this invention, Fig. 9 Including (a) the word weight change diagram of all words, (b) some words that are difficult for the model to learn, and (c) the word weight change curve of some words. After training according to the method proposed by the present invention, the weight of each word in the vocabulary presents a special pattern, as follows Fig. 9As shown in the figure. The weight of the template words shows a rapid rise and fall process in the early stage of training, such as the weight curve that has dropped to the lower limit value on the left side of the auxiliary line I. This is because the large number of template words makes them often masked in the early stage of training. Before the model learns how to predict them, their weights will rise rapidly. These template words usually have obvious prediction patterns, and the model can quickly learn how to predict them. Therefore, after a short period of time, their weights will drop rapidly until the lower limit value, and they are no longer valued by the model. However, their high occurrence rate still allows them to be occasionally masked, so that the model will not completely forget their prediction model. For long-tail words, due to their low probability of occurrence, their weights rise slowly and are never valued throughout the training process. As shown in the figure, the vocabulary weight never exceeds the auxiliary line II. The vocabulary outside these two categories of words is the main learning object of the model. The model focuses on learning them for about 2 / 3 of the training time (BC segment). Many keywords in the report are included in this type of words, such as some words with image information, such as location words and qualitative words. The present invention enables the model to focus on such words, thereby shielding the invalid information brought by the template words and long-tail words, and helping the model to learn effective visual information in the masked language modeling task. We visualized the weight curves of some words, and found that the weight curves of words such as lesions and morphology that frequently appear in the template show a trend of rapid rise and fall, while the weight curves of words such as diffuse and scar that rarely appear in the report grow slowly. The weight curves of keywords such as middle section and left lobe that represent the orientation slowly increase to the upper limit value after the model learns the prediction mode of the template words, and remain at the upper limit value until the model learns the prediction mode of these words. The figure also shows some words that are difficult for the model to understand, and the weights of these words have been maintained at the upper limit value until the end of training. Most of these words are directional words, which are used to indicate the relative position of organs. This is because the relative position of organs is artificially defined and has a certain ambiguity, making it difficult for the model to learn related position terms. The weights of position words such as left lobe and right lobe have dropped to the next value, which shows that the model has learned how to distinguish left and right based on pictures. If we rely only on medical reports, replacing these two words does not affect the semantic coherence.

[0087] refer to Fig.10, is a comparison of multi-image processing strategies. Image-text matching refers to the matching accuracy of comparative learning on the verification set. In order to verify whether the use of multiple images can provide performance improvements for pre-training and downstream tasks, two commonly used methods were deployed: using a fixed image as input and randomly sampling an image as input, and comparing them with the multi-image processing strategy proposed in the present invention. The results show that making full use of the many-to-one relationship between images and texts can significantly improve the effect of pre-training. It is worth noting that although the input of the present invention during pre-training is a synthetic image, and the input of the downstream task is a single image, and the image encoder is frozen in the downstream task, the image encoder can still apply the image feature extraction ability learned during pre-training to the downstream task, proving that the pre-trained model of the present invention can extract features with translation invariance and scale invariance-the synthetic image is obtained by shrinking and splicing a single image.

[0088] refer to Fig.11 , is the image mask analysis diagram. Compared with the mask strategy of directly removing some image blocks, the pre-training effect of the image block exchange mask strategy proposed in this invention is better. The trained model is better than the former in downstream classification and segmentation tasks. Fig.12 , which is a comparison chart of image mask strategies. The ablation experiment shows that all the modules and methods proposed in this invention have a positive effect on pre-training.

[0089] See also Fig.13 As shown, in one embodiment, an ultrasound multimodal pre-training device is provided, the device comprising: an acquisition module 10, for acquiring an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture comprises an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder;

[0090] A first encoding module 20, configured to encode the original image and the mask image respectively based on an image encoder to obtain a global representation and a local representation of the original image, and a global representation and a local representation of the mask image;

[0091] A second encoding module 30, configured to encode the original report and the mask report respectively based on a text encoder to obtain a global representation and a local representation of the original report, and a global representation and a local representation of the mask report;

[0092] A contrastive learning loss calculation module 40, used for performing contrastive learning based on the global representation of the original image and the global representation of the original report, and calculating the contrastive learning loss;

[0093] A mean square error loss calculation module 50 is used to encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image, and perform mean square error loss calculation based on the reconstructed image and the original image to obtain a mean square error loss;

[0094] A cross entropy loss calculation module 60 is used to encode based on the local representation of the mask report, the local representation of the original image and the text cross-modal encoder to obtain a prediction result of the masked vocabulary in the mask report, and calculate the cross entropy loss based on the original report and the prediction result;

[0095] The parameter updating module 70 is used to update parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, and determine the trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0096] In one embodiment, the image encoder adopts a ResNet or ViT model, the text encoder adopts a BERT model, the image cross-modal encoder adopts a BERT model, the text cross-modal encoder adopts a BERT model, and the convolutional neural network decoder adopts an upsampling convolutional neural network.

[0097] In one embodiment, the masked report is obtained by masking the original report based on the accuracy of the vocabulary.

[0098] In one embodiment, the mean square error loss calculation module 50 is used to:

[0099] Inputting the local representation of the mask image and the local representation of the original report into an image cross-modal encoder to obtain a first feature output by the image cross-modal encoder;

[0100] The first feature is input into the convolutional neural network decoder to obtain a reconstructed image output by the convolutional neural network decoder.

[0101] In one embodiment, the cross entropy loss calculation module 60 is used to:

[0102] Inputting the local representation of the mask report and the local representation of the original image into the text cross-modal encoder for encoding to obtain a second feature;

[0103] The second feature is input into the linear layer for prediction to obtain the prediction result.

[0104] In one embodiment, the parameter updating module 70 is used to:

[0105] Update parameters of the text encoder and the image encoder based on the contrastive learning loss;

[0106] Update parameters of the convolutional neural network decoder, the image cross-modal encoder, the text encoder, and the image encoder based on the mean square error loss;

[0107] Update parameters of the text cross-modal encoder, the text encoder, and the image encoder based on the cross entropy loss;

[0108] It is determined whether the ultrasound multimodal pre-training architecture after the updated parameters meets the training end condition. If so, the ultrasound multimodal pre-training architecture that meets the training end condition is used as the trained ultrasound multimodal pre-training architecture.

[0109] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, a function or step on the service side of an ultrasound multimodal pre-training method is implemented.

[0110] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.15 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of an ultrasound multimodal pre-training method.

[0111] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0112] Acquire an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image;

[0113] Encoding the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0114] Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report and the global representation and local representation of the mask report;

[0115] Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0116] Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss;

[0117] Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross entropy loss based on the original report and the prediction results;

[0118] Based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, parameters are updated to determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0119] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0120] Acquire an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image;

[0121] Encoding the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0122] Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report and the global representation and local representation of the mask report;

[0123] Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0124] Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss;

[0125] Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross entropy loss based on the original report and the prediction results;

[0126] Based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, parameters are updated to determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0127] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0130] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An ultrasonic multimodal pre-training method, characterized in that: The ultrasonic multimodal pre-training method comprises: Acquire an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image; Encoding the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image; Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report and the global representation and local representation of the mask report; Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss; Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss; Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross entropy loss based on the original report and the prediction results; Based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, parameters are updated to determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

2. The ultrasonic multimodal pre-training method according to claim 1, characterized in that: The image encoder adopts the ResNet or ViT model, the text encoder adopts the BERT model, the image cross-modal encoder adopts the BERT model, the text cross-modal encoder adopts the BERT model, and the convolutional neural network decoder adopts an upsampling convolutional neural network.

3. The ultrasonic multimodal pre-training method according to claim 2, characterized in that: The masked report is obtained by masking the original report based on the accuracy of the vocabulary.

4. The ultrasonic multimodal pre-training method according to any one of claims 2 to 3, characterized in that: The step of encoding based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain the reconstructed image includes: Inputting the local representation of the mask image and the local representation of the original report into an image cross-modal encoder to obtain a first feature output by the image cross-modal encoder; The first feature is input into the convolutional neural network decoder to obtain a reconstructed image output by the convolutional neural network decoder.

5. The ultrasonic multimodal pre-training method according to claim 4, characterized in that: The step of encoding based on the local representation of the mask report, the local representation of the original image and the text cross-modal encoder to obtain the prediction result of the masked vocabulary in the mask report includes: Inputting the local representation of the mask report and the local representation of the original image into the text cross-modal encoder for encoding to obtain a second feature; The second feature is input into the linear layer for prediction to obtain the prediction result.

6. The ultrasonic multimodal pre-training method according to claim 5, characterized in that: The steps of updating parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture and determining the trained ultrasound multimodal pre-training architecture include: Update parameters of the text encoder and the image encoder based on the contrastive learning loss; Update parameters of the convolutional neural network decoder, the image cross-modal encoder, the text encoder, and the image encoder based on the mean square error loss; Update parameters of the text cross-modal encoder, the text encoder, and the image encoder based on the cross entropy loss; It is determined whether the ultrasound multimodal pre-training architecture after the updated parameters meets the training end condition. If so, the ultrasound multimodal pre-training architecture that meets the training end condition is used as the trained ultrasound multimodal pre-training architecture.

7. An ultrasonic multimodal pre-training device, characterized in that: The ultrasound multimodal pre-training device comprises: an acquisition module, which is used to acquire an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture comprises an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder; A first encoding module, used to encode the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, and a global representation and a local representation of the mask image; A second encoding module, used for encoding the original report and the mask report respectively based on the text encoder to obtain a global representation and a local representation of the original report, and a global representation and a local representation of the mask report; A contrastive learning loss calculation module is used to perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss; A mean square error loss calculation module is used to encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image, and perform mean square error loss calculation based on the reconstructed image and the original image to obtain a mean square error loss; A cross entropy loss calculation module is used to encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross entropy loss based on the original report and the prediction results; A parameter updating module is used to update parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, and determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the ultrasound multimodal pre-training method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the ultrasound multimodal pre-training method according to any one of claims 1 to 6 are implemented.

Images