Ultrasound multimodal pre-training method and device, computer device, and storage medium

Abstract

The application relates to the fields of artificial intelligence, medicine and pre-training, and discloses an ultrasonic multimodal pre-training method and device, computer equipment and a storage medium, the method comprising the following steps: acquiring an original image, a mask image, an original report, a mask report and an ultrasonic multimodal pre-training architecture; determining a contrast learning loss, a mean square error loss and a cross-entropy loss based on the original image, the mask image, the original report, the mask report and the ultrasonic multimodal pre-training architecture; and performing parameter updating based on the contrast learning loss, the mean square error loss, the cross-entropy loss and the ultrasonic multimodal pre-training architecture to determine a trained ultrasonic multimodal pre-training architecture, wherein an image encoder in the trained ultrasonic multimodal pre-training architecture is used for a downstream task.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, medicine, and pre-training technology, and in particular to an ultrasonic multimodal pre-training method, device, computer equipment, and storage medium. Background Art

[0002] In recent years, multimodal pre-training has attracted widespread attention in both industry and academia. This technology leverages the complementary relationship between images and text to achieve performance comparable to or even better than unimodal pre-training. By fully leveraging the complementarity between images and text, multimodal pre-training can also reduce reliance on unimodal data. In particular, the medical field is a viable application scenario for multimodal pre-training. Medical images and their reports are natural image-text data pairs, requiring no additional annotation by doctors. Recent work has also demonstrated the great potential of medical multimodal pre-training. However, current work mainly focuses on radiology images and reports, with relatively little research in the ultrasound field.

[0003] Existing pre-training techniques typically use labeled images to pre-train models. However, in the medical field, due to privacy and expertise constraints, the lack of large-scale labeled medical image datasets makes it difficult to tap into the potential of pre-training in this field. Research on multimodal pre-training in medicine has primarily focused on X-rays, with relatively little research on ultrasound. Due to these significant domain differences, models pre-trained on X-rays cannot be directly transferred to ultrasound. Furthermore, ultrasound images and reports have various characteristics. For example, a single report often corresponds to multiple ultrasound images, and ultrasound reports have proprietary templates and specialized vocabulary, making many X-ray pre-training techniques inaccessible. Summary of the Invention

[0004] Based on this, it is necessary to address the technical problem that the existing technology has poor pre-training effect on ultrasound images and reports, and propose an ultrasound multimodal pre-training method, device, computer equipment and storage medium.

[0005] In a first aspect, an ultrasound multimodal pre-training method is provided, the method comprising:

[0006] Acquire an original image, a masked image, an original report, a masked report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image;

[0007] Encode the original image and the mask image based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0008] Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report, as well as the global representation and local representation of the mask report;

[0009] Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0010] Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss;

[0011] Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross-entropy loss based on the original report and the prediction results;

[0012] Parameters are updated based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine a trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks.

[0013] In a second aspect, an ultrasound multimodal pre-training device is provided, the device comprising:

[0014] An acquisition module, configured to acquire an original image, a masked image, an original report, a masked report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder;

[0015] A first encoding module, configured to encode the original image and the mask image respectively based on an image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0016] A second encoding module is used to encode the original report and the mask report based on the text encoder to obtain a global representation and a local representation of the original report, as well as a global representation and a local representation of the mask report;

[0017] A contrastive learning loss calculation module is used to perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0018] A mean square error loss calculation module is used to encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and perform mean square error loss calculation based on the reconstructed image and the original image to obtain the mean square error loss;

[0019] A cross-entropy loss calculation module is used to encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross-entropy loss based on the original report and the prediction results;

[0020] A parameter update module is used to update parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, and determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0021] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned ultrasound multimodal pre-training method when executing the computer program.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned ultrasound multimodal pre-training method are implemented.

[0023] The ultrasound multimodal pre-training method proposed in the present invention obtains an original image, a mask image, an original report, a mask report and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder and a convolutional neural network decoder. The original image is an ultrasound image, and then the original image and the mask image are encoded based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image, and the original report and the mask report are encoded based on the text encoder to obtain a global representation and a local representation of the original report, as well as a global representation and a local representation of the mask report, and then comparative learning is performed based on the global representation of the original image and the global representation of the original report. , calculate the contrastive learning loss, then encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image, and calculate the mean square error loss based on the reconstructed image and the original image to obtain the mean square error loss, then encode based on the local representation of the mask report, the local representation of the original image and the text cross-modal encoder to obtain the prediction result of the masked vocabulary in the mask report, and calculate the cross entropy loss based on the original report and the prediction result, finally update the parameters based on the contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine the trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks. The present invention can use medical images and their reports to pre-train the model, significantly improving its performance on downstream visual recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] in:

[0026] Figure 1 FIG2 is a diagram showing an application environment of an ultrasound multimodal pre-training method according to an embodiment;

[0027] Figure 2 is a flowchart of an ultrasound multimodal pre-training method in one embodiment;

[0028] Figure 3 1 is a schematic diagram of the structure of an ultrasound multimodal pre-training architecture of an ultrasound multimodal pre-training method in one embodiment;

[0029] Figure 4 A schematic diagram of generating an original image of an ultrasound multimodal pre-training method in one embodiment;

[0030] Figure 5 A graph showing the comparison of an ultrasound multimodal pre-training method with other methods in one embodiment;

[0031] Figure 6 A comparative visualization of an ultrasound multimodal pre-training method and other methods in one embodiment;

[0032] Figure 7 A visualization of segmentation and detection of an ultrasound multimodal pre-training method in one embodiment;

[0033] Figure 8 A classification visualization of an ultrasound multimodal pre-training method according to an embodiment is shown;

[0034] Figure 9 Schematic diagram of word weight analysis of an ultrasound multimodal pre-training method in one embodiment;

[0035] Figure 10 Schematic diagram showing a comparison of multi-image processing strategies for an ultrasound multimodal pre-training method in one embodiment;

[0036] Figure 11 FIG1 is an image mask analysis diagram of an ultrasound multimodal pre-training method in one embodiment;

[0037] Figure 12 A comparison diagram of image mask strategies for an ultrasound multimodal pre-training method in one embodiment;

[0038] Figure 13 is a structural block diagram of an ultrasound multimodal pre-training device in one embodiment;

[0039] Figure 14 is a structural block diagram of a computer device in one embodiment;

[0040] Figure 15 It is a structural block diagram of a computer device in another embodiment. DETAILED DESCRIPTION

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0042] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] The ultrasound multimodal pre-training method provided by the embodiment of the present invention can be applied in the following Figure 1In an application environment, the client 110 communicates with the server 120 through a network. The server 120 can obtain the original image, the mask image, the original report, the mask report and the ultrasound multimodal pre-training architecture through the client 110, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder and a convolutional neural network decoder. The original image is an ultrasound image. Then the server 120 encodes the original image and the mask image based on the image encoder to obtain the global representation and local representation of the original image, as well as the global representation and local representation of the mask image, and encodes the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report, as well as the global representation and local representation of the mask report. Then the server 120 performs comparative learning based on the global representation of the original image and the global representation of the original report to calculate. To contrastive learning loss, then the server 120 encodes based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image, and calculates the mean square error loss based on the reconstructed image and the original image to obtain the mean square error loss. Then the server 120 encodes based on the local representation of the mask report, the local representation of the original image and the text cross-modal encoder to obtain the prediction result of the masked vocabulary in the mask report, and calculates the cross entropy loss based on the original report and the prediction result. Finally, the server 120 updates the parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine the trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks. The present invention can use medical images and their reports to pre-train the model, significantly improving its performance on downstream visual recognition tasks. Among them, the client 110 can be, but is not limited to, various personal computers, laptops, smartphones, tablets and portable wearable devices. The server 120 can be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific examples.

[0045] See also Figure 2 As shown, Figure 2 A flowchart of an ultrasound multimodal pre-training method according to an embodiment of the present invention includes the following steps:

[0046] Step S101: obtaining an original image, a masked image, an original report, a masked report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder;

[0047] Among them, the original image is an ultrasound image, the mask image is an image based on the mask operation performed on the original image, the original report can be the ultrasound image report corresponding to the original image, and the mask report can refer to a report obtained by performing a mask operation on the content in the original report.

[0048] In this embodiment, referring to Figure 3 ,The proposed ultrasound multimodal pre-training architecture consists of five modules: image encoder f v , text encoder f t , image cross-modal encoder cross v , text cross-modal encoder cross t and convolutional neural network decoder d.

[0049] In one embodiment, the image encoder adopts a ResNet or ViT model, the text encoder adopts a BERT model, the image cross-modal encoder adopts a BERT model, the text cross-modal encoder adopts a BERT model, and the convolutional neural network decoder adopts an upsampling convolutional neural network.

[0050] Step S102: Encode the original image and the mask image respectively based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0051] Step S103: Encode the original report and the mask report based on the text encoder to obtain a global representation and a local representation of the original report, as well as a global representation and a local representation of the mask report;

[0052] As an example, first use the image encoder f v and the text encoder f t Encode the original image and mask image, original report and mask report respectively:

[0053] v g,i ,v l,i =f v (I i )

[0054] t g,i ,t l,i =f t (R i )

[0055] v′ g,i ,v′ l,i =f v (I′ i )

[0056] t′ g,i ,t′ l,i =f t (R′i )

[0057] Among them, i i ,R i denote the original image and original report of the i-th sample in the mini-batch, I′ i ,R′ i Represents the mask image and mask report of the i-th sample in the mini-batch, v g,i ,v l,i Characterize the global representation and local representation of the original image of the i-th sample, t g,i ,t l,i Characterize the global and local representations of the original report of the i-th sample, v′ g,i ,v′ l,i Represents the global representation and local representation of the i-th sample mask image, t′ g,i ,t′ l,i represents the global and local representations of the mask report of the i-th sample.

[0058] Step S104: performing contrastive learning based on the global representation of the original image and the global representation of the original report, and calculating the contrastive learning loss;

[0059] It's important to note that contrastive learning maps images and text into a single representational space by narrowing the distance between matching image-text pairs and widening the distance between mismatched image-text pairs. As a self-supervised learning method, contrastive learning can significantly improve the performance of pre-trained models in zero-shot and few-shot classification tasks.

[0060] In this embodiment, the global representation of the original image and the global representation of the original report, i.e., v g,i and t g,i Specifically, for a small batch of samples of size N, we first calculate the cosine similarity between each image and all reports and the cosine similarity between each report and all images, and use the calculated cosine similarity to calculate the InfoNCE loss, as shown below:

[0061]

[0062] Where τ is the temperature coefficient, s(*,*) is the similarity equation, and the total loss function of contrastive learning is as follows:

[0063]

[0064] Where λ1 is the contrastive learning loss weight.

[0065] Step S105: Encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain, and calculate the mean square error loss based on the reconstructed image and the original image to obtain the mean square error loss;

[0066] It should be noted that mask image modeling can enhance the pre-trained model's ability to understand fine-grained features. Mask images can be constructed by exchanging image patch information. Specifically, the original image is first divided into 16*16 image patches. Then, 25% of these patches are randomly selected and exchanged with any image patch in the image to generate a mask image.

[0067] Specifically, the local representation of the mask image and the local representation of the original report are input into the image cross-modal encoder to obtain the first feature output by the image cross-modal encoder; the first feature is input into the convolutional neural network decoder to obtain the reconstructed image output by the convolutional neural network decoder.

[0068] As an example, using the local representation v′ of the mask image l,i and the original reported local representation t′ l,i To reconstruct the image. Specifically, v′ l,i , t′ l,i As an image cross-modal encoder cross v Input, and t′ l,i Perform cross attention, cross v The output of will be used as the input of the image decoder d, restored to the original image size, and the masked image will be predicted, that is, the reconstructed image. The loss is defined as the mean square error between the reconstructed image and the original image:

[0069]

[0070] Among them, MSE represents the mean squared error loss, and λ2 is the loss weight for cross-modal mask image modeling.

[0071] Step S106: Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain prediction results for the masked words in the mask report, and calculate the cross-entropy loss based on the original report and the prediction results;

[0072] It should be noted that ultrasound reports contain a large number of professional terms and fixed collocations, and are a special language. Therefore, a cross-modal masked language modeling task is introduced to enhance the model's ability to understand ultrasound reports. Unlike English text, which is composed of words, Chinese text is composed of characters. If masking is performed at the character level, the fixed collocations between characters will reduce the difficulty of predicting masked text, especially in medical reports, which usually have obvious structured templates. Therefore, in order to improve the performance of masked language modeling, the present invention performs masking at the word level to obtain a masked report. Specifically, the report is first segmented using the jieba word segmenter, and then 450 common words most relevant to the report content are manually screened to form a vocabulary. For the input report, the words are first segmented according to the vocabulary, and then 25% of the words are randomly masked to obtain a masked report. 15% of the masked words are replaced with special words [MASK], 5% remain unchanged, and 5% are replaced with random words.

[0073] In this embodiment, the local representation of the mask report and the local representation of the original image are first input into the text cross-modal encoder for encoding to obtain a second feature, and then input into a linear layer for prediction based on the second feature to obtain the prediction result.

[0074] In this embodiment, the mask report is first encoded into a global representation t′ g,i , local representation t′ l,o , where the local representation t′ of the mask report l,i As a text cross-modal encoder t Input, and local representation v of the original image l,i Perform cross attention, cross t The output of is passed through a linear layer to predict the masked words. The prediction effect is characterized by cross entropy loss:

[0075] Updated text cross-modal encoder, text encoder, and image encoder.

[0076] Among them, CE is the cross entropy loss and λ3 is the loss weight for cross-modal masked language modeling.

[0077] Step S107: Parameters are updated based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

[0078] In this embodiment, the parameters of the text encoder and the image encoder are updated based on the contrastive learning loss, and then the parameters of the convolutional neural network decoder, the image cross-modal encoder, the text encoder, and the image encoder are updated based on the mean square error loss, and then the parameters of the text cross-modal encoder, the text encoder, and the image encoder are updated based on the cross entropy loss.

[0079] Finally, it is determined whether the ultrasound multimodal pre-training architecture after the updated parameters meets the training end condition. If so, the ultrasound multimodal pre-training architecture that meets the training end condition is used as the trained ultrasound multimodal pre-training architecture.

[0080] After the model pre-training is completed, the trained model weights can be directly used as the initialization weights of downstream tasks for downstream migration and fine-tuning. Taking the three downstream tasks of classification, segmentation and target detection as examples, the migration method is as follows: Classification: Use linear probe technology to linearly map the output features of the pre-trained model to the category dimension for classification; Segmentation: Use the pre-trained weights as the pre-trained weights of segmentation models such as U-Net and SETR for segmentation; Target detection: Use the pre-trained weights as the pre-trained weights of the Yolo series template detection model for target detection; After initializing expert models such as U-Net and Yolo with the pre-trained model, you can use a small amount of data to fine-tune the new model to achieve better results than retraining a new model.

[0081] In one embodiment, the masked report is generated by masking the original report based on the accuracy of vocabulary. Medical reports typically have a clear template, meaning that doctors use a unified format when writing reports, requiring only specific locations to be filled with vocabulary related to the condition. Therefore, these filled-in vocabulary is the most important linguistic information. A dynamic keyword mining technique is proposed, which aims to increase the probability of these keywords being masked in masked language modeling through self-supervision, allowing the model to spend more time learning these key vocabulary. In the dynamic keyword mining technique proposed in this invention, a weight is maintained for each word in the vocabulary to represent the word's prediction accuracy. If a word is frequently predicted correctly, it may be a template word. Such words are less difficult to predict and can be predicted directly based on context without relying on image information, so the weight of the word is reduced. If a word is frequently predicted incorrectly, its prediction may require more image information, placing higher demands on the model, so its corresponding weight is increased. During masking, the weight table is used, meaning that simple words are masked less frequently and complex keywords are masked more frequently. This allows the model to have more time to learn keywords. The weight change process can be formulated as follows:

[0082]

[0083] Where ub and lb are the upper and lower bounds of the weight change, w is the vocabulary weight, and m is a hyperparameter that controls the magnitude of the change.

[0084] In one embodiment, referring to Figure 4 A medical report usually corresponds to multiple medical images. These images correspond to the information in the report. Inputting more images can facilitate the model's modal information alignment, which will facilitate contrastive learning and modal reconstruction. Considering the needs of pre-training and downstream tasks, for a case, a 1000*1500 canvas is first constructed and then randomly divided into 6 blocks. Next, the first 6 images of the case are read from the dataset. These 6 images are randomly cropped and enhanced and then placed into the canvas. Cases with less than 6 images are replaced with completely black images. Finally, the canvas is transformed into the original 256*256 image.

[0085] It should be noted that the present invention is compared with other similar methods in ultrasound image classification, segmentation and target detection tasks. Other methods have shortcomings in these three tasks, while the present invention has balanced performance in these three methods and achieved relatively excellent performance. In visualization, the present invention can more accurately locate the lesion area, and the ROC curve also achieves the best effect. Figure 5 , the results of comparing the ultrasound multimodal pre-training method proposed in this invention with similar methods. Figure 6 , which is a comparative visualization of the ultrasound multimodal pre-training method proposed in this invention and similar methods. Figure 7 , which is the segmentation and detection visualization of the ultrasound multimodal pre-training method proposed in this invention. Figure 8 , which is the classification visualization of the ultrasound multimodal pre-training method proposed in this invention.

[0086] refer to Figure 9 ,Classification visualization of the ultrasound multimodal pre-training method proposed in this invention, Figure 9 Including (a) the word weight change diagram of all words, (b) some words that are difficult for the model to learn, and (c) the word weight change curve of some words. After training according to the method proposed by this invention, the weight of each word in the vocabulary presents a special pattern, as follows Figure 9As shown in the figure, the weights of template words rise and fall rapidly in the early stages of training, as shown by the weight curve to the left of auxiliary line I, which has already dropped to the lower bound. This is because the large number of template words often causes them to be masked in the early stages of training. Before the model learns how to predict them, their weights rise rapidly. However, these template words often have obvious prediction patterns, allowing the model to quickly learn how to predict them. Therefore, after a short period of time, their weights drop rapidly until they reach the lower bound, where they are no longer valued by the model. However, their high occurrence rate allows them to be occasionally masked, preventing the model from completely forgetting their predictions. For long-tail words, due to their low probability of occurrence, their weights rise slowly and remain undervalued throughout training, as shown in the figure, never exceeding the weight of the words marked by auxiliary line II. Words outside these two categories are the model's primary learning target, with approximately two-thirds of the training time devoted to learning them (segments BC). Many of the keywords reported in the report fall into this category, including words with image information, such as location words and qualitative words. This invention enables the model to focus on these words, thereby blocking out the ineffective information brought by template words and long-tail words, helping the model learn effective visual information in the masked language modeling task. We visualized the weight curves of some words and found that the weight curves of words frequently appearing in the template, such as "lesion" and "morphology," exhibit a rapid rise and fall, while the weight curves of words rarely appearing in reports, such as "diffuse" and "scar," show a slow increase. The weight curves of location-representing keywords, such as "middle" and "left lobe," slowly increase to an upper bound after the model learns the prediction patterns of the template words and remain at this upper bound until the model learns the prediction patterns for these words. The figure also shows some words that the model struggles to understand; their weights remain at the upper bound and do not decrease until the end of training. Most of these words are location terms, used to indicate the relative position of organs. This is because relative position is artificially defined and inherently ambiguous, making it difficult for the model to learn relevant location terms. The weights of location terms, such as "left lobe" and "right lobe," have already dropped to the lower bound, indicating that the model has learned to distinguish left from right based on images. If we rely only on medical reports, replacing these two words does not affect the semantic coherence.

[0087] refer to Figure 10, which is a comparison of multi-image processing strategies. Image-text matching refers to the matching accuracy of comparative learning on the verification set. In order to verify whether the use of multiple images can provide performance improvements for pre-training and downstream tasks, two commonly used methods were deployed: using a fixed image as input and randomly sampling an image as input, and comparing them with the multi-image processing strategy proposed in the present invention. The results show that making full use of the many-to-one relationship between images and texts can significantly improve the effect of pre-training. It is worth noting that although the input of the present invention during pre-training is a synthetic image, and the input of the downstream task is a single image, and the image encoder is frozen in the downstream task, the image encoder can still apply the image feature extraction ability learned during pre-training to the downstream task, proving that the pre-trained model of the present invention can extract features with translation invariance and scale invariance-the synthetic image is obtained by splicing a single image after reduction.

[0088] refer to Figure 11 , is an image mask analysis diagram. Compared with the mask strategy of directly removing some image blocks, the pre-training effect of the image block exchange mask strategy proposed in this paper is better. The trained model outperforms the former in downstream classification and segmentation tasks. Figure 12 , is a comparison chart of image mask strategies. The ablation experiment shows that all modules and methods proposed in this invention have a positive effect on pre-training.

[0089] See also Figure 13 As shown, in one embodiment, an ultrasound multimodal pre-training device is provided, the device comprising: an acquisition module 10, for acquiring an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture comprises an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder;

[0090] A first encoding module 20 is configured to encode the original image and the mask image based on an image encoder to obtain a global representation and a local representation of the original image, and a global representation and a local representation of the mask image;

[0091] A second encoding module 30 is configured to encode the original report and the mask report based on the text encoder to obtain a global representation and a local representation of the original report, and a global representation and a local representation of the mask report;

[0092] a contrastive learning loss calculation module 40 for performing contrastive learning based on the global representation of the original image and the global representation of the original report, and calculating the contrastive learning loss;

[0093] a mean square error loss calculation module 50 for encoding based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and performing a mean square error loss calculation based on the reconstructed image and the original image to obtain a mean square error loss;

[0094] a cross entropy loss calculation module 60 for encoding based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain prediction results for the masked words in the mask report, and calculating the cross entropy loss based on the original report and the prediction results;

[0095] The parameter update module 70 is used to update parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, and determine the trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks.

[0096] In one embodiment, the image encoder adopts a ResNet or ViT model, the text encoder adopts a BERT model, the image cross-modal encoder adopts a BERT model, the text cross-modal encoder adopts a BERT model, and the convolutional neural network decoder adopts an upsampling convolutional neural network.

[0097] In one embodiment, the masked report is obtained by masking the original report based on the accuracy of the vocabulary.

[0098] In one embodiment, the mean square error loss calculation module 50 is used to:

[0099] Inputting the local representation of the mask image and the local representation of the original report into an image cross-modal encoder to obtain a first feature output by the image cross-modal encoder;

[0100] The first feature is input into the convolutional neural network decoder to obtain a reconstructed image output by the convolutional neural network decoder.

[0101] In one embodiment, the cross entropy loss calculation module 60 is configured to:

[0102] Inputting the local representation of the mask report and the local representation of the original image into the text cross-modal encoder for encoding to obtain a second feature;

[0103] The second feature is input into the linear layer for prediction to obtain the prediction result.

[0104] In one embodiment, the parameter updating module 70 is configured to:

[0105] Updating parameters of the text encoder and the image encoder based on the contrastive learning loss;

[0106] Update parameters of the convolutional neural network decoder, the image cross-modal encoder, the text encoder, and the image encoder based on the mean squared error loss;

[0107] Updating parameters of the text cross-modal encoder, the text encoder, and the image encoder based on the cross entropy loss;

[0108] Determine whether the ultrasound multimodal pre-training architecture after the updated parameters meets the training end condition. If so, use the ultrasound multimodal pre-training architecture that meets the training end condition as the trained ultrasound multimodal pre-training architecture.

[0109] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 14 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of an ultrasound multimodal pre-training method.

[0110] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 15 As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a client side of an ultrasound multimodal pre-training method.

[0111] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0112] Acquire an original image, a masked image, an original report, a masked report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image;

[0113] Encode the original image and the mask image based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0114] Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report, as well as the global representation and local representation of the mask report;

[0115] Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0116] Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss;

[0117] Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross-entropy loss based on the original report and the prediction results;

[0118] Parameters are updated based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine a trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks.

[0119] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0120] Acquire an original image, a masked image, an original report, a masked report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image;

[0121] Encode the original image and the mask image based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image;

[0122] Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report, as well as the global representation and local representation of the mask report;

[0123] Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss;

[0124] Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss;

[0125] Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross-entropy loss based on the original report and the prediction results;

[0126] Parameters are updated based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine a trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks.

[0127] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0128] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0130] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An ultrasound multimodal pre-training method, characterized in that: The ultrasonic multimodal pre-training method comprises: Acquire an original image, a masked image, an original report, a masked report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder, and the original image is an ultrasound image; Encode the original image and the mask image based on the image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image; Encode the original report and the mask report based on the text encoder to obtain the global representation and local representation of the original report, as well as the global representation and local representation of the mask report; Perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss; Encoding is performed based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and the mean square error loss is calculated based on the reconstructed image and the original image to obtain the mean square error loss; Encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross-entropy loss based on the original report and the prediction results; Parameters are updated based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture to determine a trained ultrasound multimodal pre-training architecture, wherein the image encoder in the trained ultrasound multimodal pre-training architecture is used for downstream tasks.

2. The ultrasonic multimodal pre-training method according to claim 1, characterized in that: The image encoder adopts the ResNet or ViT model, the text encoder adopts the BERT model, the image cross-modal encoder adopts the BERT model, the text cross-modal encoder adopts the BERT model, and the convolutional neural network decoder adopts an upsampling convolutional neural network.

3. The ultrasonic multimodal pre-training method according to claim 2, characterized in that: The masked report is obtained by masking the original report based on the accuracy of the vocabulary.

4. The ultrasonic multimodal pre-training method according to any one of claims 2 to 3, characterized in that: The step of encoding based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder and the convolutional neural network decoder to obtain a reconstructed image includes: Inputting the local representation of the mask image and the local representation of the original report into an image cross-modal encoder to obtain a first feature output by the image cross-modal encoder; The first feature is input into the convolutional neural network decoder to obtain a reconstructed image output by the convolutional neural network decoder.

5. The ultrasonic multimodal pre-training method according to claim 4, characterized in that: The step of encoding based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain a prediction result for the masked vocabulary in the mask report includes: Inputting the local representation of the mask report and the local representation of the original image into the text cross-modal encoder for encoding to obtain a second feature; The second feature is input into the linear layer for prediction to obtain the prediction result.

6. The ultrasonic multimodal pre-training method according to claim 5, characterized in that: The steps of updating parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture and determining a trained ultrasound multimodal pre-training architecture include: Updating parameters of the text encoder and the image encoder based on the contrastive learning loss; Update parameters of the convolutional neural network decoder, the image cross-modal encoder, the text encoder, and the image encoder based on the mean squared error loss; Updating parameters of the text cross-modal encoder, the text encoder, and the image encoder based on the cross entropy loss; Determine whether the ultrasound multimodal pre-training architecture after the updated parameters meets the training end condition. If so, use the ultrasound multimodal pre-training architecture that meets the training end condition as the trained ultrasound multimodal pre-training architecture.

7. An ultrasonic multimodal pre-training device, characterized in that: The ultrasound multimodal pre-training device includes: an acquisition module for acquiring an original image, a mask image, an original report, a mask report, and an ultrasound multimodal pre-training architecture, wherein the ultrasound multimodal pre-training architecture includes an image encoder, a text encoder, an image cross-modal encoder, a text cross-modal encoder, and a convolutional neural network decoder; A first encoding module, configured to encode the original image and the mask image respectively based on an image encoder to obtain a global representation and a local representation of the original image, as well as a global representation and a local representation of the mask image; A second encoding module is used to encode the original report and the mask report based on the text encoder to obtain a global representation and a local representation of the original report, as well as a global representation and a local representation of the mask report; A contrastive learning loss calculation module is used to perform contrastive learning based on the global representation of the original image and the global representation of the original report, and calculate the contrastive learning loss; A mean square error loss calculation module is used to encode based on the local representation of the mask image, the local representation of the original report, the image cross-modal encoder, and the convolutional neural network decoder to obtain a reconstructed image, and perform mean square error loss calculation based on the reconstructed image and the original image to obtain the mean square error loss; A cross-entropy loss calculation module is used to encode based on the local representation of the mask report, the local representation of the original image, and the text cross-modal encoder to obtain the prediction results of the masked words in the mask report, and calculate the cross-entropy loss based on the original report and the prediction results; A parameter update module is used to update parameters based on contrastive learning loss, mean square error loss, cross entropy loss, and ultrasound multimodal pre-training architecture, and determine a trained ultrasound multimodal pre-training architecture, wherein the trained image encoder in the ultrasound multimodal pre-training architecture is used for downstream tasks.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the ultrasound multimodal pre-training method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the ultrasound multimodal pre-training method according to any one of claims 1 to 6 are implemented.

Images