Reconstruction of a mixed strategy three-dimensional medical image visual language model pre-training method

Abstract

The application discloses a three-dimensional medical image visual language model pre-training method of reconstruction mixed strategy, belongs to the technical field of medical image computation, and comprises the following steps: constructing a medical image text pair data set, a language text mask reconstruction strategy, a visual image mask reconstruction strategy, a semantic perception fusion strategy and multi-task joint learning; the large language model is fine-tuned, the fine-tuned large language model is used to extract diagnosis and attribute information in a medical report and generate efficient prompts, the large language model has strong generalization ability, and the cost of manual labeling is greatly saved; the semantic perception fusion strategy is used for combining text features obtained by a text encoder and image features obtained by an image encoder to obtain new text features, so that the text perceives diagnosis and attribute information of the image in advance, the alignment of the image and the text in an embedding space is further optimized, and the pre-training efficiency is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image computing, and particularly relates to a three-dimensional medical image visual language model pre-training method of a reconstruction mixed strategy. BACKGROUND

[0002] Visual language models are widely defined as multi-modal models that can learn on large-scale image-text pairs to improve multi-granularity downstream visual and language tasks. Visual language models are usually composed of three elements: an image encoder, a text encoder, and a learning strategy that fuses the information of the two encoders. Since the loss function is designed around the above model structure and learning strategy, the above key elements need to be tightly coupled together. In traditional methods, the most representative method is CLIP, which demonstrates great potential for learning mutual information between visual and language data. Recent studies show that fine-grained context alignment is beneficial for the model to learn more representative representations, among which the BLIP method reconstructs the text by utilizing visual semantic context. However, in the more challenging medical field, such as medical reports, the accuracy requirements are more stringent, and the above methods are difficult to meet this demand.

[0003] To solve this problem, recent medical visual language models improve the efficiency of model learning through different pre-training methods, such as a contrastive learning method of medical visual representations of image-text pairs and a few-shot self-supervised contrastive learning pre-training method, which pre-train the model by directly maximizing the mutual information between global representations. SAT proposes to align the fine-grained features of paired image patches and words. BioVIL‌ it uses paired data samples to try to understand complex medical reports. MedKLIP uses a triplet extraction module as an additional supervision signal to extract information related to medicine.

[0004] The patent document with application number 202210903886.5 discloses a pre-training method and device for a medical multi-modal model. The target medical image-text sample data bootstrap method cannot generate high-quality medical-text pairs, and the multi-modal hybrid codec MED does not perform good feature fusion, lacking perceptual ability between image and text features. This makes the pre-training model not have performance advantages in downstream tasks.

[0005] The patent document with application number 202410135051.9 discloses a medical visual language pre-training method, system and application based on context perception. The distilled report does not generate high-quality prompt text, and the multi-scale context fusion method integrates visual features and text embeddings without aligning visual and text features well in the embedding space, which makes the pre-training model not have performance advantages in downstream tasks.

[0006] Therefore, there is a need for a pre-training method that can generate high-quality text and better integrate image features and text features. SUMMARY

[0007] The purpose of the present application is to provide a visual language model pre-training method based on a large language model and a reconstruction hybrid strategy, using a pre-training method that combines a large language model to extract text information strategy, semantic perception fusion strategy, and high-order mask reconstruction task joint learning, to improve the performance of downstream medical visual or language tasks, to overcome the deficiencies of the prior art in not better integrating image and text features, and lacking effective means for perception between image and text features.

[0008] To achieve the above purpose, the present application provides the following technical solutions:

[0009] In a first aspect, the present application provides a three-dimensional medical image visual language model pre-training method using a reconstruction hybrid strategy, specifically including the following steps:

[0010] S1, constructing a medical image-text pair dataset, the medical image-text pair dataset comprising three-dimensional medical images and medical field reports;

[0011] S2, extracting and generating text information: extracting the diagnosis and attribute information of the medical field report of the medical image-text pair dataset in S1 to generate the corresponding text of the template;

[0012] S3, text feature generation: performing random mask operation on the text generated in S2 to obtain a text mask, and inputting the obtained text mask into a text encoder to generate a text feature;

[0013] S4, image feature generation: preprocessing the three-dimensional medical images of the medical image-text pair dataset in S1, and performing random mask operation on the preprocessed three-dimensional medical images to obtain a three-dimensional medical mask image, and inputting the obtained three-dimensional medical mask image into an image encoder to generate an image feature;

[0014] S5, inputting the text feature obtained in S3 into a text decoder to obtain a reconstructed text;

[0015] S6, inputting the image feature obtained in S4 into an image decoder to obtain a reconstructed image;

[0016] S7, semantic perception fusion strategy: using a cross-attention mechanism to fuse the text feature and the image feature obtained in S2 and S3 to generate a new text feature;

[0017] S8, text reconstruction task: jointly calculating a text reconstruction loss for the text feature generated in S3 and the reconstructed text obtained in S5;

[0018] S9, image reconstruction task: jointly calculate the image reconstruction loss of the image features generated in S4 and the reconstructed image obtained in S6;

[0019] S10, image-text pair task: jointly perform contrastive learning on the image features in S4 and the new text features obtained in S7, and calculate the image-text pair loss;

[0020] S11, multi-task joint learning: weighted sum of the losses obtained in S8, S9 and S10.

[0021] Further, since the three-dimensional medical data set has the characteristics of difficult data acquisition and high labeling cost, there is currently no large-scale medical image-text pair data set, so first, a large amount of publicly available three-dimensional medical image data sets and medical field report data are collected, and then the image data and medical reports are paired to construct a large-scale medical image-text pair data set, and the specific steps of S1 to construct a large-scale medical image-text pair data set include:

[0022] S11, collect and integrate different publicly available three-dimensional medical image data sets;

[0023] S12, collect medical field reports from major medical field knowledge sources;

[0024] S13, check the target in each three-dimensional medical image data set and assign a medical field report to it, first check each target in each data set and assign a medical report to it, which ensures the accuracy and clarity of the medical text pair data set.

[0025] Further, due to the complexity, diversity and professionalism of medical report data, it is difficult to select appropriate and efficient prompts from medical reports through manual methods, so it is necessary to use the powerful generalization of large language models to generate high-quality diagnostic prompts according to medical reports and templates. Secondly, considering that medical reports not only contain diagnostic information, but also contain description information of the texture and features of the corresponding medical images, therefore, the present application extracts the description information of the medical image properties in the medical report by means of a large language model, and generates available attribute prompts according to the template. The specific steps of extracting and generating text information and generating text features in S2 and S3 are:

[0026] Use a large language model and fine-tune the large language model on a publicly available large-scale medical language data set;

[0027] S22, input the medical field report and the corresponding template information to the large language model after training and fine-tuning, the large language model after training and fine-tuning can extract the corresponding diagnostic and attribute information of the medical field report and generate template corresponding diagnostic text and attribute text representation;

[0028] S23, randomly mask the generated template corresponding diagnostic text and attribute text as a prompt, wherein the fixed mask probability is 20%, obtain a text mask, and input the text mask into a text encoder to generate a text feature.

[0029] Further, due to the characteristics of sparse data, poor imaging quality, and class imbalance of three-dimensional medical image data, the three-dimensional medical image data cannot be directly used for training, and often needs to be preprocessed to generate data rich in visual information, improve the speed of model training, and make the model have good generalization ability. The specific steps of preprocessing three-dimensional medical images and generating image features in S4 are as follows:

[0030] S41, taking the center of the three-dimensional medical image as the reference, a central region is set;

[0031] S42, a point in the central region is selected as a center point for cropping to obtain a three-dimensional medical image block x, x represents a single image block;

[0032] S43, randomly mask the obtained three-dimensional medical image block x, wherein the fixed mask probability is 20%, obtain a three-dimensional medical mask image, and input the obtained three-dimensional medical mask image into an image encoder to generate an image feature.

[0033] Further, the specific steps of obtaining the text feature in S3 are as follows:

[0034] The obtained diagnostic text and attribute text are segmented into words;

[0035] Special words are added at the beginning and end of the diagnostic text sequence and the attribute text sequence;

[0036] The segmented diagnostic text sequence and attribute text sequence are converted into index identifiers corresponding to the vocabulary table in the large language model;

[0037] Create an attention mask to indicate the actual words, special words and padding in the diagnostic text sequence and attribute text sequence;

[0038] Generate a type mask to assign different type identifiers to the diagnostic text sequence and attribute text sequence in each sentence to achieve;

[0039] Create a position encoding to provide the large language model with information about the position of the word in the sentence;

[0040] Each word in the text encoder will pass through a linear projection layer to be mapped to a vector space, and the text feature is extracted.

[0041] Further, the specific steps of obtaining the image feature in S4 are as follows:

[0042] segmenting the preprocessed three-dimensional medical image into a plurality of image patches;

[0043] each image patch is mapped to a higher dimensional space by a linear layer;

[0044] the representation of each image patch is added with a position encoding, which can indicate the position of each image patch in the original image;

[0045] the image patches are serialized and input into an image encoder to obtain image features.

[0046] Further, the fusion of image features and text features in S7 includes the following steps:

[0047] in the cross-attention layer, the image features are input as queries, and the text features are input as keys and values;

[0048] the dot product between the query and the key is calculated to obtain attention weights, which are used to weight the value to generate the fused feature representation;

[0049] the fused features are generated through a feedforward network to generate the final text features.

[0050] The loss function of the new feature formed after fusion is defined as follows: the image features in S4 and the new text features obtained in S7 are jointly compared and learned, and the loss function definition of the image-text pair loss is as follows: the information noise contrast estimation loss function is a loss function for self-supervised learning, which is particularly suitable for contrast learning tasks. It learns useful representations by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. The goal of the information noise contrast estimation loss function is to maximize the similarity between matching images and texts, and to minimize the similarity between non-matching images and texts; this loss function is represented as follows:

[0051]

[0052] wherein, represents the information noise contrast estimation loss, is the feature vector of the image, is the text feature vector matched with the image is the text feature vector not matched with the image is the similarity function, is the temperature parameter, is the total number of vectors in the batch; N

[0053] ​The generated text features in S3 and the reconstructed text obtained in S5 are jointly calculated, and in the text reconstruction task, the commonly used loss function is the log-likelihood loss function, also known as the cross-entropy loss function, which is used to measure the difference between the text sequence generated by the model and the real text sequence. The loss function of the specific text reconstruction loss is denoted as:

[0054]

[0055] wherein, represents the log-likelihood loss, N represents the length of the text sequence, represents the t-th word in the real text sequence, represents the probability of the model predicting the next word under the condition of given input x and the generated sequence .

[0056] The generated image features in S4 and the reconstructed images obtained in S6 are jointly calculated, and the commonly used loss function in the image mask reconstruction task is the mean square error, which is used to measure the difference between the reconstructed image and the original image. The loss function of the image reconstruction loss is denoted as:

[0057]

[0058] wherein, represents the mean square error loss, N represents the total number of pixels in the image, represents the i-th pixel value in the original image, represents the i-th pixel value in the reconstructed image.

[0059] The loss functions of the three tasks are weighted and summed as follows:

[0060]

[0061] wherein, is the final total loss function, represents the weight of the information noise contrast estimation loss, represents the weight of the log-likelihood loss, represents the weight of the mean square error loss.

[0062] In a second aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the pre-training method of the three-dimensional medical image visual language model of the reconstruction hybrid strategy when executing the computer program.

[0063] In a third aspect, the present application provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the pre-training method of the three-dimensional medical image visual language model of the reconstruction hybrid strategy.

[0064] Compared with the prior art, the present application has the following beneficial technical effects:

[0065] The pre-training method of the three-dimensional medical image visual language model of the reconstruction hybrid strategy provided by the present application fine-tunes a large language model, extracts diagnosis and attribute information in a medical report using the fine-tuned large language model, and generates efficient prompts, and the large language model has strong generalization ability, greatly saving the cost of manual annotation. The mask reconstruction strategy of the present application includes mask reconstruction of visual images and mask reconstruction of language texts, which not only enriches the semantic knowledge learned by the network, but also improves the efficiency of pre-training. The semantic perception fusion strategy of the present application fuses the text features obtained by the text encoder and the image features to generate new text features, so that the text can perceive the diagnosis and attribute features of the image in advance, further optimizing the alignment of the image and the text in the embedding space, so that the text can perceive the image features in advance and align them, improving the efficiency of pre-training. In the present application, two branch networks and three branch tasks are designed when constructing the network. The three branch tasks are: jointly calculating the text reconstruction loss of the text features and the reconstructed text, jointly calculating the image reconstruction loss of the image features and the reconstructed image, and jointly performing contrastive learning of the image features and the newly generated text features to calculate the image-text pairing loss. The loss functions of the three branches are weighted and summed, realizing multi-task joint learning, and jointly training multiple objective functions in the training process. The network can learn more relevant semantic knowledge, enriching the content reserve of the network, facilitating the network to be used for downstream multi-task fine-tuning, and greatly improving the performance of downstream visual or language tasks. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The flowchart of the pre-training method of the three-dimensional medical image visual language model of the reconstruction hybrid strategy in the embodiments of the present application.

[0067] Figure 2 The model architecture diagram in the embodiments of the present application.

[0068] Figure 3 The schematic diagram of identifying the target that needs to be distinguished between the orientations according to the left and right of the human body in the embodiments of the present application.

[0069] Figure 4 The schematic diagram of the naming of the same anatomical target from different data sets being consistent in the embodiments of the present application.

[0070] Figure 5A schematic diagram for supplementally merging fine-grained classes and generating additional classes in the embodiment of the present application.

[0071] Figure 6 A schematic diagram for the large language model in the embodiment of the present application to extract corresponding diagnostic and attribute information therefrom and generate a template corresponding text representation.

[0072] Figure 7 A schematic diagram of a text encoder in the embodiment of the present application.

[0073] Figure 8 A schematic diagram of an image encoder in the embodiment of the present application.

[0074] Figure 9 A schematic diagram of a semantic perception fusion strategy in the embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to enable persons skilled in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0076] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0077] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clearly indicated by the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0078] It should be further understood that the term "and / or" used in the description and claims of the application means one or more of the associated listed items as well as all possible combinations of the items, and includes these combinations.

[0079] Reference Figure 1 The pre-training method of the three-dimensional medical image visual language model of the mixed strategy mainly includes the following steps:

[0080] S1, constructing a medical image-text pair data set, which includes three-dimensional medical images and medical field reports;

[0081] S2, extracting and generating text information: extracting the diagnosis and attribute information of the medical field report of the medical image-text pair data set in S1 to generate the text corresponding to the template;

[0082] S3, text feature generation: performing random mask operation on the text generated in S2 to obtain text mask, and inputting the obtained text mask into the text encoder to generate text features;

[0083] S4, image feature generation: preprocessing the three-dimensional medical images of the medical image-text pair data set in S1, and performing random mask operation on the preprocessed three-dimensional medical images to obtain three-dimensional medical mask images, and inputting the obtained three-dimensional medical mask images into the image encoder to generate image features;

[0084] S5, inputting the text features obtained in S3 into the text decoder to obtain the reconstructed text;

[0085] S6, inputting the image features obtained in S4 into the image decoder to obtain the reconstructed image;

[0086] S7, semantic perception fusion strategy: using cross attention mechanism to fuse the text features and image features obtained in S2 and S3 to generate new text features;

[0087] S8, text reconstruction task: jointly calculating the text reconstruction loss of the text features generated in S3 and the reconstructed text obtained in S5;

[0088] S9, image reconstruction task: jointly calculating the image reconstruction loss of the image features generated in S4 and the reconstructed image obtained in S6;

[0089] S10, image-text pairing task: jointly comparing the image features in S4 and the new text features obtained in S7 to calculate the image-text pairing loss;

[0090] S11, multi-task joint learning: weighted sum of the losses obtained in S8, S9 and S10.

[0091] Wherein, since the three-dimensional medical image dataset has the characteristics of difficult data acquisition and high labeling cost, there is currently no large-scale three-dimensional medical image text pair dataset constructed, therefore the present application first collects a large number of publicly available three-dimensional medical image datasets and medical field report data, then pairs the image data and medical reports to construct a large-scale medical image text pair dataset, refer to Figures 2 to 5 , the specific implementation is as follows:

[0092] S11: The present application collects and integrates 45 different publicly available three-dimensional medical image segmentation datasets, a total of 13,918 CT scan data, including 252,243 segmentation annotations across 8 major regions of the human body, the detailed information of the dataset is shown in Table 1;

[0093] Table 1: Details of 45 medical segmentation datasets

[0094]

[0095] S12: The present application collects medical report data from 2 main medical field knowledge sources, mainly including the concept, definition and relationship of segmented organs, for example as shown in Figure 2 , the detailed information of the knowledge source is shown in Table 2;

[0096] S13: The present application pairs the three-dimensional medical image data and the medical report, and executes a variety of programs to ensure the unity of the semantic information of the organs in the image and the semantic information of the medical report. The present application first checks each anatomical target in each three-dimensional medical image dataset and assigns a medical report to it, which ensures the accuracy and definiteness between the medical text pair datasets, for example, always according to the left and right of the human body to identify targets that need to be distinguished between directions, such as left lung and right lung, as shown in Figure 3 . Secondly, the naming of the same anatomical target from different datasets is consistent, for example, the i th lumbar vertebra in the TotalSegmentator dataset and the MRSpineSeg dataset is named in the format "lumbar vertebra i", as shown in Figure 4 . Finally, the same anatomical structure may be annotated at different levels in different datasets, in which case the present application merges fine-grained classes and generates additional classes as supplements to narrow the gap between datasets, for example, the sub-regions of the liver in the CouinaudLiver dataset are merged and added as a new class "liver", as shown in Figure 5 .

[0097] ​In some embodiments, due to the complexity, diversity and professionalism of medical report data, it is difficult to select appropriate and efficient prompts from medical reports by manual methods, so it is necessary to use the powerful generalization of large language models to generate high-quality diagnosis prompts according to medical reports and templates. Secondly, considering that medical reports not only contain diagnosis information, but also contain description information of corresponding medical image textures and characteristics, therefore, the present application extracts the description information of medical image attributes in the medical report by means of a large language model, and generates available attribute prompts according to the template. Referring to Figures 6 to 7 The specific implementation of extracting and generating text information and generating text features in S2 and S3 is as follows:

[0098] The open source large language model BioBERT is used, and the large language model is fine-tuned on a publicly available large-scale medical language data set; the model is trained to extract diagnosis and attribute information in medical reports;

[0099] The fine-tuned large language model is input with medical field reports and corresponding template information, and the fine-tuned large language model can extract the corresponding diagnosis and attribute information of the medical field report and generate the diagnosis text and attribute text representation corresponding to the template;

[0100] The generated diagnosis text and attribute text corresponding to the template are used as prompts for random masking operation, where the fixed masking probability is 20%, and the text mask is obtained, which is input into the text encoder to generate text features.

[0101] In some embodiments, due to the characteristics of sparse data, poor imaging quality, and class imbalance of three-dimensional medical image data, the three-dimensional medical image data cannot be directly used for training, and often needs to be preprocessed to generate data rich in visual information, improve the speed of model training, and make the model have good generalization ability. Referring to Figure 8 The specific implementation is as follows:

[0102] In some embodiments, the three-dimensional medical image data is preprocessed, including various image transformation operations, to improve the accuracy and efficiency of image analysis, and the detailed information of the preprocessing is shown in Table 3.

[0103] The specific steps of preprocessing three-dimensional medical images and generating image features in S4 are as follows:

[0104] S31: Set the center area with the image center as the reference;

[0105] S32: Select a certain point in the center region as the center point for cropping, and finally obtain a three-dimensional medical image block x, wherein x represents a single image block, R represents the image dimension, W represents the width of the image, H represents the height of the image, and D represents the depth of the image, the resolution of the three-dimensional medical image block used in the pre-training stage of the application is 32*256*256, that is, D=32, H=256, and W=256;

[0106] S33: The three-dimensional medical image block x obtained by the application is subjected to a random mask operation, wherein the mask probability is fixed at 20%, and the masked image block is subjected to image encoding to generate visual features. The visual features are input into the image encoder to obtain a reconstructed image, and the structure of the image encoder is as shown in Figure 8 , wherein the image patch size of the encoder is 4*16*16.

[0107] The semantic perception fusion strategy of the application is to combine text features and image features to enhance the transferability of the visual language model, so that the text can perceive image features in advance and align. The specific implementation is as follows:

[0108] In the image encoder shown in Figure 8 , the acquisition of image features needs to go through multiple steps:

[0109] (1) Divide the image, input the image into multiple small blocks (patches), and these small blocks are usually cuboids, such as 4*16*16 pixels.

[0110] (2) Linear mapping, each image patch is mapped to a higher-dimensional space through a linear layer (usually a convolution layer).

[0111] (3) Add position encoding, in order to enable the model to understand the spatial information in the image, the representation of each patch will be added with position encoding (positional encoding), so that the model can know the position of each image patch (patch) in the original image.

[0112] (4) Serialization, these encoded patches form a sequence, which will be input into the encoder.

[0113] In some embodiments, the size of the input three-dimensional medical image to the image encoder is 32x256x256, the three-dimensional medical image is divided into multiple image patches with a size of 4x16x16 through image segmentation, and the three-dimensional medical image is divided into 8x16x16 image blocks. Each image patch is mapped to a dimension of 768 through a linear layer, then each image block is added with position encoding indicating its position in the original image, and finally the image block is serialized and input into the image encoder. The dimension of the visual features extracted by the image encoder from the three-dimensional medical image is 2048x768.

[0114] In Figure 7 In the text encoder shown, the acquisition of text features needs to go through multiple steps:

[0115] (1) Tokenization, the obtained diagnostic text and attribute text are segmented into words, usually words or subwords;

[0116] (2) Add special words, add special words such as 'classification' and'segmentation' at the beginning and end of the diagnostic text sequence and attribute text sequence. 'Classification' is usually used for classification tasks, while'segmentation' is used to separate sentences;

[0117] (3) Convert words to unique identifiers, convert the segmented diagnostic text sequence and attribute text sequence into index identifiers corresponding to the vocabulary table in the open-source large language model BioBERT;

[0118] (4) Create attention mask, create a mask to indicate which part of the text sequence is the actual word, which is a special word or padding, which helps the model ignore the padding part when processing the sequence;

[0119] (5) Generate type mask, when processing two sentence tasks (such as question answering or sentence pair classification), it is necessary to distinguish the text sequences of the two sentences, which is usually achieved by assigning different type identifiers to the text sequences in each sentence;

[0120] (6) Create position encoding, since the Transformer architecture itself does not have the ability to capture sequence order, position encoding is added to provide the model with information about the position of the word in the sentence;

[0121] In some embodiments, when the text encoder inputs a medical report, the report is first segmented into words by word segmentation, special words are added to segment the sentence, and padding or truncation is performed to ensure that the length of the text is 2048 (the text encoder requires a text length of 2048, and the text length refers to the number of words. If the text length is less than 2048, padding is used, and if the text length is greater than 2048, truncation is performed), and then each word is converted into a unique identifier in the word table, and an attention mask, a type mask, and a position encoding are created. In the text encoder, each word is mapped to a vector space with a dimension of 768 by a linear projection layer, so the dimension of the semantic features extracted by the medical report through the text encoder is 2048x768.

[0122] In some embodiments, cross-attention mechanism is used to fuse image features and text features. The output dimension of the known image encoder is 2048x768, and the output dimension of the text encoder is 2048x768. The feature dimensions of image features and text features are the same. In the cross-attention layer, the image features are input as queries, and the text features are input as keys and values. In this way, the model can learn the alignment representation between the image and the text. By calculating the dot product between the query and the key, the attention weight is obtained, which is used to weight the value to generate the fused feature representation. The fused feature is further processed through a feed-forward network to generate the final text feature representation. The final fusion feature dimension is 2048x768.

[0123] In some embodiments, the image features in S4 and the new text features obtained in S7 are jointly used for contrastive learning, and the loss function for calculating the image-text pair loss is defined as follows: The information noise contrast estimation loss function is a loss function for self-supervised learning, which is particularly suitable for contrastive learning tasks. It learns useful representations by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. The goal of the information noise contrast estimation loss function is to maximize the similarity between matching images and texts, while minimizing the similarity between non-matching images and texts; this loss function is represented as follows:

[0124]

[0125] wherein, represents the information noise contrast estimation loss, is the feature vector of the image, is the text feature vector matching the image is the text feature vector matching the image is the text feature vector matching the image The text feature vector that does not match, sim is a similarity function, is a temperature parameter, N is the total number of vectors in the batch;

[0126] The text features generated in S3 and the reconstructed text obtained in S5 are jointly calculated. In the text reconstruction task, the commonly used loss function is the log-likelihood loss function, also known as the cross-entropy loss function. This loss function is used to measure the difference between the text sequence generated by the model and the real text sequence. The loss function of the specific text reconstruction loss is denoted as:

[0127]

[0128] where, represents the log-likelihood loss, N represents the length of the text sequence, represents the t-th word in the real text sequence, represents the probability of the model predicting the next word under the condition of given input x and previously generated sequence .

[0129] The image features generated in S4 and the reconstructed images obtained in S6 are jointly calculated. In the image mask reconstruction task, the commonly used loss function is the mean square error, which is used to measure the difference between the reconstructed image and the original image. The loss function of the image reconstruction loss is denoted as:

[0130]

[0131] where, represents the mean square error loss, N represents the total number of pixels in the image, represents the i-th pixel value in the original image, represents the i-th pixel value in the reconstructed image.

[0132] The loss functions of the three tasks are weighted and summed as follows:

[0133]

[0134] where, represents the weighted sum of the losses of the three tasks, represents the weight of the information noise contrast estimation loss, represents the weight of the log-likelihood loss, represents the weight of the mean square error loss.

[0135] The provided method is further applied in model training, as shown below:

[0136] The network's performance on medical organ segmentation was tested using a pre-trained visual encoder fine-tuned on 3 standard medical image segmentation datasets (BTCV, TotalSegmentaor v2 and TotalSegmentaor MRI), and 1%, 10% and 100% of the training data were used, respectively.

[0137] As shown in Table 2, under different training data proportions of the three datasets, the method of the application is significantly better than all CNN, Transfomer, Mamba, LSTM and SAM-based models in performance.

[0138] Table 2 Experimental results of downstream tasks (three-dimensional medical image organ segmentation)

[0139]

[0140] It is noted that in the BTCV dataset (which contains 50 abdominal CT scans from patients with metastatic liver cancer or postoperative abdominal wall hernia patients. Each scan in the dataset is taken during portal vein contrast phase, with different volume and field of view parameters. The in-image resolution in the dataset varies from 0.54 x 0.54 mm 2 to 0.98 x 0.98 mm 2 , and the slice thickness is between 2.5 mm and 5.0 mm), the method of the application exceeds the strongest opponent by 0.3% using only 1% of the labeled data, by 0.3% using 10% of the labeled data, and by 0.2% using 100% of the data.

[0141] It is worth noting that on the TotalSegmentaor v2 dataset (which is the largest publicly available CT segmentation dataset, the first version of the data was released in July 2022, and the official made a major update to the dataset in September 2023, with a small increase in image quantity and annotation class number. The total number of images increased from 1204 to 1228 (only the number of test sets increased), and the number of classes increased from 104 to 117. The currently published dataset is divided into 1082 training sets, 57 validation sets and 89 test sets (v1 version number is 65), all of which are publicly available and labeled), the method of the application achieves SOTA results when using 1% and 10% of the labeled data, but does not exceed the strongest method when using 100% of the labeled data, which indicates that the pre-trained model of the method of the application has rapid convergence.

[0142] Surprisingly, on the TotalSegmentaor MRI dataset (which contains 298 MR images and provides segmentation annotations for up to 56 different commonly used anatomical structures. Out of these, 251 MR images are from the Picture Archiving and Communication System (PACS) of the University Hospital Basel from 2011 to 2023, and the other 47 MR images are from the Imaging Data Commons (IDC) to increase the diversity of images. This dataset is derived from random sampling in daily clinical work and represents a real-world dataset that can be generalized to clinical applications. It covers a wide range of different pathologies, scanners, imaging sequences, and data from different medical institutions. It is worth noting that although the official paper mentions containing 59 classes, the public dataset only provides annotations for 56 classes, with a slight difference.) The method of the present invention achieves the accuracy of the SOTA method when using 1% of the labeled data, and exceeds all methods when using 10% and 100% of the labeled data, which indicates that the pre-trained visual model of the present method has excellent transfer performance.

[0143] Those skilled in the art will appreciate that embodiments of the present application can be supplied as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product embodied in one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) having computer usable program code embodied therein.

[0144] The present application is described in reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce an apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0145] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocksFigure 1 the function specified in the one or more blocks.

[0146] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable devices to generate a computer-implemented process, thus the instructions executed on the computer or other programmable devices provide a process for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the function specified in the one or more blocks.

[0147] The above shows and describes the basic principles and main features of the present application and the advantages of the present application, and it is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all changes falling within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0148] Furthermore, it should be understood that, although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the present specification is described in this way only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be properly combined to form other embodiments that those skilled in the art can understand. The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application, and any modification made on the basis of the technical solutions according to the present application falls within the protection scope of the claims of the present application

[0149] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A pre-training method for a 3D medical image visual language model based on a hybrid reconstruction strategy, characterized in that, Includes the following steps: S1, construct a medical image-text pair dataset, which includes three-dimensional medical images and medical field reports; S2, Extract and generate text information: Extract the diagnosis and attribute information of the medical field reports in the dataset from the medical image text pairs in S1 to generate the text corresponding to the template; S3, Text Feature Generation: Perform a random masking operation on the text generated in S2 to obtain a text mask, and input the obtained text mask into the text encoder to generate text features; S4, Image Feature Generation: Preprocess the 3D medical images in the medical image text pair dataset in S1, and perform a random masking operation on the preprocessed 3D medical images to obtain 3D medical mask images. Input the obtained 3D medical mask images into the image encoder to generate image features. S5, input the text features obtained in S3 into the text decoder to obtain the reconstructed text; S6, input the image features obtained in S4 into the image decoder to obtain the reconstructed image; S7, Semantic Aware Fusion Strategy: Use cross-attention mechanism to fuse text features and image features obtained in S3 and S4 to generate new text features; S8, Text Reconstruction Task: Jointly calculate the text reconstruction loss by combining the text features generated in S3 and the reconstructed text obtained in S5; S9, Image Reconstruction Task: Jointly calculate the image reconstruction loss by combining the image features generated in S4 and the reconstructed image obtained in S6; S10, Image-Text Pairing Task: Combine the image features from S4 and the new text features obtained in S7 for comparative learning, and calculate the image-text pairing loss; S11, Multi-task Joint Learning: Weighted summation of the losses obtained from different tasks in S8, S9 and S10; The specific steps for extracting and generating text information and text features in S2 and S3 are as follows: We used a large language model and fine-tuned it on a publicly available large-scale medical language dataset. Inputting medical field reports and corresponding template information into the trained and fine-tuned large language model, the trained and fine-tuned large language model can extract the diagnosis and attribute information of the corresponding medical field reports and generate the diagnosis text and attribute text representations corresponding to the templates; The diagnostic text and attribute text corresponding to the generated template are used as prompts for random masking, where the masking probability is fixed at 20%, to obtain the text mask. The text mask is then input into the text encoder to generate text features. The specific steps for fusing image features and text features in S7 are as follows: In the cross-attention layer, image features are used as query input, and text features are used as key and value input; Attention weights are obtained by calculating the dot product between the query and the key. These weights are used to weight the values ​​and generate the fused feature representation. The fused features are used to generate new text features through a feedforward network; The image features from S4 and the new text features obtained from S7 are jointly used for comparative learning. The loss function for calculating the image-text pairing loss is defined as follows: in, This represents the information noise contrast estimation loss. It is the feature vector of the image. Is with images Matched text feature vectors Is with images Mismatched text feature vectors, where sim is the similarity function. It's a temperature parameter. N It represents the total number of vectors in the batch.

2. The method for pre-training a 3D medical image visual language model using a reconstruction hybrid strategy according to claim 1, characterized in that, The specific steps for constructing the medical image-text pair dataset using S1 include: S11, collects and integrates different publicly available 3D medical image datasets; S12, collect medical field reports from major medical field knowledge sources; S13, examine the targets in each 3D medical image dataset and assign them a medical domain report.

3. The method for pre-training a 3D medical image visual language model using a reconstruction hybrid strategy according to claim 1, characterized in that, The specific steps for preprocessing 3D medical images and generating image features in S4 are as follows: S31, using the center of the three-dimensional medical image as a reference, sets the central region; S32, select a point in the central region as the center point for cropping to obtain a three-dimensional medical image block x, where x represents a single image block; S33, the obtained three-dimensional medical image block x is subjected to random masking operation, where the masking probability is fixed at 20%, to obtain a three-dimensional medical mask image. The obtained three-dimensional medical mask image is then input into the image encoder to generate image features.

4. The method for pre-training a 3D medical image visual language model using a reconstruction hybrid strategy according to claim 2, characterized in that, The specific steps for obtaining text features in S3 are as follows: The obtained diagnostic and attribute texts are segmented into words; Add special words at the beginning and end of the diagnostic text sequence and the attribute text sequence, respectively; The segmented diagnostic text sequence and attribute text sequence are converted into index identifiers corresponding to the vocabulary in the large language model; Create attention masks to indicate actual words, special words, and fillers in diagnostic and attribute text sequences; Generate type masks by assigning different type identifiers to the diagnostic text sequence and attribute text sequence in each sentence; Create positional encodings to provide large language models with information about the position of words in sentences; In the text encoder, each word is mapped to a vector space through a linear projection layer, and text features are extracted.

5. The method for pre-training a 3D medical image visual language model using a reconstruction hybrid strategy according to claim 3, characterized in that, The specific steps for obtaining image features in S4 are as follows: The preprocessed 3D medical mask image is divided into multiple image blocks; Each image patch is mapped to a higher-dimensional space through a linear layer; Each image patch is represented with a location code, which indicates the position of each image patch in the original image; Image patches are serialized and input into an image encoder to obtain image features.

6. The method for pre-training a 3D medical image visual language model using a reconstruction hybrid strategy according to claim 1, characterized in that: The loss function for jointly calculating the text reconstruction loss using the text features generated in S3 and the reconstructed text obtained in S5 is denoted as: in, This represents the log-likelihood loss, where N represents the length of the text sequence. This represents the t-th word in the real text sequence. This represents the sequence generated by the model given the input x and the preceding data. Predict the next word under the given conditions. The probability of; The loss function for jointly calculating the image reconstruction loss using the image features generated in S4 and the reconstructed image obtained in S6 is denoted as: in, This represents the mean squared error loss, where N represents the total number of pixels in the image. This represents the value of the i-th pixel in the original image. This represents the value of the i-th pixel in the reconstructed image; The weighted sum of the loss functions for the three tasks is defined as follows: in, The weights represent the information noise contrast estimation loss. The weights represent the log-likelihood loss. This represents the weight of the mean squared error loss.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-6.

8. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

Images