Text-guided multi-model medical image segmentation method based on MoCo

Abstract

The invention provides a text guide multi-model medical image segmentation method based on MoCo. And through the query encoder, the first momentum encoder and the second momentum encoder, combined learning is carried out by combining the medical image and the text description, and a high-precision medical image segmentation result is generated. The query encoder processes image and text data at the same time, and image features and text features are fused for segmentation; and the momentum encoder and the query encoder are subjected to parameter collaborative optimization through a momentum updating mechanism, so that the model performance is improved. And by updating the features in the query queue and the momentum queue, the learning ability and stability of the model are further enhanced. The loss function is calculated based on the similarity of the image features and the text features, and the parameters of the query encoder and the momentum encoder are optimized, so that the segmentation precision is improved. The method has high accuracy and robustness in medical image segmentation, is suitable for different medical image processing tasks, and has important application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and particularly relates to a text-guided multi-model medical image segmentation method based on MoCo. Background Art

[0002] Medical image segmentation is a key technology in intelligent healthcare and plays a crucial role in the process of disease diagnosis and treatment plan formulation. With the increasing amount of medical image data, how to effectively process and analyze these images has become an urgent problem to be solved. Most of the existing medical image segmentation methods rely on labeled data for supervised learning. However, in practical applications, it is difficult and costly to obtain labeled data. Therefore, unsupervised learning methods, especially contrastive learning, have gradually become an effective solution to this problem.

[0003] As an unsupervised learning method, contrastive learning extracts the internal structure of data by comparing the features between different samples. However, in the field of medical images, there are some challenges in directly applying traditional contrastive learning methods. For example, there are significant differences in pixel distribution and structural features between medical images and natural images, and contrastive learning methods tend to misprocess semantically similar images as negative samples, resulting in poor segmentation effects. In addition, there are often a large number of semantically similar situations in medical image tasks, such as lung lesion segmentation, thoracic organ segmentation, etc. The images in these tasks usually share similar backgrounds, and traditional contrastive learning methods may misover-distinguish these images, affecting the training effect.

[0004] In this context, the application of existing contrastive learning methods in medical image segmentation is not yet perfect and urgently needs improvement. Summary of the Invention

[0005] The purpose of the present invention is to provide a text-guided multi-model medical image segmentation method based on MoCo to overcome the above-mentioned defects existing in the prior art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] On the one hand, the present invention provides a text-guided multi-model medical image segmentation method based on MoCo, including the following steps:

[0008] Construct a query encoder, a first momentum encoder, and a second momentum encoder;

[0009] Train the query encoder, the first momentum encoder, and the second momentum encoder through contrastive learning;

[0010] After all three models are trained, when there are medical images and corresponding text data, and the video memory resources are sufficient and the time requirement is loose, the medical images and corresponding text data are input into the query encoder for image segmentation; when there are medical images and corresponding text data, and the video memory resources are tense or the time requirement is strict, the medical images and corresponding text data are input into the first momentum encoder for image segmentation; when there is only a medical image, the medical image is input into the second momentum encoder for image segmentation.

[0011] Further, the training of the query encoder, the first momentum encoder, and the second momentum encoder through contrastive learning specifically includes:

[0012] Obtain a training data set, where the training data set includes medical images and corresponding text data. The text data is the natural language description corresponding to the medical image, including anatomical structures, lesion areas, disease types, labels of segmentation areas, or other medical features in the image;

[0013] Randomly initialize the query encoder Q1, and then use the parameters in the query encoder Q1 to initialize the parameters of the parts with the same structure in the first momentum encoder and the second momentum encoder;

[0014] Receive medical images and corresponding text data through the query encoder and generate first image features and first text features;

[0015] Receive medical images and corresponding text data through the first momentum encoder and generate second image features and second text features. Update the first query queue according to the generated second image features, and update the second query queue according to the generated second text features;

[0016] Receive medical images through the second momentum encoder and generate third image features, and update the third query queue according to the generated third image features;

[0017] The first query queue, the second query queue, and the third query queue have the same and fixed length. The first query queue is used to store the second image features generated by training the first momentum encoder each time and their corresponding medical image numbers. The second query queue is used to store the second text features generated by training the first momentum encoder each time and their corresponding medical image numbers. The third query queue is used to store the third image features generated by training the second momentum encoder each time and their corresponding medical image numbers;

[0018] Calculate the similarity between the first text feature and each text feature in the second query queue, and screen out the medical image numbers corresponding to the text features with a similarity less than the first threshold to obtain a set of contrast medical image numbers;

[0019] Calculate the first loss value between the query encoder and the first momentum encoder through a loss function according to the first image feature and the corresponding second image features of each medical image number in the set of comparison medical image numbers in the first query queue;

[0020] Calculate the second loss value between the query encoder and the second momentum encoder through a loss function according to the first image feature and the corresponding third image features of each medical image number in the set of comparison medical image numbers in the third query queue;

[0021] If the first loss value is greater than the second loss value, update the parameters of the query encoder through gradient descent according to the first loss value, and update the parameters of the first momentum encoder by combining gradient descent and momentum update according to the updated parameters of the query encoder;

[0022] If the first loss value is less than the second loss value, update the parameters of the query encoder through gradient descent according to the second loss value, and update the parameters of the second momentum encoder by combining gradient descent and momentum update according to the updated parameters of the query encoder;

[0023] If the first loss value is equal to the second loss value, update the parameters of the query encoder through gradient descent according to the first loss value, and update the parameters of the first momentum encoder and the second momentum encoder by combining gradient descent and momentum update according to the updated parameters of the query encoder.

[0024] Further, the query encoder is a multi-modal model that can simultaneously process medical images and corresponding text descriptions, and generate a segmented image of the medical image through joint learning of image features and text features;

[0025] The query encoder includes a first encoder, a first feature fusion module, and a first decoder;

[0026] The first encoder includes an independent first image feature extraction module and a first text feature extraction module. The first image feature extraction module includes an image embedding layer and four consecutive Swin-Transformer layers connected in sequence. The image embedding layer includes a convolutional layer and a position encoding layer. The Swin-Transformer layer includes a fully connected layer, a window attention layer, and a residual layer connected in sequence. The first text feature extraction module includes a frozen BERT encoder layer. The first feature fusion module includes four consecutive conjugate cross-attention layers. The conjugate cross-attention layer includes a first cross-attention layer, a second cross-attention layer, a splicing layer, and a channel attention layer. The first decoder includes four consecutive Swin-Transformer layers;

[0027] The four consecutive Swin-Transformer layers of the first image feature extraction module are respectively connected to the four consecutive conjugate cross-attention layers of the first feature fusion module in a corresponding manner; the four consecutive conjugate cross-attention layers of the first feature fusion module are respectively connected to the four consecutive Swin-Transformer layers of the first decoder in a corresponding manner; the output of the fourth Swin-Transformer layer of the first image feature extraction module is connected to the input of the channel attention layer, and the output of the channel attention layer is connected to the fourth Swin-Transformer layer of the first decoder.

[0028] Further, the query encoder receives the medical image and the corresponding text data and generates the first image feature and the first text feature, specifically including:

[0029] Input the medical image into the image embedding layer of the query encoder. After being processed by the convolutional layer and the position encoding layer in the image embedding layer of the query encoder, it is input into four consecutive Swin-Transformer layers for processing. After being processed by the fully connected layer, window attention layer, and residual layer of the Swin-Transformer layer, the four Swin-Transformer layers respectively output the first-layer output image feature I 1 ∈N / 4×2C, the second-layer output image feature I 2 ∈N / 16×4C, the third-layer output image feature I 2 ∈N / 64×8C, the fourth-layer output image feature I 4 ∈N / 256×16C;

[0030] Input the corresponding text data into the BERT encoder layer of the query encoder for text feature extraction. Input the extracted feature T into the first conjugate cross-attention layer of the first feature fusion module of the query encoder. The first cross-attention layer of the first conjugate cross-attention layer uses the feature T output by the BERT encoder layer as the query Q, and uses the first-layer output image feature I 1 ∈N / 4×2C of the first decoder as the key K and value V for cross-attention calculation, and outputs the first-layer text-interested image feature M 1 , input the first-layer text-interested image feature M 1 into the second cross-attention layer of the first conjugate cross-attention layer. The second cross-attention layer of the first conjugate cross-attention layer uses the first-layer text-interested image feature M 1 as the query Q, and uses the first-layer output image feature I 1 ∈N / 4×2C of the first decoder as the key K and value V for cross-attention calculation, and outputs the first-layer fusion feature M 12 , input the first-layer fusion feature M 12The processed result is input into the subsequent conjugate cross-attention layer, and the fused features of the second layer M 22 , the fused features of the third layer M 32 , and the fused features of the fourth layer M 42 are output respectively. The fused features of the first layer M 12 , the fused features of the second layer M 22 , the fused features of the third layer M 32 , and the fused features of the fourth layer M 42 are respectively concatenated with the output image features of the first layer I 1 ∈ N / 4×2C, the output image features of the second layer I 2 ∈ N / 16×4C, the output image features of the third layer I 2 ∈ N / 64×8C, and the output image features of the fourth layer I 4 ∈ N / 256×16C, and then input into a 1×1 channel convolution to obtain the first final fused output feature M 13 , the second final fused output feature M 23 , the third final fused output feature M 33 , and the fourth final fused output feature M 43 respectively;

[0031] The fused features of the fourth layer M 42 are used as the first text feature T1 output by the query encoder;

[0032] The output image features of the fourth layer I 4 ∈ N / 256×16C are input into the channel attention layer for processing. The processed features are input into the fourth Swin-Transformer layer of the first decoder, and the first final fused output feature M 13 , the second final fused output feature M 23 , the third final fused output feature M 33 , and the fourth final fused output feature M 43 are respectively input into four consecutive Swin-Transformer layers of the corresponding first decoder for processing. The output features of the first Swin-Transformer layer of the first decoder are dimensionally reduced through a fully connected layer to output the first image feature I1.

[0033] Further, the first momentum encoder is a multi-modal model that can simultaneously process medical images and corresponding text descriptions, and generate a segmented image of the medical image through joint learning of image features and text features;

[0034] The first momentum encoder includes a second encoder, a second feature fusion module, and a second decoder. The structure of the first momentum encoder is similar to that of the query encoder, except that the Swin-Transformer layers of the second encoder of the first momentum encoder, the conjugate cross-attention layers of the second feature fusion module, and the Swin-Transformer layers of the second decoder are reduced from four layers to three layers;

[0035] The processing process of the first momentum encoder for medical images and corresponding text data is the same as that of the query encoder. The third-layer fusion feature M output by the third-layer conjugate cross-attention layer of the second feature fusion module 32 is used as the second text feature T2 output by the query encoder. The feature output by the first-layer Swin-Transformer layer of the second decoder is dimensionally reduced through a fully connected layer to output the second image feature I2.

[0036] Furthermore, the second momentum encoder is a single-modal model that can process medical images and generate a segmented image of the medical image through the image features;

[0037] The second momentum encoder includes a third encoder, a third decoder, and a channel attention layer. The third encoder is the same as the first encoder of the query encoder, and the third decoder is the same as the first decoder of the query encoder. The four consecutive Swin-Transformer layers of the third encoder of the second momentum encoder are respectively connected to the four consecutive Swin-Transformer layers of the third decoder of the second momentum encoder. The output of the fourth Swin-Transformer layer of the third encoder is connected to the input of the channel attention layer, and the output of the channel attention layer is connected to the fourth Swin-Transformer layer of the third decoder;

[0038] The second momentum encoder reduces the dimension of the feature output by the first-layer Swin-Transformer layer of the third decoder through a fully connected layer to output the third image feature I3.

[0039] Furthermore, updating the first query queue according to the generated second image feature and updating the second query queue according to the generated second text feature specifically include: enqueuing the generated second image feature into the first query queue, dequeuing the image feature at the tail of the first query queue, enqueuing the generated second text feature into the second query queue, and dequeuing the text feature at the tail of the second query queue;

[0040] Updating the third query queue according to the generated third image feature specifically includes: enqueuing the generated third image feature into the third query queue and dequeuing the image feature at the tail of the third query queue.

[0041] Further, the loss function is as follows:

[0042]

[0043] where L is the loss function, I1 is the first image feature output by the query encoder, and I j is the image feature output by the j-th momentum encoder, j = 1 or 2, N is the number of medical image numbers in the set of contrast medical image numbers, and I k is the corresponding image feature in the query queue of the k-th medical image number in the set of contrast medical image numbers. sim(I1, I j ) is the cosine similarity between the image feature I1 and the image feature I j , and τ is a hyperparameter.

[0044] Further, the parameter update formula of the query encoder is as follows:

[0045]

[0046] where is the parameter of the query encoder in the (t + 1)-th iterative training, is the parameter of the query encoder in the t-th iterative training, η is the learning rate, is the gradient of the loss function with respect to the parameters of the query encoder.

[0047] Further, the parameter update formulas of the first momentum encoder and the second momentum encoder are as follows:

[0048]

[0049] where is the parameter of the momentum encoder in the (T + 1)-th iterative training, is the parameter of the momentum encoder in the T-th iterative training, m is the momentum coefficient, is the parameter of the query encoder in the (T + 1)-th iterative training.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] (1) The present invention trains multiple models through a contrastive learning method. Especially in the field of medical images, unlabeled data is used for learning, solving the problem of scarce labeled data. This method effectively improves the learning efficiency and the generalization ability of the model during the training process through the joint processing of medical images and text descriptions.

[0052] (2) By using the combination of a query encoder, a first momentum encoder, and a second momentum encoder, the present invention can flexibly select a suitable model for image segmentation according to different situations of sufficient or scarce video memory resources and different time requirements, enabling efficient segmentation under different computing resources and time constraints, and having strong adaptability.

[0053] (3) The conjugate cross-attention layer and Swin-Transformer structure adopted by the present invention, combined with the fusion of image and text features, can effectively extract key information of medical images, reduce noise interference in traditional methods, and significantly improve the accuracy and stability of image segmentation.

[0054] (4) The present invention manages image features and text features through a queue mechanism and uses a contrastive learning training method based on MoCo, which significantly improves the model training efficiency and can quickly and accurately perform medical image segmentation on large-scale medical image datasets, with strong practicality.

[0055] (5) The multi-model method proposed by the present invention can not only handle multi-modal tasks that combine medical images and text data, but also handle image data alone, and is applicable to various medical image segmentation tasks, such as lung lesion segmentation, thoracic organ segmentation, etc., enhancing the application scenarios of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is the flowchart of the method of the present invention;

[0057] Figure 2 is the block diagram of MoCo-based text-guided multi-model training of the present invention;

[0058] Figure 3 is the model diagram of the query encoder of the present invention;

[0059] Figure 4 is the model diagram of the first momentum encoder of the present invention;

[0060] Figure 5 is the model diagram of the second momentum encoder of the present invention;

[0061] Figure 6 is the schematic diagram of the differences between medical image contrastive learning and natural image contrastive learning. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0063] Embodiment 1:

[0064] This embodiment provides a text-guided multi-model medical image segmentation method based on MoCo. Figure 1 As shown, the following steps are included:

[0065] constructing a query encoder, a first momentum encoder, and a second momentum encoder;

[0066] training the query encoder, the first momentum encoder, and the second momentum encoder by contrastive learning;

[0067] After all three models are trained, when there are medical images and corresponding text data, and the video memory resources are sufficient and the time requirements are loose, the medical images and the corresponding text data are input into the query encoder for image segmentation; when there are medical images and corresponding text data, and the video memory resources are tight or the time requirements are strict, the medical images and the corresponding text data are input into the first momentum encoder for image segmentation; when there are only medical images, the medical images are input into the second momentum encoder for image segmentation.

[0068] Furthermore, the query encoder, the first momentum encoder, and the second momentum encoder are trained by contrastive learning, such as Figure 2 As shown, specifically including:

[0069] Acquire a training data set, wherein the training data set includes a medical image and corresponding text data, wherein the text data is a natural language description corresponding to the medical image, including an anatomical structure, a lesion area, a disease type, a label of a segmented area, or other medical features in the image;

[0070] The training dataset includes medical images and their corresponding text descriptions, which provide detailed information about the anatomical structure, lesion area, disease type, etc. The image itself provides information about the shape and structure, while the text provides information at the semantic level. Through joint training, the query encoder can not only extract features from the image, but also fill in the gaps in image understanding through text features, making the model's understanding ability more comprehensive. This training method is particularly suitable for medical images, because the details in medical images can be very complex, and relying solely on image features is sometimes not enough for accurate segmentation and classification. Text descriptions can provide very important supplementary information.

[0071] Randomly initialize the query encoder Q1, and then use the parameters in the query encoder Q1 to initialize the parameters of the parts of the first momentum encoder and the second momentum encoder that have the same structure;

[0072] In the initialization stage, the parameters of the query encoder are randomly initialized, and then the parameters of the same parts in the first momentum encoder and the second momentum encoder are initialized with these parameters. This approach can ensure similar initial conditions among the three models and promote and optimize each other during the training process, thereby improving the stability and training efficiency of the models.

[0073] Receive medical images and corresponding text data through the query encoder and generate first image features and first text features;

[0074] Receive medical images and corresponding text data through the first momentum encoder and generate second image features and second text features. Update the first query queue according to the generated second image features and update the second query queue according to the generated second text features;

[0075] Receive medical images through the second momentum encoder and generate third image features. Update the third query queue according to the generated third image features;

[0076] The lengths of the first query queue, the second query queue, and the third query queue are the same and fixed. The first query queue is used to store the second image features generated by training the first momentum encoder each time and their corresponding medical image numbers. The second query queue is used to store the second text features generated by training the first momentum encoder each time and their corresponding medical image numbers. The third query queue is used to store the third image features generated by training the second momentum encoder each time and their corresponding medical image numbers;

[0077] After the query encoder generates image features and text features, the first momentum encoder and the second momentum encoder also generate corresponding image features and text features respectively. Through these features, update the query queue (the feature set of images and texts) so that the model can remember the features generated during the previous training process and continuously track the relationship between images and texts. This design helps the model gradually learn the relationship between images and texts and optimize according to historical feature information, thereby reducing the overfitting risk of the model when training on new samples.

[0078] Calculate the similarity between the first text feature and each text feature in the second query queue, and filter out the medical image numbers corresponding to the text features with a similarity less than the first threshold to obtain a set of comparison medical image numbers;

[0079] By calculating the similarity between text features and screening out the image numbers corresponding to the features with low similarity, the model can compare the correlations between different images and texts, thereby strengthening the model's understanding of the differences between features. The design of the loss function, based on the similarity between images and texts, ensures that the model pays more attention to the relative positions and semantic matches of images and texts during the learning process by calculating the contrast loss between the image features and the image features in the query queue.

[0080] The purpose of screening through text features is to reduce the computational amount and complexity during the model training process, while improving the learning efficiency and accuracy of the model. Specifically, the screening of text features plays the following key roles:

[0081] 1. Improve the matching degree of image and text features: Medical images usually contain a large amount of detailed information, and the understanding of this detailed information often requires text descriptions to provide context or semantic explanations. Text features contain information such as anatomical structures, lesion areas, and disease types in the images. By first screening using text features, it can help the model better understand which images are associated with specific text descriptions. This semantic-based screening can enhance the matching degree between images and texts, thereby avoiding mis-matching of irrelevant images and texts and improving the feature fusion effect of the model.

[0082] 2. Reduce the computational burden: Medical image datasets are usually very large, containing a large amount of image and text data. If all image features are comprehensively compared during each training, not only is the computational amount huge, but it may also cause the model to be difficult to converge. By first using text features to screen out the images similar to the target text description, it can significantly reduce the number of images that need to be calculated subsequently, thereby saving computational resources and improving the training efficiency.

[0083] 3. Optimize the quality of contrast learning: During the process of contrast learning, the model usually guides learning by calculating the similarity between different samples. By first using text features for screening, it can remove those images that are semantically irrelevant or have low similarity from the training set. This can ensure that only the images and texts with strong correlations are matched during the contrast learning process, so that the model can more intensively learn the true relationship between images and texts, avoid interference from too many irrelevant features, and improve the accuracy and effect of learning.

[0084] 4. Precise positioning and noise reduction: Much information in medical images is very important for specific tasks, while other information may be noise or irrelevant. For example, when performing medical image segmentation, the focus may be on the lesion area, and the image features of other parts may be less important. Text descriptions can usually precisely indicate the key areas of the image, and text screening can help the model pay more attention to those areas that are medically significant, thereby improving the accuracy of image segmentation.

[0085] 5. Enhance the robustness of the model: By pre-screening text features first, the model can be made more robust during training, especially in cases where there are significant differences between the image and text modalities, reducing the model's incorrect learning due to irrelevant information. This semantic-based pre-screening can effectively improve the model's adaptability to complex scenarios, making it perform more consistently under different tasks and data distributions.

[0086] Calculate the first loss value between the query encoder and the first momentum encoder through a loss function according to the first image feature and the corresponding second image features of each medical image number in the set of comparative medical image numbers in the first query queue;

[0087] Calculate the second loss value between the query encoder and the second momentum encoder through a loss function according to the first image feature and the corresponding third image features of each medical image number in the set of comparative medical image numbers in the third query queue;

[0088] If the first loss value is greater than the second loss value, update the parameters of the query encoder through gradient descent according to the first loss value, and update the parameters of the first momentum encoder by combining gradient descent and momentum update according to the updated parameters of the query encoder;

[0089] If the first loss value is less than the second loss value, update the parameters of the query encoder through gradient descent according to the second loss value, and update the parameters of the second momentum encoder by combining gradient descent and momentum update according to the updated parameters of the query encoder;

[0090] If the first loss value is equal to the second loss value, update the parameters of the query encoder through gradient descent according to the first loss value, and update the parameters of the first momentum encoder and the second momentum encoder by combining gradient descent and momentum update according to the updated parameters of the query encoder.

[0091] Through this contrastive learning, the query encoder, the first momentum encoder, and the second momentum encoder can be adaptively updated according to the loss value of each time. If the loss value of the query encoder is greater than that of the first momentum encoder, the parameters of the query encoder will be updated according to gradient descent, and at the same time, the parameters of the first momentum encoder will be adjusted; if the loss value of the second momentum encoder is smaller, the parameters of the query encoder and the second momentum encoder will be adjusted. This mechanism ensures that the model will adjust the parameters according to which model performs better in each iteration, thus achieving dynamic optimization.

[0092] If the two loss values are equal, it indicates that the relationship between the image and text features is equally important for the two momentum encoders, and the model will update the parameters of all encoders according to the loss values of both. This flexible update method enables the entire training process to make full use of the advantages of each model, ensuring that the three encoders can effectively improve the performance of the segmentation task under different conditions.

[0093] Furthermore, the query encoder is a multimodal model that can simultaneously process medical images and corresponding text descriptions, and generate a segmented image of the medical image through joint learning of image features and text features;

[0094] The query encoder is designed to be able to simultaneously process medical images and text descriptions, which is a typical method in multimodal learning. In medical image analysis, the image itself contains a large amount of visual information, while the text description can provide a deeper interpretation and context of the image content. By enabling the model to process these two types of data simultaneously, the model can understand the image content from multiple perspectives, which is crucial for medical image analysis because medical images often contain complex anatomical structures, lesion areas, etc., and single visual features may be difficult to comprehensively understand these contents.

[0095] As Figure 3 shown, the query encoder includes a first encoder, a first feature fusion module, and a first decoder;

[0096] The first encoder includes an independent first image feature extraction module and a first text feature extraction module. The first image feature extraction module includes an image embedding layer and four consecutive Swin-Transformer layers connected in sequence. The image embedding layer includes a convolutional layer and a position encoding layer. The Swin-Transformer layer includes a fully connected layer, a window attention layer, and a residual layer connected in sequence. The first text feature extraction module includes a frozen BERT encoder layer. The first feature fusion module includes four consecutive conjugate cross-attention layers. The conjugate cross-attention layer includes a first cross-attention layer, a second cross-attention layer, a splicing layer, and a channel attention layer. The first decoder includes four consecutive Swin-Transformer layers.

[0097] The four consecutive Swin-Transformer layers of the first image feature extraction module are respectively connected to the four consecutive conjugate cross-attention layers of the first feature fusion module in a corresponding manner. The four consecutive conjugate cross-attention layers of the first feature fusion module are respectively connected to the four consecutive Swin-Transformer layers of the first decoder in a corresponding manner. The output of the fourth Swin-Transformer layer of the first image feature extraction module is connected to the input of the channel attention layer, and the output of the channel attention layer is connected to the fourth Swin-Transformer layer of the first decoder.

[0098] Further, the query encoder receives a medical image and corresponding text data and generates a first image feature and a first text feature, which specifically includes:

[0099] Input the medical image into the image embedding layer of the query encoder. After being processed by the convolutional layer and the position encoding layer of the image embedding layer of the query encoder, it is input into four consecutive Swin-Transformer layers for processing. After being processed by the fully connected layer, the window attention layer, and the residual layer of the Swin-Transformer layer, the four Swin-Transformer layers respectively output the first-layer output image feature I 1 ∈N / 4×2C, the second-layer output image feature I 2 ∈N / 16×4C, the third-layer output image feature I 2 ∈N / 64×8C, the fourth-layer output image feature I 4 ∈N / 256×16C;

[0100] The corresponding text data is input into the BERT encoder layer of the query encoder for text feature extraction. The extracted feature T is input into the first layer conjugate cross-attention layer of the first feature fusion module of the query encoder. The first cross-attention layer of the first layer conjugate cross-attention layer uses the feature T output by the BERT encoder layer as the query Q, and uses the first layer output image feature I of the first decoder 1 ∈N / 4×2C as the key K and value V for cross-attention calculation, and outputs the first layer text-interested image feature M 1 The first layer text-interested image feature M 1 is input into the second cross-attention layer of the first layer conjugate cross-attention layer. The second cross-attention layer of the first layer conjugate cross-attention layer uses the first layer text-interested image feature M 1 as the query Q, and uses the first layer output image feature I of the first decoder 1 ∈N / 4×2C as the key K and value V for cross-attention calculation, and outputs the first layer fusion feature M 12 The first layer fusion feature M 12 is input into the subsequent conjugate cross-attention layers for processing, and outputs the second layer fusion feature M 22 , the third layer fusion feature M 32 , the fourth layer fusion feature M 42 The first layer fusion feature M 12 , the second layer fusion feature M 22 , the third layer fusion feature M 32 , the fourth layer fusion feature M 42 are respectively concatenated with the first layer output image feature I 1 ∈N / 4×2C, the second layer output image feature I 2 ∈N / 16×4C, the third layer output image feature I 2 ∈N / 64×8C, the fourth layer output image feature I 4 ∈N / 256×16C, and then input into a 1×1 channel convolution to obtain the first final fusion output feature M 13 , the second final fusion output feature M 23 , the third final fusion output feature M 33 , the fourth final fusion output feature M 43 ;

[0101] The fourth layer fusion feature M 42 is used as the first text feature T1 output by the query encoder;

[0102] The fourth layer output image feature I 4Processed by the ∈N / 256×16C input channel attention layer, and input the processed features into the fourth Swin-Transformer layer of the first decoder, and output the first final fusion output feature M 13 and the second final fusion output feature M 23 and the third final fusion output feature M 33 and the fourth final fusion output feature M 43 They are respectively input into four consecutive Swin-Transformer layers of the corresponding first decoder for processing. The output features of the first Swin-Transformer layer of the first decoder are reduced in dimension through a fully connected layer to output the first image feature I1.

[0103] In the structural design of the query encoder, the first encoder processes medical images and text data through independent image feature extraction modules and text feature extraction modules. The image feature extraction module uses an image embedding layer and four consecutive Swin-Transformer layers to extract image features. Swin-Transformer performs well in computer vision tasks because it can effectively capture the detailed features of images through local window attention mechanisms. The BERT encoder layer is used to extract text features. As a pre-trained language model, BERT can handle natural language texts well, especially when dealing with text descriptions in the medical field, it can capture the details in medical language better.

[0104] By extracting image features and text features, the first feature fusion module further fuses these two types of features through multiple conjugate cross-attention layers. The role of the cross-attention layer is to calculate the similarity between image features and text features, and weight and fuse image and text features based on this similarity. In this way, when generating medical image segmentation images, the model can take into account both the visual information of the image and the semantic information in the text description, thereby improving the accuracy and robustness of the model. The design of the conjugate cross-attention layer can ensure the effective flow of information between images and texts, and further strengthen their correlation through layer-by-layer fusion.

[0105] In addition, the introduction of the channel attention layer enables the model to dynamically adjust the weights of features, thereby focusing on the regions most important for the segmentation task. Medical image segmentation tasks usually require accurate calibration of lesion regions or anatomical structures. The channel attention layer enhances the model's attention to key regions by automatically selecting the features most important for the task.

[0106] During the decoding process of the query encoder, the image and text features after feature fusion are input into the decoder and processed through four consecutive Swin-Transformer layers to gradually recover the features of the segmented image. Through this decoding structure, the model can not only better understand the visual information of the image and the semantic information of the text, but also generate accurate medical image segmentation results based on the fused information.

[0107] Further, as Figure 4 shown, the first momentum encoder is a multi-modal model that can simultaneously process medical images and corresponding text descriptions, and generate a segmented image of the medical image through the joint learning of image features and text features;

[0108] The first momentum encoder includes a second encoder, a second feature fusion module, and a second decoder. The structure of the first momentum encoder is similar to that of the query encoder, except that the number of Swin-Transformer layers in the second encoder of the first momentum encoder, the conjugate cross-attention layer in the second feature fusion module, and the Swin-Transformer layer in the second decoder are reduced from four layers to three layers;

[0109] The processing process of the first momentum encoder for medical images and corresponding text data is the same as that of the query encoder. The third-layer fusion feature M output by the third conjugate cross-attention layer of the second feature fusion module 32 is used as the second text feature T2 output by the query encoder. The feature output by the first Swin-Transformer layer of the second decoder is dimension-reduced through a fully connected layer to output the second image feature I2.

[0110] The purpose of this step is to build a multi-modal model (the first momentum encoder) similar to the query encoder, but make some modifications to its structure to adapt to different training requirements and resource limitations. Specifically, the first momentum encoder generates a segmented image of the medical image through the joint learning of the features of the medical image and the text description. The design and modification intention of this model is to improve the efficiency, processing speed, and adaptability of the model in different environments.

[0111] First of all, the structures of the first momentum encoder and the query encoder are very similar. Both are multi-modal models that can process medical images and corresponding text data. The main difference between the two is that the first momentum encoder has made some simplifications in its structure, especially in reducing the number of Swin-Transformer layers from four layers to three layers. The purpose of this adjustment is to reduce the amount of computation, especially in cases where the video memory requirement is high or the computing resources are limited, to provide higher efficiency.

[0112] The design of reducing the number of Swin-Transformer layers is to optimize the use of computing resources during training. Swin-Transformer is an architecture based on the self-attention mechanism. Although it is very powerful in processing image features, adding each layer will significantly increase the computational complexity and video memory consumption. Under certain resource constraints, by reducing the number of layers, a relatively efficient feature extraction ability can be maintained while reducing computational and memory overheads. This enables the first momentum encoder to still maintain good performance under relatively tight computing resources.

[0113] When processing medical images and text data, the processing process of the first momentum encoder is the same as that of the query encoder. It still processes image and text data through the image feature extraction module and the text feature extraction module respectively, and fuses the features of these two modalities. Through the conjugate cross-attention layer in the second feature fusion module, the image and text features are weighted and fused to generate a more informative and semantically related feature representation.

[0114] After the fusion process is completed, the first momentum encoder will output a text feature and an image feature. The text feature is obtained after being processed by the third layer of conjugate cross-attention layer, while the image feature is output after being further processed by the Swin-Transformer layer of the second decoder. Such a design ensures the in-depth fusion of text features and image features, and uses the fused information to generate a segmented image of the medical image. By reducing the number of Swin-Transformer layers, the computational and video memory burdens are alleviated, enabling the model to run on resource-constrained devices and increasing its practical application flexibility. Through the simplification of the structure, the model can be used in different computing environments, such as for inference and training on hardware with less video memory or limited computing resources. In addition, the similarity between the first momentum encoder and the query encoder also facilitates the transfer learning of the model, and the pre-trained weights of the query encoder can be easily transferred to the first momentum encoder to accelerate the training process.

[0115] Furthermore, as Figure 5 shown, the second momentum encoder is a single-modal model that can process medical images and generate a segmented image of the medical image through image features;

[0116] The second momentum encoder includes a third encoder, a third decoder, and a channel attention layer. The third encoder is the same as the first encoder of the query encoder, and the third decoder is the same as the first decoder of the query encoder. Four consecutive Swin-Transformer layers of the third encoder of the second momentum encoder are respectively connected to four consecutive Swin-Transformer layers of the third decoder of the second momentum encoder. The output of the fourth Swin-Transformer layer of the third encoder is connected to the input of the channel attention layer, and the output of the channel attention layer is connected to the fourth Swin-Transformer layer of the third decoder;

[0117] The second momentum encoder reduces the dimension of the output features of the first Swin-Transformer layer of the third decoder through a fully connected layer and outputs the third image feature I3.

[0118] The second momentum encoder is a single-modal model that focuses on processing medical images and generates segmentation images of medical images by extracting image features. Different from the multi-modal design of the first momentum encoder and the query encoder, the second momentum encoder only processes image data. The purpose of this design is to provide an efficient solution that focuses on single-modal tasks under resource constraints.

[0119] Structurally, the second momentum encoder inherits the design of the image feature extraction and decoding parts in the query encoder, but its complexity is simplified. Specifically, the third encoder of the second momentum encoder is the same as the first encoder of the query encoder, and the third decoder is the same as the first decoder of the query encoder. This means that the second momentum encoder utilizes the trained image processing capabilities in the query encoder but no longer needs to process text data, thus simplifying the model's structure and computational requirements.

[0120] During the processing, the second momentum encoder extracts image features through four consecutive Swin-Transformer layers, and the design of these layers is the same as the image feature extraction part in the query encoder. Swin-Transformer is a self-attention architecture that can efficiently capture local and global information and is suitable for processing complex image data. The output features of the four Swin-Transformer layers are gradually refined to ensure the step-by-step extraction of multi-level information of the image from low level to high level.

[0121] It is worth noting that the output of the fourth layer of the Swin-Transformer of the third encoder is connected to the channel attention layer. This design utilizes the channel attention mechanism to further strengthen the feature representation. The channel attention layer can adjust its weights according to the importance of different channels in the image, thereby enhancing the model's attention to key features and optimizing the image segmentation effect. The channel attention mechanism helps improve the model's response to different feature regions, thus enhancing the ability to understand and process image details.

[0122] The third decoder of the second momentum encoder reduces the dimension of the output features of its first Swin-Transformer layer through a fully connected layer and finally outputs the third image feature I_3. The dimension reduction process can compress high-dimensional feature vectors into lower-dimensional vectors, which not only reduces the computational burden but also maintains the effectiveness of image features. Through this process, the model converts high-level image features into more compact and effective representations, which is crucial for subsequent medical image segmentation tasks. Compared with the query encoder and the first momentum encoder, the second momentum encoder simplifies the model structure, omits the part for processing text data, and reduces the demand for computing resources. In the case of only image data, the second momentum encoder can be trained and inferred quickly and efficiently. The second momentum encoder only processes image data, avoiding the complexity of multimodal fusion. The model is more focused on the extraction and segmentation of image features and is suitable for tasks that mainly rely on medical images for analysis.

[0123] Furthermore, updating the first query queue according to the generated second image feature and updating the second query queue according to the generated second text feature specifically include: enqueueing the generated second image feature into the first query queue, dequeueing the image feature at the tail of the first query queue, enqueueing the generated second text feature into the second query queue, and dequeueing the text feature at the tail of the second query queue;

[0124] Updating the third query queue according to the generated third image feature specifically includes: enqueueing the generated third image feature into the third query queue and dequeueing the image feature at the tail of the third query queue.

[0125] By updating the features in the query queue, the model can always use the latest and most relevant features for learning while maintaining the length of the queue and the dynamic update of the data. The design idea of this approach is to ensure that the model can efficiently and flexibly handle new data during the training process and avoid the negative impact of outdated data on the model performance.

[0126] Furthermore, the loss function is:

[0127]

[0128] Among them, L is the loss function, I1 is the first image feature output by the query encoder, and I j is the image feature output by the j-th momentum encoder, where j = 1 or 2, N is the number of medical image numbers in the set of contrast medical image numbers, and I k is the corresponding image feature in the query queue of the k-th medical image number in the set of contrast medical image numbers. sim(I1, I j ) is the cosine similarity between the image feature I1 and the image feature I j , and τ is a hyperparameter.

[0129] The form of the loss function is actually a "soft" contrast loss function, where the denominator is the sum of all image feature similarities, ensuring that when optimizing, not only the similarity between the query image and a certain image feature is considered, but also the influence of all contrast images is considered. In this way, the model can effectively learn the ability to distinguish different image features during training, rather than just matching a specific image feature.

[0130] Furthermore, the parameter update formula of the query encoder is:

[0131]

[0132] Where is the parameter of the query encoder in the (t + 1)-th iterative training, is the parameter of the query encoder in the t-th iterative training, η is the learning rate, is the gradient of the loss function with respect to the parameters of the query encoder.

[0133] Furthermore, the parameter update formulas of the first momentum encoder and the second momentum encoder are:

[0134]

[0135] Where is the parameter of the momentum encoder in the (t + 1)-th iterative training, is the parameter of the momentum encoder in the t-th iterative training, m is the momentum coefficient, is the parameter of the query encoder in the (t + 1)-th iterative training.

[0136] The first momentum encoder and the second momentum encoder adopt a momentum update method combined with the parameters of the query encoder. The momentum update method is an extension of the gradient descent method, aiming to accelerate convergence by introducing "momentum", avoid local minima, and smooth the oscillations during training.

[0137] Example 2:

[0138] The parts not mentioned in this embodiment are the same as those in Embodiment 1.

[0139] This embodiment provides a text-guided multi-model training method based on MoCo. By using an improved MoCo contrastive learning method, two momentum encoders are set up. Combining backpropagation and momentum update, a loss function is generated using text similarity to reduce the excessive contrast between medical image segmentations. The method includes the following steps:

[0140] S1. Preprocess the medical natural language processing dataset;

[0141] S2. The text data corresponding to the pictures is passed through a tokenizer and a BERT encoder to obtain word vectors, and the word vectors are completed according to the statistical data in the dataset;

[0142] S3. Input the medical image and the corresponding text into the query encoder and the momentum encoder for three forward propagations to obtain three image features. Calculate the similarity between the features extracted by the text encoder and the features in the text feature queue to generate a mask;

[0143] S4. Based on the mask obtained in step S3, generate two sets of loss functions loss_1 and loss_2; compare the values of the two sets of loss functions, and backpropagate to update the parameters of the query encoder corresponding to the set with the larger loss function. Momentum update the parameters of the part of the momentum encoder corresponding to the structure of the query encoder and the unique part of the momentum encoder updated by backpropagation, which is generally a fully connected layer;

[0144] The image processing module of the query encoder consists of Swin-Transformer blocks, including sliding windows, serial multi-head self-attention (Multi-Head Self-Attention), feed-forward neural network (Feed-Forward Neural Network), etc. Take the image features as the input of the Swin-Transformer block, and use layer normalization at the input end of each layer of the Swin-Transformer to stabilize the training process.

[0145] The text processing module of the query encoder and the feature fusion model are composed of a conjugate cross-attention module stacked by cross-attention, 1x1 channel convolution, and an image feature splicing layer.

[0146] The structure of momentum encoder 1 is similar to that of the query encoder. The only difference lies in the depth of the network, which decreases from 4 layers of Swin-Transformer blocks to 3 layers of Swin-Transformer blocks.

[0147] The structure of momentum encoder 2 is the structure of the query encoder removing the text processing module and the feature fusion module, and can only process single-modal inputs.

[0148] Input an image into the above three models, then compare the output results of the two momentum encoder models with the output of the query encoder we trained. Backpropagate and update the momentum encoder for the model with a worse loss function value. After the model is trained for a certain number of rounds, the three models will show similar effects. Use the first model when memory resources are abundant and the inference time requirement is not strict; use the second model when memory resources are scarce and the inference time requirement is high; use the third model when there is no multi-modal data input. The training method proposed in this patent can obtain three models with similar effects through one training, and select different models for deployment according to different requirements.

[0149] The following takes an actual application as an example for illustration, including the following steps:

[0150] Step1, preprocess the medical image and the corresponding text dataset;

[0151] Step2, input the medical image and the corresponding text into the query encoder network and the momentum encoder network;

[0152] Step3, divide the image and text dataset according to a preset ratio, which is randomly divided into 7:1:2 for training, validation, and testing respectively;

[0153] Step4, calculate the similarity between positive and negative samples of image features, calculate the similarity between positive and negative sample text features, generate a mask according to the similarity, and calculate the loss function between the three groups of features according to the similarity and the mask;

[0154] Step5, select the group of momentum encoders with a larger loss value, backpropagate the query encoder, update the same parameters of the momentum encoder and the query encoder by momentum, and backpropagate the last MLP layer of the momentum encoder

[0155] The parameters of the model are updated using the Adam optimization strategy, and an early stopping strategy is used, with the threshold set at 50 rounds. For the batch size of one training, due to the limitation of video memory, it is set to 40 for all.

[0156] Step6, perform supervised fine-tuning on the model trained by contrastive learning

[0157] Step6, input the test set of the dataset into the trained image segmentation model to obtain the segmentation result.

[0158] Among them, the segmentation result is evaluated by the Dice similarity coefficient (DSC) and the average intersection over union.

[0159] After multiple rounds of experiments, we compared the results of MoCo training with the models of our method. As shown in Tables 1 and 2, experiments were conducted on the QaTa-Covid19+ dataset and the SIIM Pneumothorax dataset respectively.

[0160] Table 1 Recognition accuracy of three models for positive samples after training with the MoCo framework

[0161]

[0162] Table 2 Recognition accuracy of three models for positive samples in the multi-model contrastive learning training method

[0163]

[0164]

[0165] We fine-tuned the model pre-trained in the contrastive learning stage to obtain the following experimental results. Compared with supervised learning, the model trained with the MoCo unsupervised learning framework even reduces the training effectiveness, making the model less effective than when all supervised learning is performed. However, in simple-structured single-modal models, the segmentation effect is improved after MoCo training, obtaining results better than those of multi-modal models. Since we introduced text information as a guide in the contrastive learning method and filtered out similar medical images through text, the excessive contrast between medical images was reduced, as Figure 6 shown.

[0166] Table 3 Performance of the models trained with the multi-model simultaneous training method and the MoCo training framework in medical image segmentation tasks

[0167]

[0168] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0169] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A text-guided multi-model medical image segmentation method based on MoCo, characterized in that It includes the following steps: Construct a query encoder, a first momentum encoder, and a second momentum encoder; Train the query encoder, the first momentum encoder, and the second momentum encoder through contrastive learning; After all three models are trained, when there are medical images and corresponding text data, and the video memory resources are sufficient and the time requirement is loose, input the medical images and corresponding text data into the query encoder for image segmentation; when there are medical images and corresponding text data, and the video memory resources are tight or the time requirement is strict, input the medical images and corresponding text data into the first momentum encoder for image segmentation; when there is only a medical image, input the medical image into the second momentum encoder for image segmentation.

2. A text-guided multi-model medical image segmentation method based on MoCo according to claim 1, characterized in that, The training of the query encoder, the first momentum encoder, and the second momentum encoder through contrastive learning specifically includes: Obtain a training data set, where the training data set includes medical images and corresponding text data, and the text data is the natural language description corresponding to the medical images, including anatomical structures, lesion regions, disease types, labels of segmented regions, or other medical features in the images; Randomly initialize the query encoder Q1, and then use the parameters in the query encoder Q1 to initialize the parameters of the parts with the same structure in the first momentum encoder and the second momentum encoder; Receive the medical image and corresponding text data through the query encoder and generate a first image feature and a first text feature; Receive the medical image and corresponding text data through the first momentum encoder and generate a second image feature and a second text feature. Update the first query queue according to the generated second image feature, and update the second query queue according to the generated second text feature; Receive the medical image through the second momentum encoder and generate a third image feature, and update the third query queue according to the generated third image feature; The lengths of the first query queue, the second query queue, and the third query queue are the same and fixed. The first query queue is used to store the second image features generated by training the first momentum encoder each time and their corresponding medical image numbers. The second query queue is used to store the second text features generated by training the first momentum encoder each time and their corresponding medical image numbers. The third query queue is used to store the third image features generated by training the second momentum encoder each time and their corresponding medical image numbers; Calculate the similarity between the first text feature and each text feature in the second query queue, and screen out the medical image numbers corresponding to the text features with a similarity less than the first threshold to obtain a set of contrastive medical image numbers; Calculate the first loss value between the query encoder and the first momentum encoder through a loss function according to the first image feature and the corresponding second image features of each medical image number in the set of contrastive medical image numbers in the first query queue; Calculate the second loss value between the query encoder and the second momentum encoder through a loss function according to the first image feature and the corresponding third image features of each medical image number in the set of contrastive medical image numbers in the third query queue; If the first loss value is greater than the second loss value, update the parameters of the query encoder by gradient descent according to the first loss value, and update the parameters of the first momentum encoder by a method combining gradient descent and momentum update according to the parameters of the updated query encoder; If the first loss value is less than the second loss value, update the parameters of the query encoder by gradient descent according to the second loss value, and update the parameters of the second momentum encoder by a method combining gradient descent and momentum update according to the parameters of the updated query encoder; If the first loss value is equal to the second loss value, update the parameters of the query encoder by gradient descent according to the first loss value, and update the parameters of the first momentum encoder and the second momentum encoder by a method combining gradient descent and momentum update according to the parameters of the updated query encoder.

3. A text-guided multi-model medical image segmentation method based on MoCo according to claim 1, characterized in that, The query encoder is a multi-modal model that can simultaneously process medical images and corresponding text descriptions, and generate a segmented image of the medical image through joint learning of image features and text features; The query encoder includes a first encoder, a first feature fusion module, and a first decoder; The first encoder includes an independent first image feature extraction module and a first text feature extraction module. The first image feature extraction module includes an image embedding layer and four consecutive Swin-Transformer layers connected in sequence. The image embedding layer includes a convolutional layer and a position encoding layer. The Swin-Transformer layer includes a fully connected layer, a window attention layer, and a residual layer connected in sequence. The first text feature extraction module includes a frozen BERT encoder layer. The first feature fusion module includes four consecutive conjugate cross-attention layers. The conjugate cross-attention layer includes a first cross-attention layer, a second cross-attention layer, a splicing layer, and a channel attention layer. The first decoder includes four consecutive Swin-Transformer layers; The four consecutive Swin-Transformer layers of the first image feature extraction module are respectively connected to the four consecutive conjugate cross-attention layers of the first feature fusion module in correspondence; the four consecutive conjugate cross-attention layers of the first feature fusion module are respectively connected to the four consecutive Swin-Transformer layers of the first decoder in correspondence; the output of the fourth Swin-Transformer layer of the first image feature extraction module is connected to the input of the channel attention layer, and the output of the channel attention layer is connected to the fourth Swin-Transformer layer of the first decoder.

4. A text-guided multi-model medical image segmentation method based on MoCo according to claim 2 or 3, characterized in that, Receiving the medical image and the corresponding text data by the query encoder and generating the first image feature and the first text feature specifically includes: Input the medical image into the image embedding layer of the query encoder. After being processed by the convolutional layer and the position encoding layer, the image embedding layer of the query encoder is input into four consecutive Swin-Transformer layers for processing. After being processed by the fully connected layer, the window attention layer and the residual layer of the Swin-Transformer layer, the four Swin-Transformer layers respectively output the first-layer output image feature I 1 ∈ N / 4×2C, the second-layer output image feature I 2 ∈ N / 16×4C, the third-layer output image feature I 2 ∈ N / 64×8C, the fourth-layer output image feature I 4 ∈ N / 256×16C; The corresponding text data is input into the BERT encoder layer of the query encoder for text feature extraction. The extracted feature T is input into the first layer of the conjugate cross-attention layer of the first feature fusion module of the query encoder. The first cross-attention layer of the first layer of the conjugate cross-attention layer uses the feature T output by the BERT encoder layer as the query Q, and uses the first-layer output image feature I of the first decoder 1 ∈N / 4×2C as the key K and value V for cross-attention calculation, and outputs the first-layer text-interested image feature M 1 The first-layer text-interested image feature M 1 is input into the second cross-attention layer of the first layer of the conjugate cross-attention layer. The second cross-attention layer of the first layer of the conjugate cross-attention layer uses the first-layer text-interested image feature M 1 as the query Q, and uses the first-layer output image feature I of the first decoder 1 ∈N / 4×2C as the key K and value V for cross-attention calculation, and outputs the first-layer fusion feature M 12 The first-layer fusion feature M 12 is input into the subsequent conjugate cross-attention layers for processing, and the second-layer fusion feature M 22 , the third-layer fusion feature M 32 , and the fourth-layer fusion feature M 42 are output respectively. The first-layer fusion feature M 12 , the second-layer fusion feature M 22 , the third-layer fusion feature M 32 , and the fourth-layer fusion feature M 42 are respectively concatenated with the first-layer output image feature I 1 ∈N / 4×2C, the second-layer output image feature I 2 ∈N / 16×4C, the third-layer output image feature I 2 ∈N / 64×8C, and the fourth-layer output image feature I 4 ∑N / 256×16C, and then input into a 1×1 channel convolution to obtain the first final fusion output feature M 13 , the second final fusion output feature M 23 , the third final fusion output feature M 33 , and the fourth final fusion output feature M 43 ; Take the fourth-layer fused feature M 42 as the first text feature T1 output by the query encoder; Input the fourth-layer output image feature I 4 into the input channel attention layer of N / 256×16C, input the processed feature into the fourth Swin-Transformer layer of the first decoder, and input the first final fusion output feature M 13 , the second final fusion output feature M 23 , the third final fusion output feature M 33 , and the fourth final fusion output feature M 43 into the four consecutive Swin-Transformer layers of the corresponding first decoder for processing respectively. Dimension reduction of the feature output by the first Swin-Transformer layer of the first decoder is performed through a fully connected layer, and the first image feature I1 is output.

5. A text-guided multi-model medical image segmentation method based on MoCo according to claim 2, characterized in that, The first momentum encoder is a multi-modal model that can simultaneously process medical images and corresponding text descriptions, and generate a segmented image of the medical image through joint learning of image features and text features; The first momentum encoder includes a second encoder, a second feature fusion module, and a second decoder. The structure of the first momentum encoder is similar to that of the query encoder, except that the Swin-Transformer layer of the second encoder, the conjugate cross-attention layer of the second feature fusion module, and the Swin-Transformer layer of the second decoder of the first momentum encoder are reduced from four layers to three layers; The processing process of the first momentum encoder for medical images and corresponding text data is the same as that of the query encoder, and the third-layer fusion feature M output by the third-layer conjugate cross-attention layer of the second feature fusion module 32 is used as the second text feature T2 output by the query encoder. The feature output by the first-layer Swin-Transformer layer of the second decoder is dimensionally reduced through a fully connected layer to output the second image feature I2.

6. A text-guided multi-model medical image segmentation method based on MoCo according to claim 2, characterized in that, The second momentum encoder is a single-modal model that can process medical images and generate a segmented image of the medical image through image features; The second momentum encoder includes a third encoder, a third decoder, and a channel attention layer. The third encoder is the same as the first encoder of the query encoder, and the third decoder is the same as the first decoder of the query encoder. Four consecutive Swin-Transformer layers of the third encoder of the second momentum encoder are respectively connected to four consecutive Swin-Transformer layers of the third decoder of the second momentum encoder. The output of the fourth Swin-Transformer layer of the third encoder is connected to the input of the channel attention layer, and the output of the channel attention layer is connected to the fourth Swin-Transformer layer of the third decoder; The second momentum encoder reduces the dimension of the features of the output features of the first Swin-Transformer layer of the third decoder through a fully connected layer and outputs the third image feature I3.

7. A text-guided multi-model medical image segmentation method based on MoCo according to claim 2, wherein Updating the first query queue according to the generated second image feature and updating the second query queue according to the generated second text feature specifically include: enqueueing the generated second image feature into the first query queue, dequeueing the image feature at the end of the first query queue, enqueueing the generated second text feature into the second query queue, and dequeueing the text feature at the end of the second query queue; Updating the third query queue according to the generated third image feature specifically includes: enqueueing the generated third image feature into the third query queue and dequeueing the image feature at the end of the third query queue.

8. A text-guided multi-model medical image segmentation method based on MoCo according to claim 2, wherein The loss function is: where, L is the loss function, I1 is the first image feature output by the query encoder, and I j is the image feature output by the j-th momentum encoder, j = 1 or 2, N is the number of medical image numbers in the set of contrast medical image numbers, and I k is the corresponding image feature in the query queue of the k-th medical image number in the set of contrast medical image numbers, and sim(I1, I j ) is the cosine similarity between the image feature I1 and the image feature I j , and τ is a hyperparameter.

9. A text-guided multi-model medical image segmentation method based on MoCo according to claim 2, wherein, The parameter update formula of the query encoder is: Among them, is the parameter of the query encoder in the (t + 1)-th iteration training, is the parameter of the query encoder in the t-th iteration training, and η is the learning rate. is the gradient of the loss function with respect to the parameters of the query encoder.

10. A text-guided multi-model medical image segmentation method based on MoCo according to claim 2, characterized in that, The parameter update formula of the first momentum encoder and the second momentum encoder is: Among them, is the parameter of the momentum encoder in the (t + 1)-th iteration training, is the parameter of the momentum encoder in the t-th iteration training, and m is the momentum coefficient. is the parameter of the query encoder in the (t + 1)-th iteration training.

Images