Medical image segmentation method and device based on text driving and affinity learning

Abstract

The invention discloses a medical image segmentation method and device based on text driving and affinity learning, and relates to the field of weak supervised learning, and the method comprises the steps: constructing and training a medical image segmentation model based on text driving, and obtaining a trained medical image segmentation model; acquiring a medical image to be segmented and corresponding professional knowledge and text labels, inputting the medical image and the corresponding professional knowledge and text labels into the trained medical image segmentation model, encoding the text labels and the professional knowledge by using a label encoder and a professional knowledge encoder respectively, and guiding image features to pay attention to related regions of target tissues through a knowledge attention module. And further generating an initial pseudo tag, calculating through an affinity prediction module to obtain an affinity prediction value, and correcting the initial pseudo tag by using the affinity prediction value to obtain a pseudo tag corresponding to the medical image to be segmented. The problems that CAM activation of an existing weak supervision method is insufficient, and an image label cannot provide enough supervision information are solved.

Description

Technical Field

[0001] The present invention relates to the field of weakly supervised learning, and in particular to a medical image segmentation method and device based on text-driven and affinity learning. Background Art

[0002] In the field of computer-aided diagnosis (CAD), the automatic segmentation technology of histopathological images plays a vital role. It can help doctors quickly locate abnormal tissue areas and conduct quantitative analysis of tumor microenvironment, thereby providing important basis for cancer grading and prognosis, and significantly improving the diagnostic efficiency of clinicians. However, the fully supervised method is highly dependent on a large amount of high-quality labeled data. Especially when professional medical knowledge is required, it is a time-consuming and labor-intensive task for doctors to manually perform pixel-level fine annotation of histopathological images. Therefore, it is urgent to develop an automated tool based on weak supervision methods to segment and track regions of interest (ROI) in histopathological images. However, it is worth noting that factors such as the morphological homogeneity, obvious overlap, and low color contrast of histopathological images lead to differences between the same categories and similarities between different categories, which brings great challenges to the segmentation technology based on weak supervision. In addition, the tissue structure in histopathological images may be randomly arranged and scattered, which makes the identification of complete tissues and regions of interest extremely difficult.

[0003] The current mainstream method based on weakly supervised segmentation is to use class activation mapping (CAM) to locate the attention area, and then generate pseudo labels to train the segmentation network. However, this method has a significant problem: the CAM generated using image-level labels can only highlight the most discriminative areas, but cannot locate the complete object, resulting in defective pseudo labels. Therefore, many studies have tried different methods to improve the quality of CAM, thereby improving the performance of weakly supervised semantic segmentation. Despite this, these improved CAM variants still face challenges in capturing complete tissues. The main problem is that in medical images, the symptoms and manifestations of subtypes are difficult to fully describe with abstract semantic categories, and the supervision information that relies solely on image-level labels is often insufficient to accurately locate the complete target area.

[0004] Recently, the contrastive language image pre-training model (CLIP) has achieved remarkable success. It is pre-trained on 400 million image-text pairs from the Internet to predict whether an image and a text fragment match, thereby demonstrating strong capabilities in zero-shot classification. This dataset-agnostic model has the flexibility of direct transfer. More importantly, the powerful text-to-image generation capability demonstrated by CLIP, such as DALL-E2, reveals the deep connection between the corresponding elements in the text and the image. Given these advantages, researchers have tried to introduce CLIP in weakly supervised natural image segmentation. Xie et al. successfully activated more complete object regions and suppressed the relevant interference with open background regions by introducing natural language supervision. They proposed a novel cross-lingual image matching (CLIMS) framework for weakly supervised semantic segmentation. Lin et al. re-examined the text input in the weakly supervised semantic segmentation setting and carefully designed two text-driven strategies: clarity-based prompt selection and synonym fusion. However, the application of CLIP in the field of weakly supervised histopathology image segmentation is very scarce. On the other hand, the Multi-Head Self-Attention (MHSA) layer in Transformer can capture the semantic affinity between patches, which can be used to improve the rough pseudo-labels. However, the affinity captured in MHSA is still inaccurate, and directly applying MHSA predicted affinity to modify labels does not work well in practice. Summary of the invention

[0005] The purpose of this application is to propose a medical image segmentation method and device based on text-driven and affinity learning to address the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a medical image segmentation method based on text-driven and affinity learning, comprising the following steps:

[0007] A text-driven medical image segmentation model is constructed and trained to obtain a trained medical image segmentation model. The trained medical image segmentation model includes an image visual encoder, a label encoder, an expertise encoder, a knowledge attention module, and an affinity prediction module. The image visual encoder includes four stages of Transformer blocks connected in sequence. A fully connected layer is connected behind the expertise encoder and the label encoder.

[0008] The medical image to be segmented and its corresponding professional knowledge and text labels are obtained and input into the trained medical image segmentation model. The medical image to be segmented passes through the image visual encoder to obtain the hierarchical feature maps output by the Transformer blocks of the four stages respectively. The professional knowledge passes through the professional knowledge encoder to obtain the knowledge feature. The hierarchical feature map of the fourth stage and the knowledge feature after passing through the fully connected layer pass through the knowledge attention module to obtain the first image feature. The text label passes through the label encoder and the fully connected layer to obtain the label feature. The hierarchical feature map of the third stage and the hierarchical feature map of the fourth stage are added to obtain the second image feature. The similarity between the first image feature and the second image feature and the label feature is calculated respectively to obtain the first similarity feature map and the second similarity feature map. The first similarity feature map and the second similarity feature map are added, and then the argmax operation is performed to obtain the initial pseudo label. The hierarchical feature map of the first stage and the hierarchical feature map of the fourth stage are added to obtain the third image feature. The third image feature passes through the affinity prediction module to obtain the affinity prediction value. The affinity prediction value is used to refine the initial pseudo label through random walk to obtain the pseudo label corresponding to the medical image to be segmented.

[0009] Preferably, the Transformer block adopts the structure of the Transformer block in the SegFormer network model, the label encoder adopts the MedCLIP model, and the expertise encoder adopts the ClinicalBert model.

[0010] Preferably, the knowledge attention module includes a splicing layer, two multi-head self-attention layers and a segmentation layer connected in sequence. The hierarchical feature map of the fourth stage and the knowledge features after passing through the fully connected layer are input into the splicing layer, connected on the token dimension to obtain mixed features. The mixed features pass through two multi-head self-attention layers to obtain a token sequence. The token sequence is input into the segmentation layer, split and reconstructed to obtain the first image feature.

[0011] Preferably, the similarities between the first image feature and the second image feature and the label feature are calculated respectively to obtain a first similarity feature map and a second similarity feature map, the first similarity feature map and the second similarity feature map are added, and then an argmax operation is performed to obtain an initial pseudo label, which specifically includes:

[0012] The similarity between the first image feature and the label feature is calculated to obtain the first similarity feature map, as shown in the following formula:

[0013] ;

[0014] in, represents the first image feature, Represents label features, represents a first similarity feature map;

[0015] The similarity between the second image feature and the label feature is calculated to obtain the second similarity feature map, as shown in the following formula:

[0016] ;

[0017] in, represents the second image feature, represents the second similarity feature map;

[0018] The first similarity feature map and the second similarity feature map are added to obtain the fusion feature, as shown in the following formula:

[0019] ;

[0020] in, Indicates fusion features;

[0021] Perform the argmax operation on the fused features to obtain the initial pseudo label, as shown in the following formula:

[0022] ;

[0023] in, represents the argmax function, represents the initial pseudo-label.

[0024] Preferably, the third image feature is passed through an affinity prediction module to obtain an affinity prediction value, which specifically includes:

[0025] The affinity prediction module includes a multi-head self-attention layer and a multi-layer perceptron connected in sequence. The third image feature is input into the multi-head self-attention layer in the affinity prediction module to obtain the affinity matrix, as shown in the following formula:

[0026] ;

[0027] in, represents the third image feature, represents a multi-head self-attention layer, represents the affinity matrix;

[0028] ;

[0029] in, represents a multi-layer perceptron, Represents the affinity matrix The transposed matrix of represents the affinity prediction value.

[0030] Preferably, during the training of the medical image segmentation model, the parameters of the label encoder and the expertise encoder are frozen, and the total loss function used in the training of the medical image segmentation model is It is expressed as:

[0031] ;

[0032] in, represents the classification loss function, represents the weight coefficient, represents the affinity loss function;

[0033] ;

[0034] in, represents the first similarity loss function, represents the second similarity loss function;

[0035] The expression of the first similarity loss function is as follows:

[0036] ;

[0037] in, represents the class label of the nth sample, Represents the first type of prediction vector obtained after the first similar feature map passes through the global average pooling layer, represents the sigmoid activation function, and N represents the total number of samples in the training data;

[0038] The expression of the second similarity loss function is as follows:

[0039] ;

[0040] in, represents the class label of the nth sample, Represents the second type of prediction vector obtained after the second similar feature map passes through the global average pooling layer;

[0041] The expression of affinity loss function is as follows:

[0042] ;

[0043] in, Represents pixels and pixels The predicted affinity value of and Represent the positive sample set and the negative sample set respectively, and Respectively and The number of samples in the initial pseudo label If the pixel and pixels If they have the same semantics, they are marked as positive samples and put into the positive sample set; otherwise, they are marked as negative samples and put into the negative sample set.

[0044] In a second aspect, the present invention provides a medical image segmentation device based on text-driven and affinity learning, comprising:

[0045] A model building module is configured to build and train a text-driven medical image segmentation model to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes an image visual encoder, a label encoder, an expertise encoder, a knowledge attention module, and an affinity prediction module, wherein the image visual encoder includes four stages of Transformer blocks connected in sequence, and a fully connected layer is connected behind the expertise encoder and the label encoder;

[0046] The prediction module is configured to obtain the medical image to be segmented and its corresponding professional knowledge and text labels and input them into the trained medical image segmentation model. The medical image to be segmented passes through the image visual encoder to obtain the hierarchical feature maps output by the Transformer blocks of the four stages respectively. The professional knowledge passes through the professional knowledge encoder to obtain the knowledge feature. The hierarchical feature map of the fourth stage and the knowledge feature after passing through the fully connected layer pass through the knowledge attention module to obtain the first image feature; the text label passes through the label encoder and the fully connected layer to obtain the label feature; the hierarchical feature map of the third stage and the hierarchical feature map of the fourth stage are added to obtain the second image feature, the similarity of the first image feature and the second image feature with the label feature is calculated respectively to obtain the first similarity feature map and the second similarity feature map, the first similarity feature map and the second similarity feature map are added, and then the argmax operation is performed to obtain the initial pseudo label; the hierarchical feature map of the first stage and the hierarchical feature map of the fourth stage are added to obtain the third image feature, the third image feature passes through the affinity prediction module to obtain the affinity prediction value, and the affinity prediction value is used to refine the initial pseudo label through random walk to obtain the pseudo label corresponding to the medical image to be segmented.

[0047] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0049] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) The text-driven and affinity-learning-based medical image segmentation method proposed in this paper introduces text supervision (including text labels and expertise) and affinity learning to address the limitations of weakly supervised semantic segmentation on histopathology images. Since image-level labels alone cannot provide sufficient supervision information, text supervision can provide additional guidance for the model. Therefore, a label encoder and an expertise encoder are used to encode text labels and expertise. And the expertise is used to guide the model to focus on the relevant areas of the target tissue through the knowledge attention module.

[0052] (2) The text-driven and affinity-learning-based medical image segmentation method proposed in the present invention introduces an affinity prediction module to learn reliable affinity prediction values, and uses the learned affinity prediction values ​​to correct the initial pseudo-labels through random walks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of a medical image segmentation method based on text-driven and affinity learning according to an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a training process of a medical image segmentation model of a medical image segmentation method based on text-driven and affinity learning according to an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a medical image segmentation device based on text-driven and affinity learning according to an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Figure 1 A medical image segmentation method based on text-driven and affinity learning provided by an embodiment of the present application is shown, comprising the following steps:

[0060] S1, construct and train a text-driven medical image segmentation model to obtain a trained medical image segmentation model. The trained medical image segmentation model includes an image vision encoder, a label encoder, an expertise encoder, a knowledge attention module and an affinity prediction module. The image vision encoder includes four stages of Transformer blocks connected in sequence. The expertise encoder and the label encoder are each connected to a fully connected layer.

[0061] In a specific embodiment, the Transformer block adopts the structure of the Transformer block in the SegFormer network model, the label encoder adopts the MedCLIP model, and the professional knowledge encoder adopts the ClinicalBert model.

[0062] In a specific embodiment, the knowledge attention module includes a splicing layer, two multi-head self-attention layers and a segmentation layer connected in sequence. The hierarchical feature map of the fourth stage and the knowledge features after passing through the fully connected layer are input into the splicing layer, connected in the token dimension to obtain mixed features. The mixed features pass through two multi-head self-attention layers to obtain a token sequence. The token sequence is input into the segmentation layer, split and reconstructed to obtain the first image feature.

[0063] Specifically, refer to Figure 2, the text-driven medical image segmentation model (CLIPAL) proposed in the embodiment of the present application is intended to improve the accuracy of weakly supervised histopathology image segmentation. CLIPAL uses two kinds of text information: first, the text label of the image type in the segmentation task provides basic category guidance for the image feature points: second, the external description of each type of performance is introduced, which is rich in the expertise provided by experts about the subtype tissue morphology, color and relationship with other tissues. In CLIPAL, each text label is matched with the feature point in the image space using the pre-trained MedCLIP model to extract the corresponding first image feature. This matching process is based on the similarity between features. The higher the similarity, the greater the possibility that the location belongs to the corresponding semantic category. Discriminative information is explored from professional knowledge to help accurately identify and locate complete pathological tissues by jointly modeling long-range correlations between images and texts. In addition, the affinity inherent in Transformer is further used to improve the initial pseudo-label. Through random walk propagation, the initial pseudo-label is corrected using affinity prediction values, which can diffuse object areas and suppress erroneously activated areas.

[0064] Furthermore, the medical image segmentation model mainly consists of five parts: three encoders (an image vision encoder, a label encoder, and an expertise encoder), a knowledge attention module, and an affinity prediction module. The image vision encoder is a hierarchical encoder consisting of four stages. The Transformer block in each stage adopts the structure of the Transformer block in the SegFormer network model, which includes an efficient self-attention layer and overlapping block merging. The Transformer block in each stage generates multi-level and multi-scale features of a given input image, which provide high-resolution coarse features and low-resolution fine-grained features. Specifically, given a size of The input image is encoded into image features and the resolution is The hierarchical feature map of the i-th stage ,in .

[0065] The label encoder encodes the text labels in the dataset into label features, denoted as ,in represents the number of classes in the dataset, Represents the dimension of the label feature. Since the label feature will be used to calculate the similarity with the image feature and the image feature, it is very important to choose a suitable image-text pre-trained language model. In the embodiment of the present application, the MedCLIP model is used as the label encoder, which is a model fine-tuned on the ROCO dataset based on CLIP.

[0066] The expertise encoder is responsible for embedding the expertise describing the subtype into the knowledge feature, denoted as . The knowledge features guide the image features to focus on the area related to the target tissue. In order to encode the expertise describing the subtype into more general semantic features, the embodiment of the present application uses the ClinicalBert model as an expertise encoder. The ClinicalBert model is a language model fine-tuned on the MIMIC-II dataset based on BioBert.

[0067] To improve training efficiency, the weights of the label encoder and expertise encoder are frozen, but a fully connected layer (FC) is added after them to adjust the features extracted from the label encoder and expertise encoder.

[0068] In order to enhance the model's understanding of the color, morphology, and relationship between different tissues, pathology experts provide expertise on different subtypes and encode them into knowledge features through the expertise encoder. The knowledge attention module uses these external knowledge features to guide image features to focus on the relevant areas of the target tissue. The knowledge attention module consists of a splicing layer, two multi-head self-attention modules, and a segmentation layer. And the knowledge features after the fully connected layer Connect on the token dimension to get mixed features The mixed features are input into the multi-head self-attention module in the knowledge attention module. Finally, the output token is split through the segmentation layer, and the part corresponding to the hierarchical feature map of the fourth stage is taken out as the first image feature, and the hierarchical feature map of the third stage and the hierarchical feature map of the fourth stage are added to form the second image feature. In order to save computing resources, the knowledge attention module is only added after the last layer of the image visual encoder.

[0069] In a specific embodiment, the similarities between the first image feature and the second image feature and the label feature are calculated respectively to obtain a first similarity feature map and a second similarity feature map, the first similarity feature map and the second similarity feature map are added, and then an argmax operation is performed to obtain an initial pseudo label, which specifically includes:

[0070] The similarity between the first image feature and the label feature is calculated to obtain the first similarity feature map, as shown in the following formula:

[0071] ;

[0072] in, represents the first image feature, Represents label features, represents a first similarity feature map;

[0073] The similarity between the second image feature and the label feature is calculated to obtain the second similarity feature map, as shown in the following formula:

[0074] ;

[0075] in, represents the second image feature, represents the second similarity feature map;

[0076] The first similarity feature map and the second similarity feature map are added to obtain the fusion feature, as shown in the following formula:

[0077] ;

[0078] in, Indicates fusion features;

[0079] Perform the argmax operation on the fused features to obtain the initial pseudo label, as shown in the following formula:

[0080] ;

[0081] in, represents the argmax function, represents the initial pseudo-label.

[0082] Specifically, the embodiment of the present application uses the inner product to calculate the similarity between the image feature and the label feature. The image feature includes the first image feature and the second image feature. Therefore, it is necessary to calculate the similarity between the first image feature and the second image feature and the label feature respectively. In order to make full use of the information of the network at different stages, CLIPAL adds and fuses the first similarity feature map and the second similarity feature map. Finally, the argmax operation is performed on the fused feature to obtain the initial pseudo label. .

[0083] In a specific embodiment, the third image feature is passed through an affinity prediction module to obtain an affinity prediction value, which specifically includes:

[0084] The affinity prediction module includes a multi-head self-attention layer and a multi-layer perceptron connected in sequence. The third image feature is input into the multi-head self-attention layer in the affinity prediction module to obtain the affinity matrix, as shown in the following formula:

[0085] ;

[0086] in, represents the third image feature, represents a multi-head self-attention layer, represents the affinity matrix;

[0087] ;

[0088] in, represents a multi-layer perceptron, Represents the affinity matrix The transposed matrix of represents the affinity prediction value.

[0089] Specifically, the multi-head self-attention layer (MHSA) can capture the semantic affinity between patches, which can be used to improve the rough initial pseudo-labels. However, the affinity captured in MHSA is still inaccurate, and directly applying MHSA as affinity to correct the initial pseudo-labels does not work well in practice. Therefore, an affinity prediction module is introduced, assuming that the output feature of MHSA is recorded as In the affinity prediction model, the output features of the multi-head self-attention layer are directly linearly combined through a multi-layer perceptron to generate affinity prediction values. In essence, the self-attention mechanism is a directed graph model, and the affinity matrix should be symmetric because nodes sharing the same semantics should be equal. To perform such a transformation, just replace and its transpose.

[0090] In a specific embodiment, during the training process of the medical image segmentation model, the parameters of the label encoder and the expertise encoder are frozen, and the total loss function used by the medical image segmentation model during the training process is It is expressed as:

[0091] ;

[0092] in, represents the classification loss function, represents the weight coefficient, represents the affinity loss function;

[0093] ;

[0094] in, represents the first similarity loss function, represents the second similarity loss function;

[0095] The expression of the first similarity loss function is as follows:

[0096] ;

[0097] in, represents the class label of the nth sample, Represents the first type of prediction vector obtained after the first similar feature map passes through the global average pooling layer, represents the sigmoid activation function, and N represents the total number of samples in the training data;

[0098] The expression of the second similarity loss function is as follows:

[0099] ;

[0100] in, represents the class label of the nth sample, Represents the second type of prediction vector obtained after the second similar feature map passes through the global average pooling layer;

[0101] The expression of affinity loss function is as follows:

[0102] ;

[0103] in, Represents pixels and pixels The predicted affinity value of and Represent the positive sample set and the negative sample set respectively, and Respectively and The number of samples in the initial pseudo label If the pixel and pixels If they have the same semantics, they are marked as positive samples and put into the positive sample set; otherwise, they are marked as negative samples and put into the negative sample set.

[0104] Specifically, in order to obtain a more favorable affinity prediction value , a key step is to derive a reliable pseudo-affinity tag As supervision. Pseudo affinity labels Is Specifically, in If the pixel and pixels have the same semantics, labeling them with pseudo-affinity Set to positive, otherwise their pseudo-affinity labels is negative. Therefore, the pseudo affinity label of the pixel pair can be The samples are classified into positive sample sets and negative sample sets, and the affinity prediction values ​​are calculated in different ways for the positive sample sets and negative sample sets, respectively, so as to realize affinity learning. and pixels In the case of the same local window, the affinity of distant pixel pairs is ignored.

[0105] The affinity loss function is used to force the network to learn highly confident semantic affinity relationships from MHSA. On the other hand, since the affinity prediction value It is a linear combination of MHSA. The affinity loss function is also beneficial to the learning of self-attention, which helps the network to further discover the overall object area.

[0106] S2, obtain the medical image to be segmented and its corresponding professional knowledge and text labels and input them into the trained medical image segmentation model, the medical image to be segmented passes through the image visual encoder to obtain the hierarchical feature maps output by the Transformer blocks of the four stages respectively, the professional knowledge passes through the professional knowledge encoder to obtain the knowledge feature, the hierarchical feature map of the fourth stage and the knowledge feature after passing through the fully connected layer pass through the knowledge attention module to obtain the first image feature; the text label passes through the label encoder and the fully connected layer to obtain the label feature; the hierarchical feature map of the third stage and the hierarchical feature map of the fourth stage are added to obtain the second image feature, the similarity between the first image feature and the second image feature and the label feature is calculated respectively to obtain the first similarity feature map and the second similarity feature map, the first similarity feature map and the second similarity feature map are added, and then the argmax operation is performed to obtain the initial pseudo label; the hierarchical feature map of the first stage and the hierarchical feature map of the fourth stage are added to obtain the third image feature, the third image feature passes through the affinity prediction module to obtain the affinity prediction value, and the affinity prediction value is used to refine the initial pseudo label through random walk to obtain the pseudo label corresponding to the medical image to be segmented.

[0107] Specifically, after the trained medical image segmentation model is deployed, the medical image to be segmented and its corresponding professional knowledge and text labels can be input into the trained medical image segmentation model, and the introduction of text information (including text labels and professional knowledge) provides additional guidance for the model to improve the CAM activation effect. At the same time, the affinity prediction module calculates the affinity prediction value, which is used to refine the initial pseudo-label to generate a more refined pseudo-label.

[0108] Considering that fully supervised methods often have a high dependence on a large amount of high-quality labeled data, CLIPAL shows significant advantages. It effectively reduces the dependence on pixel-level fine annotation and can achieve accurate image segmentation only by using image-level annotation and text supervision, providing new possibilities for the automatic analysis and interpretation of pathological tissue images.

[0109] Further references Figure 3As an implementation of the methods shown in the above figures, the present application provides an embodiment of a medical image segmentation device based on text-driven and affinity learning. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0110] The present application embodiment provides a medical image segmentation device based on text-driven and affinity learning, including:

[0111] A model building module 1 is configured to build and train a text-driven medical image segmentation model to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes an image visual encoder, a label encoder, an expertise encoder, a knowledge attention module, and an affinity prediction module, wherein the image visual encoder includes four stages of Transformer blocks connected in sequence, and a fully connected layer is connected to the back of the expertise encoder and the label encoder;

[0112] The prediction module 2 is configured to obtain the medical image to be segmented and its corresponding professional knowledge and text labels and input them into the trained medical image segmentation model. The medical image to be segmented passes through the image visual encoder to obtain the hierarchical feature maps output by the Transformer blocks of the four stages respectively. The professional knowledge passes through the professional knowledge encoder to obtain the knowledge feature. The hierarchical feature map of the fourth stage and the knowledge feature after passing through the fully connected layer pass through the knowledge attention module to obtain the first image feature; the text label passes through the label encoder and the fully connected layer to obtain the label feature; the hierarchical feature map of the third stage and the hierarchical feature map of the fourth stage are added to obtain the second image feature, the similarity of the first image feature and the second image feature with the label feature is calculated respectively to obtain the first similarity feature map and the second similarity feature map, the first similarity feature map and the second similarity feature map are added, and then the argmax operation is performed to obtain the initial pseudo label; the hierarchical feature map of the first stage and the hierarchical feature map of the fourth stage are added to obtain the third image feature, the third image feature passes through the affinity prediction module to obtain the affinity prediction value, and the affinity prediction value is used to refine the initial pseudo label through random walk to obtain the pseudo label corresponding to the medical image to be segmented.

[0113] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4 As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0114] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0115] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0116] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0117] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0118] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0119] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0120] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0121] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0122] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0123] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0124] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0125] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0126] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0127] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image segmentation method based on text-driven and affinity learning, characterized in that: The following steps are involved: Constructing and training a text-driven medical image segmentation model to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes an image vision encoder, a label encoder, an expertise encoder, a knowledge attention module, and an affinity prediction module, wherein the image vision encoder includes four stages of Transformer blocks connected in sequence, and each of the expertise encoder and the label encoder is connected to a fully connected layer; A medical image to be segmented and its corresponding professional knowledge and text labels are obtained and input into the trained medical image segmentation model, the medical image to be segmented passes through the image visual encoder to obtain hierarchical feature maps output by the Transformer blocks of four stages respectively, the professional knowledge passes through the professional knowledge encoder to obtain knowledge features, the hierarchical feature map of the fourth stage and the knowledge features after passing through the fully connected layer pass through the knowledge attention module to obtain the first image features; the text label passes through the label encoder and the fully connected layer to obtain the label feature; The hierarchical feature map of the third stage and the hierarchical feature map of the fourth stage are added to obtain a second image feature, and the similarities between the first image feature and the second image feature and the label feature are respectively calculated to obtain a first similarity feature map and a second similarity feature map, and the first similarity feature map and the second similarity feature map are added, and then an argmax operation is performed to obtain an initial pseudo-label; the hierarchical feature map of the first stage and the hierarchical feature map of the fourth stage are added to obtain a third image feature, and the third image feature is passed through the affinity prediction module to obtain an affinity prediction value, and the initial pseudo-label is refined by random walk using the affinity prediction value to obtain a pseudo-label corresponding to the medical image to be segmented.

2. The medical image segmentation method based on text-driven and affinity learning according to claim 1, characterized in that: The Transformer block adopts the structure of the Transformer block in the SegFormer network model, the label encoder adopts the MedCLIP model, and the expertise encoder adopts the ClinicalBert model.

3. The medical image segmentation method based on text-driven and affinity learning according to claim 1, characterized in that: The knowledge attention module includes a splicing layer, two multi-head self-attention layers and a segmentation layer connected in sequence. The hierarchical feature map of the fourth stage and the knowledge features after passing through the fully connected layer are input into the splicing layer, connected in the token dimension to obtain mixed features. The mixed features pass through two multi-head self-attention layers to obtain a token sequence. The token sequence is input into the segmentation layer, split and reconstructed to obtain the first image feature.

4. The medical image segmentation method based on text-driven and affinity learning according to claim 1, characterized in that: The similarities between the first image feature and the second image feature and the label feature are calculated respectively to obtain a first similarity feature map and a second similarity feature map, the first similarity feature map and the second similarity feature map are added, and then an argmax operation is performed to obtain an initial pseudo label, specifically including: The similarity between the first image feature and the label feature is calculated to obtain a first similarity feature map, as shown in the following formula: ； in, represents the first image feature, Represents label features, represents a first similarity feature map; The similarity between the second image feature and the label feature is calculated to obtain a second similarity feature map, as shown in the following formula: ； in, represents the second image feature, represents the second similarity feature map; The first similarity feature map and the second similarity feature map are added to obtain a fusion feature, as shown in the following formula: ； in, Indicates fusion features; Perform argmax operation on the fused features to obtain the initial pseudo label, as shown in the following formula: ； in, represents the argmax function, represents the initial pseudo-label.

5. The medical image segmentation method based on text-driven and affinity learning according to claim 1, characterized in that: The third image feature is passed through the affinity prediction module to obtain an affinity prediction value, which specifically includes: The affinity prediction module includes a multi-head self-attention layer and a multi-layer perceptron connected in sequence. The third image feature is input into the multi-head self-attention layer in the affinity prediction module to obtain an affinity matrix, as shown in the following formula: ； in, represents the third image feature, represents a multi-head self-attention layer, represents the affinity matrix; ； in, represents a multi-layer perceptron, Represents the affinity matrix The transposed matrix of represents the affinity prediction value.

6. The medical image segmentation method based on text-driven and affinity learning according to claim 1, characterized in that: During the training of the medical image segmentation model, the parameters of the label encoder and the expertise encoder are frozen, and the total loss function used in the training of the medical image segmentation model is It is expressed as: ； in, represents the classification loss function, represents the weight coefficient, represents the affinity loss function; ； in, represents the first similarity loss function, represents the second similarity loss function; The expression of the first similarity loss function is as follows: ； in, represents the class label of the nth sample, Represents the first type of prediction vector obtained after the first similar feature map passes through the global average pooling layer, represents the sigmoid activation function, and N represents the total number of samples in the training data; The expression of the second similarity loss function is as follows: ； in, represents the class label of the nth sample, Represents the second type of prediction vector obtained after the second similar feature map passes through the global average pooling layer; The expression of the affinity loss function is as follows: ； in, Represents pixels and pixels The predicted affinity value of and Represent the positive sample set and the negative sample set respectively, and Respectively and The number of samples in the initial pseudo label If the pixel and pixels If they have the same semantics, they are marked as positive samples and put into the positive sample set; otherwise, they are marked as negative samples and put into the negative sample set.

7. A medical image segmentation device based on text-driven and affinity learning, characterized in that: include: A model building module is configured to build and train a text-driven medical image segmentation model to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes an image vision encoder, a label encoder, an expertise encoder, a knowledge attention module, and an affinity prediction module, wherein the image vision encoder includes four stages of Transformer blocks connected in sequence, and each of the expertise encoder and the label encoder is connected to a fully connected layer; The prediction module is configured to obtain the medical image to be segmented and its corresponding professional knowledge and text labels and input them into the trained medical image segmentation model, the medical image to be segmented passes through the image visual encoder to obtain the hierarchical feature maps output by the Transformer blocks of the four stages respectively, the professional knowledge passes through the professional knowledge encoder to obtain the knowledge features, the hierarchical feature map of the fourth stage and the knowledge features after passing through the fully connected layer pass through the knowledge attention module to obtain the first image features; the text label passes through the label encoder and the fully connected layer to obtain the label feature; The hierarchical feature map of the third stage and the hierarchical feature map of the fourth stage are added to obtain a second image feature, and the similarities between the first image feature and the second image feature and the label feature are respectively calculated to obtain a first similarity feature map and a second similarity feature map, and the first similarity feature map and the second similarity feature map are added, and then an argmax operation is performed to obtain an initial pseudo-label; the hierarchical feature map of the first stage and the hierarchical feature map of the fourth stage are added to obtain a third image feature, and the third image feature is passed through the affinity prediction module to obtain an affinity prediction value, and the initial pseudo-label is refined by random walk using the affinity prediction value to obtain a pseudo-label corresponding to the medical image to be segmented.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images