Weak supervision medical image segmentation method based on edge prior information and multi-modal fusion

Abstract

The invention discloses a weak supervision medical image segmentation method based on edge prior information and multi-modal fusion, and the method comprises the following steps: S1, extracting the edge information of an original image through employing an edge detection operator which is efficient and accurate in calculation according to the corresponding relation between the original image and a label; and S2, constructing a text prompt by using the category label of the image, and obtaining a classification score according to the similarity between the image feature and the text prompt feature. Generating a class activation graph by using reverse gradient propagation weighted fusion, and then fusing the class activation graph with edge information to obtain a pseudo tag; s3, supervising the training of the main network according to the output consistency of the double auxiliary networks by using an interlaced learning network architecture and a knowledge distillation strategy; and Z-score standardization is carried out on the prediction of the teacher network and the student network before T-softmax operation, so that the learning ability of the student network is improved. Compared with a traditional method, the method has the advantages that the required annotation data is greatly reduced, the annotation cost is remarkably reduced, an efficient and economical solution is provided for the field of medical image segmentation, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention belongs to the field of computer-aided medical image technology, and in particular relates to a weakly supervised medical image segmentation method based on edge prior information and multimodal fusion. Background Art

[0002] Accurate segmentation of medical images is a core component of clinical diagnosis and treatment, directly impacting the accuracy of disease diagnosis and the development of treatment plans. In tumor diagnosis, segmentation technology can clearly define the relationship between lesion boundaries and surrounding tissues, providing millimeter-level anatomical reference for surgical planning. In radiotherapy, precise delineation of the target area maximizes the protection of normal tissue and improves local control.

[0003] In recent years, as the difficulty of acquiring medical image data and the cost of labeling have continued to increase, more and more researchers have turned their attention to weakly supervised medical image segmentation methods. Compared with traditional fully supervised methods, weakly supervised methods have shown significant advantages. Fully supervised methods usually rely on a large number of precisely annotated pixel-level labels. The acquisition of these labels not only requires professional doctors to spend a lot of time and energy, but is also costly. Weakly supervised methods, on the other hand, use more readily available labels, such as image-level annotations, bounding box annotations, or scribble-style annotations. These weak labels are relatively simple to obtain, which can greatly reduce the cost and time of labeling. At the same time, they can also alleviate the problem of data scarcity to a certain extent, providing new ideas for the development of the field of medical image segmentation.

[0004] Current weakly supervised segmentation methods are based on class activation maps (CAMs) generated by image classification networks. However, CAMs are not accurate enough for segmentation because they tend to focus on the most discriminative regions of an object while ignoring other parts. Most CAM-based methods typically focus on the most discriminative object parts, resulting in limited segmentation accuracy. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide an accurate and efficient weakly supervised medical image segmentation method to achieve more precise medical image segmentation.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A weakly supervised medical image segmentation method based on edge prior information and multimodal fusion, the key steps of which are:

[0008] S1: Use the original image and labels to calculate an efficient and accurate edge detection operator to extract the edge information of the image;

[0009] S2: Use the image category label to construct the text hint, obtain the classification score based on the similarity between the image features and the text features, use the directional gradient propagation weighted fusion to generate the class activation map, and fuse it with the edge information to obtain the pseudo label;

[0010] S3 uses interleaved learning and knowledge distillation strategies to supervise the training of the main network based on the consistency of the outputs of the two auxiliary networks; and performs Z-score normalization on the prediction results of the teacher network and the student network before the T-softmax operation to improve the learning ability of the student network.

[0011] Through the above design, the use of prior edge information and multimodal fusion can extract rich semantic features, help generate more accurate pseudo-labels, and avoid noise in the fitting labels.

[0012] A further method is as follows: the specific contents of step S1 are:

[0013] The weakly supervised medical image segmentation method based on edge prior information and multimodal fusion according to claim 1 is characterized in that the specific content of step S1 is:

[0014] S11: Reduce the dimension of the original image to obtain a single-channel grayscale image, and denote it and the corresponding label as I∈R H×W and L∈{0,1} H×W .

[0015] S12: Use a sliding window w of size c×c and step size s to extract the sub-regions of the original image and the sub-regions of the corresponding label image of the original image, and slide in the left, right, up, and down directions to obtain the sub-region sets of L and X in turn.

[0016] S13: Label sub-area represents the jth subregion extracted from the lth labeled image. The subregions of the label can be divided into three categories. When the sum of the pixels of a subregion is c×c, it is recorded as a foreground subregion, and the foreground subregions of all labels are used to construct a foreground subregion set: k represents the pixel value of a sub-region. When the sum of the pixels of a sub-region is 0, it is recorded as a background sub-region, and the foreground sub-regions of all labels are used to construct a background sub-region set: When the sum of the pixels of a sub-region is greater than 0 and less than c×c, it is recorded as an edge sub-region, and the foreground sub-regions of all labels are used to construct an edge sub-region set: Similarly, the sub-regions extracted from the original image can be divided into a foreground sub-region set, a background sub-region set, and an edge sub-region set. Correspondingly, the set of original foreground sub-regions is recorded as The set of background sub-regions is denoted as and the marginal subregion set

[0017] S14: Will be The q subregions of the set are selected, and each subregion is expanded into a one-dimensional vector and connected to obtain a vector of size q×c 2 The fusion feature matrix M of the sub-region I .

[0018] S15: From Extract q sub-regions from the set, take out the central pixel value of each sub-region and connect them to get a matrix M of size q×1 L .

[0019] S16: Multiply the image pseudo-inverse fusion feature matrix by the label center matrix to obtain a c 2 ×1 matrix. operator=((M I ) T ·M I ) -1 ·(M I ) T ·M L

[0020] S17: Convert the operator into a c×c matrix, which is the final edge detection operator.

[0021] S18: Using an edge detection operator, extract edge information of the image data.

[0022] By adopting the above steps, the edge information of the image can be extracted accurately and efficiently, providing prior information for the generation of pseudo labels.

[0023] Furthermore, the specific content of step S2 is:

[0024] S21: Use the image class-level labels of the data to construct a text prompt: Aphoto of{CLASS}, which is input into the text encoder.

[0025] S22: The text encoder uses the BERT encoding structure to convert the input token sequence into an embedding vector and adds the position encoding to the embedding vector to capture the position information of the token in the sequence.

[0026] S23: Resize the image data to [256×256×3] and randomly crop it to [224×224×3]. Divide the preprocessed image into [16×16×3] patches, and get M=N after division. 2 =(224÷16) 2= 196 patches. For each patch of size [16×16×3], it is mapped to a one-dimensional vector [196×768] through projection transformation. Before entering the image encoder, a token specifically used for classification operation is inserted, and the token vector of dimension size [1×768] is concatenated with the image patch vector, and finally the image patch is transformed into dimension size [197×768]. Then the positional encoding E is added. pos , the latitude of the position code is equal to the latitude of tokens, and they are directly superimposed to obtain X0 with a shape size of [197×768] as the input of the image encoder. The specific form is:

[0027]

[0028] S24: The image encoder uses multiple consecutive TransformerBlocks, each of which consists of a multi-head self-attention mechanism (MHSA), a multi-layer perceptron (MLP), a residual connection, and a layer normalization (LN). Token sequence X l-1 After the lth layer Transform Block becomes X l :

[0029] X l =MLP(LN(MHSA(LN(X l-1 ))+X l-1 ))+MHSA(LN(X l-1 ))+X l-1

[0030] S25: By calculating the pooled image feature f I and text features f T Calculate the cosine similarity to get the classification score y, the specific form is:

[0031]

[0032] S26: Starting from the output node, the classification score y is calculated by gradient back propagation feature map f c The weight of channel c is obtained by weighted summing of the feature channels to obtain the foreground class activation map and perform ReLU activation:

[0033]

[0034] S27: Upsample the obtained class activation maps of different scales to the same size. The three obtained class activation maps are regarded as three sequences of query (Q), key (K), and value (V), and the representation of each image is linearly transformed to obtain three matrices, which are respectively used as query matrix (Q_matrix), key matrix (K_matrix), and value matrix (V_matrix). Calculate the dot product between each query matrix and the other two key matrices, and normalize them by the scaling factor to obtain three attention weight matrices. Multiply each attention weight matrix with the corresponding value matrix to obtain three attention-weighted value matrices. Use multiple attention heads, each head has its own learned weights. Cascade or concatenate the outputs of multiple attention heads to obtain the final output. Let Q = X1W Q , K=V=X2W k , The calculation of cross attention is as follows:

[0035]

[0036] The fused class activation map is multiplied with the generated edge extraction information to obtain the initial pseudo label.

[0037] Furthermore, in order to avoid noise in the fitted pseudo-labels, the specific content of step S3 is:

[0038] S31: Using the network structure of interleaved learning, according to the auxiliary network Aux 1 and auxiliary network Aux 2 The similarity of the output is assigned an uncertainty weight W to each pixel i,j :

[0039]

[0040] Optimize the loss function L of the main network ce :

[0041]

[0042] S32: A dual-teacher distillation strategy is adopted, where the average soft prediction of the teacher network is used to supervise the student network, and the teacher and student predictions are Z-score normalized before T-softmax.

[0043] S33: Perform T-softmax operation on the normalized prediction results, and optimize the entire network by optimizing the KL loss of the two teacher networks and the student network:

[0044]

[0045] S34: For the test image, we use the trained model to fuse the predicted class activation map and edge prior information to generate pseudo labels, then test the segmentation network to obtain the results.

[0046] The beneficial effects of the present invention are:

[0047] This weakly supervised medical image segmentation method enriches semantic information through multimodal fusion and fuses multi-scale class activation maps through cross-attention. By utilizing interleaved learning and knowledge distillation strategies, it filters out noise in the generated pseudo-labels, helping to improve the model's robustness and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the method of the present invention;

[0049] Figure 2 Schematic diagram of computing edge extraction operator;

[0050] Figure 3 Schematic diagram of generating pseudo labels using edge prior information and multimodal fusion;

[0051] Figure 4 Schematic diagram for training a segmentation network using interleaved learning and knowledge distillation strategies; DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0053] from Figure 1 It can be seen that the weakly supervised medical image segmentation method based on edge prior information and multimodal fusion provided in the embodiment includes the following steps:

[0054] S100: Select some image data from the dataset, perform dimensionality reduction on the image data, and obtain a single-channel grayscale image, which and the corresponding label are respectively recorded as I∈R H×W and L∈{0,1} H×W The calculation process of edge extraction operator is as follows: Figure 2 As shown in the figure, a sliding window w with a size of c×c and a step size of s is used to slide from left to right and from top to bottom to extract the sub-regions of the original image and the corresponding sub-regions of the label image of the original image, and obtain the sub-region sets of L and X in turn. Denotes the jth subregion extracted from the lth labeled image. The subregions of the label can be divided into three categories: when the sum of the pixels of a subregion is c×c, it is recorded as a foreground subregion, and the foreground subregions of all labels are constructed into a foreground subregion set: k represents the pixel value of a sub-region; when the sum of the pixels of a sub-region is 0, it is recorded as a background sub-region, and the foreground sub-regions of all labels are used to construct a background sub-region set: When the sum of the pixels of a sub-region is greater than 0 and less than c×c, it is recorded as an edge sub-region, and the foreground sub-regions of all labels are used to construct an edge sub-region set: Similarly, the sub-regions extracted from the original image can be divided into foreground sub-regions, background sub-regions and edge sub-regions. Correspondingly, the original foreground sub-region set is recorded as The background sub-region set is recorded as and the marginal subregion set

[0055] S200: From Select q subregions of the set proportionally from the set, expand each subregion into a one-dimensional vector, and connect them to get a vector of size q×c 2 The fusion feature matrix M of the sub-region I .from Extract the corresponding q sub-regions of the set, take out the central pixel value of each sub-region and connect them to get a matrix M of size q×1 L Multiply the image pseudo-inverse fusion feature matrix by the label center matrix to obtain a c 2 ×1 matrix: operator = ((M I ) T ·M I ) -1 ·(M I ) T ·M L , convert the operator into a c×c matrix, which is the final edge detection operator. Using the obtained edge detection operator, the edge information of the image data in the dataset is extracted.

[0056] S300: Use the image category label to construct the text prompt, obtain the classification score according to the similarity between the image features and the text features, use the reverse gradient propagation weighted fusion to generate the class activation map, and fuse it with the edge information to obtain the pseudo label; the pseudo label generation module is as follows Figure 3 As shown in Figure 1, the class-level labels of the image data are used to construct a text prompt: Aphoto of{CLASS}, which is input into the text encoder. The text encoder uses the BERT encoding structure to convert the input token sequence into an embedding vector and add the position encoding to the embedding vector to capture the position information of the token in the sequence.

[0057] S400: Resize the image data to [256×256×3] and randomly crop it to [224×224×3]. Divide the pre-processed image into [16×16×3] patches, and get M=N after division. 2 =(224÷16) 2 = 196 patches. For each patch of size [16×16×3], it is mapped to a one-dimensional vector [196×768] through projection transformation. Before entering the image encoder, a token specifically used for classification operation is inserted, and the token vector of dimension size [1×768] is concatenated with the image patch vector, and finally the image patch is transformed into dimension size [197×768]. Then the positional encoding E is added. pos , the latitude of the position code is equal to the latitude of tokens, and they are directly superimposed, so the data latitude remains unchanged, and the shape of X0 with a size of [197×768] is obtained as the input of the image encoder. The specific form is:

[0058]

[0059] S500: The image encoder uses multiple consecutive Transformer Blocks, each of which consists of a multi-head self-attention mechanism (MHSA), a multi-layer perceptron (MLP), a residual connection, and a layer normalization (LN). Token sequence X l-1 After the lth layer Transform Block becomes X l :

[0060] X l =MLP(LN(MHSA(LN(X l-1 ))+X l-1 ))+MHSA(LN(X l-1 ))+X l-1

[0061] By calculating the pooled image feature f I and text features f T Calculate the cosine similarity to get the classification score y, the specific form is:

[0062] Starting from the output nodes of the last three layers, the classification score y is calculated by gradient back propagation to the feature map f c The weight of channel c is obtained by weighted summing of the feature channels to obtain the foreground class activation map and perform ReLU activation:

[0063] S600: Upsample the three class activation maps of different scales to the same size and treat them as three sequences of query (Q), key (K), and value (V). Perform a linear transformation on the representation of each image to obtain three matrices, which are used as the query matrix (Q_matrix), key matrix (K_matrix), and value matrix (V_matrix). Calculate the dot product between each query matrix and the other two key matrices and normalize them by the scaling factor.

[0064] Get three attention weight matrices. Multiply each attention weight matrix by the corresponding value matrix to get three attention weighted value matrices. Use multiple attention heads, each with its own learned weights. Cascade or concatenate the outputs of multiple attention heads to get the final output. Let Q = X1W Q , K=V=X2W k , The calculation of cross attention is as follows:

[0065]

[0066] The initial pseudo-label is obtained by multiplying the fused class activation map with the edge information generated by the edge extraction operator.

[0067] S700, finally uses interleaved learning and knowledge distillation strategies, such as Figure 4 As shown, according to the auxiliary network Aux 1 and auxiliary network Aux 2 The similarity of the output t, assigning uncertainty weight W to each pixel i,j :

[0068]

[0069] And optimize the loss function L of the main network ce :

[0070]

[0071] A dual-teacher distillation strategy is adopted, in which the average soft prediction of the teacher network is used to supervise the student network, and the predictions of the teacher and the student are normalized by Z-score before T-softmax. The normalized prediction results are subjected to T-softmax operation, and the distillation loss L of the teacher network and the student network is used. kd , optimize and train the entire network:

[0072]

[0073] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0074] The weakly supervised medical image segmentation method based on edge prior information and multimodal fusion provided by the present invention designs a model based on the characteristics of medical images, combines edge prior information and multimodal features, and fuses multi-scale class activation maps to solve the problem of insufficient accuracy in generating pseudo-labels. Secondly, the present invention uses interleaved learning and knowledge distillation strategies to further target the noise in the generated pseudo-labels and supervise them through an auxiliary network to avoid noise in the fitted labels.

[0075] The above description is an example of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A weakly supervised medical image segmentation method based on edge prior information and multimodal fusion. Characterized by: Follow these steps: S1: Use the original image and labels to calculate an efficient and accurate edge detection operator to extract the edge information of the original image; S2: Use the category label of the original image to construct the text hint, obtain the classification score based on the similarity between the image features and the text hint features, use the back gradient propagation weighted fusion to generate the class activation map, and fuse it with the edge information to obtain the pseudo label; S3 uses interleaved learning and knowledge distillation strategies to supervise the training of the main network based on the consistency of the outputs of the two auxiliary networks; and performs Z-score normalization on the prediction results of the teacher network and the student network before the T-softmax operation to improve the learning ability of the student network.

2. The weakly supervised medical image segmentation method based on edge prior information and multimodal fusion according to claim 1, characterized in that: The specific content of step S1 is: S11: Reduce the dimension of the original image to obtain a single-channel grayscale image, and denote it and the corresponding label as I∈R H×W and L∈{0,1} H×W . S12: Using a sliding window w of size c×c and step size s, extract the sub-regions of the original image and the sub-regions of the corresponding label image of the original image, and slide from left to right and from top to bottom to obtain the sub-region sets of L and X in turn. S13: Label sub-area Denotes the jth subregion extracted from the lth labeled image. The subregions of the label can be divided into three categories. When the sum of the pixels of a subregion is c×c, it is recorded as a foreground subregion, and the foreground subregions of all labels are constructed into a foreground subregion set: k represents the pixel value of a sub-region. When the sum of the pixels of a sub-region is 0, it is recorded as a background sub-region, and the foreground sub-regions of all labels are used to construct a background sub-region set: When the sum of the pixels of a sub-region is greater than 0 and less than c×c, it is recorded as an edge sub-region, and the foreground sub-regions of all labels are used to construct an edge sub-region set: Similarly, the sub-regions extracted from the original image can be divided into foreground sub-regions, background sub-regions and edge sub-regions. Correspondingly, the original foreground sub-region set is recorded as The background sub-region set is recorded as and the marginal subregion set S14: Will be The q subregions of the set are selected, and each subregion is expanded into a one-dimensional vector and connected to obtain a vector of size q×c 2 The fusion feature matrix M of the sub-region I . S15: From Extract q sub-regions from the set, take out the central pixel value of each sub-region and connect them to get a matrix M of size q×1 L . S16: Multiply the image pseudo-inverse fusion feature matrix by the label center matrix to obtain a c 2 ×1 matrix, operator=((M I ) T ·M I ) -1 ·(M I ) T ·M L . S17: Convert the operator into a c×c matrix, which is the final edge detection operator. S18: Using an edge detection operator, extract edge information of the image data.

3. The weakly supervised medical image segmentation method based on edge prior information and multimodal fusion according to claim 1, characterized in that: The specific content of step S2 is: S21: Use the image class-level labels of the data to construct a text prompt: Aphoto of{CLASS}, which is input into the text encoder. S22: The text encoder uses the BERT encoding structure to convert the input token sequence into an embedding vector and adds the position encoding to the embedding vector to capture the position information of the token in the sequence. S23: Resize the image data to [256×256×3] and randomly crop it to [224×224×3]. Divide the preprocessed image into [16×16×3] patches, and get M=N after division. 2 =(224÷16) 2 = 196 patches. For each patch of size [16×16×3], it is mapped to a one-dimensional vector [196×768] through projection transformation. Before entering the image encoder, a token specifically used for classification operation is inserted, and the token vector of dimension size [1×768] is concatenated with the image patch vector, and finally the image patch is transformed into dimension size [197×768]. Then the positional encoding E is added. pos , the latitude of the position code is equal to the latitude of tokens, and they are directly superimposed, so the data latitude remains unchanged, and the shape of X0 with a size of [197×768] is obtained as the input of the image encoder. The specific form is: S24: The image encoder uses multiple consecutive TransformerBlocks, each of which consists of a multi-head self-attention mechanism (MHSA), a multi-layer perceptron (MLP), a residual connection, and a layer normalization (LN). Token sequence X l-1 After the lth layer Transform Block becomes X l : X l =MLP(LN(MHSA(LN(X l-1 ))+X l-1 ))+MHSA(LN(X l-1 ))+X l-1 S25: By calculating the pooled image feature f I and text features f T Calculate the cosine similarity to get the classification score y, the specific form is: S26: Starting from the output node, the classification score y is calculated by gradient back propagation feature map f c The weight of channel c is obtained by weighted summing of the feature channels to obtain the foreground class activation map and perform ReLU activation: S27: Upsample the obtained class activation maps of different scales to the same size. The three obtained class activation maps are regarded as three sequences of query (Q), key (K), and value (V), and the representation of each image is linearly transformed to obtain three matrices, which are respectively used as query matrix (Q_matrix), key matrix (K_matrix), and value matrix (V_matrix). Calculate the dot product between each query matrix and the other two key matrices, and normalize them by the scaling factor to obtain three attention weight matrices. Multiply each attention weight matrix with the corresponding value matrix to obtain three attention-weighted value matrices. Use multiple attention heads, each head has its own learned weights. Cascade or concatenate the outputs of multiple attention heads to obtain the final output. Let Q = X1W Q , K=V=X2W k , The calculation of cross attention is as follows: The fused class activation map is multiplied with the generated edge extraction information to obtain the initial pseudo label.

4. The weakly supervised medical image segmentation method based on edge prior information and multimodal fusion according to claim 1, characterized in that: The specific content of step S3 is: S31: Using the network structure of interleaved learning, according to the auxiliary network Aux 1 and auxiliary network Aux 2 The similarity of the output t, assigning uncertainty weight W to each pixel i,j : And optimize the loss function L of the main network ce : S32: A dual-teacher distillation strategy is adopted, where the average soft prediction of the teacher network is used to supervise the student network, and the teacher and student predictions are Z-score normalized before T-softmax. S33: Perform T-softmax operation on the normalized prediction results, and optimize the entire network by optimizing the prediction losses of the teacher network and the student network: S34: For the test image, we use the trained model to fuse the predicted class activation map and edge prior information to generate pseudo labels, then test the segmentation network to obtain the results.

Images