Multi-perturbation consistency medical image segmentation method based on 2+1 correction mode

Abstract

The present invention discloses a multi-perturbation consistency medical image segmentation method based on a 2+1 correction mode, comprising the following steps: S1, constructing an image segmentation model; S2, obtaining two differently enhanced images; S3, inputting the differently enhanced images into the image segmentation model to obtain 6 segmentation results; S4, respectively inputting the 6 segmentation results into a student difference guidance correction learning module to obtain a difference loss and an error loss; S5, inputting the segmentation results of the student network into an adaptive bidirectional displacement module to obtain 4 new samples; S6, respectively inputting the 4 new samples into different student networks to perform dice loss calculation; S7, summing up all losses to obtain a total loss; S8, using the stochastic gradient descent of the total loss to train a student model A that can correctly segment medical images. The present invention can solve the problem that only limited situations can be processed using a single perturbation, and can achieve the quality of consistent learning that can still be guaranteed when multiple perturbations are used.

Description

Technical Field

[0001] The present invention relates to the technical field of semi-supervised medical image segmentation, and in particular to a multi-disturbance consistency medical image segmentation method based on a 2+1 correction mode. Background Art

[0002] In the existing technology, semi-supervised medical image segmentation methods have reduced a lot of costs and manpower compared to fully supervised medical image segmentation methods. Therefore, the high accuracy of semi-supervised segmentation is pursued by researchers in this field. With the deepening of research, many new frameworks and methods have been proposed in the field of semi-supervised medical image segmentation, such as bidirectional copy and paste (BCP) and mutual correction framework (MCF). These methods significantly improve the performance of existing segmentation models by integrating semi-supervised learning and consistency regularization. Recently, the student-teacher model combined with consistency learning has been very popular in the field of semi-supervised medical image segmentation. The pseudo-labels generated by the teacher model can be used as supervisory signals for the output of the student model. Its core strategy is to force the model to produce consistent predictions under different perturbations.

[0003] There are still some shortcomings in the above semi-supervised medical image segmentation method:

[0004] (1) Although the semi-supervised image segmentation method based on the average teacher model effectively utilizes a large number of unlabeled images, it is affected by incorrect pseudo-labels, resulting in segmentation errors.

[0005] (2) Consistency-based semi-supervised segmentation methods either focus on utilizing a specific single perturbation and can only handle limited situations, or use multiple perturbations simultaneously but find it difficult to ensure the quality of consistent learning. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-perturbation consistency medical image segmentation method based on a 2+1 correction mode, which can solve the problem that only limited situations can be handled using a single perturbation, and can achieve the quality of consistency learning while using multiple perturbations.

[0007] To achieve the above object, the present invention provides a multi-perturbation consistency medical image segmentation method based on a 2+1 correction mode, comprising the following steps:

[0008] S1. Build an image segmentation model consisting of a teacher network T and two different student networks A and B;

[0009] S2: Image X is weakly enhanced and strongly enhanced to obtain two differently enhanced images X w and X s ;

[0010] S3, the imageX w and X s Input them into the teacher network T, student network A and student network B respectively, and get 6 segmentation results;

[0011] S4. Divide the six segmentation results of the teacher network T, the student network A and the student network B into two groups, one group O wA , O wB and O wT , and another group O sA , O sB and O sT , and input these two sets of segmentation results into the student difference guidance correction learning module to obtain the difference loss and error loss ;

[0012] S5. Divide the four segmentation results of student network A and student network B into two groups. O wA , O sB Form a group, O sA , O wB They are grouped together and input into the adaptive bidirectional displacement module to obtain 4 new samples;

[0013] S6. Then input the 4 new samples into student network A and student network B respectively, and get 8 outputs. Divide the 8 outputs into 4 groups, and divide the outputs of the same new sample through different student networks into one group, perform dice loss calculation, and get the adaptive displacement loss. ;

[0014] S7. Sum the losses to get the total loss L all ;

[0015] S8. Use stochastic gradient descent of total loss to train a student model A that can correctly segment medical images.

[0016] Preferably, in S1, the student network A applies perturbation 1 to inject noise into the output feature map z of the encoder, and the student network B applies perturbation 2 to discard some activations of the output feature map z of the encoder.

[0017] Preferably, the perturbation 1 is as follows: uniformly sample the noise tensor N~u(-0.3, 0.3) of the same size as the feature map z, adjust its amplitude by multiplying it with the feature map z, and then inject the noise into the feature map z to obtain ;

[0018] The specific perturbation 2 is: first, the threshold After uniform sampling, a mask is generated after summing the channel dimension and normalizing the feature map z to obtain z'. , and then use this to get the perturbed version .

[0019] Preferably, in S4, O wA , O wB and O wT For example, O wA , O wB and O wT After Softmax and argmax operations, we get M wA , M wB and M wT ; M wA and M wB The difference mask is obtained by XOR operation ; M wA and M wT The error mask is obtained by XOR operation ; M wB and M wT The error mask is obtained by XOR operation ; and XOR to get different error masks ; and XOR to get different error masks ; Similarly, O sA , O sB and O sT The same operation was performed to obtain , and , and finally useO wA , O wB , O wT , and Get the difference loss , the formula is as follows:

[0020] ;

[0021] use O wA , O wB , O wT , 、 , , and uniformly distributed u to obtain the error loss , the formula is as follows:

[0022] ;

[0023] In the formula is a mask, calculated as , the s and w in the mask represent strong enhancement and weak enhancement respectively.

[0024] Preferably, in S5, O wA , O sB For example, O wA , O sB After Softmax operation, we get P wA , P sB , the segmentation prediction after input perturbation, feature perturbation and network perturbation P wA , P sB Divide it into 16 blocks and give each block an index from 0 to 15. P wA Do inter-block attention on the 16 blocks in the image to get the least relevant block, write down its index, and then compare this block with P sB Perform inter-block attention on the 16 blocks in the block to find the least relevant block and write down the index. X w and X s It is also divided into 16 blocks and each block is labeled with an index from 0 to 15.P wA The index of the block found in corresponds to X w The blocks with the same index in P sB The index of the block found in corresponds to X s The blocks with the same index in X s Replace the found block X w The blocks found, new samples obtained ; Conversely, the same P sB Do inter-block attention on the 16 blocks in the image to get the least relevant block, write down its index, and then compare this block with P wA Perform inter-block attention on the 16 blocks in the block to find the least relevant block and write down the index. X w Replace the found block X s The blocks found, new samples obtained Similarly, O sA , O wB Perform the same operation on the new sample and .

[0025] Preferably, in S6, The formula is as follows:

[0026] ;

[0027] In the formula c Indicates the total number of categories in the dataset.

[0028] Preferably, in S7, the total loss L all The formula is as follows:

[0029] ;

[0030] ;

[0031] ;

[0032] In the formula L ce is the cross entropy loss, L dice It is the dice loss, and the dice loss calculation formula is: , where |X| and |Y| represent the true label and predicted mask of the segmentation, respectively. L mse is the squared error loss, L kl is the KL divergence loss, ,in Indicates that all components are uniform distribution of , denote labeled data and unlabeled data respectively.

[0033] Therefore, the present invention adopts the above-mentioned multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode, which can solve the problem that only limited situations can be processed using a single perturbation, and can achieve the quality of consistency learning while using multiple perturbations.

[0034] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the overall process of an embodiment of the multi-disturbance consistency medical image segmentation method based on the 2+1 correction mode of the present invention;

[0036] Figure 2 It is a flow chart of the student difference guidance and correction learning module of the present invention;

[0037] Figure 3 It is a schematic diagram of the process of the adaptive bidirectional displacement module of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0039] Embodiment 1

[0040] like Figure 1 As shown, the present invention provides a multi-perturbation consistency medical image segmentation method based on a 2+1 correction mode, comprising the following steps:

[0041] S1. Build an image segmentation model consisting of a teacher network T and two different student networks A and B.

[0042] Student network A applies perturbation 1 which injects noise into the output feature map z of the encoder, and student network B applies perturbation 2 which discards some activations of the output feature map z of the encoder.

[0043] Perturbation 1 is as follows: uniformly sample the noise tensor Nu(-0.3, 0.3) of the same size as the feature map z, adjust its amplitude by multiplying it with the feature map z, and then inject the noise into the feature map z to obtain ;

[0044] The specific perturbation 2 is: first, the threshold After uniform sampling, a mask is generated after summing the channel dimension and normalizing the feature map z to obtain z'. , and then use this to get the perturbed version .

[0045] S2: Image X is weakly enhanced and strongly enhanced to obtain two differently enhanced images X w and X s .

[0046] S3, the image X w and X s They are input into the teacher network T, student network A and student network B respectively, and 6 segmentation results are obtained.

[0047] S4. Divide the six segmentation results of the teacher network T, the student network A and the student network B into two groups, one group O wA , O wB and O wT , and another group O sA , O sB and O sT , and input these two sets of segmentation results into the student difference guidance correction learning module to obtain the difference loss and error loss .

[0048] Specifically: Figure 2 As shown, O wA , O wB and O wT For example, O wA , O wB and O wT After Softmax and argmax operations, we get M wA ,M wB and M wT ; M wA and M wB The difference mask is obtained by XOR operation ; M wA and M wT The error mask is obtained by XOR operation ; M wB and M wT The error mask is obtained by XOR operation ; and XOR to get different error masks ; and XOR to get different error masks ; Similarly, O sA , O sB and O sT The same operation was performed to obtain , and , and finally use O wA , O wB , O wT , and Get the difference loss , the formula is as follows:

[0049] ;

[0050] use O wA , O wB , O wT , 、 , , and uniformly distributed u to obtain the error loss , the formula is as follows:

[0051] ;

[0052] In the formula is a mask, calculated as , the s and w in the mask represent strong enhancement and weak enhancement respectively.

[0053] S5. Divide the four segmentation results of student network A and student network B into two groups. O wA , O sB Form a group, O sA , O wB They are grouped together and input into the adaptive bidirectional displacement module respectively to obtain 4 new samples.

[0054] Specifically: Figure 3 As shown, O wA , O sB For example, O wA , O sB After Softmax operation, we get P wA , P sB , the segmentation prediction after input perturbation, feature perturbation and network perturbation P wA , P sB Divide it into 16 blocks and give each block an index from 0 to 15. P wA Do inter-block attention on the 16 blocks in the image to get the least relevant block, write down its index, and then compare this block with P sB Perform inter-block attention on the 16 blocks in the block to find the least relevant block and write down the index. X w and X s It is also divided into 16 blocks and each block is labeled with an index from 0 to 15. P wA The index of the block found in corresponds to X w The blocks with the same index in P sB The index of the block found in corresponds to X s The blocks with the same index in X s Replace the found block X w Found blocks, get new samples ; Conversely, the same P sBDo inter-block attention on the 16 blocks in the image to get the least relevant block, write down its index, and then compare this block with P wA Perform inter-block attention on the 16 blocks in the block to find the least relevant block and write down the index. X w Replace the found block X s Found blocks, get new samples Similarly, O sA , O wB Perform the same operation on the new sample and .

[0055] S6. Then input the 4 new samples into student network A and student network B respectively, and get 8 outputs. Divide the 8 outputs into 4 groups, and divide the outputs of the same new sample through different student networks into one group, perform dice loss calculation, and get the adaptive displacement loss. .

[0056] The formula is as follows:

[0057] ;

[0058] In the formula c Indicates the total number of categories in the dataset.

[0059] S7. Sum the losses to get the total loss L all .

[0060] Total loss L all The formula is as follows:

[0061] ;

[0062] ;

[0063] ;

[0064] In the formula L ce is the cross entropy loss, L dice It is the dice loss, and the dice loss calculation formula is: , where |X| and |Y| represent the true label and predicted mask of the segmentation, respectively. L mse is the squared error loss, L kl is the KL divergence loss, ,in u Indicates that all components are uniform distribution of , denote labeled data and unlabeled data respectively.

[0065] S8. Use stochastic gradient descent of total loss to train a student model A that can correctly segment medical images.

[0066] Therefore, the present invention adopts the above-mentioned multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode, which can solve the problem that only limited situations can be processed using a single perturbation, and can achieve the quality of consistency learning while using multiple perturbations.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-perturbation consistency medical image segmentation method based on a 2+1 correction mode, characterized in that: The following steps are involved: S1. Build an image segmentation model consisting of a teacher network T and two different student networks A and B; S2: Image X is weakly enhanced and strongly enhanced to obtain two differently enhanced images X w and X s ; S3, the image X w and X s Input them into the teacher network T, student network A and student network B respectively, and get 6 segmentation results; S4. Divide the six segmentation results of the teacher network T, the student network A and the student network B into two groups, one group O wA , O wB and O wT , and another group O sA , O sB and O sT , and input these two sets of segmentation results into the student difference guidance correction learning module to obtain the difference loss and error loss ; S5. Divide the four segmentation results of student network A and student network B into two groups. O wA , O sB Form a group, O sA , O wB They are grouped together and input into the adaptive bidirectional displacement module to obtain 4 new samples; S6. Then input the 4 new samples into student network A and student network B respectively, and get 8 outputs. Divide the 8 outputs into 4 groups, and divide the outputs of the same new sample through different student networks into one group, perform dice loss calculation, and get the adaptive displacement loss. ; S7. Sum the losses to get the total loss L all ; S8. Use stochastic gradient descent of total loss to train a student model A that can correctly segment medical images.

2. The multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode according to claim 1 is characterized in that: In S1, the student network A applies perturbation 1 to inject noise into the output feature map z of the encoder, and the student network B applies perturbation 2 to discard some activations of the output feature map z of the encoder.

3. The multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode according to claim 2 is characterized in that: Perturbation 1 is specifically: for the noise tensor N~ with the same size as the feature map z u (-0.3, 0.3) is uniformly sampled, and after adjusting its amplitude by multiplying it with the feature map z, the noise is injected into the feature map z to obtain ; The specific perturbation 2 is: first, the threshold After uniform sampling, a mask is generated after summing the channel dimension and normalizing the feature map z to obtain z'. , and then use this to get the perturbed version .

4. The multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode according to claim 1, characterized in that: In S4, O wA , O wB and O wT For example, O wA , O wB and O wT After Softmax and argmax operations, we get M wA , M wB and M wT ; M wA and M wB The difference mask is obtained by XOR operation ; M wA and M wT The error mask is obtained by XOR operation ; M wB and M wT The error mask is obtained by XOR operation ; and XOR to get different error masks ; and XOR to get different error masks ; Similarly, O sA , O sB and O sT The same operation was performed to obtain , and , and finally use O wA , O wB , O wT , and Get the difference loss , the formula is as follows: ； use O wA , O wB , O wT , 、 , , and uniform distribution u Get error loss , the formula is as follows: ； In the formula M differr is a mask, calculated as M differr = M diff M err ,The s and w in the mask represent strong enhancement and weak enhancement, respectively; L mse is the mean square error loss, L kl is the KL divergence loss, ,in u Indicates that all components are of uniform distribution.

5. The multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode according to claim 4 is characterized in that: In S5, O wA , O sB For example, O wA , O sB After Softmax operation, we get P wA , P sB , the segmentation prediction after input perturbation, feature perturbation and network perturbation P wA , P sB Divide it into 16 blocks and give each block an index from 0 to 15. P wA Do inter-block attention on the 16 blocks in the image to get the least relevant block, write down its index, and then compare this block with P sB Perform inter-block attention on the 16 blocks in the block to find the least relevant block and write down the index. X w and X s It is also divided into 16 blocks and each block is labeled with an index from 0 to 15. P wA The index of the block found in corresponds to X w The blocks with the same index in P sB The index of the block found in corresponds to X s The blocks with the same index in X s Replace the found block X w The blocks found, new samples obtained ; Conversely, the same P sB Do inter-block attention on the 16 blocks in the image to get the least relevant block, write down its index, and then compare this block with P wA Perform inter-block attention on the 16 blocks in the block to find the least relevant block and write down the index. X w Replace the found block X s The blocks found, new samples obtained Similarly, O sA , O wB Perform the same operation on the new sample and .

6. The multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode according to claim 5, characterized in that: In S6, The formula is as follows: ； In the formula C Indicates the total number of categories in the data set; L dice It is the dice loss, and the dice loss calculation formula is: , where |X| and |Y| represent the true label and predicted mask of the segmentation, respectively.

7. The multi-perturbation consistency medical image segmentation method based on the 2+1 correction mode according to claim 6, characterized in that: In S7, the total loss L all The formula is as follows: ； ； ； In the formula L ce is the cross entropy loss, the superscript ， denote labeled data and unlabeled data respectively.

Images