Semi-supervised medical image segmentation method and device

Abstract

The invention discloses a semi-supervised medical image segmentation method and device, and the method comprises the steps: constructing a double-branch medical image segmentation network which comprises a teacher sub-network and a student sub-network which are the same in structure; outputting a segmentation map based on the segmentation network; the consistency loss of entropy weighting is designed, and the consistency constraint is constructed in the uncertainty region of the segmentation image; carrying out one-hot coding on a segmentation image output by the student sub-network to generate a pseudo label, and constructing dynamically weighted pseudo supervision loss to endow higher weights for more confidence pixels in the pseudo label so as to provide more effective supervision for a deterministic region; forming supervised loss by cross entropy loss and dice loss, forming unsupervised loss by consistency loss of entropy weighting and pseudo-supervised loss of dynamic weighting, and obtaining an overall loss function of the network by combining the supervised loss and the unsupervised loss; and training based on the total loss function, and finally outputting the segmented medical image. The device comprises a processor and a memory. According to the invention, a more reliable supervision signal is provided for the network, and the medical image segmentation performance is finally improved.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and image segmentation, and in particular to a semi-supervised medical image segmentation method and device. Background Art

[0002] With the advancement of medical imaging technologies such as computed tomography (CT) and magnetic resonance imaging (MRI), medical image segmentation has attracted widespread attention from researchers. This technology aims to accurately divide organs or lesion areas from medical images to provide effective support for doctors' subsequent diagnosis. In recent years, with the innovation of deep learning technology, medical image segmentation methods based on deep learning have made significant progress. Among them, medical image segmentation methods based on supervised learning have achieved excellent segmentation results by training on large-scale data with real annotations. However, there is a large amount of unlabeled data in actual medical scenarios. Manually labeling large-scale unlabeled medical image data is not only time-consuming and laborious, but also requires extremely high professionalism of the annotator, which greatly limits the practical application of medical image segmentation methods based on supervised learning.

[0003] In order to alleviate the dependence on large-scale labeled data, researchers began to explore medical image segmentation methods based on semi-supervised learning, aiming to use a small amount of labeled data and a large amount of unlabeled data to improve the performance of medical image segmentation in label-restricted scenarios. According to the way of using unlabeled data, existing semi-supervised medical image segmentation methods can be divided into two categories: consistency regularization methods and entropy minimization methods. Among them, the consistency regularization method usually adds different random perturbations to the same unlabeled data and uses it as the network input, and encourages the network to produce consistent predictions for inputs with different perturbations to achieve effective use of unlabeled data. For example: Xu et al. proposed a consistency regularization method based on fuzzy theory, which uses the average teacher model to ensure the consistency of the output of the teacher network and the student network in the uncertainty region, and improves the segmentation accuracy of the left atrium and brain tumors. However, this type of method only calculates the loss in the uncertainty region and does not make full use of the information in the deterministic region to supervise network learning. In contrast, the entropy minimization method can make full use of the information in the deterministic region, thereby providing more effective supervision for the network. Specifically, this type of method usually trains an initial segmentation model on labeled data, then generates pseudo labels based on the deterministic regions in the segmentation results output by the model, and finally uses the generated pseudo labels to optimize network learning, thereby achieving effective use of unlabeled data. For example: Xu et al. proposed an entropy minimization method with adaptive threshold adjustment, which dynamically adjusts the threshold of the entropy value during the training process to generate pseudo labels with higher confidence, thereby improving the medical image segmentation results. Although the above-mentioned consistency regularization method and entropy minimization method respectively achieve effective use of information in uncertainty regions and deterministic regions, these methods usually treat all pixels in the uncertainty region or the deterministic region equally, while ignoring the differences between different pixels. This will result in insufficient attention to pixels at fuzzy edges or small branches in the uncertainty region, and insufficient use of high-confidence pixels in the deterministic region, resulting in unreliable supervision signals provided by the network, which ultimately affects the segmentation performance. Summary of the invention

[0004] Although existing semi-supervised medical image segmentation methods have made certain progress, these methods usually treat all pixels in the uncertainty region or the deterministic region equally, while ignoring the differences between different pixels. This will lead to insufficient attention to pixels at fuzzy edges or small branches in the uncertainty region, and insufficient utilization of high-confidence pixels in the deterministic region, resulting in unreliable supervision signals provided by the network, which ultimately affects the segmentation performance. Based on this, the present invention proposes a semi-supervised medical image segmentation method and device, which fully exploits the differences between different pixels in the uncertainty region and the deterministic region to adaptively assign weights to different pixels when constructing constraints, thereby providing the network with a more reliable supervision signal, and ultimately achieving improved medical image segmentation performance, as described below:

[0005] In a first aspect, a semi-supervised medical image segmentation method is provided, the method comprising:

[0006] Construct a dual-branch medical image segmentation network, including: a teacher sub-network and a student sub-network with the same structure; output a segmentation map based on the segmentation network;

[0007] Design an entropy-weighted consistency loss to construct consistency constraints in the uncertainty region of the segmentation map;

[0008] The segmentation map output by the student subnetwork is one-hot encoded to generate a pseudo label, and a dynamically weighted pseudo-supervision loss is constructed to give higher weights to more confident pixels in the pseudo label to provide more effective supervision for the deterministic area;

[0009] The cross entropy loss and dice loss are combined into supervised loss, the entropy-weighted consistency loss and the dynamic-weighted pseudo-supervised loss are combined into unsupervised loss, and the supervised loss and unsupervised loss are combined to obtain the overall loss function of the network;

[0010] Training is performed based on the overall loss function and the segmented medical image is finally output.

[0011] Among them, the entropy weighted consistency loss is:

[0012]

[0013]

[0014] in, The weight map representing the uncertainty region, L c represents the entropy-weighted consistency loss, Sigmoid(·) represents the Sigmoid function, and MSE(·) represents the mean square error function. The entropy-weighted consistency loss enables the network to pay more attention to challenging pixels with low confidence in the uncertainty area, thereby improving the segmentation accuracy.

[0015] Among them, the dynamically weighted pseudo-supervision loss is:

[0016]

[0017]

[0018] Among them, L d represents the dynamically weighted pseudo-supervisory loss, A weight map representing the deterministic region, Indicates that the loss is calculated only in the deterministic region, and Dice(·) represents the dice loss. The dynamically weighted pseudo-supervised loss is used to give higher weights to more confident pixels in the pseudo-label to improve the supervision accuracy of the deterministic region.

[0019] Among them, the overall loss function is:

[0020]

[0021] L u =L c +L d

[0022]

[0023] Among them, L total represents the overall loss function, L s represents the supervision loss, L u represents the unsupervised loss, t represents the current training step number, t max Indicates the total number of training steps; L c represents the entropy-weighted consistency loss, L d represents the dynamically weighted pseudo-supervisory loss; The input image representing the labeled data is passed through the student subnetwork Net s The predicted segmentation result, Y i represents the true value of the corresponding segmentation map, and CE(·) represents the cross entropy loss. The multi-loss function mechanism is used to make more efficient use of labeled data and unlabeled data, thereby improving the supervision accuracy of labeled data and unlabeled data.

[0024] In a second aspect, a semi-supervised medical image segmentation device comprises: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0025] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0026] The beneficial effects of the technical solution provided by the present invention are:

[0027] 1) The present invention proposes a semi-supervised medical image segmentation method, which mines the differences between different pixels in a region to adaptively assign weights to different pixels when constructing constraints, thereby providing a more reliable supervisory signal for the network, and ultimately achieving improved medical image segmentation performance;

[0028] 2) The present invention designs an entropy-weighted consistency loss, which encourages the network to pay more attention to challenging pixels by establishing a numerically decreasing mapping relationship between the entropy value of the uncertainty region and the loss weight, thereby providing more effective supervision on the uncertainty region;

[0029] 3) The present invention designs a dynamically weighted pseudo-supervision loss, which dynamically adjusts the weights of different pixels in the pseudo-label according to the changes in confidence, providing a more reliable pseudo-supervision signal for network learning, thereby further improving the segmentation performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Flowchart of a semi-supervised medical image segmentation method. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0032] Based on the above analysis, an embodiment of the present invention proposes a semi-supervised medical image segmentation method, which effectively mines the differences between different pixels in the region to encourage the network to focus more on challenging pixels such as edges and small branches in the uncertainty region. In addition, by increasing the weights of pixels with higher confidence in the pseudo-labels in the deterministic region, a more reliable supervisory signal is provided to the network, thereby improving the segmentation performance of the network.

[0033] 1. Constructing a dual-branch medical image segmentation network

[0034] The embodiment of the present invention constructs a dual-branch medical image segmentation network, which adopts a teacher-student network architecture, including: a teacher sub-network and a student sub-network with the same structure. The dual-branch medical image segmentation network takes an unlabeled image as input and outputs two different segmentation maps. Specifically, the input unlabeled image Transformed into two enhanced images I′ through data enhancement iand I″ i , and then sent to the teacher sub-network Net t and the student subnetwork Net s In order to obtain two segmentation graphs S′ i and S″i 。 Among them, both the teacher subnetwork and the student subnetwork are composed of an encoder and a decoder. The encoder contains multiple 3D convolutional layers (for example, five 3D convolutional layers), each of which is accompanied by a 3D maximum pooling layer. Through convolution and pooling operations, the encoder can gradually reduce the spatial dimension of the feature map and increase the number of channels to achieve the extraction of multi-scale features of the image. The decoder contains multiple 3D transposed convolutional layers (for example: four 3D transposed convolutional layers) to upsample the feature map, thereby gradually restoring the image resolution. In addition, jump connections are made between the shallow and deep features of the encoder and decoder to achieve effective fusion of features at different levels.

[0035] 2. Designing Entropy-Weighted Consistency Loss

[0036] Segmentation map S′ based on the output of a dual-branch medical image segmentation network i and S″ i , the embodiment of the present invention designs an entropy-weighted consistency loss, in the segmentation graph S′ i and S″ i Specifically, we first construct a consistency constraint based on S′ i and S″ i A joint count matrix is ​​constructed to represent S′ i and S″ i The joint count matrix is ​​then normalized to obtain the uncertainty map In the uncertainty diagram In the figure, a pixel value of 1 indicates that the segmentation result corresponding to the pixel is reliable, and a pixel value of 0 indicates that the segmentation result corresponding to the pixel is unreliable.

[0037] At the same time, based on the teacher sub-network Net t Output segmentation map S′ i , calculate the corresponding entropy graph E′ i , the formula of this process is as follows:

[0038]

[0039] Where T represents the total number of pixels in the image, b represents each pixel, c represents the predicted category, and S′ i,c Represents S′ i The predicted value corresponding to category c. E′ i The value range of each pixel in is 0-1, and its value represents S′i The entropy value of the corresponding pixel in . The higher the entropy value, the lower the confidence; the lower the entropy value, the higher the confidence.

[0040] Generally speaking, the pixel confidence at the edge and small branches in the image is low, which is more challenging for the segmentation task and has a greater impact on the final segmentation performance. Therefore, in order to improve the segmentation performance, an entropy-weighted consistency loss is designed to encourage the network to pay more attention to these challenging pixels. The formula of this loss is expressed as follows:

[0041]

[0042]

[0043] in, The weight map representing the uncertainty region, L c represents the entropy-weighted consistency loss, Sigmoid(·) represents the Sigmoid function, and MSE(·) represents the mean square error function. Since the Sigmoid function is a monotonically increasing function, it can assign greater weights to pixels with lower confidence, that is, pixels with larger entropy values, so that the network pays more attention to challenging pixels with low confidence in the uncertainty area, thereby improving the segmentation accuracy.

[0044] 3. Designing Dynamically Weighted Pseudo-Supervisory Loss

[0045] In order to further achieve effective supervision of the deterministic region, the embodiment of the present invention designs a dynamically weighted pseudo-supervision loss. Specifically, based on the student subnetwork Net s Output segmentation map S″ i , first one-hot encode it to generate a pseudo label P i The formula for the above process is as follows:

[0046] P i =argmax(S″ i )

[0047] Here, argmax(·) represents the operation of selecting the index corresponding to the maximum value.

[0048] At the same time, calculate S″ i The corresponding entropy graph E″ i :

[0049]

[0050] Among them, E″ i The value of each pixel in the equation ranges from 0 to 1, and these values ​​directly reflect the value of S″ iThe entropy value of the corresponding pixel in , thus characterizing the confidence level of the pixel to a certain extent.

[0051] Considering the pseudo label P i and segmentation map S″ i The corresponding relationship, E″ i It is also possible to achieve P i An accurate representation of the confidence level of each pixel in the image. Intuitively speaking, the higher the confidence of a pixel, the more accurate its corresponding pseudo label is, and it should be given a higher weight when used as a supervisory signal. Based on this, an embodiment of the present invention constructs a dynamically weighted pseudo-supervisory loss, and the formula of the loss is expressed as follows:

[0052]

[0053]

[0054] Among them, L d represents the dynamically weighted pseudo-supervisory loss, A weight map representing the deterministic region, Indicates that the loss is calculated only in the deterministic region, and Dice(·) indicates the dice loss. As the entropy value decreases (i.e., the confidence increases), it increases, so more confident pixels in the pseudo-label can be given higher weights, which helps to improve the supervision accuracy of the certainty area.

[0055] 4. Constructing a multi-loss function mechanism

[0056] For semi-supervised segmentation tasks, the training process needs to make joint use of labeled data and unlabeled data. Therefore, the embodiment of the present invention constructs a multi-loss function mechanism, which combines supervised loss and unsupervised loss to form an overall loss function, thereby achieving efficient use of the two types of data. Specifically, the supervised loss includes cross entropy loss and dice loss. s The formula is as follows:

[0057]

[0058] Among them, L s represents the supervision loss, The input image representing the labeled data is passed through the student subnetwork Net s The predicted segmentation result, Y i represents the corresponding segmentation map truth value, and CE(·) represents the cross entropy loss.

[0059] In addition, the entropy-weighted consistency loss L c and the dynamically weighted pseudo-supervisory loss L d Together they form the unsupervised loss. Unsupervised loss Lu The formula is as follows:

[0060] L u =L c +L d

[0061] Finally, by combining the supervised loss and the unsupervised loss, the overall loss function of the network is defined as follows:

[0062]

[0063] Among them, t represents the current training step number, t max Represents the total number of training steps.

[0064] 5. Training a Semi-Supervised Medical Image Segmentation Network

[0065] The embodiment of the present invention uses the overall loss function L total Optimize the student subnetwork Net s During the training process, the teacher sub-network Net is trained using an exponential moving average strategy. t Update to integrate information from different training steps. The formula for the above process is as follows:

[0066] θ′ t =αθ t +(1-α)θ s

[0067] Among them, θ′ t Represents the updated teacher sub-network Net t The parameter θ t and θ s Respectively represent the teacher sub-network Net before updating t and the student subnetwork Net s , α represents the smoothing coefficient of the exponential moving average. In the embodiment of the present invention, α is set to 0.99.

[0068] The example of the present invention only uses the trained teacher sub-network Net t Test to obtain the final semi-supervised medical image segmentation result. Specifically, the medical image to be tested is sent to the trained teacher sub-network Net t , to obtain the final segmentation result. The formula of the above process is as follows:

[0069]

[0070] Among them, S represents the result of semi-supervised segmentation of any medical image I in the test set. Represents the trained teacher sub-network Net t , whose network parameter is θ′t The final output of the proposed semi-supervised medical image segmentation network is the above segmentation result S, which can be used to assist doctors in diagnosis to improve diagnostic efficiency.

[0071] A semi-supervised medical image segmentation device, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to perform the following method steps:

[0072] Construct a dual-branch medical image segmentation network, including: a teacher sub-network and a student sub-network with the same structure; output a segmentation map based on the segmentation network;

[0073] Design an entropy-weighted consistency loss to construct consistency constraints in the uncertainty region of the segmentation map;

[0074] The segmentation map output by the student subnetwork is one-hot encoded to generate a pseudo label, and a dynamically weighted pseudo-supervision loss is constructed to give higher weights to more confident pixels in the pseudo label to provide more effective supervision for the deterministic area;

[0075] The cross entropy loss and dice loss are combined into supervised loss, the entropy-weighted consistency loss and the dynamic-weighted pseudo-supervised loss are combined into unsupervised loss, and the supervised loss and unsupervised loss are combined to obtain the overall loss function of the network;

[0076] Training is performed based on the overall loss function and the segmented medical image is finally output.

[0077] Among them, the entropy-weighted consistency loss is:

[0078]

[0079]

[0080] in, The weight map representing the uncertainty region, L c represents the entropy-weighted consistency loss, Sigmoid(·) represents the Sigmoid function, and MSE(·) represents the mean square error function. The entropy-weighted consistency loss enables the network to pay more attention to challenging pixels with low confidence in the uncertainty area, thereby improving the segmentation accuracy.

[0081] Among them, the dynamically weighted pseudo-supervision loss is:

[0082]

[0083]

[0084] Among them, L d represents the dynamically weighted pseudo-supervisory loss, A weight map representing the deterministic region, Indicates that the loss is calculated only in the deterministic region, and Dice(·) represents the dice loss. The dynamically weighted pseudo-supervised loss is used to give higher weights to more confident pixels in the pseudo-label to improve the supervision accuracy of the deterministic region.

[0085] Among them, the overall loss function is:

[0086]

[0087] L u =L c +L d

[0088]

[0089] Among them, L total represents the overall loss function, L s represents the supervision loss, L u represents the unsupervised loss, t represents the current training step number, t max Indicates the total number of training steps; L c represents the entropy-weighted consistency loss, L d represents the dynamically weighted pseudo-supervisory loss; The input image representing the labeled data is passed through the student subnetwork Net s The predicted segmentation result, Y i represents the true value of the corresponding segmentation map, CE(·) represents the cross entropy loss; the multi-loss function mechanism is used to make more efficient use of labeled data and unlabeled data and improve the supervision accuracy of labeled data and unlabeled data.

[0090] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0091] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0092] The data signal is transmitted between the memory and the processor via a bus, which is not described in detail in the embodiment of the present invention.

[0093] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0094] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0095] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0096] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0097] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0098] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0099] Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A semi-supervised medical image segmentation method, characterized in that: The method comprises: Construct a dual-branch medical image segmentation network, including: a teacher sub-network and a student sub-network with the same structure; including: outputting a segmentation map based on the segmentation network; designing an entropy-weighted consistency loss and constructing a consistency constraint in the uncertainty region of the segmentation map; Design an entropy-weighted consistency loss to construct consistency constraints in the uncertainty region of the segmentation map; The segmentation map output by the student subnetwork is one-hot encoded to generate a pseudo label, and a dynamically weighted pseudo-supervision loss is constructed to give higher weights to more confident pixels in the pseudo label to provide more effective supervision for the deterministic area; The cross entropy loss and dice loss are combined into supervised loss, the entropy-weighted consistency loss and the dynamic-weighted pseudo-supervised loss are combined into unsupervised loss, and the supervised loss and unsupervised loss are combined to obtain the overall loss function of the network; Training is performed based on the overall loss function and the segmented medical image is finally output.

2. A semi-supervised medical image segmentation method according to claim 1, characterized in that: The entropy-weighted consistency loss is: in, The weight map representing the uncertainty region, L c represents the entropy-weighted consistency loss, Sigmoid(·) represents the Sigmoid function, and MSE(·) represents the mean square error function. The entropy-weighted consistency loss enables the network to pay more attention to challenging pixels with low confidence in the uncertainty area, thereby improving the segmentation accuracy.

3. A semi-supervised medical image segmentation method according to claim 1, characterized in that: The dynamically weighted pseudo-supervision loss is: Among them, L d represents the dynamically weighted pseudo-supervisory loss, A weight map representing the deterministic region, Indicates that the loss is calculated only in the deterministic region, and Dice(·) represents the dice loss. The dynamically weighted pseudo-supervisory loss improves the supervision accuracy of the deterministic region by giving higher weights to more confident pixels in the pseudo-label.

4. A semi-supervised medical image segmentation method according to claim 1, characterized in that: The overall loss function is: L u =L c +L d Among them, L total represents the overall loss function, L s represents the supervision loss, L u represents the unsupervised loss, t represents the current training step number, t max Indicates the total number of training steps; L c represents the entropy-weighted consistency loss, L d represents the dynamically weighted pseudo-supervisory loss; The input image with labeled data is passed through the student subnetwork Net s The predicted segmentation result, Y i represents the true value of the corresponding segmentation map, and CE(·) represents the cross entropy loss. The multi-loss function mechanism is used to make more efficient use of labeled data and unlabeled data, thereby improving the supervision accuracy of labeled data and unlabeled data.

5. A semi-supervised medical image segmentation device, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 4.

Images