Medical image segmentation model training method and device based on scribble annotation and terminal

Abstract

The application discloses a medical image segmentation model training method and device based on scribble annotation and a terminal, and uses two weight-shared medical image segmentation models to construct a double-network structure. A standard enhanced view and a composite enhanced view are obtained through processing of an input medical image via an enhancement operation, and the standard enhanced view and the composite enhanced view are respectively taken as inputs of the double-network structure to obtain pseudo labels and prediction labels. A consistency regularization loss that measures the similarity between the pseudo labels and the prediction labels is calculated. In the model training process, a loss function including the consistency regularization is minimized to optimize model parameters. Compared with the prior art, the application uses the weight-shared double-network structure to perform consistency regularization training, maximizes the similarity of predictions of different enhanced views of the same image, has good optimization effect, and can complete network parameter optimization in one training, without the need to perform multiple iterative training, and has high training efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, in particular to a medical image segmentation model training method and device based on scribble annotation and a terminal. BACKGROUND

[0002] Currently, a data-driven deep learning method is usually used for medical image segmentation, and a large amount of fully annotated data (i.e., complete annotation of the shape of a target organ) is needed to train a medical image segmentation model. However, the fully annotated data of a three-dimensional medical image needs to be annotated layer by layer by a professional person in the imaging department, and a large amount of data annotation is costly and time-consuming. Scribble annotation has high annotation freedom, contains accurate annotated pixels, and does not introduce annotation noise, and is suitable for annotating medical anatomical structures. Training a medical image segmentation model using scribble annotation data can effectively reduce the burden of manual annotation.

[0003] Currently, there are various deep learning training methods based on scribble annotation for image segmentation, mainly including an iterative training method, a training method based on regularization constraints, and a training method based on a generative adversarial network.

[0004] The iterative training method causes the network model to fit to incorrect annotation information and needs multiple rounds of training, which is low in efficiency and poor in optimization effect. The regularization constraint-based training method has limited optimization effect on medical images because the pixel value difference between adjacent tissues in a medical image is very common. The training method based on the generative adversarial network still needs to use fully annotated data as positive samples for discriminator training and is prone to training collapse.

[0005] Therefore, the optimization effect is poor when training a medical image segmentation model based on scribble annotation data in the prior art. SUMMARY

[0006] The main purpose of the present application is to provide a medical image segmentation model training method based on scribble annotation, which aims to solve the problem of poor optimization effect when training a medical image segmentation model based on scribble annotation data in the prior art.

[0007] In order to achieve the above purpose, the present application provides a medical image segmentation model training method based on scribble annotation, which comprises the following steps:

[0008] performing a first image enhancement operation on the scribble-annotated medical image to obtain a standard enhanced view, and performing a second image enhancement operation on the standard enhanced view to obtain a composite enhanced view;

[0009] input the standard enhanced view and the composite enhanced view into one network in a double network structure respectively to obtain pseudo labels and prediction labels, each network in the double network structure is a medical image segmentation model, and the two medical image segmentation models share weights;

[0010] calculate the similarity between the pseudo labels and the prediction labels to obtain a consistency regularization loss;

[0011] construct a loss function including the consistency regularization loss;

[0012] train the medical image segmentation model according to the loss function until the loss function converges to obtain a trained medical image segmentation model.

[0013] Optionally, the training method further comprises calculating an entropy minimization loss of the pseudo labels, and the loss function further comprises the entropy minimization loss when being constructed.

[0014] Optionally, when constructing the loss function, the consistency regularization loss and the entropy minimization loss are weighted according to an exponential form of an ascending function, and an expression of the ascending function is r(t)=exp(-η(1-t / T)), wherein t is a training round, T and η are hyperparameters.

[0015] Optionally, the training method further comprises calculating a cross-entropy loss of extracted features of the labeled pixels in the standard enhanced view to obtain a partial cross-entropy loss; and the loss function further comprises the partial cross-entropy loss when being constructed.

[0016] Optionally, the first image enhancement operation on the medical image with scribble annotation to obtain the standard enhanced view and the second image enhancement operation on the standard enhanced view to obtain the composite enhanced view comprise:

[0017] performing a geometric enhancement operation and a noise enhancement operation on the medical image with scribble annotation to obtain the standard enhanced view;

[0018] performing a color distortion enhancement operation on the standard enhanced view to obtain the composite enhanced view.

[0019] Optionally, the training method further comprises a feature pool for storing features of the labeled pixels.

[0020] extracting features of the labeled pixels in the standard enhanced view to obtain hidden features;

[0021] obtaining features of the labeled pixels based on weights of the hidden features and the labeled pixels;

[0022] dynamically updating the feature pool based on the features of the labeled pixels;

[0023] The partial cross-entropy between the hidden features and the graffiti annotation and the cross-entropy between the unit matrix formed by the annotation of the feature pool and the feature pool are calculated based on the same mapping module, and an auxiliary loss and a feature pool loss are obtained respectively; when the loss function is constructed, the auxiliary loss and the feature pool loss are also included.

[0024] Optionally, the feature based on the labeled pixel is used to dynamically update the feature pool, including:

[0025] According to the set momentum coefficient and the feature of the labeled pixel, the momentum moving average method is used to update the feature pool.

[0026] In order to achieve the above purpose, the application further provides a medical image segmentation model training device based on graffiti annotation, the device comprises:

[0027] The enhancement operation module is used for performing first image enhancement operation on the medical image after graffiti annotation to obtain a standard enhanced view, and performing second image enhancement operation on the standard enhanced view to obtain a composite enhanced view.

[0028] The network mapping module is used for inputting the standard enhanced view and the composite enhanced view into one network in a double network structure respectively to obtain pseudo labels and prediction labels, each network in the double network structure is a medical image segmentation model, and the two medical image segmentation models share weights.

[0029] The consistency regularization loss module is used for calculating the similarity between the pseudo labels and the prediction labels to obtain a consistency regularization loss.

[0030] The loss function module is used for constructing a loss function including the consistency regularization loss.

[0031] The network parameter updating module is used for training the medical image segmentation model according to the loss function until the loss function converges, and obtaining a trained medical image segmentation model.

[0032] In order to achieve the above purpose, the application further provides an intelligent terminal, the intelligent terminal comprises a memory, a processor, and a medical image segmentation model training program based on graffiti annotation stored in the memory and executable on the processor, and the medical image segmentation model training program based on graffiti annotation implements the steps of any one of the above medical image segmentation model training methods based on graffiti annotation when executed by the processor.

[0033] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a medical image segmentation model training program based on graffiti annotation, wherein when the medical image segmentation model training program based on graffiti annotation is executed by a processor, it implements any of the steps of the above-described medical image segmentation model training method based on graffiti annotation.

[0034] As described above, this invention uses two medical image segmentation models to construct a dual-network structure with shared weights. The medical image after graffiti annotation is augmented to obtain a standard augmented view and a composite augmented view. Each augmented view is then input into a separate medical image segmentation model to obtain pseudo-labels and predicted labels. A consistency regularization loss, used to measure the similarity between pseudo-labels and predicted labels, is calculated. The medical image segmentation model is trained and optimized based on this consistency regularization loss function. Compared to existing technologies, this invention uses a weight-shared dual-network structure and consistency regularization training to maximize the predicted similarity of different augmented views of the same image during training, resulting in better optimization. Furthermore, network parameter optimization is completed in a single training iteration, eliminating the need for multiple iterations and improving training efficiency. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 It is a doodle illustration of abdominal organs and heart structure;

[0037] Figure 2 This is a flowchart illustrating an embodiment of the medical image segmentation model training method based on graffiti annotation proposed in this invention.

[0038] Figure 3 yes Figure 2 Schematic diagram of the embodiment;

[0039] Figure 4 yes Figure 2 A schematic diagram of the process for obtaining feature pool loss in the embodiment;

[0040] Figure 5 This is a schematic diagram of the structure of the medical image segmentation model training device based on graffiti annotation provided in an embodiment of the present invention;

[0041] Figure 6 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0043] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", when used in this specification, including the accompanying claims, includes the presence of said feature, item, component or element but does not exclude the presence of one or more other features, items, components, elements, and / or groups thereof.

[0044] It is also to be understood that the terminology in the specification of the application is used for the purpose of describing specific embodiments only and is not intended to limit the application. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0045] It will be further understood that the terms "and / or", "includes" and / or "comprising", when used in this specification, including the following claims, shall not be construed as closed, limiting or restrictive.

[0046] As used in this specification and the appended claims, the term "if" can be construed to mean "when" or "once" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be construed to mean "once it is determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]", depending on the context.

[0047] The technical solutions in the embodiments of the present application are clearly and completely described below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0048] In the following description, a lot of specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the concept of the present application, therefore, the present application is not limited to the specific embodiments disclosed below.

[0049] Since a large amount of data is expensive and time-consuming, the medical image segmentation model supervised by the weakly annotated data can effectively reduce the burden of manual annotation. According to statistics, it takes about 240 seconds to annotate each image with complete annotation data, and it only takes 20 to 35 seconds to annotate the image with weak annotation data, saving about 9 to 12 times of annotation time.

[0050] As shown in Figure 1 The scribble annotation is a kind of weak annotation, compared with all the pixels in the complete annotation data having corresponding labels, the scribble annotation data only contains a small amount of annotated pixels, and most of the pixels have no corresponding labels; the scribble annotation has high annotation freedom, contains accurate annotated pixels and does not introduce annotation noise, and is suitable for annotating medical anatomical structures.

[0051] Since the pixel value difference between adjacent tissues of the medical image is small, it is difficult to distinguish organs or structures with similar pixel value distribution. When training the medical image segmentation model by using the scribble data, the existing training method has the problem that the annotated pixels contain errors during iterative updating, which leads to fitting to the wrong information and poor optimization effect.

[0052] The present application provides a medical image segmentation model training method based on scribble annotation, which uses scribble annotation to supervise the training of the medical image segmentation model, does not need to use complete annotation data, and can effectively reduce the cost of medical image data annotation. By using the weight-shared double network structure, consistent regularization training is performed to maximize the similarity of the predictions of different views of the same image. The optimization effect is good, and the network parameter optimization is completed in one training, without the need for multiple iterative training, so the training efficiency is high. It is suitable for segmentation of various medical anatomical structures, such as abdominal organs, head and neck organs, prostate and heart structures, etc.

[0053] Exemplary method

[0054] The embodiment of the present application provides a medical image segmentation model training method based on scribble annotation, which is deployed on an intelligent terminal and used for image segmentation of medical images of abdominal organs. Specifically, as shown in Figure 2 The training method comprises the following steps:

[0055] Step S100: performing a first image enhancement operation on the scribble annotated medical image to obtain a standard enhanced view, and performing a second image enhancement operation on the standard enhanced view to obtain a composite enhanced view;

[0056] Specifically, due to the small data set of medical images, the problem of insufficient training samples is widespread. In order to expand the training data set, first, the medical images after scribble annotation are subjected to image enhancement operation.

[0057] In the prior art, such as the iterative training method, the training method based on regularization constraint and the training method based on generative adversarial network, a single image enhancement operation is performed on the same medical image to obtain a single enhanced view. However, under the concept of double network structure, two different image enhancement operations are adopted on the same medical image and the semantic consistency between different views is limited, that is, the enhanced views obtained by the two different image enhancement operations can output the same semantic label during subsequent feature extraction, so as to ensure that the consistency regularization effectively regulates the network parameter optimization.

[0058] The common image enhancement operations include the following types: geometric enhancement, noise enhancement and pixel enhancement. Geometric enhancement, also known as spatial enhancement, is used for geometric transformation operation on the image, such as scaling, flipping, rotating, shearing and translation. Pixel enhancement is used for adjusting the pixel attributes of the image, such as brightness, contrast, saturation and hue. Noise enhancement is used to add noise points in the image. When performing image enhancement operation, multiple types of enhancement operations can be combined, or multiple image enhancement operations can be performed under the same type.

[0059] In this embodiment, the medical image after scribble annotation is subjected to standard enhancement operation, which includes geometric enhancement operation and noise enhancement operation, to obtain a standard enhanced view. Then, pixel enhancement operation, i.e. composite enhancement operation, is performed on the medical image after standard enhancement operation to obtain a composite enhanced view. That is, the composite enhancement operation is applied to the standard enhanced view ω(x) again.

[0060] Reference Figure 3 As shown in the figure, the standard enhancement function ω(·) represents the standard enhancement operation, and the specific steps of image enhancement are as follows: first, normalize the medical image to have a mean value of 0 and a variance of 1; then sequentially perform multiple operations on the image, and the parameter value of each operation is subject to uniform distribution within a certain range and occurs with a certain probability, including scaling, elastic deformation, rotation, horizontal and vertical mirroring, and Gaussian noise and various operations.

[0061] The composite augmentation function β(·) represents a composite augmentation operation. This embodiment employs color distortion augmentation, applying it to consistency training based on medical images. The specific steps are as follows: Adjusting the image brightness involves adding a random value to the pixel intensity value, where the random value follows a uniform distribution U(-0.8, +0.8); Adjusting the image contrast involves multiplying the pixel intensity value by a random value, where the random value follows a uniform distribution U(1-0.8, 1+0.8), and then limiting the pixel intensity value to the upper and lower limits of the original pixel intensity value; Gamma augmentation is performed on the image, first applying minimum-maximum normalization, and then taking a random power of the pixel value, where the random number follows a uniform distribution U(1-0.8, 1+0.8). Here, δ = 1 represents the augmentation intensity. These three data augmentation operations are performed sequentially, each with a probability of 0.8.

[0062] The input image x is processed by the standard data augmentation function ω(·) and the composite augmentation function β(·) to produce two augmented views: the standard augmented view ω(x) and the composite augmented view β°ω(x).

[0063] This embodiment uses the standard enhancement function ω(·) and the composite enhancement function β(·) to generate two different views of the same image, and can ensure that consistency regularization effectively regulates network parameter optimization.

[0064] Optionally, in the composite enhancement function, one or more composite forms of color enhancement functions (such as Gaussian blur function or blend enhancement function, or a combination thereof) can be added. The Gaussian blur function smooths the image using a Gaussian convolution kernel; the blend enhancement function linearly weights the target image and another random image from the training dataset.

[0065] Step S200: Input the standard augmented view and the composite augmented view into one of the networks in the dual network structure to obtain pseudo labels and predicted labels respectively. Each network in the dual network structure is a medical image segmentation model, and the weights are shared between the two medical image segmentation models.

[0066] Specifically, the existing technology involves iteratively optimizing labeled data by: fixing the labeled data and updating the network parameters; and fixing the network parameters and updating the labeled data. Even after optimization by traditional image processing algorithms, the predicted labels of training images still contain incorrect labeled pixels, which can cause the network to fit to incorrect information. Furthermore, it requires multiple steps of updating network parameters and optimizing predicted labels, involving multiple training and label processing and storage operations, which is complex and time-consuming.

[0067] To address the aforementioned issues, the training framework in this embodiment includes two medical image segmentation models, forming a dual-network structure. Each network in the dual-network structure includes a medical image segmentation model, such as... Figure 3f θ (·) The network parameter optimization is completed in one training using the double network structure, non-iterative training can be realized, and network generalization to multiple labeled data (i.e., each training image corresponds to multiple labeled data) is allowed, effectively limiting the network fitting of the wrong labeled information.

[0068] The network weight parameters of the two medical image segmentation models are the same during the training process, i.e., the weight sharing between the two medical image segmentation models. The standard enhanced view ω(x) is input into the medical image segmentation model, and the generated prediction result is the pseudo label (i.e., the predicted complete labeled data); the composite enhanced view β°ω(x) is input into another medical image segmentation model, and the generated prediction result is the predicted label.

[0069] It should be noted that the network type and parameters of the medical image segmentation model are not specifically limited, and can be adjusted according to the attributes of different medical image data sets. The applicable network types include: "U" shaped network (UNet), "V" shaped network (VNet), residual network (ResNet) and the like; the modifiable network parameters include: the number of network layers, the number of channels of features, the size and step of convolution kernel, normalization algorithm (such as batch, layer or group normalization, etc.), rectifier function (such as ReLU or LeakyReLU, etc.), down-sampling algorithm (such as maximum pooling or step convolution) and scale, up-sampling algorithm (such as linear interpolation or transposed convolution) and scale, etc.

[0070] Step S300: calculating the similarity between the pseudo label and the predicted label to obtain a consistency regularization loss;

[0071] Step S400: constructing a loss function including the consistency regularization loss;

[0072] Specifically, the training methods based on regularization constraint in the prior art, such as Graph Cut and Dense Conditional Random Field, add a regularization term to the loss function of network training by constructing the regularization term. In the loss function, the partial cross-entropy function punishes the prediction of the labeled pixels (i.e., the pixels labeled by scribbles), and limits the prediction of the unlabeled pixels. The regularization term mainly adopts: the normalized cut is constructed as the regularization term; or the dense conditional random field and the segmentation technology combining the dense conditional random field and the normalized cut (kernel cut) are constructed as the regularization term. However, since the regularization term relies on the regional characteristics (such as the probability density function of the pixel value) of the image to distinguish different targets, it is difficult to distinguish organs or structures with similar pixel value distribution; and relies on the difference in pixel values between adjacent pixels to ensure the smoothness of the predicted label, that is, when adjacent pixels have different labels, the penalty obtained by a large difference in pixel values is small, and the penalty obtained by a small difference in pixel values is large. However, the case that the pixel value difference between adjacent tissues is small is very common in medical images, and therefore, the regularization term based on the graph cut may cause the edge region of the predicted label to overflow or be damaged (due to the small pixel value difference, the regularization term is excessively smooth). The regularization term based on the dense conditional random field algorithm optimizes the predicted label of the network, and since the predicted label of the network itself has error information, the optimization effect of the regularization term is limited.

[0073] The consistency regularization loss is used to predict the similarity between the label and the pseudo label. During network optimization, the consistency regularization loss is used to maximize the cross-entropy similarity between the pseudo label and the composite enhanced view prediction value, directly supervises the semantic label of the predicted pixel, and is superior to the regularization term (which limits the predicted label according to the pixel value characteristics, and does not directly punish the semantics of the predicted label). The consistency regularization loss is expressed as:

[0074]

[0075] wherein β(·) is a composite enhancement function. In the above formula, the pseudo label can be used for back propagation of the gradient for parameter updating.

[0076] Currently, the pseudo label is detached from the computational graph when consistency regularization is applied (i.e. there is no edge pointing to the pseudo label in the computational graph), and the gradient cannot be back propagated. The present application extends the consistency regularization to the image segmentation algorithm based on scribble annotation. The consistency regularization is applied to each training image, allowing the pseudo label to back propagate the gradient and update the parameters of the corresponding network. The input of the cross-entropy function in the consistency regularization loss (the pseudo label and the prediction of the composite augmented view) can back propagate the gradient, which makes the two inputs in the loss more consistent, ensuring the coherence and effectiveness of the network parameter update.

[0077] Optionally, in addition to the form of cross-entropy similarity, the consistency regularization loss can also use l1 loss, mean square error loss and KL divergence.

[0078] The formula of the l1 loss is:

[0079] The formula of the mean square error loss is:

[0080] The loss of the KL divergence is:

[0081] The consistency regularization loss is the key to limiting the semantic consistency of different predictions of the same image, network parameter update and fitting multiple labels, so the consistency regularization loss needs to be added to the loss function to train and optimize the network model. Of course, other loss terms can also be included in the loss function to construct the loss function together with the consistency regularization loss.

[0082] Further, the present embodiment also includes calculating the cross-entropy loss of the features extracted from the labeled pixels in the standard augmented view, obtaining the partial cross-entropy loss, and using the partial cross-entropy loss in the loss function to train the medical image segmentation model to generate the pseudo label.

[0083] Using the partial cross-entropy loss Training a single network is the baseline algorithm of the scribble annotation based segmentation technology. The partial cross-entropy loss includes the labeled pixels (i.e. scribble annotated pixels) in the summation term, but erases the unlabeled pixels (i.e. the penalty term for the unlabeled pixels is zero, with no gradient back propagation). The partial cross-entropy loss is represented as:

[0084]

[0085] wherein, represents the input image, and N represents the number of pixels in the image x. represents the scribble annotation, wherein K is the number of classes (including the background class). In the scribble annotation y, for the labeled pixels, It is a one-hot encoded vector; for unlabeled pixels, It is a zero vector. Therefore, in this setting, The cross-entropy of unlabeled pixels in the array is zero, and no gradients will be backpropagated. Let f represent the output vector, where f θ (·) represents a network with parameter θ, and ω(·) is the standard augmentation function, which predicts the standard augmented view. The generated pseudo-labels. CrossEntropy(p,q) = -p·logsoftmax(q) represents the cross-entropy function, where p and q are K-dimensional vectors. Let q... ′ =softmax(q), then the vector q ′ The i-th channel is represented as:

[0086] By using a dual-network structure with weight sharing, simultaneously optimizing the network using consistency regularization loss and training pseudo-labels using partial cross-entropy loss, the shortcomings of using partial cross-entropy loss alone can be effectively overcome. Training network f θ The performance is poor when (·).

[0087] Furthermore, unlike applying consistency regularization which uses predefined processing algorithms (such as setting thresholds and "sharpening" probability distributions) to obtain low-entropy labels, this invention uses entropy minimization loss to train the medical image segmentation model end-to-end, enabling the model to output low-entropy pseudo-labels to ensure their effectiveness. Under low-entropy label supervision, the decision boundary of the medical image segmentation model is located in a low-density region, which is beneficial for class differentiation. Therefore, this embodiment also includes calculating the entropy minimization loss of the pseudo-labels. Minimize the loss of entropy Used in the loss function.

[0088] Specifically, low-entropy pseudo-labels are pixel prediction vectors that approximate one-hot encoding. Using low-entropy pseudo-labels for consistency training causes the model's decision boundary to tend to lie in low-density regions, resulting in significant differences (i.e., obvious distinctions) in the mapping vectors of pixels from different categories. To obtain low-entropy pseudo-labels, entropy minimization loss is used. Measuring the uncertainty of a probability distribution: When the probability distribution is close to one-hot encoding (i.e., highly deterministic), the entropy minimizes the loss. The penalty is small; when the probability distribution tends to be uniform (i.e., the values ​​of each channel tend to be equal, and the determinism is weak), the entropy minimizes the loss. The penalty is significant. Entropy minimization loss. Represented as:

[0089]

[0090] wherein, is a pseudo label, and

[0091] Therefore, the penalty term of the pseudo label in the training process of the embodiment includes a part of cross-entropy loss and an entropy minimization loss The loss function is wherein r(t) is a rising function, and the expression of the rising function is: r(t) = exp(-η(1-t / T)), wherein t is a training round, T and η are hyperparameters. In the first T training rounds, r(t) rises from a very small positive value to 1, and then remains the value of 1 until the end of the training; the rising speed is controlled by η, and a larger η makes the rising process slower. In the embodiment, T is set to 80, and η is set to 8.

[0092] Step S500: training the medical image segmentation model according to the loss function until the loss function converges, to obtain the trained medical image segmentation model.

[0093] Specifically, after the loss function is constructed, the medical image segmentation model is trained by using the gradient descent method until the loss function converges, to obtain the trained medical image segmentation model. When used, the medical image of a specific anatomical structure is input into the trained medical image segmentation model, and the segmentation result of the medical image can be obtained.

[0094] As described above, the embodiment uses two medical image segmentation models to construct a double-network structure and the two medical image segmentation models share weights, and two kinds of standard enhancement operations and a composite enhancement operation are performed on the medical image with doodle annotation, to generate two different views of the same image. On this basis, the two enhanced views are each input into a medical image segmentation model to obtain a pseudo label and a predicted label, and a consistency regularization loss for measuring the similarity between the pseudo label and the predicted label is calculated, and the medical image segmentation model is trained and optimized according to the consistency regularization loss. Since the double-network structure is used, the “pseudo label generation” and the “network optimization” can be performed simultaneously in the training process, allowing the network to generalize to multiple labels and avoiding multiple iterative training; and the inputs (pseudo labels and predictions of the composite enhanced view) of the cross-entropy function in the consistency regularization loss can all backpropagate gradients, making the two inputs in the loss more consistent, ensuring the coherence and effectiveness of the network parameter update; and the medical image segmentation model outputs a low-entropy pseudo label, and under the supervision of the low-entropy label, the decision boundary of the network is located in the low-density region, which is conducive to class distinction.

[0095] It should be noted that the weight-shared double network structure and the consistency regularization loss of the application can better optimize the medical image segmentation model, and the entropy minimization loss, the partial cross-entropy loss and the composite enhancement operation can be combined or added arbitrarily (for example, all penalty terms are applied or only a certain penalty term is used), which can improve the performance of training the medical image segmentation model.

[0096] Based on the fact that there are few labeled pixels in the scribble annotation data, the training is prone to overfitting to such pixels, which affects the shape of the final predicted label (that is, the decision boundary of the model is overfitted to the scribble annotation pixels, so that the predicted label of the training image only appears near the scribble annotation, and the shape is irregular). The application further provides a feature pool for storing annotated pixel features, which can reduce the overfitting of the network to the scribble annotation pixels by using the feature pool to regularize the feature training.

[0097] Specifically, as shown in Figure 3 , the embodiment further introduces a feature pool after the output of the encoder of the left network (or in the auxiliary path) to regularize the feature learning. As shown in Figure 4 , the steps include:

[0098] Step A100: feature extraction is performed on the annotated pixels in the standard enhanced view to obtain hidden features;

[0099] Step A200: based on the weights of the hidden features and the annotated pixels, the features of the annotated pixels are obtained;

[0100] Specifically, the feature pool contains integrated feature vectors of each semantic category. These feature vectors are momentum moving averages of annotated pixel features. The feature pool is constructed based on the encoder features where f e (·) represents the encoder.

[0101] First, the feature f e (ω(x)) is upsampled, and then mapped to the hidden feature Consider that the index set pointing to the annotated pixels in the image annotation y is Then the feature of the annotated pixels is calculated by the following formula:

[0102]

[0103]

[0104] where s ik is a pixel The importance weight scalar. sim(p,q)=p / ‖p‖·q / ‖q‖ represents the inner product of p and q in L2 normalization, i.e., cosine similarity. From formula (5), we know that s ik With M k and z i The cosine similarity is inversely proportional to the feature M. k Feature vector z of pixels with low similarity i It has a high weight.

[0105] Step A300: Dynamically update the feature pool based on the features of labeled pixels;

[0106] Specifically, the formula for feature pool update is as follows:

[0107] M k ←αM k +(1-α)m k

[0108] in Initialize as a zero vector, α = 0.9 is the momentum coefficient, m k Features for labeled pixels. During training, m is used in the form of a momentum moving average. k Update the feature pool.

[0109] Step A400: Based on the same mapping module, calculate the partial cross-entropy between the hidden features and the graffiti annotations, as well as the cross-entropy between the identity matrix formed by the annotations in the feature pool and the feature pool, to obtain the auxiliary loss and the feature pool loss respectively; the auxiliary loss and the feature pool loss are also included when constructing the loss function.

[0110] Specifically, the same mapping module g(·) is used to map the hidden features z and the feature pool M. The mapping results are then input into two loss functions to obtain the auxiliary loss and the feature pool loss, respectively. The auxiliary loss and the feature pool loss are then added to the loss function to optimize the training of the medical image segmentation model.

[0111] The two loss functions are auxiliary loss functions. and feature pooling loss function in Let be the identity matrix, representing the labels of the feature pool. Feature pool loss. The weights of g(·) and the output matrix g(z) affect the hidden feature z, thereby regularizing feature learning.

[0112] Optionally, although the feature pool in this embodiment uses the encoder output of the left backbone network as hidden features, multi-scale encoder features can also be used simultaneously; feature pool regularization feature learning can also be added to the right backbone network; and other similarity measures and feature pool update methods can also be used.

[0113] After considering the auxiliary loss function and the feature pool loss function, the loss function of the embodiment is where r(t) is a rising function, and λ1 and λ2 are coefficients for balancing the importance of loss terms.

[0114] As described above, unlike the existing iterative training method, the training method based on regularization constraint, and the training method based on the generative adversarial network, the feature pool introduced in the embodiment can store the integrated features in the form of momentum moving average, and use the integrated features to regularize the encoder feature learning, thereby reducing the overfitting of network parameters to scribble labeled pixels.

[0115] Exemplary apparatus

[0116] As shown in Figure 5 Corresponding to the medical image segmentation model training method based on scribble labeling, the embodiment of the application also provides a medical image segmentation model training device based on scribble labeling, the device comprises:

[0117] The enhancement operation module 600 is configured to perform a first image enhancement operation on the medical image after scribble labeling to obtain a standard enhanced view, and perform a second image enhancement operation on the standard enhanced view to obtain a composite enhanced view.

[0118] The network mapping module 610 is configured to input the standard enhanced view and the composite enhanced view into one network of a double network structure respectively to obtain pseudo labels and prediction labels, each network of the double network structure is a medical image segmentation model, and the two medical image segmentation models share weights.

[0119] The consistency regularization loss module 620 is configured to calculate the similarity between the pseudo labels and the prediction labels to obtain a consistency regularization loss.

[0120] The loss function module 630 is configured to construct a loss function comprising the consistency regularization loss.

[0121] The network parameter updating module 640 is configured to train the medical image segmentation model according to the loss function until the loss function converges, and obtain a trained medical image segmentation model.

[0122] In the embodiment, the medical image segmentation model training device based on scribble labeling described above can refer to the corresponding description in the medical image segmentation model training method based on scribble labeling described above, and will not be described here.

[0123] Based on the above embodiment, the application further provides an intelligent terminal, and the principle block diagram thereof can be as shown in Figure 6The smart terminal shown in the figure includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the smart terminal is used to provide computing and control capabilities. The memory of the smart terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a medical image segmentation model training program based on scribble annotation. The internal memory provides an environment for the operating system and the medical image segmentation model training program based on scribble annotation in the non-volatile storage medium. The network interface of the smart terminal is used to communicate with external terminals through network connection. The medical image segmentation model training program based on scribble annotation is executed by the processor to implement the steps of any one of the medical image segmentation model training methods based on scribble annotation. The display screen of the smart terminal can be a liquid crystal display screen or an electronic ink display screen.

[0124] Those skilled in the art can understand that, Figure 6 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the smart terminal to which the present application scheme is applied. The specific smart terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0125] In one embodiment, a smart terminal is provided, which includes a memory, a processor, and a medical image segmentation model training program based on scribble annotation stored on the memory and executable on the processor. When the medical image segmentation model training program based on scribble annotation is executed by the processor, the following operation instructions are performed:

[0126] performing a first image enhancement operation on the scribble-annotated medical image to obtain a standard enhanced view, and performing a second image enhancement operation on the standard enhanced view to obtain a composite enhanced view;

[0127] inputting the standard enhanced view and the composite enhanced view into one network of a double-network structure respectively to obtain pseudo-labels and prediction labels respectively, each network of the double-network structure being a medical image segmentation model, and the two medical image segmentation models sharing weights;

[0128] calculating the similarity between the pseudo-labels and the prediction labels to obtain a consistency regularization loss;

[0129] constructing a loss function including the consistency regularization loss;

[0130] training the medical image segmentation model according to the loss function until the loss function converges, to obtain a trained medical image segmentation model.

[0131] Optionally, the entropy minimization loss of the pseudo label is calculated, and the loss function is constructed by further including the entropy minimization loss.

[0132] Optionally, when constructing the loss function, the consistency regularization loss and the entropy minimization loss are weighted according to an exponential form of the rising function, and the expression of the rising function is r(t)=exp(-η(1-t / T)), wherein t is a training round, and T and η are hyperparameters.

[0133] Optionally, the cross-entropy loss of the extracted features of the labeled pixels in the standard augmented view is calculated to obtain a partial cross-entropy loss, and the loss function is constructed by further including the partial cross-entropy loss.

[0134] Optionally, the first image enhancement operation is performed on the medical image with graffiti annotation to obtain a standard augmented view, and the second image enhancement operation is performed on the standard augmented view to obtain a composite augmented view, including:

[0135] The geometric enhancement operation and the noise enhancement operation are performed on the medical image with graffiti annotation to obtain a standard augmented view.

[0136] The color distortion enhancement operation is performed on the standard augmented view to obtain a composite augmented view.

[0137] Optionally, a feature pool for storing the features of the labeled pixels is further provided, and the training method further includes:

[0138] The feature extraction is performed on the labeled pixels in the standard augmented view to obtain hidden features.

[0139] The features of the labeled pixels are obtained based on the hidden features and the weights of the labeled pixels.

[0140] The feature pool is dynamically updated based on the features of the labeled pixels.

[0141] Based on the same mapping module, the partial cross-entropy between the hidden features and the graffiti annotation and the cross-entropy between the unit matrix formed by the annotation of the feature pool and the feature pool are calculated to obtain an auxiliary loss and a feature pool loss, respectively, and the loss function is constructed by further including the auxiliary loss and the feature pool loss.

[0142] Optionally, the feature pool is dynamically updated based on the features of the labeled pixels, including:

[0143] The feature pool is updated by using the momentum moving average method according to the set momentum coefficient and the features of the labeled pixels.

[0144] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a medical image segmentation model training program based on scribble annotation, and the medical image segmentation model training program based on scribble annotation, when executed by a processor, implements the steps of any medical image segmentation model training method based on scribble annotation provided by the embodiment of the present application.

[0145] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the execution sequence, and the execution sequence of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the above method embodiments, which will not be described here.

[0147] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0148] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0149] In the embodiments of the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the above-described apparatus / terminal device embodiments are merely schematic, for example, the division of the above modules or units is only a logical function division, and an actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0150] The integrated modules / units described above, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the above-mentioned computer program can be stored in a computer readable storage medium, and when the processor executes the computer program, the steps of the above-mentioned various method embodiments can be implemented. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The above-mentioned computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. capable of carrying the above-mentioned computer program code. It should be noted that the above-mentioned computer readable storage medium contains contents which can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0151] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand; it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for training a medical image segmentation model based on scribbled annotation, characterized in that, The training method comprises: performing a first image enhancement operation on the medical image with graffiti annotation to obtain a standard enhanced view, and performing a second image enhancement operation on the standard enhanced view to obtain a composite enhanced view; inputting the standard enhanced view and the composite enhanced view into one network of a double-network structure respectively to obtain pseudo-labels and prediction labels respectively, each network of the double-network structure being a medical image segmentation model, and the two medical image segmentation models sharing weights; calculating the similarity between the pseudo-labels and the prediction labels to obtain a consistency regularization loss; constructing a loss function comprising the consistency regularization loss; training the medical image segmentation model according to the loss function until the loss function converges to obtain a trained medical image segmentation model; a feature pool for storing features of annotated pixels is further provided, and the training method further comprises: extracting features of the annotated pixels in the standard enhanced view to obtain hidden features; obtaining features of the annotated pixels based on the weights of the hidden features and the annotated pixels; dynamically updating the feature pool based on the features of the annotated pixels; mapping the hidden features and the feature pool based on the same mapping function, calculating the partial cross-entropy between the mapping results of the hidden features and the graffiti annotation and the cross-entropy between a unit matrix representing the annotation of the feature pool and the mapping results of the feature pool to obtain an auxiliary loss and a feature pool loss respectively; and constructing the loss function further comprises the auxiliary loss and the feature pool loss. The dynamic updating of the feature pool based on the features of the annotated pixels comprises: updating the feature pool by using a momentum moving average method according to a set momentum coefficient and the features of the annotated pixels. 2.The scribble-based medical image segmentation model training method of claim 1, wherein, Further comprising calculating an entropy minimization loss of the pseudo-labels, and constructing the loss function further comprises the entropy minimization loss. 3.The scribble-based medical image segmentation model training method of claim 2, wherein, In constructing the loss function, the consistency regularization loss and the entropy minimization loss are weighted according to an exponential form of an ascending function, the expression of which is: wherein is a training epoch, and is a hyperparameter. 4.The scribble-based medical image segmentation model training method of claim 1, wherein, Further comprising calculating a cross-entropy loss of the extracted features of the annotated pixels in the standard enhanced view to obtain a partial cross-entropy loss; and constructing the loss function further comprises the partial cross-entropy loss. 5.The scribble-based medical image segmentation model training method of claim 1, wherein, The first image enhancement operation on the medical image with graffiti annotation to obtain a standard enhanced view and the second image enhancement operation on the standard enhanced view to obtain a composite enhanced view comprise: performing a geometric enhancement operation and a noise enhancement operation on the medical image with graffiti annotation to obtain the standard enhanced view; performing a color distortion enhancement operation on the standard enhanced view to obtain the composite enhanced view. 6.A medical image segmentation model training apparatus based on scribbled annotation, characterized by, The device comprises: an enhancement operation module configured to perform a first image enhancement operation on a medical image with graffiti annotation to obtain a standard enhanced view, and perform a second image enhancement operation on the standard enhanced view to obtain a composite enhanced view; a network mapping module configured to input the standard enhanced view and the composite enhanced view into one network of a double-network structure respectively to obtain pseudo-labels and prediction labels respectively, each network of the double-network structure being a medical image segmentation model, and the two medical image segmentation models sharing weights; a consistency regularization loss module configured to calculate the similarity between the pseudo-labels and the prediction labels to obtain a consistency regularization loss; The loss function module is configured to construct a loss function including the consistency regularization loss. The network parameter updating module is configured to train the medical image segmentation model according to the loss function until the loss function converges, and obtain a trained medical image segmentation model. There is also a feature pool for storing features of the labeled pixels. The features of the labeled pixels in the standard enhanced view are extracted to obtain hidden features. Based on the hidden features and the weights of the labeled pixels, the features of the labeled pixels are obtained. Based on the features of the labeled pixels, the feature pool is dynamically updated. Based on the same mapping function, the hidden features and the feature pool are mapped, the partial cross-entropy between the mapping results of the hidden features and the scribble labels and the cross-entropy between the unit matrix representing the feature pool labels and the mapping results of the feature pool are calculated, and the auxiliary loss and the feature pool loss are obtained respectively; when constructing the loss function, the auxiliary loss and the feature pool loss are also included. The features of the labeled pixels are used to dynamically update the feature pool, including: According to the set momentum coefficient and the features of the labeled pixels, the momentum moving average method is used to update the feature pool.

7. Intelligent terminal, characterized in that The intelligent terminal includes a memory, a processor, and a medical image segmentation model training program based on scribble annotation stored on the memory and executable on the processor. When the medical image segmentation model training program based on scribble annotation is executed by the processor, the steps of the medical image segmentation model training method based on scribble annotation according to any one of claims 1-5 are implemented.

8. A computer readable storage medium, characterized in that, The computer readable storage medium stores a medical image segmentation model training program based on scribble annotation. When the medical image segmentation model training program based on scribble annotation is executed by the processor, the steps of the medical image segmentation model training method based on scribble annotation according to any one of claims 1-5 are implemented.

Images