Image segmentation method and apparatus

Abstract

The application discloses an image segmentation method and device, and relates to the technical field of image processing. The image segmentation method comprises the following steps: acquiring an image to be segmented, wherein the image to be segmented comprises an annotated image and an unannotated image; inputting the annotated image and the unannotated image into an image segmentation model to obtain an image segmentation result corresponding to the image to be segmented; wherein the image segmentation model is obtained by training an initial segmentation model based on an image training set, the initial segmentation model comprises a learnable network branch and a non-learnable network branch, a learning prototype of the learnable network branch is consistent with a non-learnable prototype of the non-learnable network branch, and outputs of the learnable network branch and the non-learnable network branch are consistent. The application effectively improves the accuracy of image segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image segmentation method and device. BACKGROUND

[0002] Semantic segmentation is to assign a semantic label to each pixel in an image, so as to realize accurate segmentation of the image. At present, in order to save the time and cost of manual labeling, the semi-supervised learning method of using a small amount of labeled images and a large amount of unlabeled images to improve the model performance is more favored by people. However, the existing semi-supervised learning method is sensitive to noise from the wrong prediction pixels, which leads to noise accumulation, and thus is prone to cause low accuracy of image segmentation.

[0003] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide an image segmentation method, which aims to solve the technical problem of low accuracy of image segmentation under the semi-supervised learning method.

[0005] To achieve the above purpose, the present application provides an image segmentation method, which comprises the following steps:

[0006] Obtaining an image to be segmented, wherein the image to be segmented comprises a labeled image and an unlabeled image;

[0007] Inputting the labeled image and the unlabeled image into an image segmentation model to obtain an image segmentation result corresponding to the image to be segmented;

[0008] Wherein, the image segmentation model is obtained by training an initial segmentation model based on an image training set, the initial segmentation model comprises a learnable network branch and a non-learnable network branch, the learning prototype of the learnable network branch is consistent with the non-learnable prototype of the non-learnable network branch, and the outputs of the learnable network branch and the non-learnable network branch are consistent.

[0009] Optionally, before the step of inputting the labeled image and the unlabeled image into the image segmentation model to obtain the image segmentation result corresponding to the image to be segmented, the image segmentation method comprises:

[0010] Obtaining a segmentation loss value, model parameters and an image training set of the initial segmentation model at a current iteration number;

[0011] inputting the image training set into the initial segmentation model based on the model parameters, to generate a learning prototype of the learnable network branch and a non-learnable prototype of the non-learnable network branch, and to output a first prediction result of the learnable network branch and a second prediction result of the non-learnable network branch;

[0012] determining a prototype loss and an auxiliary training loss of the initial segmentation model according to the learnable prototype and the non-learnable prototype;

[0013] determining a branch loss of the initial segmentation model according to a current iteration number of the initial segmentation model, the first prediction result and the second prediction result;

[0014] calculating a total loss of the initial segmentation model according to the segmentation loss value, the prototype loss, the auxiliary training loss and the branch loss;

[0015] updating the non-learnable prototype according to the learnable prototype and the non-learnable prototype to obtain a new non-learnable model;

[0016] updating the model parameters of the initial segmentation model according to the total loss until the initial segmentation model converges to obtain an image segmentation model.

[0017] Optionally, before the step of obtaining the segmentation loss value, the model parameters and the image training set of the initial segmentation model at the current iteration number, the method comprises:

[0018] obtaining an image training set, wherein the image training set comprises a labeled image training set, and the labeled image training set comprises a first image training sample and a label corresponding to the first image training sample;

[0019] inputting the first image training sample into an initial segmentation model to obtain a predicted label;

[0020] calculating a segmentation loss value of the initial segmentation model at the current iteration number based on a preset segmentation loss function, the predicted label and the label.

[0021] Optionally, the image training set comprises a labeled image training set and an unlabeled image training set, and the step of inputting the image training set into the initial segmentation model based on the model parameters to obtain the learning prototype of the learnable network branch and the non-learnable prototype of the non-learnable network branch comprises:

[0022] performing image enhancement on the unlabeled image training set to obtain an enhanced image training set;

[0023] The labeled image training set and the unlabeled image training set are input into the learnable network branch of the initial segmentation model to generate a learnable prototype of the learnable network branch.

[0024] The labeled image training set and the enhanced image training set are input into the non-learnable network branch of the initial segmentation model to generate the non-learnable prototype of the non-learnable network branch.

[0025] Optionally, the step of inputting the labeled image training set and the unlabeled image training set into the learnable network branch of the initial segmentation model to generate a learnable prototype of the learnable network branch includes:

[0026] The labeled image training set and the unlabeled image training set are input into the first encoder in the learnable network branch of the initial segmentation model to obtain the first multi-scale feature map;

[0027] Obtain the first segmentation mask corresponding to the labeled image training set and the unlabeled image training set, and convert the first segmentation mask into a first symbolic distance map;

[0028] Based on the first multi-scale feature map and the first symbolic distance map, a learnable prototype is generated in the learnable network branch.

[0029] Optionally, the step of inputting the labeled image training set and the enhanced image training set into the non-learnable network branch of the initial segmentation model to generate the non-learnable prototype of the non-learnable network branch includes:

[0030] The labeled image training set and the enhanced image training set are input into the second encoder in the non-learnable network branch of the initial segmentation model to obtain the second multi-scale feature map.

[0031] Obtain the second segmentation mask corresponding to the labeled image training set and the enhanced image training set, and convert the second segmentation mask into a second symbolic distance map;

[0032] Based on the second multi-scale feature map and the second symbolic distance map, an unlearnable prototype in the unlearnable network branch is generated.

[0033] Optionally, the step of determining the prototype loss and auxiliary training loss of the initial segmentation model based on the learnable prototype and the non-learnable prototype includes:

[0034] Prototype consistency calculation is performed on the learnable prototype of the initial segmentation model to obtain the prototype loss of the initial segmentation model;

[0035] According to the learnable prototype and the unlearnable prototype, an average value of a pixel cross-entropy loss value of the image training set is calculated, and the average value is taken as an auxiliary training loss.

[0036] Optionally, the step of performing prototype consistency calculation on the learnable prototype of the initial segmentation model to obtain a prototype loss of the initial segmentation model comprises:

[0037] According to the learnable prototype, a positive sample set and a negative sample set of the learnable prototype are determined.

[0038] A preset temperature hyperparameter is obtained, and a basic consistency loss of the learnable prototype is calculated according to the preset temperature hyperparameter, the positive sample set and the negative sample set.

[0039] An uncertainty weight of the learnable prototype is obtained.

[0040] The basic consistency loss is weighted and summed according to the uncertainty weight to obtain the prototype loss of the initial segmentation model.

[0041] Optionally, the step of determining the branch loss of the initial segmentation model according to the current iteration number of the initial segmentation model, the first prediction result and the second prediction result comprises:

[0042] According to the first prediction result and the second prediction result, a corresponding output consistency loss is calculated.

[0043] According to the first prediction result and the second prediction result, a first prediction entropy of the learnable network branch and a second prediction entropy of the unlearnable network branch are determined.

[0044] According to the first prediction entropy and the second prediction entropy, an average uncertainty value between the learnable network branch and the unlearnable network branch is calculated.

[0045] According to the average uncertainty value, the output consistency loss and a preset trade-off coefficient, a branch loss of the initial segmentation model is calculated.

[0046] In addition, to achieve the above-mentioned purposes, the present application also provides an image segmentation device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program implements the steps of the method according to any one of the above-mentioned embodiments when executed by the processor.

[0047] In addition, to achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of the above.

[0048] The image segmentation method provided in the application comprises the following steps: obtaining an image to be segmented, wherein the image to be segmented comprises an annotated image and an unannotated image; inputting the annotated image and the unannotated image into an image segmentation model to obtain an image segmentation result corresponding to the image to be segmented; wherein the image segmentation model is obtained by training an initial segmentation model based on an image training set, the initial segmentation model comprises a learnable network branch and a non-learnable network branch, a learning prototype of the learnable network branch is consistent with a non-learnable prototype of the non-learnable network branch, and outputs of the learnable network branch and the non-learnable network branch are consistent. The consistency of the learning prototype of the learnable network branch and the non-learnable prototype of the non-learnable network branch in the initial segmentation model and the consistency of the outputs of the learnable network branch and the non-learnable network branch are ensured in the semi-supervised training process, so that the consistency of the prediction of the image segmentation model is strengthened, the uncertainty of the prediction is reduced, and the accuracy of image segmentation is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A device structure schematic diagram of a hardware running environment related to an embodiment of the application;

[0050] Figure 2 A flowchart of a first embodiment of the image segmentation method of the application;

[0051] Figure 3 A flowchart of a second embodiment of the image segmentation method of the application;

[0052] Figure 4 A scene schematic diagram of the image segmentation method of the application;

[0053] Figure 5 A scene diagram of the non-learnable prototype updating related to the application;

[0054] Figure 6 A scene schematic diagram of the learnable prototype and the non-learnable prototype related to an embodiment of the application;

[0055] Figure 7 A sample example diagram of the ACDC data set related to an embodiment of the application;

[0056] Figure 8 A sample example diagram of the ISIC data set related to an embodiment of the application;

[0057] Figure 9 A sample example graph of the BU dataset involved in the embodiments of the present application;

[0058] Figure 10 A sample example graph of the BM dataset involved in the embodiments of the present application;

[0059] Figure 11 A sample example graph of the BU dataset involved in the embodiments of the present application;

[0060] Figure 12 A sample example graph of the BU dataset involved in the embodiments of the present application;

[0061] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0063] The term "and / or" in the present application is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone.

[0064] The terms "first" and "second" and the like in the description and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe the specific order of the target objects.

[0065] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present concepts in a particular manner.

[0066] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. For example, a plurality of processing units means two or more processing units; a plurality of systems means two or more systems.

[0067] In order to better describe the technical solutions of the present application, some related contents of the prior art are given as follows:

[0068] In recent years, image segmentation technology has attracted widespread attention in various fields. For example, in the medical field, fully supervised medical image segmentation has made great progress. For example, the Fully Convolutional Network (FCN) is a milestone method in image segmentation. It is the first time to replace the fully connected layer with the convolutional layer for dense prediction, realizing end-to-end segmentation. U-Net extracts multi-scale features for medical image segmentation based on the encoder-decoder structure. Inspired by these representative methods, some variants and new architectures of U-Net achieve advanced performance. These excellent supervised segmentation techniques rely on a large number of annotated images, and due to the limitation of professional clinical knowledge and expensive cost, a large number of labeled images are not easy to obtain. In contrast, large-scale unlabeled medical images are easily available. Therefore, some research focuses on how to use unlabeled images for segmentation.

[0069] Semi-supervised learning methods use a small number of labeled images and a large number of unlabeled images to improve the performance of the model, and have achieved impressive results in image classification and semantic segmentation. The learning strategies of semi-supervised image segmentation include collaborative training using different views, consistency regularization using slight perturbations, pseudo-label technology using predictions as supervision information for training, entropy minimization strategy, etc. However, these methods only focus on the same position relationship between images, ignoring the information between pixels between labeled and unlabeled images.

[0070] To explore the information between image pixels, contrastive learning is proposed for image segmentation, aiming to improve the representation ability of the encoder, so that similar objects in the embedding space are close to each other, and dissimilar objects are far away from each other. The key strategy of contrastive learning is to define similar samples and dissimilar samples. In the image-level classification task, MoCo (Momentum Contrast) and SimCLR (A Simple Framework for Contrastive Learning of Visual Representations) get different views through different transformations. However, semantic segmentation needs to classify each pixel into the corresponding label. In order to solve this problem, pixel-level contrastive learning is used to improve the segmentation accuracy. GLC (Global-Local Context) uses global and local features of labeled images for contrastive learning. ContrastiveSeg (Cross-image Pixel-level Contrastive Learning) proposes to explore cross-image pixels to construct contrastive samples for pixel-level sampling from different labeled images. LEC uses predicted pseudo-labels to determine the class of samples. PCL (Prototypical Contrastive Learning) constructs contrastive samples for unsupervised prototype contrastive learning at the image level. ProtoSeg (Prototype Perspective) designs contrastive samples from pixel prototypes in labeled images, which is only applicable to supervised learning. 2 Seg uses data augmentation to construct consistency learning samples and pseudo-label construction to construct contrastive learning samples. In U 2 In PCL, unreliable pseudo-labels are used to sample negative samples for contrastive learning.

[0071] Since the predicted pseudo-labels may not be consistent with the true semantic labels, pixel contrastive learning in the semi-supervised case is sensitive to noise from mispredicted pixels, leading to noise accumulation. In addition, the memory and computational cost of pixel-level contrastive learning is O (N 2 )(N is the image size), and a complex filtering mechanism is usually needed when sampling negative and positive samples. PPS introduces prototypes into contrastive learning and uses the prediction of prototypes as pseudo-labels for contrastive learning. However, this method simply treats pixels in the same class as prototypes without fully exploiting the internal information of images. Existing semi-supervised learning methods are sensitive to noise from mispredicted pixels, leading to noise accumulation, which in turn can easily lead to low accuracy of image segmentation.

[0072] AsFigure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0073] Specifically, the image segmentation device in this application can be a PC (Personal Computer), a portable computer, a server, or other similar devices.

[0074] like Figure 1 As shown, the image segmentation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0075] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the image segmentation device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0076] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and computer programs.

[0077] exist Figure 1 In the image segmentation device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the image segmentation device of this application can be set in the image segmentation device, and the image segmentation device calls the computer program stored in the memory 1005 through the processor 1001 and executes the image segmentation method provided in the embodiment of this application.

[0078] It should be understood that the above description is only an example for better understanding the technical solutions of the embodiments and is not the only limitation of the embodiments.

[0079] With reference to Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a scene of a first embodiment of an image segmentation method of the present application.

[0080] The first embodiment of the present application provides an image segmentation method, which comprises the following steps:

[0081] In step S100, an image to be segmented is acquired, wherein the image to be segmented comprises an annotated image and an unannotated image.

[0082] In step S200, the annotated image and the unannotated image are input into an image segmentation model to obtain an image segmentation result corresponding to the image to be segmented.

[0083] The image segmentation model is obtained by training an initial segmentation model based on an image training set, wherein the initial segmentation model comprises a learnable network branch and a non-learnable network branch, the learning prototype of the learnable network branch is consistent with the non-learnable prototype of the non-learnable network branch, and the outputs of the learnable network branch and the non-learnable network branch are consistent.

[0084] In the embodiment, it should be noted that the image to be segmented comprises an annotated image and an unannotated image, wherein the annotated image is an image with a target annotation label. For example, the annotated image contains targets such as skin spots and wounds, and the corresponding annotation labels can be annotated respectively. Exemplarily, the annotation label can be embodied in the form of a segmentation mask, which is used to describe the area occupied by the target in the annotated image. Exemplarily, the annotated image and the unannotated image can contain the same target. That is, the annotated image and the unannotated image both contain the target corresponding to the same annotation label to be recognized. Thus, the image segmentation model can segment the target in the unannotated image based on the annotated image.

[0085] It should be noted in this embodiment that the image segmentation model is obtained by training an initial segmentation model based on an image training set. The initial segmentation model includes a learnable network branch and a non-learnable network branch. The learning prototype of the learnable network branch is consistent with the non-learnable prototype of the non-learnable network branch, and the outputs of the learnable network branch and the non-learnable network branch are consistent. It can be understood that the image training set includes a labeled image training set and an unlabeled image training set. Therefore, after training the initial segmentation model based on the image training set, the segmentation loss value, the prototype loss, the auxiliary training loss, and the branch loss of the initial segmentation model are determined, and the total loss of the initial segmentation model is calculated. The model parameters of the initial segmentation model are updated based on the total loss until the initial segmentation model converges to obtain the image segmentation model.

[0086] In the first embodiment of the present application, a to-be-segmented image is obtained, wherein the to-be-segmented image includes a labeled image and an unlabeled image. The labeled image and the unlabeled image are input into an image segmentation model to obtain an image segmentation result corresponding to the to-be-segmented image. The image segmentation model is obtained by training an initial segmentation model based on an image training set. The initial segmentation model includes a learnable network branch and a non-learnable network branch. The learning prototype of the learnable network branch is consistent with the non-learnable prototype of the non-learnable network branch, and the outputs of the learnable network branch and the non-learnable network branch are consistent. In the semi-supervised training process, the consistency of the learning prototype of the learnable network branch and the non-learnable prototype of the non-learnable network branch, and the consistency of the outputs of the learnable network branch and the non-learnable network branch are ensured. Therefore, the consistency of the prediction of the image segmentation model is strengthened, the uncertainty of the prediction is reduced, and the accuracy of image segmentation is effectively improved.

[0087] Further, with reference to Figure 3 The second embodiment of the present application provides an image segmentation method. Based on the first embodiment, before the step of inputting the labeled image and the unlabeled image into the image segmentation model to obtain the image segmentation result corresponding to the to-be-segmented image, the image segmentation method includes:

[0088] Step A10, obtaining the segmentation loss value, model parameters, and image training set of the initial segmentation model at the current iteration number;

[0089] Step A20, inputting the image training set into the initial segmentation model based on the model parameters, generating a learning prototype of the learnable network branch and a non-learnable prototype of the non-learnable network branch, and outputting a first prediction result of the learnable network branch and a second prediction result of the non-learnable network branch;

[0090] Step A30, determining a prototype loss and an auxiliary training loss of the initial segmentation model according to the learnable prototype and the non-learnable prototype;

[0091] Step A40, determining a branch loss of the initial segmentation model according to a current iteration number of the initial segmentation model, the first prediction result and the second prediction result;

[0092] Step A50, calculating a total loss of the initial segmentation model according to the segmentation loss value, the prototype loss, the auxiliary training loss and the branch loss;

[0093] Step A60, updating the non-learnable prototype according to the learnable prototype and the non-learnable prototype to obtain a new non-learnable model;

[0094] Step A70, updating model parameters of the initial segmentation model according to the total loss until the initial segmentation model converges to obtain an image segmentation model.

[0095] In this embodiment, it should be noted that the initial segmentation model is composed of an encoder (Encoder) and a decoder (Decoder), and the initial segmentation model is divided into two network branches, i.e., a learnable network branch and a non-learnable network branch, both of which are composed of an encoder and a decoder. Therefore, referring to Figure 4 , Figure 4 A scene diagram of the image segmentation method. Figure 4 A scene diagram of training the initial segmentation model to obtain an image segmentation model. Figure 4In the figure, the arrow Data Flow is the data flow direction, the arrow sampling Flow is the sampling flow direction, the arrow stop grad is the stop gradient, the arrow EMA (Exponential Moving Average, exponential moving average) Transfer is the exponential moving average transformation, and the arrow Loss Function is the loss function calculation. The Encoder is the encoder, the Decoder is the decoder, the Labeled is the labeled image training set in the image training set, and the Unlabeled is the unlabeled image training set in the image training set. By inputting the first image training sample Labeled in the labeled image training set into the encoder Encoder and the decoder Decoder of the initial segmentation model, the prediction label Prediction is obtained; and based on the preset segmentation loss function, the prediction label and the labeled label, the segmentation loss value l sup of the initial segmentation model at the current iteration number is calculated. Then, by sharing the model parameters (Share), the model parameters of the initial segmentation model at the current iteration number are obtained, and based on the model parameters, the image training set (i.e., the labeled image training set Labeled and the unlabeled image training set Unlabeled) is input into the initial segmentation model to obtain the corresponding multi-size feature map (FeatureMap) after the encoder. The image training set is directly input into the learnable network branch of the initial segmentation model, and the labeled image training set Labeled and the enhanced unlabeled image training set Unlabeled are input into the initial segmentation model after EMA Transfer conversion. Then, based on the corresponding multi-size feature map (Feature Map), the learnable prototype of the learnable network branch and the non-learnable prototype of the non-learnable network branch can be generated, and the learnable prototype and the non-learnable prototype constitute the corresponding prototype layer (Prototypical Layer). And output the corresponding first prediction result and second prediction result (Prediction). Wherein by sampling the multi-size feature map (Feature Map), and then performing feature comparison (Contrastive Feature) on the sampled features, the anchor feature AnchorFeature (i.e., the learnable prototype) is obtained, and the prototype loss l pc of all learnable prototypes is calculated. Then, according to the learnable prototype and the non-learnable prototype, the auxiliary training loss l aux brought by the second prediction result output by the non-learnable network branch for training the learnable network branch is calculated. Finally, by the first prediction result and the second prediction result, the branch loss l c of the initial segmentation model is determined.

[0096] As an example, the present application can be trained by obtaining an image training set, wherein the image training set includes a labeled image training set, and the labeled image training set includes a first image training sample and a corresponding labeled label of the first image training sample. The first image training sample is input into the encoder and decoder of the initial segmentation model to obtain a predicted label; and then based on a preset segmentation loss function, the predicted label and the labeled label, a segmentation loss value of the initial segmentation model at a current iteration number is calculated. Then by obtaining the model parameters of the initial segmentation model at the current iteration number, based on the model parameters, the image training set is input into the initial segmentation model to generate a learning prototype of the learnable network branch and a non-learnable prototype of the non-learnable network branch, and output a first prediction result of the learnable network branch and a second prediction result of the non-learnable network branch; according to the learnable prototype and the non-learnable prototype, a prototype loss and an auxiliary training loss of the initial segmentation model are determined. Wherein the prototype loss is a loss value representing the uncertainty and consistency of the learnable prototype and the non-learnable prototype, and the auxiliary training loss is a loss value representing that the second prediction result of the non-learnable network branch assists the training of the learnable network branch. Then according to the current iteration number of the initial segmentation model, the first prediction result and the second prediction result, a branch loss of the initial segmentation model is determined. Wherein the branch loss is a loss value representing the uncertainty and consistency between the first prediction result and the second prediction result of the learnable network branch and the non-learnable network branch. Thus, according to the segmentation loss value, the prototype loss, the auxiliary training loss and the branch loss, the total loss of the initial segmentation model is calculated. Wherein, the target loss function of the initial segmentation model for calculating the total loss includes a segmentation loss term, a prototype loss term, an auxiliary training loss term and a branch loss term. Exemplarily, the calculation formula of the target loss function is as follows:

[0097] l total =l sup +λ c l c +λ aux l aux +λ pc l pc

[0098] Wherein, l total is the total loss, l sup is the segmentation loss value, l c is the prototype loss value, l aux is the auxiliary training loss value, and l pc is the branch loss term. λ c , λ aux and λpc is a trade-off hyperparameter. The auxiliary hyperparameter λ aux and the branch hyperparameter λ pc can be obtained by experiment or using empirical values. Exemplarily, λ aux = 0.3 and λ pc = 0.01 can be set. The prototype hyperparameter λ c is determined by the current iteration number of the initial segmentation model, and the specific formula is as follows:

[0099]

[0100] where t represents the current iteration number.

[0101] Further, the embodiment updates the unlearnable prototype according to the learnable prototype and the unlearnable prototype, and obtains a new unlearnable model. Referring to Figure 5 , Figure 5 is a scene diagram related to the unlearnable prototype updating of the present application. The initial segmentation model is input with the labeled image training set, the unlabeled image training set, and the enhanced image training set obtained after image enhancement on the unlabeled image training set, to obtain a multi-scale feature map (i.e. Features in the figure). Then the multi-scale feature map is pooled (Pool) to obtain the average feature map (i.e. Prototype). α1 and α2 in the figure represent the learnable prototype and the unlearnable prototype respectively. The unlearnable prototype α2 of the unlearnable network branch is updated by the following formula, as shown below:

[0102] α2 = (1 - μ) α1 + μ α2

[0103] where μ is a trade-off parameter, and the value of the trade-off parameter can be obtained by experiment or using empirical values. Exemplarily, μ can be set to 0.99. The prototype layer α2 composed of the unlearnable prototype α2 can be a tensor with a size of C x K x PH. PH is the dimension of the embedding space, and c is the c-th class prototype, each class corresponding to K prototypes. represents a feature map from the encoder of the unlearnable network branch, is the feature of the i-th pixel in the feature map. Thus, the feature v i closest to the c-th class prototype can be found in the training to update the unlearnable prototype α2. In combination with the features of the feature map and the prototype layer α2, the prototype updating mechanism of the unlearnable prototype is:

[0104] α2 = (μ + γ) α2 + (1 - μ) α1 + (1 - γ) v i

[0105] wherein g is an update parameter, the value of which can be obtained by experiment or an empirical value, and exemplarily, g can be set as g = 0.999. Through the above prototype updating mechanism, the non-learnable prototype a2 not only absorbs the information of the learnable prototype a1, but also adjusts the prototype according to the relationship between the features v i in the embedding space. In this way, the prototype does not deviate from the center of the corresponding same class, and the prototype learned by the non-learnable prototype a2 is more effective, which helps to further improve the potential of prototype contrast learning.

[0106] Finally, the model parameters of the initial segmentation model are updated according to the total loss until the initial segmentation model converges, and an image segmentation model is obtained. The condition for the convergence of the initial segmentation model is that the total loss tends to zero or the current iteration number of the initial segmentation model reaches a preset threshold (such as 8000, 10000, 12000, etc.). The image segmentation model obtained in this way has consistency between the learning prototype of the learnable network branch and the non-learnable prototype of the non-learnable network branch, and consistency between the outputs of the learnable network branch and the non-learnable network branch.

[0107] Further, in another embodiment, the step of obtaining the segmentation loss value of the initial segmentation model at the current iteration number, the model parameters and the image training set includes:

[0108] Step B10, obtaining an image training set, wherein the image training set includes a labeled image training set, and the labeled image training set includes a first image training sample and a label corresponding to the first image training sample;

[0109] Step B20, inputting the first image training sample into the initial segmentation model to obtain a predicted label;

[0110] Step B30, calculating the segmentation loss value of the initial segmentation model at the current iteration number based on a preset segmentation loss function, the predicted label and the label.

[0111] In this embodiment, it should be noted that the preset segmentation loss function includes a cross-entropy loss term and a Dice (dice) loss term, wherein the Dice loss term is used to measure the similarity between the predicted label and the label.

[0112] Exemplarily, let denote the labeled image training set in the image training set, n l is the number of samples of the labeled image training set, let denote the unlabeled image training set in the image training set, n u is the number of samples of the unlabeled image training set. x i ,xj ∈ R H*W denotes a training image in an image training set with size H*W, y i ,∈{0,1} H*W*K denotes a label with k-class pixel-level of a corresponding image x i in the image training set. The initial segmentation network h is composed of an encoder f and a decoder g, so that h=f·g. Therefore, y i '=h(x i ) is the prediction of the model for the image x i in the image training set. The preset segmentation loss function is as follows:

[0113]

[0114] wherein l sup is the segmentation loss, L CE is the cross-entropy loss, L Dice is the Dice loss, y i is the labeled label, and y i ' is the predicted label.

[0115] Therefore, in the embodiment, an image training set is obtained, wherein the image training set includes a labeled image training set, and the labeled image training set includes a first image training sample and a labeled label corresponding to the first image training sample. The first image training sample is input into an initial segmentation model to obtain a predicted label. Based on a preset segmentation loss function, the predicted label and the labeled label, a segmentation loss value of the initial segmentation model at a current iteration number is calculated. Thus, the performance of the initial segmentation model at the current iteration number can be determined.

[0116] Further, in another embodiment, the image training set includes a labeled image training set and an unlabeled image training set. The step of inputting the image training set into the initial segmentation model based on the model parameters to obtain a learning prototype of the learnable network branch and a non-learnable prototype of the non-learnable network branch includes:

[0117] Step A21, image enhancement is performed on the unlabeled image training set to obtain an enhanced image training set;

[0118] Step A22, the labeled image training set and the unlabeled image training set are input into the learnable network branch of the initial segmentation model to generate a learnable prototype of the learnable network branch;

[0119] Step A23, the labeled image training set and the enhanced image training set are input into the non-learnable network branch of the initial segmentation model to generate a non-learnable prototype of the non-learnable network branch.

[0120] It should be noted in this embodiment that the image enhancement processing includes flip (Fip), cropping (Crop), color jitter (Color Jitter) and the like, and then EMA transformation is performed.

[0121] As an example, after the unannotated image training set is subjected to image enhancement to obtain an enhanced image training set, the annotated image training set and the unannotated image training set are input into the learnable network branch of the initial segmentation model to generate a learnable prototype of the learnable network branch. The annotated image training set and the enhanced image training set are input into the non-learnable network branch of the initial segmentation model to generate a non-learnable prototype of the non-learnable network branch. Thus, the sample amount input into the initial segmentation model can be enriched while maintaining the consistency of the unannotated image training set and the enhanced image training set, and the robustness of the model can be improved to a certain extent.

[0122] The step of inputting the annotated image training set and the unannotated image training set into the learnable network branch of the initial segmentation model to generate a learnable prototype of the learnable network branch includes:

[0123] Step B10, inputting the annotated image training set and the unannotated image training set into a first encoder in the learnable network branch of the initial segmentation model to obtain a first multi-scale feature map;

[0124] Step B20, obtaining a first segmentation mask corresponding to the annotated image training set and the unannotated image training set, and converting the first segmentation mask into a first signed distance map;

[0125] Step B30, generating a learnable prototype in the learnable network branch according to the first multi-scale feature map and the first signed distance map.

[0126] It should be noted in this embodiment that the first segmentation mask includes an annotation label corresponding to a first image training sample of a first image sample in the annotated image training set, and a first prediction result output by the learnable network branch after the unannotated image training set is input into the learnable network branch of the initial segmentation model.

[0127] As an example, the annotated image training set and the unannotated image training set are input into a first encoder in the learnable network branch of the initial segmentation model to obtain a first multi-scale feature map. The first multi-scale feature map is M∈R H*W*Dwhere H*W is the size of the images in the annotated image training set and the unannotated image training set, and D is the feature dimension. Since the feature dimension D of the feature map generated by the encoder is usually large, in order to reduce the memory overhead, the feature dimension of M can be reduced from D to a preset dimension (such as 64, 128, 256, etc., and 128 is preferred) by a projection layer. The preset dimension is 128, that is, the first multi-scale feature map is M f ∈R H*W*128 After obtaining the first segmentation mask corresponding to the annotated image training set and the unannotated image training set, the first segmentation mask is converted into a first signed distance map. The prototype (including the learnable prototype and the non-learnable prototype) in the embodiment is defined as the average feature of all pixels with the same boundary distance in the embedding space, where the embedding space is a space composed of feature vectors of all pixels in the images in the annotated image training set and the unannotated image training set. The distance between each position (pixel) in the image and the segmentation boundary is calculated by the following formula:

[0128]

[0129] where x, y are two different pixels in the first segmentation mask, is the boundary in the first segmentation mask, represents the outline of the target, S in and S out represent the area of the target and the other area, respectively. T(x) is the conversion from the segmentation mask to the signed distance map. Let represent the signed distance map of the c-th class in the first segmentation mask, represent the signed distance map The value of the pixel at the index i, where i ∈ {1, 2, …, H*W}. Since the prototype (including the learnable prototype and the non-learnable prototype) in the embodiment is defined as the average feature of all pixels with the same boundary distance in the embedding space, the integrated prototype is the mean feature of the same semantic sample features, which is calculated from the signed distance map , that is:

[0130]

[0131] where is the feature of the pixel at the index i of the feature map M f , and the prototype p c,j is the average feature of all pixel features of the c-th class with the same distance from the j-th pixel in the signed distance map. Therefore, according to the first multi-scale feature map and the first signed distance map, the learnable prototype in the learnable network branch is generated.

[0132] The step of inputting the annotated image training set and the enhanced image training set into the non-learnable network branch of the initial segmentation model to generate a non-learnable prototype of the non-learnable network branch comprises:

[0133] Step C10, inputting the annotated image training set and the enhanced image training set into a second encoder in the non-learnable network branch of the initial segmentation model to obtain a second multi-scale feature map;

[0134] Step C20, obtaining a second segmentation mask corresponding to the annotated image training set and the enhanced image training set, and converting the second segmentation mask into a second signed distance map;

[0135] Step C30, generating a non-learnable prototype in the non-learnable network branch according to the second multi-scale feature map and the second signed distance map.

[0136] Similarly, the second segmentation mask includes the annotation label corresponding to the first image training sample in the annotated image training set, and the second prediction result output by the non-learnable network branch after inputting the enhanced image training set into the non-learnable network branch of the initial segmentation model.

[0137] As an example, a second multi-scale feature map is obtained by inputting the annotated image training set and the enhanced image training set into a second encoder in the non-learnable network branch of the initial segmentation model. The second multi-scale feature map is M∈R H*W*D , where H*W is the size of the input image in the annotated image training set and the enhanced image training set, and D is the feature dimension. Since the feature dimension D of the feature map generated by the encoder is usually large. Therefore, in order to reduce the memory overhead, the feature dimension of M can be reduced from D to a preset dimension (such as 64, 128, 256, etc., and 128 is preferred) by a projection layer. The second multi-scale feature map is M f ∈R H*W*128 After obtaining the second segmentation mask corresponding to the annotated image training set and the enhanced image training set, and converting the second segmentation mask into a second signed distance map. The prototype (including the learnable prototype and the non-learnable prototype) in this embodiment is defined as the average feature of all pixels with equal boundary distance in the embedding space, where the embedding space is a space composed of feature vectors of all pixels in the annotated image training set and the unannotated image training set. The distance between each position (pixel) in the image and the segmentation boundary is calculated by the following formula:

[0138]

[0139] where x, y are two different pixels in the second segmentation mask, is the boundary in the second segmentation mask, representing the contour of the target, S in and S out represent the region of the target and the other region, respectively. T(x) is the conversion from the segmentation mask to the signed distance map. Let denote the signed distance map of the c-th class in the second segmentation mask, denote the signed distance map the value of the pixel at index i in the signed distance map, where i∈{1, 2, …, H×W}. Since the prototypes (including the learnable prototypes and the non-learnable prototypes) in this embodiment are defined as the average features of all pixels with the same distance to the boundary in the embedding space, the integrated prototype is the mean feature of the semantically identical sample features, which is calculated by the signed distance map , that is:

[0140]

[0141] where is the feature of the pixel at index i in the feature map M f , and the prototype p c,j is the average feature of all pixel features of the c-th class with the same distance to the pixel at index j in the signed distance map. Therefore, according to the second multi-scale feature map and the second signed distance map, the learnable prototypes in the learnable network branch are generated.

[0142] wherein the step of determining the prototype loss and the auxiliary training loss of the initial segmentation model according to the learnable prototypes and the non-learnable prototypes comprises:

[0143] Step A31, performing prototype consistency calculation on the learnable prototypes of the initial segmentation model to obtain the prototype loss of the initial segmentation model.

[0144] Step A32, calculating the average value of the pixel cross-entropy loss value of the image training set according to the learnable prototypes and the non-learnable prototypes, and taking the average value as the auxiliary training loss.

[0145] In this embodiment, it can be understood that the purpose of prototype contrast learning is to enhance the similarity between similar semantic prototypes in the image and to push prototypes of different semantic categories far away, as shown in Figure 6 . Figure 6 is a scene diagram of the learnable prototypes and the non-learnable prototypes involved in the embodiments of the present application. Figure 6Prototypes represented by similar color depths (corresponding triangles, circles, and rhombuses of the same color depth) are consistent, while different color depths represent different prototypes with separability. Since semantically similar prototypes should be close together and selected, and semantically different prototypes should be far apart and sampled, this application calculates the prototype consistency of the learnable prototypes of the initial segmentation model to obtain the prototype loss of the initial segmentation model. This prototype loss is then used as a component of the total loss of the initial segmentation model to improve the discriminative power of the prototypes and make intra-class features in the embedding space more compact and inter-class features more separable, thus enhancing the consistency of similar prototypes and the separability of different prototypes.

[0146] Based on the learnable prototype and the non-learnable prototype, the average pixel cross-entropy loss value of the image training set is calculated, and this average value is used as an auxiliary training loss to further mine the internal information of the unlabeled image training set. For example,

[0147] All prototypes form the prototype set P = {p c,j |c∈{1,2,…,C},j∈D c}, where set D c The set D contains all values ​​of the symbolic distance graph in class c. c It can be represented as:

[0148]

[0149] In this embodiment, the anchor features used for contrastive learning are obtained from the prototype set P. That is, each prototype in the prototype set P serves as an anchor feature.

[0150] For any prototype p in the prototype set P c, j Each of them has its corresponding positive sample set. And the prototype p c,j The negative samples are taken from the negative sample set. For the image training set, let v i ∈R PH This represents the feature vector of pixel i mapped by the encoder. PH is the dimension of the embedding space; for example, PH = 2^30. In the embedding space, there are a total of K*Q prototypes, where K is the number of classes of prototypes and Q is the number of prototypes per class.

[0151] make Representing the first of the prototypes i If there are 100 pixels, then Indicates negative sample v iThe sample set. Then, assuming that the prototype of matching pixel i is taken as a positive sample and other prototypes are taken as negative samples, the formula for calculating the cross-entropy loss (i.e., pixel cross-entropy loss) of pixel i in the image training set is:

[0152]

[0153] The predicted outputs of the non-learnable network branches are used as pseudo-labels to aid the training of the learnable branches. In summary, x∈R H*W The auxiliary training loss can be defined as:

[0154]

[0155] Therefore, the outputs of the non-learnable network branches and the learnable network branches will be affected by the auxiliary training loss. aux Make consistent predictions to effectively mine the internal information of unlabeled images.

[0156] The step of performing prototype consistency calculation on the learnable prototype of the initial segmentation model to obtain the prototype loss of the initial segmentation model includes:

[0157] Step D10: Based on the learnable prototype, determine the positive sample set and negative sample set of the learnable prototype;

[0158] Step D20: Obtain the preset temperature hyperparameter, and calculate the basic consistency loss of the learnable prototype based on the preset temperature hyperparameter, the positive sample set, and the negative sample set.

[0159] Step D30: Obtain the uncertainty weights of the learnable prototype;

[0160] Step D40: Based on the uncertainty weights, the basic consistency loss is weighted and summed to obtain the prototype loss of the initial segmentation model.

[0161] In this embodiment, it should be noted that the preset temperature hyperparameter is a parameter selected according to specific needs. For example, the preset temperature hyperparameter can be 0.05.

[0162] In this embodiment, it should also be noted that the key operation of prototype comparison learning is to maintain the prototype pc. ,j Positive sample set of the prototype The distance between them is closer, and the prototype p c,j Negative sample set that deviates from the prototype For example, for any prototype pc,j in the prototype set P, there is a corresponding set of positive samples. The negative samples of the prototype pc,j are taken from the negative sample set.

[0163] Thus, the positive sample set and the negative sample set of the learnable prototype can be determined according to the learnable prototype.

[0164] To achieve this goal, the embodiment obtains a preset temperature hyperparameter, and calculates a basic consistency loss of the learnable prototype according to the preset temperature hyperparameter, the positive sample set and the negative sample set, wherein the basic consistency loss is used to represent an uncertainty weighted contrast consistency loss of the prototype, and a calculation formula of the basic consistency loss is:

[0165]

[0166] H(p c,j )=-g(p c,j )logg(p c,j )

[0167]

[0168] Wherein pc,j, p+ and p- are the learnable prototype, the positive sample set and the negative sample set of the learnable prototype, respectively, and the basic consistency loss L pc (p c ,j) is normalized to [0,1], and τ is a preset temperature hyperparameter for controlling the probability distribution of the vector, according to the experiment, τ can be set to 0.05. Wherein, Indicates the contrast consistency loss when pc,j is used as an anchor point, and enhances the relationship between the prototype pc, j And the positive sample to improve the consistency of the intra-class prototype.

[0169] We can calculate the prototype contrast loss by taking all prototypes as anchor points. In addition, the present application assigns a greater uncertainty weight to the prototype with high uncertainty through the uncertainty weighted loss, wherein the high uncertainty is positively correlated with the uncertainty weight. The prototype loss l pc Indicates the uncertainty weighted sum of the basic consistency loss Lp c Of all prototypes. Thus, the present application obtains the uncertainty weight of the learnable prototype, and then weights and sums the basic consistency loss according to the uncertainty weight, to obtain the prototype loss of the initial segmentation model. The initial segmentation model of the present application improves the discriminative ability of the prototype through the prototype loss l pc , and makes the intra-class features in the embedding space more compact and the inter-class features more separable.

[0170] Wherein, the step of determining the branch loss of the initial segmentation model according to the current iteration number of the initial segmentation model, the first prediction result and the second prediction result comprises:

[0171] Step S41, according to the first prediction result and the second prediction result, the corresponding output consistency loss is calculated;

[0172] Step S42, according to the first prediction result and the second prediction result, the first prediction entropy of the learnable network branch and the second prediction entropy of the non-learnable network branch are determined;

[0173] Step S43, according to the first prediction entropy and the second prediction entropy, the average uncertainty value between the learnable network branch and the non-learnable network branch is calculated;

[0174] Step S44, according to the average uncertainty value, the output consistency loss and the preset trade-off coefficient, the branch loss of the initial segmentation model is calculated.

[0175] As an example, the corresponding output consistency loss can be calculated according to the first prediction result and the second prediction result. Wherein, the first prediction result can include a first prediction probability map p l , the second prediction result can include a second prediction probability map p u . The calculation formula of the output consistency loss of the first prediction probability map and the second prediction probability map is as follows:

[0176] l con =L dis (p l ,p u )

[0177] Wherein, l con represents the output consistency loss, L dis represents the mean square error (MSE).

[0178] In order to further reduce the prediction uncertainty of the network and mine more reliable information from the unlabeled image training set, an uncertainty consistency loss is designed to minimize the prediction uncertainty. Let be the prediction probability vector of the i-th pixel, where K is the total number of classes of the prototype. The first prediction entropy of the i-th pixel is calculated by the following formula:

[0179]

[0180] wherein k∈{0, 1,..., K-1} represents the k-th class. Wherein ε is a decimal greater than zero, to avoid the case of log0, which leads to the inability to calculate the number. In order to minimize the uncertainty of the output between the two network branches (the learnable network branch and the non-learnable network branch). The target loss function used in the initial segmentation model of the present application to calculate the total loss includes a branch loss term. The calculation formula of the branch loss term is:

[0181] l c =l con +λ u l u

[0182]

[0183] wherein, and are the first predicted entropy of the learnable network branch and the second predicted entropy of the non-learnable network branch at the i-th pixel, respectively. The average uncertainty value l u is obtained by averaging the first predicted entropy of the learnable network branch and the second predicted entropy of the non-learnable network branch (i.e. the uncertainty of the output of the two network branches). The preset trade-off coefficient λ u is a coefficient for balancing between uncertainty and consistency loss, which can be set to λ u = 0.1, for example. Thus, the initial segmentation model of the present application can effectively reduce the uncertainty of the prediction, further improve the prediction accuracy of the pseudo label (i.e. the second prediction result) output by the non-learnable branch, and thus realize accurate image segmentation.

[0184] The comparison results of the present application and six existing semi-supervised segmentation methods are as follows:

[0185] Firstly, the data sets used in the experiment include ACDC data set, ISIC data set, breast ultrasound (BU) data set and breast MRI (BM) data set.

[0186] The ACDC data set contains 1902 short-axis cardiac MR images from 100 patients. For example, the data set can be divided into a training set (1312 images) and a validation set (590 images) in a ratio of 7:3. See Figure 7 , Figure 7 for the sample example image of the ACDC data set involved in the embodiments of the present application.

[0187] The ISIC data set includes 2594 skin images. We use 1815 images for training and 779 images for validation. See Figure 8 , Figure 8A sample example image of the ISIC dataset involved in the embodiments of the present application.

[0188] The Breast Ultrasound (BU) dataset is a dataset composed of 977 breast ultrasound images. We split it into a training set and a validation set, where the training set includes 684 images and the validation set includes 293 images. See Figure 9 , Figure 9 A sample example image of the BU dataset involved in the embodiments of the present application.

[0189] The Breast MRI (BM) dataset is a dataset composed of 2000 breast MRI images, where 1400 images constitute the training set and 600 images constitute the validation set. See Figure 10 , Figure 10 A sample example image of the BM dataset involved in the embodiments of the present application.

[0190] In the experiments, two semi-supervised experimental settings were used. The first case is to use 10% labeled images and 90% unlabeled images as the training set. The second case is to use 5% labeled images and 95% unlabeled images as the training set.

[0191] This experiment will use seven different semi-supervised segmentation methods on the ACDC, ISBI, BU and BM datasets. These methods include MT (Mean Teacher), UA-MT (Uncertainty-aware Mean Teacher), EM (Entropy Minimization), CCT (Cross-consistency Training), CPS (Cross-pseudo label Supervision), U 2 PL (using unreliable pseudo labels) and the image segmentation model of the present application (i.e. Ours).

[0192] Referring to Figure 11 , Figure 11 A segmentation result image of different methods involved in the embodiments of the present application on the ACDC dataset. Figure 11 Some visual results of different methods using 136 labeled images and 1176 unlabeled images on the ACDC dataset are shown. We selected four methods, namely MT, CPS, U 2 PL and the image segmentation model of the present application for comparison. According to Figure 11 It can be seen that the Dice of the present application is 87.70, which is significantly higher than other methods. The prediction performance of the image segmentation model of the present application is not only significantly higher than MT, CPS and U 2 PL, but also the visual results are closer to the true label, with fewer missing and unexpected areas.

[0193] In addition, the performance comparison of various semi-supervised segmentation methods under two semi-supervised settings can be referred to Table 1 below:

[0194] Table 1 Performance of various semi-supervised segmentation methods under two semi-supervised settings

[0195]

[0196] In Table 1, |D l | represents the number of images in the labeled image training set. SupOnly represents the performance of the baseline model trained only on the labeled image training set. The experimental results of seven semi-supervised segmentation methods on the ACDC, ISBI, BU and BM datasets under two semi-supervised experimental settings are shown by Table 1. When using 10% of the labeled images, the image segmentation model of the present application is superior to other methods. In particular, when the training samples of labeled images in the training set are reduced (only 5% of the labeled images), the image segmentation model of the present application is very competitive and significantly better than other methods.

[0197] The quantitative results of the ablation study of the image segmentation model of the present application on the ACDC dataset are shown in Table 2 as follows:

[0198] Table 2 Ablation experimental results of the present application on the ACDC dataset

[0199]

[0200] Table 2 shows the ablation experimental results of the image segmentation model of the present application on the ACDC dataset, using 136 labeled images and 1176 unlabeled images. By using all the components listed in Table 2 (i.e. each loss term in the total loss) base consistency loss l con , branch loss l un , auxiliary training loss l aux and prototype loss l pc , the image segmentation model of the present application achieves the best performance. The accuracy of Dice and IoU shows that each component used in the image segmentation model of the present application has a positive contribution and can continuously improve the performance of the model.

[0201] The effects of different encoder layers in the image segmentation model of the present application are shown in Table 3 as follows:

[0202] Table 3 Effects of different encoder layers

[0203]

[0204] Table 3 shows the results from different encoder layers in the image segmentation model of this application. In the image segmentation model of this application, the intermediate feature maps (i.e., Conv3) of the learnable network branch are used as the first and second multi-scale feature maps to generate corresponding learnable and non-learnable prototypes for comparative learning. Compared to the intermediate feature maps, larger feature maps do not contain more high-level features, while smaller feature maps lose image details due to their low resolution. Therefore, using the intermediate layer (i.e., Conv3 29×29) in the image segmentation model, which contains more semantic information, can produce the best performance.

[0205] Reference Figure 12 , Figure 12 This is a visualization of pixel features obtained using different methods on the ACDC dataset, as described in the embodiments of this application. Figure 12 The dimensionality of pixel features in the training set can be reduced to 64 to 256 using the t-SNE algorithm, with 128 being the preferred value. The intensity of color can represent pixel categories. Figure 12 In the middle (a), there is no contrastive learning (SupOnly), and in (b), there is U. 2 PL, (c) is the method of this application. (and) Figure 12 Comparing (a) and (b) in the text, it is clear that... Figure 12 (c) in this paper has better intra-class compactness and inter-class separability, which shows the effectiveness of the prototype contrastive learning approach adopted in this application for image segmentation tasks.

[0206] Furthermore, this application also proposes a computer storage medium storing a computer program. When the computer program is executed by a processor, it implements the operations in the image segmentation method provided in the above embodiments. The specific steps will not be described in detail here.

[0207] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity / operation / object from another, and do not necessarily require or imply any such actual relationship or order between these entities / operations / objects; the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0208] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part can refer to the part of the method embodiment. The device embodiment described above is only illustrative, and the units described as separate components can or can not be physically separated. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present application. Those skilled in the art can understand and implement it without creative labor.

[0209] The above sequence numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0210] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, vehicle, or network device, etc.) execute the method described in each embodiment of the present application.

[0211] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image segmentation method, characterized in that, The image segmentation method includes the following steps: Obtain the image to be segmented, wherein the image to be segmented includes labeled images and unlabeled images; The labeled image and the unlabeled image are input into the image segmentation model to obtain the image segmentation result corresponding to the image to be segmented. Prior to the step of inputting the labeled image and the unlabeled image into the image segmentation model to obtain the image segmentation result corresponding to the image to be segmented, the image segmentation method includes: Obtain the segmentation loss value, model parameters, and image training set of the initial segmentation model at the current iteration number; Based on the model parameters, the image training set is input into the initial segmentation model to generate a learning prototype of the learnable network branch and a non-learnable prototype of the non-learnable network branch, and outputs the first prediction result of the learnable network branch and the second prediction result of the non-learnable network branch. Prototype consistency calculation is performed on the learnable prototype of the initial segmentation model to obtain the prototype loss of the initial segmentation model; Based on the learnable prototype and the non-learnable prototype, the average pixel cross-entropy loss value of the image training set is calculated, and the average value is used as the auxiliary training loss. The branch loss of the initial segmentation model is determined based on the current iteration number of the initial segmentation model, the first prediction result, and the second prediction result. The total loss of the initial segmentation model is calculated based on the segmentation loss value, the prototype loss, the auxiliary training loss, and the branch loss. Based on the learnable prototype and the non-learnable prototype, the non-learnable prototype is updated to obtain a new non-learnable model; Based on the total loss, update the model parameters of the initial segmentation model until the initial segmentation model converges to obtain the image segmentation model; The image segmentation model is obtained by training an initial segmentation model based on an image training set. The initial segmentation model includes a learnable network branch and a non-learnable network branch. The learning prototype of the learnable network branch is consistent with the non-learnable prototype of the non-learnable network branch, and the outputs of the learnable network branch and the non-learnable network branch are consistent.

2. The image segmentation method as described in claim 1, characterized in that, Before the step of obtaining the segmentation loss value, model parameters, and image training set of the initial segmentation model at the current iteration number, the following steps are included: Obtain an image training set, wherein the image training set includes a labeled image training set, and the labeled image training set includes a first image training sample and a label corresponding to the first image training sample; Input the first image training sample into the initial segmentation model to obtain the predicted label; Based on the preset segmentation loss function, the predicted label, and the labeled label, the segmentation loss value of the initial segmentation model at the current iteration number is calculated.

3. The image segmentation method as described in claim 1, characterized in that, The image training set includes a labeled image training set and an unlabeled image training set. The step of inputting the image training set into the initial segmentation model based on the model parameters to obtain the learning prototype of the learnable network branch and the non-learnable prototype of the non-learnable network branch includes: Image augmentation is performed on the unlabeled image training set to obtain an augmented image training set; The labeled image training set and the unlabeled image training set are input into the learnable network branch of the initial segmentation model to generate a learnable prototype of the learnable network branch. The labeled image training set and the enhanced image training set are input into the non-learnable network branch of the initial segmentation model to generate the non-learnable prototype of the non-learnable network branch.

4. The image segmentation method as described in claim 3, characterized in that, The step of inputting the labeled image training set and the unlabeled image training set into the learnable network branch of the initial segmentation model to generate a learnable prototype of the learnable network branch includes: The labeled image training set and the unlabeled image training set are input into the first encoder in the learnable network branch of the initial segmentation model to obtain the first multi-scale feature map; Obtain the first segmentation mask corresponding to the labeled image training set and the unlabeled image training set, and convert the first segmentation mask into a first symbolic distance map; Based on the first multi-scale feature map and the first symbolic distance map, a learnable prototype is generated in the learnable network branch.

5. The image segmentation method as described in claim 3, characterized in that, The step of inputting the labeled image training set and the enhanced image training set into the non-learnable network branch of the initial segmentation model to generate the non-learnable prototype of the non-learnable network branch includes: The labeled image training set and the enhanced image training set are input into the second encoder in the non-learnable network branch of the initial segmentation model to obtain the second multi-scale feature map. Obtain the second segmentation mask corresponding to the labeled image training set and the enhanced image training set, and convert the second segmentation mask into a second symbolic distance map; Based on the second multi-scale feature map and the second symbolic distance map, an unlearnable prototype in the unlearnable network branch is generated.

6. The image segmentation method as described in claim 5, characterized in that, The step of calculating the prototype consistency of the learnable prototype of the initial segmentation model to obtain the prototype loss of the initial segmentation model includes: Based on the learnable prototype, determine the positive and negative sample sets of the learnable prototype; Obtain a preset temperature hyperparameter, and calculate the basic consistency loss of the learnable prototype based on the preset temperature hyperparameter, the positive sample set, and the negative sample set. Obtain the uncertainty weights of the learnable prototype; Based on the uncertainty weights, the basic consistency loss is weighted and summed to obtain the prototype loss of the initial segmentation model.

7. The image segmentation method as described in claim 1, characterized in that, The step of determining the branch loss of the initial segmentation model based on the current iteration number of the initial segmentation model, the first prediction result, and the second prediction result includes: Based on the first prediction result and the second prediction result, the corresponding output consistency loss is calculated; Based on the first prediction result and the second prediction result, determine the first prediction entropy of the learnable network branch and the second prediction entropy of the non-learnable network branch; The average uncertainty value between the learnable network branch and the non-learnable network branch is calculated based on the first prediction entropy and the second prediction entropy. The branch loss of the initial segmentation model is calculated based on the average uncertainty value, the output consistency loss, and the preset tradeoff coefficient.

8. An image segmentation device, characterized in that, The image segmentation device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 7.

Images