Training method of image segmentation model, image segmentation method, device and equipment

Abstract

The application discloses a kind of training method of image segmentation model, image segmentation method, device and equipment, belong to artificial intelligence technical field.Method includes: obtaining annotated image set and unannotated image set, annotated image set includes first image and the annotated image of first image, unannotated image set includes second image;According to first network model, the predicted image of first image is obtained;According to second network model, the predicted image of second image is obtained;Based on the reference image of second image and the annotated image of first image, the reference image of second image is determined;Based on the predicted image of first image, the annotated image of first image, the predicted image of second image and the reference image of second image, according to first network model, determine image segmentation model.The embodiment of the application reduces the number of images that need to be labeled, reduces time consumption and labor consumption, reduces cost, improves model training speed, thereby improves image segmentation efficiency.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and particularly relate to a training method of an image segmentation model, an image segmentation method, an apparatus and a device. BACKGROUND

[0002] With the development of artificial intelligence technology, network models are applied more and more widely, for example, the network model is applied in the technical field of image processing, and a target region in an image is identified through the network model. Such a network model for identifying a target region in an image is called an image segmentation model.

[0003] In the related art, a large number of images need to be obtained, and training samples are obtained by manually labeling the images, and an image segmentation model is obtained based on the training samples. However, for technical fields such as medical imaging, a large amount of time and manpower is required to manually label images, which increases the cost and also affects the training speed of the image segmentation model, thereby affecting the image segmentation efficiency. SUMMARY

[0004] Embodiments of the present application provide a training method of an image segmentation model, an image segmentation method, an apparatus and a device, which can be used to solve the problems of large consumption of time and manpower, slow training speed of the image segmentation model and low image segmentation efficiency in the related art. The technical solution includes the following contents.

[0005] In one aspect, the embodiments of the present application provide a training method of an image segmentation model, which comprises:

[0006] obtaining a labeled image set and an unlabeled image set, the labeled image set comprising a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set comprising a second image;

[0007] obtaining a predicted image of the first image according to a first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image;

[0008] obtaining a predicted image of the second image according to a second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image;

[0009] determining a reference image of the second image based on the second image and the labeled image of the first image, the reference image of the second image being an image segmentation result obtained by calculating the second image;

[0010] determine an image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image.

[0011] In another aspect, an embodiment of the present application provides an image segmentation method, the method comprising:

[0012] obtaining an image to be segmented;

[0013] obtaining an image segmentation result of the image to be segmented according to an image segmentation model, the image segmentation model being obtained based on the training method of any of the image segmentation models.

[0014] In another aspect, an embodiment of the present application provides a training device of an image segmentation model, the device comprising:

[0015] an obtaining module, configured to obtain a labeled image set and an unlabeled image set, the labeled image set comprising a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set comprising a second image;

[0016] the obtaining module is further configured to obtain a predicted image of the first image according to a first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image;

[0017] the obtaining module is further configured to obtain a predicted image of the second image according to a second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image;

[0018] a determining module, configured to determine a reference image of the second image based on the second image and the labeled image of the first image, the reference image of the second image being an image segmentation result obtained by calculating the second image;

[0019] the determining module is further configured to determine an image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image.

[0020] In a possible implementation, the determining module is configured to determine the foreground prototype of the first image and the background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being a reference feature of a first region in the first image, and the background prototype of the first image being a reference feature of a region other than the first region in the first image; and determine the reference image of the second image based on the foreground prototype of the first image, the background prototype of the first image, and the second image.

[0021] In a possible implementation, the determining module is configured to obtain a feature map of the first image, the feature map of the first image being used to represent semantic information of the first image; and determine the foreground prototype of the first image and the background prototype of the first image based on the feature map of the first image and the labeled image of the first image.

[0022] In a possible implementation, the determining module is configured to obtain a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; and determine the reference image of the second image based on the foreground prototype of the first image, the background prototype of the first image, and the feature map of the second image.

[0023] In a possible implementation, the determining module is configured to determine the loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image; and determine the image segmentation model according to the first network model based on the loss value of the first network model.

[0024] In a possible implementation, the determining module is configured to determine a first loss value based on the predicted image of the first image and the labeled image of the first image; determine a second loss value based on the predicted image of the second image and the reference image of the second image; and determine the loss value of the first network model based on the first loss value and the second loss value.

[0025] In a possible implementation, the determining module is configured to determine model parameters of the first network model and model parameters of the second network model; and determine the loss value of the first network model based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

[0026] In a possible implementation, the model parameters of the second network model are determined based on the model parameters of the first network model.

[0027] In a possible implementation, the determining module is further configured to determine a reference image of the first image based on the predicted image of the first image and the predicted image of the second image, the reference image of the first image being an image segmentation result obtained by performing calculation on the first image.

[0028] The determining module is further configured to determine the loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image, and the reference image of the second image.

[0029] In a possible implementation, the determining module is configured to determine a foreground prototype of the second image and a background prototype of the second image based on the predicted image of the second image, the foreground prototype of the second image being a reference feature of a second region in the second image, and the background prototype of the second image being a reference feature of a region other than the second region in the second image; and determine the reference image of the first image based on the foreground prototype of the second image, the background prototype of the second image, and the first image.

[0030] In a possible implementation, the determining module is configured to obtain a feature map of the second image; and determine the foreground prototype of the second image and the background prototype of the second image based on the feature map of the second image and the predicted image of the second image.

[0031] In a possible implementation, the determining module is configured to obtain a feature map of the first image; and determine the reference image of the first image based on the foreground prototype of the second image, the background prototype of the second image, and the feature map of the first image.

[0032] In a possible implementation, the determining module is configured to determine a first loss value based on the predicted image of the first image and the labeled image of the first image; determine a second loss value based on the predicted image of the second image and the reference image of the second image; determine a third loss value based on the labeled image of the first image and the reference image of the first image; and determine the loss value of the first network model based on the first loss value, the second loss value, and the third loss value.

[0033] In another aspect, an embodiment of the present application provides a training device of an image segmentation model, the device comprising:

[0034] A first obtaining module is configured to obtain a to-be-segmented image.

[0035] The second acquisition module is configured to acquire an image segmentation result of the image to be segmented according to the image segmentation model.

[0036] In a possible implementation, the second acquisition module is configured to acquire a feature map of the image to be segmented according to the image segmentation model, where the feature map of the image to be segmented is used to represent semantic information of the image to be segmented; and the image segmentation result of the image to be segmented is determined based on the feature map of the image to be segmented.

[0037] In another aspect, an electronic device is provided, which includes a processor and a memory. The memory stores at least one piece of program code. The at least one piece of program code is loaded and executed by the processor, so that the electronic device implements the training method of the image segmentation model or the image segmentation method according to any of the above embodiments.

[0038] In another aspect, a computer-readable storage medium is also provided. The computer-readable storage medium stores at least one piece of program code. The at least one piece of program code is loaded and executed by a processor, so that the computer implements the training method of the image segmentation model or the image segmentation method according to any of the above embodiments.

[0039] In another aspect, a computer program or computer program product is also provided. The computer program or computer program product stores at least one piece of computer instruction. The at least one piece of computer instruction is loaded and executed by a processor, so that the computer implements the training method of the image segmentation model or the image segmentation method according to any of the above embodiments.

[0040] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0041] The technical solutions provided by the embodiments of the present application are based on the annotated image of the first image and the second image, to determine the reference image of the second image, to realize the propagation of the image segmentation result obtained by annotation in the annotated data set to the unannotated data set, thereby reducing the number of images to be annotated, reducing the time consumption and the labor consumption, and reducing the cost. Then, the image segmentation model is obtained through the predicted image of the first image and the annotated image, and the predicted image of the second image and the reference image, to accelerate the training speed of the image segmentation model and improve the efficiency of image segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0043] Figure 1 is a schematic diagram of an implementation environment of a training method of an image segmentation model or an image segmentation method provided by an embodiment of the present application.

[0044] Figure 2 is a flowchart of a training method of an image segmentation model provided by an embodiment of the present application.

[0045] Figure 3 is a flowchart of an image segmentation method provided by an embodiment of the present application.

[0046] Figure 4 is a schematic diagram of training of an image segmentation model provided by an embodiment of the present application.

[0047] Figure 5 is a schematic diagram of an image segmentation result of a brain tumor image provided by an embodiment of the present application.

[0048] Figure 6 is a schematic diagram of an image segmentation result of a kidney image provided by an embodiment of the present application.

[0049] Figure 7 is a schematic diagram of a structure of a training device of an image segmentation model provided by an embodiment of the present application.

[0050] Figure 8 is a schematic diagram of a structure of an image segmentation device provided by an embodiment of the present application.

[0051] Figure 9 is a schematic diagram of a structure of a terminal device provided by an embodiment of the present application.

[0052] Figure 10 is a schematic diagram of a structure of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail with reference to the drawings.

[0054] Figure 1 is a schematic diagram of an implementation environment of a training method of an image segmentation model or an image segmentation method provided by an embodiment of the present application, like Figure 1The implementation environment shown includes an electronic device 11. The training method of an image segmentation model or the image segmentation method in the embodiments of the present application can be executed by the electronic device 11. Exemplarily, the electronic device 11 can include at least one of a terminal device or a server.

[0055] The terminal device can be at least one of a smartphone, a game console, a desktop computer, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, and a laptop computer.

[0056] The server can be a server, or a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center, and the embodiments of the present application do not limit this. The server can be connected to the terminal device through a wired network or a wireless network. The server can have functions of data processing, data storage, and data transmission, and the embodiments of the present application do not limit this.

[0057] The training method of an image segmentation model or the image segmentation method of the optional embodiments of the present application is implemented based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology and application system of using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0058] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation, etc.

[0059] Computer Vision (CV) is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics so that the computer processing becomes images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, Optical Character Recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional (3D) object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0060] Based on the above implementation environment, the embodiment of the present application provides a training method of an image segmentation model, so as to Figure 2 The flowchart of the training method of the image segmentation model provided by the embodiment of the present application is taken as an example. The method can be executed by the electronic device 11 in Figure 1 As shown in Figure 2 The method includes steps 201 to 205.

[0061] Step 201, obtaining a labeled image set and an unlabeled image set, the labeled image set including a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, the unlabeled image set including a second image.

[0062] Optionally, the number of the first images is at least one, and there is a labeled image of any first image. Optionally, the number of the second images is also at least one. It can be understood that the number of the second images is greater than, equal to, or less than the number of the first images. Since the processing methods of the first images are the same, the processing methods of the first images will be introduced from the perspective of the first images in each optional embodiment of the present application. Since the processing methods of the second images are also the same, the processing methods of the second images will be introduced from the perspective of the second images in each optional embodiment of the present application.

[0063] In the embodiment of the present application, the training data set includes a labeled image set and an unlabeled image set, the labeled image set including M first images and a labeled image of each first image, the unlabeled image set including N-M second images, and the number of the first images and the number of the second images being N. Wherein, N and M are positive integers.

[0064] The labeled image set can be represented as X l(i) is the i-th first image, Y l(i) is the i-th first image, and the unlabeled image set can be represented as X u(i) is the i-th second image.

[0065] Optionally, the first image and the second image are both two-dimensional images or both three-dimensional images, wherein the minimum segmentation unit of the two-dimensional image is a pixel, and the minimum segmentation unit of the three-dimensional image is a voxel (i.e., a volume element). In the embodiments of the present application, the first image and the second image are both three-dimensional images, and the implementation principle of the first image and the second image being both two-dimensional images is similar to that of the first image and the second image being both three-dimensional images, which will not be described here.

[0066] In the embodiments of the present application, X l(i) ,X u(i) ∈R H×W×D , Y l(i) ∈{0, 1} H×W×D R represents a real number, H represents the height of the three-dimensional image, W represents the width of the three-dimensional image, and D represents the depth of the three-dimensional image.

[0067] In step 202, a predicted image of the first image is obtained according to a first network model, and the predicted image of the first image is an image segmentation result obtained by predicting the first image.

[0068] Optionally, the first network model includes an encoder and a decoder. The first image is input into the first network model, the feature map of the first image is extracted by the encoder of the first network model, and the predicted image of the first image is determined based on the feature map of the first image by the decoder of the first network model. The feature map of the first image includes the voxel features of each voxel in the first image, and therefore, the feature map of the first image can represent the semantic information of the first image.

[0069] In the embodiments of the present application, the model parameters of the first network model are denoted as θ, and the model parameters of the first network model include the model parameters of the encoder and the model parameters of the decoder.

[0070] In step 203, a predicted image of the second image is obtained according to a second network model, and the predicted image of the second image is an image segmentation result obtained by predicting the second image.

[0071] Optionally, the second network model comprises an encoder and a decoder. The second image is input into the second network model, a feature map of the second image is extracted by the encoder of the second network model, and a predicted image of the second image is determined by the decoder of the second network model based on the feature map of the second image. The feature map of the second image comprises features of each voxel in the second image, and therefore, the feature map of the second image can represent semantic information of the second image.

[0072] In the embodiments of the present application, the model parameters of the second network model are denoted as The model parameters of the second network model comprise model parameters of the encoder and model parameters of the decoder. The model parameters of the second network model can be the same as the model parameters of the first network model (i.e., the second network model is the first network model), or the model parameters of the second network model can be different from the model parameters of the first network model (i.e., the second network model and the first network model are two different models, for example, the first network model is a student model, and the second network model is a teacher model).

[0073] In a possible implementation, the model parameters of the second network model are determined based on the model parameters of the first network model.

[0074] In the embodiments of the present application, the first network model can be a student model, and the second network model can be a teacher model. The model parameters of the teacher model are determined based on the model parameters of the student model, which realizes the construction of a self-integrated teacher model based on a student model, and can improve the accuracy of the prediction result of the teacher model.

[0075] Optionally, the model parameters of the second network model are adjusted based on the model parameters of the first network model according to an Exponential Moving Average (EMA) method, and the model parameters of the second network model are obtained. The model parameters of the second network model can be represented by formula (1) as shown below.

[0076]

[0077] wherein, is the model parameters of the second network model, is the model parameters of the third network model, θ t is the model parameters of the first network model, and α is a decay rate. The embodiments of the present application do not limit the value of the decay rate, and for example, the value of the decay rate is 0.99.

[0078] It can be understood that, since the first network model includes an encoder and a decoder, the second network model also includes an encoder and a decoder, and the second network model is obtained by adjusting the model parameters of the third network model, therefore, the third network model also includes an encoder and a decoder. In formula (1), The model parameters of the encoder (or the decoder) of the second network model can be adjusted. The model parameters of the encoder (or the decoder) of the third network model can be adjusted. t The model parameters of the encoder (or the decoder) of the first network model can be adjusted.

[0079] In step 204, a reference image of the second image is determined based on the second image and the labeled image of the first image, and the reference image of the second image is an image segmentation result obtained by calculating the second image.

[0080] In the embodiments of the present application, the reference image of the second image can be determined based on the second image, the first image and the labeled image of the first image. That is, for any second image, the reference image of the second image can be determined based on the second image, any first image and the labeled image of the first image.

[0081] In a possible implementation, the reference image of the second image is determined based on the second image and the labeled image of the first image, including: determining a foreground prototype of the first image and a background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being a reference feature of the first region in the first image, and the background prototype of the first image being a reference feature of the region other than the first region in the first image; determining the reference image of the second image based on the foreground prototype of the first image, the background prototype of the first image and the second image.

[0082] Optionally, the first image includes a first region and a region other than the first region. The first region in the first image can be referred to as a foreground region of the first image, and the foreground region is a region where a target object (such as a vehicle, a building, a tumor, a kidney, etc.) is located. The region other than the first region in the first image can be referred to as a background region of the first image, and the background region is a region that does not contain the target object. Generally, the image segmentation processing of the first image is to segment the foreground region of the first image from the first image.

[0083] In the embodiments of the present application, each voxel of the foreground region has a similar voxel feature, and one voxel feature can be used to express any voxel of the foreground region, which can be referred to as a reference feature. Based on the same principle, each voxel of the background region also has a similar voxel feature, and another reference feature can be used to express any voxel of the background region.

[0084] In the embodiments of the present application, the reference feature of the foreground region of the first image is referred to as the foreground prototype of the first image, and the reference feature of the background region of the first image is referred to as the background prototype of the first image. The foreground prototype of the first image and the background prototype of the first image can be determined based on the first image and the labeled image of the first image.

[0085] Optionally, the foreground prototype of the first image and the background prototype of the first image are determined based on the labeled image of the first image, comprising: obtaining a feature map of the first image, the feature map of the first image being used to represent semantic information of the first image; determining the foreground prototype of the first image and the background prototype of the first image based on the feature map of the first image and the labeled image of the first image.

[0086] In the embodiments of the present application, the first image is input into the first network model, and the feature map of the first image is extracted by the encoder of the first network model. The feature map of the first image is cross-multiplied with the labeled image of the first image to obtain a feature segmentation map of the first image, the feature segmentation map being a feature map in which the voxel features of the foreground region and the voxel features of the background region are segmented. The foreground prototype of the first image is determined based on the voxel features of the foreground region of the first image, and the background prototype of the first image is determined based on the voxel features of the background region of the first image.

[0087] The operation of determining the foreground prototype of the first image and the background prototype of the first image can also be referred to as a mask average pooling operation. The mask average pooling operation is to mask the voxel features of the foreground region (or the voxel features of the background region) extracted by the encoder by using the segmentation label, so as to obtain the background prototype (or the foreground prototype), so as to measure the similarity with the unlabeled image set based on the foreground prototype and the background prototype in the labeled image set by non-parametric metric learning, thereby segmenting the unlabeled image set.

[0088] In the embodiments of the present application, the feature map of the kth first image X l(k) is denoted as F l(k) , the labeled image of the kth first image is denoted as Y l(k) , the foreground region of the kth first image is denoted as C fg , and the spatial position of any voxel in the kth first image is denoted as (x, y, z). The foreground prototype of the first image is determined by the mask average pooling operation according to formula (2) as follows, and the background prototype of the first image is determined according to formula (3) as follows.

[0089]

[0090] wherein p l(fg) is the foreground prototype of the first image, K is the number of the first images, a voxel feature of a voxel with a spatial position of (x, y, z) in a feature map of the kth first image, a voxel information of a voxel with a spatial position of (x, y, z) in a labeled image of the kth first image, 1[·] is an indicator function with a return value of 1 when a condition is met, p l(bg) a background prototype of the first image.

[0091] It should be noted that the feature map of the first image needs to be up-sampled by trilinear interpolation so that the feature map of the first image after up-sampling is consistent in size with the labeled image of the first image. In this embodiment of the present application, the specific manner of trilinear interpolation is not limited.

[0092] Optionally, based on the foreground prototype of the first image, the background prototype of the first image and the second image, the reference image of the second image is determined, comprising: obtaining a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; based on the foreground prototype of the first image, the background prototype of the first image and the feature map of the second image, the reference image of the second image is determined.

[0093] In this embodiment of the present application, the second image is input into the second network model, and the feature map of the second image is extracted by the encoder of the second network model. Then, based on the foreground prototype of the first image, the background prototype of the first image and the feature map of the second image, the similarity between the feature map of the second image and the foreground prototype of the first image or the similarity between the feature map of the second image and the background prototype of the first image is calculated to obtain a similarity result. Subsequently, the similarity result is normalized by using a normalization exponential function (for example, a Softmax function) to obtain the reference image of the second image.

[0094] The operation of determining the reference image of the second image described above can also be referred to as an operation based on a non-parametric metric learning mechanism. The similarity with the unlabeled image set is measured by non-parametric metric learning, so that the unlabeled image set is segmented.

[0095] In this embodiment of the present application, the function used when calculating the similarity is denoted as a distance function d(·), and the feature map of the second image is denoted as F u , and the spatial position of any voxel in the feature map of the second image is denoted as (x, y, z). The foreground prototype of the first image is denoted as p l(fg) , the background prototype of the first image is denoted as p l(bg) , and P l ={p l(fg)}∪{p l(bg)}. For each P l(j∈(fg,bg)) ∈P l , the reference image of the second image is determined according to the formula (4) as follows.

[0096]

[0097] wherein, is a voxel feature of a voxel with spatial position (x, y, z) in the reference image of the second image, exp is an exponential function, d(·) is a distance function, is a voxel feature of a voxel with spatial position (x, y, z) in the feature map of the second image, p l(j) is a foreground prototype of the first image or a background prototype of the first image, P l includes the foreground prototype of the first image and the background prototype of the first image, ∑ is a symbol of a summation function, and a is a proportional coefficient. The specific value of the proportional coefficient is not limited in the embodiments of the present application. For example, the proportional coefficient is 20.

[0098] It should be noted that the feature map of the second image needs to be up-sampled by trilinear interpolation. In the embodiments of the present application, the specific manner of trilinear interpolation is not limited. In addition, the distance function is not limited in the embodiments of the present application. For example, the distance function is a cosine distance function.

[0099] In step 205, based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image, the image segmentation model is determined according to the first network model.

[0100] In the embodiments of the present application, the first network model is updated once based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image, to obtain an updated first network model. If the training end condition is met, the updated first network model is taken as the image segmentation model. If the training end condition is not met, the updated first network model is taken as the first network model for the next training, and the steps of steps 201 (or step 202) to step 205 are re-executed until the image segmentation model is obtained.

[0101] The embodiments of the present application do not limit the training end condition. For example, the training end condition is that the training times reach a target number. The value of the target number can be flexibly set according to artificial experience or application scenarios, which is not limited herein.

[0102] In a possible implementation, based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image, the image segmentation model is determined according to the first network model, including: based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image, a loss value of the first network model is determined; based on the loss value of the first network model, the image segmentation model is determined according to the first network model.

[0103] In the embodiments of the present application, the loss value of the first network model can be determined based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image. Then, the model parameters of the first network model are adjusted according to the loss value of the first network model, so as to realize one-time updating processing of the first network model, obtain an updated first network model, and obtain an image segmentation model based on the updated first network model.

[0104] Optionally, the loss value of the first network model is determined based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image, comprising: determining a first loss value based on the predicted image of the first image and the labeled image of the first image; determining a second loss value based on the predicted image of the second image and the reference image of the second image; and determining the loss value of the first network model based on the first loss value and the second loss value.

[0105] In the embodiments of the present application, the predicted image of the first image and the labeled image of the first image are input into the first loss function to obtain the first loss value. The first loss function is also called a supervised loss function, and the embodiments of the present application do not limit the first loss function. The first loss function includes at least one function type. For example, the first loss value is determined according to the formula (5) as follows.

[0106] L s = 0.5 * L ce (Y l , P l ) + 0.5 * L Dice (Y l , P l ) Formula (5)

[0107] Wherein, L s is the first loss value, L ce is the function symbol of the cross-entropy loss function, Y l is the predicted image of the first image, P l is the labeled image of the first image, L Dice is the function symbol of the Dice loss function, and the Dice loss function is a set similarity measure function, which can be used to calculate the similarity between two samples.

[0108] In the embodiments of the present application, the predicted image of the second image and the reference image of the second image are input into the second loss function to obtain the second loss value. The second loss function is also called a forward prototype consistency loss function, and the embodiments of the present application do not limit the second loss function. For example, the second loss value is determined according to the formula (6) as follows.

[0109] L fpc = L mse (P l2u , P u ) Formula (6)

[0110] wherein, L fpc is a second loss value, L mse is a function symbol of a Mean-Square Error (MSE) loss function, P l2u is a reference image of the second image, and P u is a predicted image of the second image.

[0111] After the first loss value and the second loss value are determined, the loss value of the first network model is determined based on the first loss value and the second loss value. Optionally, the loss value of the first network model is determined based on the first loss value, a weight of the first loss value, the second loss value, and a weight of the second loss value, and the application embodiments do not limit the weight of the first loss value and the weight of the second loss value.

[0112] Optionally, the loss value of the first network model is determined based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image, and includes: determining the model parameters of the first network model and the model parameters of the second network model; and determining the loss value of the first network model based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

[0113] In the application embodiments, the model parameters of the first network model and the model parameters of the second network model are obtained first. The relationship between the model parameters of the second network model and the model parameters of the first network model has been described above and will not be repeated here.

[0114] In determining the first loss value, the first loss value is determined based on the model parameters of the first network model, the predicted image of the first image, and the labeled image of the first image. In determining the second loss value, the second loss value is determined based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the second image, and the reference image of the second image. Then, the loss value of the first network model is determined based on the first loss value and the second loss value.

[0115] In a possible implementation, after obtaining the predicted image of the second image according to the second network model, the method further includes: determining a reference image of the first image based on the first image and the predicted image of the second image, the reference image of the first image being an image segmentation result obtained by performing calculation on the first image; and determining the loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image, including: determining the loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image, and the reference image of the second image.

[0116] In the embodiments of the present application, the reference image of the first image can be determined based on the first image, the second image, and the predicted image of the second image. That is, for any first image, the reference image of the first image can be determined based on the any first image, any second image, and the predicted image of the any second image.

[0117] Optionally, determining the reference image of the first image based on the first image and the predicted image of the second image includes: determining a foreground prototype of the second image and a background prototype of the second image based on the predicted image of the second image, the foreground prototype of the second image being a reference feature of the second region in the second image, and the background prototype of the second image being a reference feature of a region other than the second region in the second image; and determining the reference image of the first image based on the foreground prototype of the second image, the background prototype of the second image, and the first image.

[0118] The second image includes a second region and a region other than the second region. The second region in the second image can be referred to as a foreground region of the second image, and the foreground region is a region where a target object (such as a vehicle, a building, a tumor, a kidney, etc.) is located. The region other than the second region in the second image can be referred to as a background region of the second image, and the background region is a region that does not contain the target object. Generally, the image segmentation processing of the second image is to segment the foreground region of the second image from the second image.

[0119] In the embodiments of the present application, each voxel of the foreground region has a similar voxel feature, and one voxel feature can be used to express any voxel of the foreground region, which can be referred to as a reference feature. Based on the same principle, each voxel of the background region also has a similar voxel feature, and another reference feature can be used to express any voxel of the background region.

[0120] In the embodiments of the present application, the reference feature of the foreground region of the second image is referred to as the foreground prototype of the second image, and the reference feature of the background region of the second image is referred to as the background prototype of the second image. The foreground prototype of the second image and the background prototype of the second image can be determined based on the second image and the predicted image of the second image.

[0121] Optionally, the foreground prototype of the second image and the background prototype of the second image are determined based on the predicted image of the second image, including: obtaining a feature map of the second image; and determining the foreground prototype of the second image and the background prototype of the second image based on the feature map of the second image and the predicted image of the second image.

[0122] In the embodiments of the present application, the second image is input into the second network model, and a feature map of the second image is extracted by an encoder of the second network model. The feature map of the second image is cross-multiplied with the predicted image of the second image to obtain a feature segmentation map of the second image, and the feature segmentation map is a feature map in which the voxel features of the foreground region and the voxel features of the background region are segmented. The foreground prototype of the second image is determined based on the voxel features of the foreground region of the second image, and the background prototype of the second image is determined based on the voxel features of the background region of the second image.

[0123] The operation of determining the foreground prototype of the second image and the background prototype of the second image described above can also be referred to as a mask average pooling operation. Through the mask average pooling operation, the foreground prototype and the background prototype in the unannotated image set are used to measure the similarity with the annotated image set through non-parametric metric learning, so as to segment the annotated image set.

[0124] In the embodiments of the present application, the predicted image of the second image satisfies the formula (7) as follows

[0125]

[0126] wherein, is the predicted image of the second image, when j∈bg, P u(j) is the background region in the predicted image of the second image (i.e., the background region of the second image), when j∈fg, P u(j) is the foreground region in the predicted image of the second image (i.e., the foreground region of the second image), and argmax is a symbol of a function for parameterizing a function.

[0127] In the embodiments of the present application, the feature map of the kth second image X u(k) is denoted as F u(k) , and the predicted image of the kth second image is denoted as the foreground region of the kth second image is denoted as C fg, the spatial position of any voxel in the kth second image is denoted as (x, y, z), the foreground prototype of the second image is determined through the mask average pooling operation according to formula (8-1) as shown below, and the background prototype of the second image is determined according to formula (8-2) as shown below.

[0128]

[0129] wherein p u(fg) is the foreground prototype of the second image, K is the number of second images, is the voxel feature of the voxel with the spatial position (x, y, z) in the feature map of the kth second image, is the voxel information of the voxel with the spatial position (x, y, z) in the predicted image of the kth second image, 1[·] is an indicator function with a return value of 1 when the condition is met, p u(bg) is the background prototype of the second image.

[0130] It should be noted that the feature map of the second image needs to be up-sampled through trilinear interpolation to make the size of the up-sampled second image consistent with that of the predicted image of the second image. In this embodiment of the present application, the specific manner of trilinear interpolation is not limited.

[0131] Optionally, the reference image of the first image is determined based on the foreground prototype of the second image, the background prototype of the second image and the first image, comprising: obtaining the feature map of the first image; determining the reference image of the first image based on the foreground prototype of the second image, the background prototype of the second image and the feature map of the first image.

[0132] In this embodiment of the present application, the first image is input into the first network model, and the feature map of the first image is extracted by the encoder of the first network model. Then, the similarity between the feature map of the first image and the foreground prototype of the second image or the similarity between the feature map of the first image and the background prototype of the second image is calculated based on the foreground prototype of the second image, the background prototype of the second image and the feature map of the first image, to obtain a similarity result. Subsequently, the similarity result is normalized by using a normalized exponential function (for example, a Softmax function) to obtain the reference image of the first image.

[0133] The above operation of determining the reference image of the first image can also be referred to as an operation based on a non-parametric metric learning mechanism. The similarity with the labeled image set is measured by non-parametric metric learning, so as to segment the labeled image set.

[0134] In this embodiment of the present application, the function used when calculating the similarity is denoted as a distance function d(·), and the feature map of the first image is denoted as F l, the spatial position of any voxel in the feature map of the first image is denoted as (x, y, z). The foreground prototype of the second image is p u(fg) , the background prototype of the second image is p u(bg) , and P u ={p u(fg)}∪{p u(bg)}. For each P u(j∈(fg,bg)) ∈P u , the reference image of the first image is determined according to formula (9) as follows.

[0135]

[0136] wherein, is the voxel feature of the voxel with the spatial position (x, y, z) in the reference image of the first image, exp is an exponential function, d(·) is a distance function, F l (x,y,z) is the voxel feature of the voxel with the spatial position (x, y, z) in the feature map of the first image, p u(j) is the foreground prototype of the second image or the background prototype of the second image, P u includes the foreground prototype of the second image and the background prototype of the second image, and ∑ is the symbol of a summation function. α is a proportionality coefficient, and the specific value of the proportionality coefficient is not limited in the embodiments of the present application. For example, the proportionality coefficient is 20.

[0137] It should be noted that the feature map of the first image needs to be up-sampled by trilinear interpolation. In the embodiments of the present application, the specific manner of trilinear interpolation is not limited. In addition, the distance function is not limited in the embodiments of the present application. For example, the distance function is a cosine distance function.

[0138] In the above manner, the reference image of the first image can be determined. Then, based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image and the reference image of the second image, the loss value of the first network model is determined.

[0139] Optionally, based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image and the reference image of the second image, the loss value of the first network model is determined, including: determining a first loss value based on the predicted image of the first image and the labeled image of the first image; determining a second loss value based on the predicted image of the second image and the reference image of the second image; determining a third loss value based on the labeled image of the first image and the reference image of the first image; and determining the loss value of the first network model based on the first loss value, the second loss value and the third loss value.

[0140] In the embodiments of the present application, the predicted image of the first image and the labeled image of the first image are input into the first loss function to obtain a first loss value, and the predicted image of the second image and the reference image of the second image are input into the second loss function to obtain a second loss value. The related description of the first loss value and the second loss value can be seen from the description of the first loss value and the second loss value above, and will not be repeated here. In the embodiments of the present application, the labeled image of the first image and the reference image of the first image are input into the third loss function to obtain a third loss value. The third loss function is also called a backward prototype consistency loss function, and the embodiments of the present application do not limit the third loss function. For example, the third loss value is determined according to the formula (10) as follows.

[0141] L bpc ce (Y l ,P u2l ) Formula (10)

[0142] Wherein, L bpc is the third loss value, L ce is the function symbol of the cross-entropy loss function, Y l is the labeled image of the first image, and P u2l is the reference image of the first image.

[0143] After the first loss value, the second loss value and the third loss value are determined, the loss value of the first network model is determined based on the first loss value, the second loss value and the third loss value. Optionally, the loss value of the first network model is determined according to the formula (11) as follows.

[0144] L=L s +λL c , wherein L c =L fpc +βL bpc Formula (11)

[0145] Wherein, L is the loss value of the first network model, L s is the first loss value, L c is the prototype consistency loss value, L fpc is the second loss value, L bpc is the third loss value, λ is the coefficient of balancing L s and L c , β is the hyperparameter of balancing L fpc and L bpc , and the embodiments of the present application do not limit the values of λ and β.

[0146] For example, β is 10, and λ satisfies: ​where λ(t) (i.e. λ) is a Gaussian function related to the number of training, t is the number of training, w max is the final consistency weight, e is a natural number, t max is the target number of training, such a λ design can avoid being dominated by the very unreliable consistency objective at the beginning of training.

[0147] In the embodiments of the present application, when determining the loss value of the first network model, the model parameters of the first network model and the model parameters of the second network model can be obtained first, and then the loss value of the first network model is determined based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image and the reference image of the second image. The relationship between the model parameters of the second network model and the model parameters of the first network model has been described above and will not be repeated here.

[0148] Optionally, the loss value of the first network model is determined based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image and the reference image of the second image.

[0149] When determining the first loss value, the first loss value is determined based on the model parameters of the first network model, the predicted image of the first image and the labeled image of the first image. When determining the second loss value, the second loss value is determined based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the second image and the reference image of the second image. When determining the third loss value, the third loss value is determined based on the model parameters of the first network model, the model parameters of the second network model, the labeled image of the first image and the reference image of the first image. Then, the loss value of the first network model is determined based on the first loss value, the second loss value and the third loss value.

[0150] In a possible implementation, the model parameters of the first network model are adjusted according to the loss value of the first network model to implement one update process of the first network model, and an updated first network model is obtained. If the training end condition is met, the updated first network model is taken as the image segmentation model. If the training end condition is not met, the model parameters of the second network model can be determined based on the model parameters of the updated first network model to implement one update process of the second network model, and an updated second network model is obtained. The model parameters of the second network model can be determined in the manner of the above formula (1). Then, the updated first network model is taken as the first network model for the next training, the updated second network model is taken as the second network model for the next training, and the steps of steps 201 (or 202) to 205 are re-executed until the image segmentation model is obtained.

[0151] The method determines the reference image of the second image based on the labeled image of the second image and the first image, implements the propagation of the image segmentation result obtained by labeling in the labeled data set to the unlabeled data set, thereby reducing the number of images that need to be labeled, reducing the time consumption and labor consumption, and reducing the cost. Then, the image segmentation model is obtained through the predicted image and the labeled image of the first image and the predicted image and the reference image of the second image, the training speed of the image segmentation model is accelerated, and the efficiency of image segmentation is improved.

[0152] Based on the above implementation environment, the embodiment of the application provides an image segmentation method. Figure 3 As shown in the flowchart of the image segmentation method provided by the embodiment of the application, the method can be executed by the electronic device 11 in the electronic device 10. Figure 1 As shown in the flowchart of the image segmentation method provided by the embodiment of the application, the method can be executed by the electronic device 11 in the electronic device 10. Figure 3 The method includes steps 301 to 302.

[0153] Step 301: An image to be segmented is obtained.

[0154] The image to be segmented is not limited in the embodiment of the application, and is exemplarily a medical image, a photographic image, or the like.

[0155] Step 302: An image segmentation result of the image to be segmented is obtained according to an image segmentation model.

[0156] The image segmentation model is obtained based on the training method of the image segmentation model in the above embodiment.

[0157] In the embodiment of the application, the image to be segmented is input into the image segmentation model, and the image segmentation result predicted by the image segmentation model is output.

[0158] Optionally, the image segmentation result of the to-be-segmented image is obtained according to the image segmentation model, including: a feature map of the to-be-segmented image is obtained according to the image segmentation model, the feature map of the to-be-segmented image is used to represent semantic information of the to-be-segmented image; and the image segmentation result of the to-be-segmented image is determined based on the feature map of the to-be-segmented image.

[0159] In the embodiments of the present application, the image segmentation model includes an encoder and a decoder. The to-be-segmented image is input into the image segmentation model, the feature map of the to-be-segmented image is extracted by the encoder of the image segmentation model, and the image segmentation result of the to-be-segmented image is determined by the decoder of the image segmentation model based on the feature map of the to-be-segmented image. The feature map of the to-be-segmented image includes the features of each voxel in the to-be-segmented image, and therefore, the feature map of the to-be-segmented image can represent the semantic information of the to-be-segmented image.

[0160] The image segmentation model in the above method is obtained through the predicted image and the labeled image of the first image, and the predicted image and the reference image of the second image, wherein the reference image of the second image is determined based on the second image and the labeled image of the first image. The image segmentation result obtained by labeling in the labeled data set is propagated to the unlabeled data set, thereby reducing the number of images that need to be labeled, reducing the time consumption and the labor consumption, and reducing the cost. The image segmentation model is obtained based on the reference image of the second image, which speeds up the training speed of the image segmentation model and improves the efficiency of image segmentation.

[0161] The training method of the image segmentation model and the image segmentation method are described from the perspective of a scene. In the embodiments of the present application, the scene is a scene of segmenting medical images (such as kidney images), that is, the image segmentation model is trained using medical images, and the medical images are segmented using the image segmentation model.

[0162] In the embodiments of the present application, 210 Computed Tomography (CT) kidney images of objects are collected. For each CT kidney image of an object, the following preprocessing is performed to obtain a preprocessed kidney image: the CT kidney image is resampled to a resolution of 1 cubic millimeter (mm 3 ), the CT intensity is truncated to [75, 175] Hounsfield Units (HU), then the CT intensity is normalized, and a Region Of Interest (ROI) is cropped to obtain a 3D patch region centered on the kidney, that is, a preprocessed kidney image is obtained.

[0163] The preprocessed kidney images of the 210 objects are divided into three groups in the embodiment of the application, one group is a sample image set used for training the image segmentation model, the sample image set includes preprocessed kidney images of 150 objects, another group of images is a verification image set used for verifying the segmentation effect of the image segmentation model, the verification image set includes preprocessed kidney images of 10 objects, and the remaining group of images is a test image set used for testing the segmentation result of the image segmentation model, the test image set includes preprocessed kidney images of 50 objects. Wherein, the number of kidney images is expanded by a data expansion method, which includes but is not limited to random cropping, flipping and rotating.

[0164] The sample image set used for training the image segmentation model includes an annotated image set and an unannotated image set. The annotated image set is denoted as S L The unannotated image set is denoted as S U The embodiment of the application adjusts the loss value of the first network model based on the annotated image set, the unannotated image set, the model parameters of the first network model and the model parameters of the second network model according to formula (12) as follows until the image segmentation model is obtained.

[0165]

[0166] Wherein, is the loss value obtained based on the annotated image set, is the loss value obtained based on the annotated image set and the unannotated image set. θ is the model parameter of the first network model, is the model parameter of the second network model, and λ is a Gaussian function related to the number of training times.

[0167] In the embodiment of the application, the annotated image set includes a first image and an annotated image of the first image, and the unannotated image set includes a second image, both the first image and the second image are kidney images. Please refer to Figure 4 , Figure 4 is a training schematic diagram of an image segmentation model provided by the embodiment of the application.

[0168] The first image is denoted as l, and the first network model comprises an encoder and a decoder. The first image is input into the encoder of the first network model, and features of the first image are extracted by the encoder of the first network model. In one aspect, a predicted image of the first image is determined by the decoder of the first network model according to a feature map of the first image, and in another aspect, a feature segmentation map of the first image is obtained by performing cross multiplication processing on the feature map of the first image and a labeled image of the first image, the feature segmentation map of the first image being a feature map in which voxel features of a foreground region of the first image and voxel features of a background region of the first image have been segmented. Then, a foreground / background prototype of the first image is determined based on the feature segmentation map of the first image, and a set of foreground / background prototypes of the first image is constructed.

[0169] The second image is denoted as u, and the second network model comprises an encoder and a decoder. Model parameters of the encoder of the second network model are determined according to model parameters of the encoder of the first network model by using an exponential moving average method, and model parameters of the decoder of the second network model are determined according to model parameters of the decoder of the first network model by using the exponential moving average method. The second image is input into the encoder of the second network model, and features of the second image are extracted by the encoder of the second network model. In one aspect, a predicted image of the second image is determined by the decoder of the second network model according to a feature map of the second image, and in another aspect, a reference image of the second image is obtained by performing similarity calculation processing on the feature map of the second image and the set of foreground / background prototypes of the first image, and the process of determining the reference image of the second image can be referred to as a forward process.

[0170] Optionally, cross multiplication processing is performed on the predicted image of the second image and the feature map of the second image to obtain a feature segmentation map of the second image, the feature segmentation map of the second image being a feature map in which voxel features of a foreground region of the second image and voxel features of a background region of the second image have been segmented. Then, a foreground / background prototype of the second image is determined based on the feature segmentation map of the second image, and a set of foreground / background prototypes of the second image is constructed. Subsequently, similarity calculation processing is performed on the feature map of the first image and the set of foreground / background prototypes of the second image to obtain a reference image of the first image, and the process of determining the reference image of the first image can be referred to as a reverse (or backward) process.

[0171] Then, based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image, and the reference image of the second image, a loss value of the first network model is determined to adjust the model parameters of the first network model according to the loss value of the first network model. Through multiple adjustments of the model parameters of the first network model, an image segmentation model can be obtained, which is denoted as an image segmentation model of the present embodiment.

[0172] It should be noted that the implementation of training the first network model to obtain the image segmentation model can be seen from the related description of steps 201 to 205, and the implementation principles of the two are similar, which will not be repeated here.

[0173] In the present embodiment, the first network model is also trained using the labeled image set to obtain another image segmentation model, which is denoted as a fully supervised image segmentation model. When training the image segmentation model and the fully supervised image segmentation model of the present embodiment, the first network model is trained based on a graphics processing unit (GPU) and a stochastic gradient descent (SGD) optimizer, wherein the weight decay = 0.0001 and the momentum = 0.9. The sample size (i.e., Batch) is set to 4, including 2 first images and 2 second images. The target training times are 20000. The learning rate is initialized to 0.01 and decayed by the power of 0.9 after each training. The image size input to the first network model is a 96x96x96 voxel image.

[0174] After obtaining the above two image segmentation models, a to-be-segmented image is obtained, and the to-be-segmented image is input into each image segmentation model to obtain an image segmentation result. The to-be-segmented image is a medical image. Please refer to Figure 5 and Figure 6 , Figure 5 is a brain tumor image segmentation result schematic diagram provided by the present embodiment, Figure 6 is a kidney image segmentation result schematic diagram provided by the present embodiment.

[0175] Figure 5 (1) in is a brain tumor image, Figure 5 (2) in is a labeled image of the brain tumor image (the dashed line represents the labeling result), Figure 5 (3) in is an image segmentation result of the brain tumor image obtained by using the image segmentation model of the present embodiment (the dashed line represents the image segmentation result), Figure 5(4) is an image segmentation result of the brain tumor image in (3) obtained by using the fully supervised image segmentation model (the dashed line represents the image segmentation result). Figure 5 (5) is another brain tumor image, Figure 5 (6) is a labeled image of the brain tumor image in (5) (the dashed line represents the labeling result), Figure 6 (7) is an image segmentation result of the brain tumor image in (5) obtained by using the image segmentation model of the embodiment of the present application (the dashed line represents the image segmentation result), Figure 6 (8) is an image segmentation result of the brain tumor image in (5) obtained by using the fully supervised image segmentation model (the dashed line represents the image segmentation result).

[0176] Figure 6 (1) is a kidney image, Figure 6 (2) is an image segmentation result of the kidney image in (1) obtained by using the image segmentation model of the embodiment of the present application (the dashed line represents the image segmentation result), Figure 6 (3) is an image segmentation result of the kidney image in (1) obtained by using the fully supervised image segmentation model (the dashed line represents the image segmentation result). Figure 6 (4) is another kidney image, Figure 7 (5) is an image segmentation result of the kidney image in (4) obtained by using the image segmentation model of the embodiment of the present application (the dashed line represents the image segmentation result), Figure 7 (6) is an image segmentation result of the kidney image in (4) obtained by using the fully supervised image segmentation model (the dashed line represents the image segmentation result).

[0177] In order to compare the image segmentation effects of different models, the embodiment of the present application trains seven network models by using three sample image sets, to obtain various image segmentation models. The proportions of labeled images in the three sample image sets are 5%, 10% and 20% respectively. Each sample image set can train seven image segmentation models, which are a fully supervised image segmentation model, a mean-teacher self-integrated (MT) image segmentation model, an uncertainty-aware mean-teacher self-integrated (UAMT) image segmentation model, an entropy minimization method (Entropy Mini) image segmentation model, a deep adversarial network (DAN) image segmentation model, an interpolation consistency training (ICT) image segmentation model and the image segmentation model of the embodiment of the present application.

[0178] In the test phase, the embodiment of the present application uses a sliding window with a voxel step of 64x64x64, and uses four indexes to comprehensively evaluate the above seven image segmentation models, the four indexes are respectively a Dice coefficient (an index for measuring similarity), a Jaccard coefficient, a 95% Hausdorff distance (95HD), and an Average Surface Distance (ASD), and the comprehensive evaluation results are shown in Table 1.

[0179] Table 1

[0180]

[0181] As can be seen from Table 1, the Dice coefficient and the Jaccard coefficient of the image segmentation model of the embodiment of the present application are both larger, and the 95HD and the ASD are both smaller. Since the segmentation effect of the image segmentation model is proportional to the Dice coefficient and the Jaccard coefficient, and is inversely proportional to the 95HD and the ASD, therefore, according to Table 1, it can be obviously concluded that the segmentation effect of the image segmentation model of the embodiment of the present application is better than the segmentation effects of the other six image segmentation models.

[0182] Figure 7 As shown in FIG. 7, the device includes: Figure 8 As shown in FIG. 7, the device includes:

[0183] The acquisition module 701 is configured to acquire a labeled image set and an unlabeled image set, the labeled image set including a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set including a second image.

[0184] The acquisition module 701 is further configured to acquire a predicted image of the first image according to the first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image.

[0185] The acquisition module 701 is further configured to acquire a predicted image of the second image according to the second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image.

[0186] The determination module 702 is configured to determine a reference image of the second image based on the second image and the labeled image of the first image, the reference image of the second image being an image segmentation result obtained by calculating the second image.

[0187] The determining module 702 is further configured to determine the image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

[0188] In a possible implementation, the determining module 702 is configured to determine the foreground prototype of the first image and the background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being reference features of the first region in the first image, and the background prototype of the first image being reference features of regions other than the first region in the first image; and determine the reference image of the second image based on the foreground prototype of the first image, the background prototype of the first image, and the second image.

[0189] In a possible implementation, the determining module 702 is configured to obtain a feature map of the first image, the feature map of the first image being used to represent semantic information of the first image; and determine the foreground prototype of the first image and the background prototype of the first image based on the feature map of the first image and the labeled image of the first image.

[0190] In a possible implementation, the determining module 702 is configured to obtain a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; and determine the reference image of the second image based on the foreground prototype of the first image, the background prototype of the first image, and the feature map of the second image.

[0191] In a possible implementation, the determining module 702 is configured to determine a loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image; and determine the image segmentation model according to the first network model based on the loss value of the first network model.

[0192] In a possible implementation, the determining module 702 is configured to determine a first loss value based on the predicted image of the first image and the labeled image of the first image; determine a second loss value based on the predicted image of the second image and the reference image of the second image; and determine the loss value of the first network model based on the first loss value and the second loss value.

[0193] In a possible implementation, the determining module 702 is configured to determine model parameters of the first network model and model parameters of the second network model; and determine the loss value of the first network model based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

[0194] In a possible implementation, the model parameters of the second network model are determined based on the model parameters of the first network model.

[0195] In a possible implementation, the determining module 702 is further configured to determine a reference image of the first image based on the predicted image of the first image and the predicted image of the second image, the reference image of the first image being an image segmentation result obtained by performing calculation on the first image.

[0196] The determining module 702 is further configured to determine the loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image, and the reference image of the second image.

[0197] In a possible implementation, the determining module 702 is configured to determine a foreground prototype of the second image and a background prototype of the second image based on the predicted image of the second image, the foreground prototype of the second image being a reference feature of the second region in the second image, and the background prototype of the second image being a reference feature of a region other than the second region in the second image; and determine the reference image of the first image based on the foreground prototype of the second image, the background prototype of the second image, and the first image.

[0198] In a possible implementation, the determining module 702 is configured to obtain a feature map of the second image; and determine the foreground prototype of the second image and the background prototype of the second image based on the feature map of the second image and the predicted image of the second image.

[0199] In a possible implementation, the determining module 702 is configured to obtain a feature map of the first image; and determine the reference image of the first image based on the foreground prototype of the second image, the background prototype of the second image, and the feature map of the first image.

[0200] In a possible implementation, the determining module 702 is configured to determine a first loss value based on the predicted image of the first image and the labeled image of the first image; determine a second loss value based on the predicted image of the second image and the reference image of the second image; determine a third loss value based on the labeled image of the first image and the reference image of the first image; and determine the loss value of the first network model based on the first loss value, the second loss value, and the third loss value.

[0201] The apparatus described above determines the reference image of the second image based on the second image and the labeled image of the first image, implements propagation of an image segmentation result obtained by labeling in a labeled data set to an unlabeled data set, thereby reducing the number of images that need to be labeled, reducing time consumption and labor consumption, and reducing costs. Then, the image segmentation model is obtained based on the predicted image and the labeled image of the first image, the predicted image and the reference image of the second image, thereby accelerating the training speed of the image segmentation model and improving the efficiency of image segmentation.

[0202] It should be understood that the above Figure 8The apparatus provided is only exemplified by the above-mentioned division of each functional module in realizing its functions. In actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided by the above-mentioned embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0203] Figure 8 As shown in a structural schematic diagram of an image segmentation device provided by an embodiment of the application, the device comprises Figure 9 As shown, the device comprises

[0204] The first acquisition module 801 is configured to acquire an image to be segmented.

[0205] The second acquisition module 802 is configured to acquire an image segmentation result of the image to be segmented according to an image segmentation model, wherein the image segmentation model is obtained based on the training method of any one of the above-mentioned image segmentation models.

[0206] In a possible implementation, the second acquisition module 802 is configured to acquire a feature map of the image to be segmented according to the image segmentation model, wherein the feature map of the image to be segmented is used to represent semantic information of the image to be segmented; and the image segmentation result of the image to be segmented is determined based on the feature map of the image to be segmented.

[0207] The image segmentation model in the above-mentioned device is obtained through a predicted image and a labeled image of a first image, and a predicted image and a reference image of a second image, wherein the reference image of the second image is determined based on the labeled image of the first image and the second image, so as to propagate the image segmentation result obtained by labeling in the labeled data set to the unlabeled data set, thereby reducing the number of images that need to be labeled, reducing the time consumption and the labor consumption, and reducing the cost. The image segmentation model is obtained based on the reference image of the second image, so as to accelerate the training speed of the image segmentation model and improve the efficiency of image segmentation.

[0208] It should be understood that the above-mentioned Figure 9 The apparatus provided is only exemplified by the above-mentioned division of each functional module in realizing its functions. In actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided by the above-mentioned embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0209] Figure 10A structure block diagram of a terminal device 900 provided by an example embodiment of the present application is shown. The terminal device 900 can be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer, or a desktop computer. The terminal device 900 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0210] Generally, the terminal device 900 includes a processor 901 and a memory 902.

[0211] The processor 901 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 901 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 901 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 901 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0212] The memory 902 can include one or more computer-readable storage media, which can be non-transitory. The memory 902 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction for being executed by the processor 901 to implement a training method of an image segmentation model or an image segmentation method provided by the method embodiments of the present application.

[0213] In some embodiments, the terminal device 900 can further optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 903 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 904, a display screen 905, a camera component 906, an audio circuit 907 and a power supply 909.

[0214] The peripheral device interface 903 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902 and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902 and the peripheral device interface 903 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0215] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 904 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 904 can also include NFC (Near Field Communication) related circuitry, which is not limited in the present application.

[0216] The display screen 905 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 is further configured to capture touch signals on or above the surface of the display screen 905. The touch signals can be input to the processor 901 as control signals for processing. In this case, the display screen 905 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 905 can be one, disposed on the front panel of the terminal device 900; in other embodiments, the display screen 905 can be at least two, respectively disposed on different surfaces of the terminal device 900 or in a folding design; in other embodiments, the display screen 905 can be a flexible display screen, disposed on a curved surface or a folding surface of the terminal device 900. Even, the display screen 905 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 905 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0217] The camera assembly 906 is configured to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 906 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0218] The audio circuit 907 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 901 for processing, or input to the radio frequency circuit 904 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal device 900. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 907 can also include a headphone jack.

[0219] The power supply 909 is used to supply power to each component in the terminal device 900. The power supply 909 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0220] In some embodiments, the terminal device 900 further includes one or more sensors 910. The one or more sensors 910 include, but are not limited to, an acceleration sensor 911, a gyroscope sensor 912, a pressure sensor 913, an optical sensor 915, and a proximity sensor 916.

[0221] The acceleration sensor 911 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the terminal device 900. For example, the acceleration sensor 911 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 911. The acceleration sensor 911 can also be used for game or user motion data collection.

[0222] The gyroscope sensor 912 can detect the body direction and rotation angle of the terminal device 900, and the gyroscope sensor 912 can collect 3D actions of the user on the terminal device 900 in cooperation with the acceleration sensor 911. The processor 901 can realize the following functions according to the data collected by the gyroscope sensor 912: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0223] The pressure sensor 913 can be disposed on the side bezel of the terminal device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is disposed on the side bezel of the terminal device 900, it can detect the user's grip signal on the terminal device 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is disposed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0224] An optical sensor 915 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 915.

[0225] The proximity sensor 916, also known as a distance sensor, is typically located on the front panel of the terminal device 900. The proximity sensor 916 is used to detect the distance between the user and the front of the terminal device 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front of the terminal device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 916 detects that the distance between the user and the front of the terminal device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.

[0226] Those skilled in the art will understand that ​ The structure shown does not constitute a limitation on the terminal device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0227] ​A structural diagram of a server provided in the embodiments of the present application is shown in FIG. 10. The server 1000 can have a great difference due to different configurations or performances, and can include one or more processors 1001 and one or more memories 1002. The one or more memories 1002 store at least one piece of program code, which is loaded and executed by the one or more processors 1001 to implement the training method of the image segmentation model or the image segmentation method provided in any of the above method embodiments. The processor 1001 is a CPU, for example. Of course, the server 1000 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for implementing device functions, which are not described herein.

[0228] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one piece of program code. The at least one piece of program code is loaded and executed by a processor to enable an electronic device to implement any of the above training methods of the image segmentation model or the image segmentation methods.

[0229] Optionally, the above computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0230] In an exemplary embodiment, a computer program or computer program product is also provided, which stores at least one computer instruction. The at least one computer instruction is loaded and executed by a processor to enable a computer to implement any of the above training methods of the image segmentation model or the image segmentation methods.

[0231] It should be understood that “multiple” mentioned herein refers to two or more. “And / or” describes the association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship.

[0232] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0233] The above merely describes exemplary embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training an image segmentation model, characterized in that, The method comprises: obtaining a labeled image set and an unlabeled image set, the labeled image set comprising a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set comprising a second image; obtaining a predicted image of the first image according to a first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image; obtaining a predicted image of the second image according to a second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image; determining a foreground prototype of the first image or a background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being a reference feature of a first region in the first image, and the background prototype of the first image being a reference feature of a region other than the first region in the first image; obtaining a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; performing similarity calculation processing on the foreground prototype of the first image or the background prototype of the first image and the feature map of the second image to obtain a reference image of the second image, the reference image of the second image being an image segmentation result obtained by calculating the second image; determining an image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

2. The method of claim 1, wherein, The method comprises: obtaining a labeled image set and an unlabeled image set, the labeled image set comprising a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set comprising a second image; obtaining a predicted image of the first image according to a first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image; 3. The method according to claim 1 or 2, characterized in that, obtaining a predicted image of the second image according to a second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image; determining a foreground prototype of the first image or a background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being a reference feature of a first region in the first image, and the background prototype of the first image being a reference feature of a region other than the first region in the first image; obtaining a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; 4. The method of claim 3, wherein, performing similarity calculation processing on the foreground prototype of the first image or the background prototype of the first image and the feature map of the second image to obtain a reference image of the second image, the reference image of the second image being an image segmentation result obtained by calculating the second image; determining an image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image. The method comprises: obtaining a labeled image set and an unlabeled image set, the labeled image set comprising a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set comprising a second image; obtaining a predicted image of the first image according to a first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image; obtaining a predicted image of the second image according to a second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image; determining a foreground prototype of the first image or a background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being a reference feature of a first region in the first image, and the background prototype of the first image being a reference feature of a region other than the first region in the first image; obtaining a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; performing similarity calculation processing on the foreground prototype of the first image or the background prototype of the first image and the feature map of the second image to obtain a reference image of the second image, the reference image of the second image being an image segmentation result obtained by calculating the second image; determining an image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image. The method comprises: obtaining a labeled image set and an unlabeled image set, the labeled image set comprising a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set comprising a second image; obtaining a predicted image of the first image according to a first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image; obtaining a predicted image of the second image according to a second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image; determining a foreground prototype of the first image or a background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being a reference feature of a first region in the first image, and the background prototype of the first image being a reference feature of a region other than the first region in the first image; obtaining a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; performing similarity calculation processing on the foreground prototype of the first image or the background prototype of the first image and the feature map of the second image to obtain a reference image of the second image, the reference image of the second image being an image segmentation result obtained by calculating the second image; determining an image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

5. The method of claim 3, wherein, The loss value of the first network model is determined based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image. The model parameters of the first network model and the model parameters of the second network model are determined. The loss value of the first network model is determined based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

6. The method of claim 5, wherein, The model parameters of the second network model are determined based on the model parameters of the first network model.

7. The method of claim 3, wherein, After obtaining the predicted image of the second image according to the second network model, the method further includes: The reference image of the first image is determined based on the predicted image of the first image and the predicted image of the second image, and the reference image of the first image is an image segmentation result obtained by calculating the first image. The loss value of the first network model is determined based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image. The loss value of the first network model is determined based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image, and the reference image of the second image.

8. The method of claim 7, wherein, The reference image of the first image is determined based on the predicted image of the first image and the predicted image of the second image, and the reference image of the first image is an image segmentation result obtained by calculating the first image. The foreground prototype of the second image and the background prototype of the second image are determined based on the predicted image of the second image, and the foreground prototype of the second image is a reference feature of a second region in the second image, and the background prototype of the second image is a reference feature of a region other than the second region in the second image; The reference image of the first image is determined based on the foreground prototype of the second image, the background prototype of the second image, and the first image.

9. The method of claim 8, wherein, The foreground prototype of the second image and the background prototype of the second image are determined based on the feature map of the second image and the predicted image of the second image. The reference image of the first image is determined based on the foreground prototype of the second image, the background prototype of the second image, and the feature map of the first image. The feature map of the first image is obtained.

10. The method of claim 8, wherein, The reference image of the first image is determined based on the foreground prototype of the second image, the background prototype of the second image, and the feature map of the first image. The loss value of the first network model is determined based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image, and the reference image of the second image. ​ 11. The method of claim 7, wherein, ​ determine a first loss value based on the predicted image of the first image and the labeled image of the first image; determine a second loss value based on the predicted image of the second image and the reference image of the second image; determine a third loss value based on the labeled image of the first image and the reference image of the first image; determine a loss value of the first network model based on the first loss value, the second loss value and the third loss value.

12. An image segmentation method characterized by, The method comprises: obtaining a to-be-segmented image; obtaining an image segmentation result of the to-be-segmented image according to an image segmentation model, the image segmentation model being obtained based on the method of any one of claims 1 to 11.

13. The method of claim 12, wherein, The image segmentation result of the to-be-segmented image according to the image segmentation model comprises: obtaining a feature map of the to-be-segmented image according to the image segmentation model, the feature map of the to-be-segmented image being used to represent semantic information of the to-be-segmented image; determining the image segmentation result of the to-be-segmented image based on the feature map of the to-be-segmented image.

14. A training device for an image segmentation model, characterized in that, The device comprises: an obtaining module, configured to obtain a labeled image set and an unlabeled image set, the labeled image set comprising a first image and a labeled image of the first image, the labeled image being an image segmentation result obtained by labeling the first image, and the unlabeled image set comprising a second image; the obtaining module is further configured to obtain a predicted image of the first image according to a first network model, the predicted image of the first image being an image segmentation result obtained by predicting the first image; the obtaining module is further configured to obtain a predicted image of the second image according to a second network model, the predicted image of the second image being an image segmentation result obtained by predicting the second image; a determining module, configured to determine a foreground prototype of the first image or a background prototype of the first image based on the labeled image of the first image, the foreground prototype of the first image being a reference feature of a first region in the first image, and the background prototype of the first image being a reference feature of a region other than the first region in the first image; obtain a feature map of the second image, the feature map of the second image being used to represent semantic information of the second image; and perform similarity calculation processing on the foreground prototype of the first image or the background prototype of the first image and the feature map of the second image to obtain a reference image of the second image, the reference image of the second image being an image segmentation result obtained by calculating the second image; the determining module is further configured to determine an image segmentation model according to the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image and the reference image of the second image.

15. The apparatus of claim 14, wherein, the determining module is configured to obtain a feature map of the first image, the feature map of the first image being used to represent semantic information of the first image; and determine the foreground prototype of the first image or the background prototype of the first image based on the feature map of the first image and the labeled image of the first image.

16. The apparatus of claim 14 or 15, wherein, The determining module is configured to determine the loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image. The image segmentation model is determined according to the first network model based on the loss value of the first network model.

17. The apparatus of claim 16, wherein, The determining module is configured to determine a first loss value based on the predicted image of the first image and the labeled image of the first image, determine a second loss value based on the predicted image of the second image and the reference image of the second image, and determine the loss value of the first network model based on the first loss value and the second loss value.

18. The apparatus of claim 16, wherein, The determining module is configured to determine the model parameters of the first network model and the model parameters of the second network model, and determine the loss value of the first network model based on the model parameters of the first network model, the model parameters of the second network model, the predicted image of the first image, the labeled image of the first image, the predicted image of the second image, and the reference image of the second image.

19. The apparatus of claim 18, wherein, The model parameters of the second network model are determined based on the model parameters of the first network model.

20. The apparatus of claim 16, wherein, The determining module is further configured to determine the reference image of the first image based on the predicted images of the first image and the second image, the reference image of the first image being an image segmentation result obtained by performing calculation on the first image. The determining module is further configured to determine the loss value of the first network model based on the predicted image of the first image, the labeled image of the first image, the reference image of the first image, the predicted image of the second image, and the reference image of the second image.

21. The apparatus of claim 20, wherein, The determining module is configured to determine the foreground prototype of the second image and the background prototype of the second image based on the predicted image of the second image, the foreground prototype of the second image being the reference feature of a second region in the second image, and the background prototype of the second image being the reference feature of other regions in the second image except the second region. The reference image of the first image is determined based on the foreground prototype of the second image, the background prototype of the second image, and the first image.

22. The apparatus of claim 21, wherein, The determining module is configured to obtain the feature map of the second image, and determine the foreground prototype of the second image and the background prototype of the second image based on the feature map of the second image and the predicted image of the second image.

23. The apparatus of claim 21, wherein, The determining module is configured to obtain the feature map of the first image, and determine the reference image of the first image based on the foreground prototype of the second image, the background prototype of the second image, and the feature map of the first image.

24. The apparatus of claim 20, wherein, The determining module is configured to determine a first loss value based on the predicted image of the first image and the labeled image of the first image, determine a second loss value based on the predicted image of the second image and the reference image of the second image, determine a third loss value based on the labeled image of the first image and the reference image of the first image, and determine the loss value of the first network model based on the first loss value, the second loss value and the third loss value.

25. An image segmentation apparatus characterized by comprising: The apparatus comprises: A first obtaining module is configured to obtain a to-be-segmented image. A second obtaining module is configured to obtain an image segmentation result of the to-be-segmented image according to an image segmentation model, wherein the image segmentation model is obtained based on the method of any one of claims 1 to 11.

26. The apparatus of claim 25, wherein, The second obtaining module is configured to obtain a feature map of the to-be-segmented image according to the image segmentation model, wherein the feature map of the to-be-segmented image is used to represent semantic information of the to-be-segmented image, and the image segmentation result of the to-be-segmented image is determined based on the feature map of the to-be-segmented image.

27. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one program code, which is loaded and executed by the processor, so that the electronic device implements the image segmentation model training method of any one of claims 1 to 11 or implements the image segmentation method of any one of claims 12 to 13.

28. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one program code, which is loaded and executed by the processor, so that the computer implements the image segmentation model training method of any one of claims 1 to 11 or implements the image segmentation method of any one of claims 12 to 13.

29. A computer program product, characterised in that, The computer program product stores at least one computer instruction, which is loaded and executed by the processor, so that the computer implements the image segmentation model training method of any one of claims 1 to 11 or implements the image segmentation method of any one of claims 12 to 13.

Images