Training method of image segmentation network, and image segmentation method, device and equipment

Abstract

Embodiments of the present disclosure provide a training method of an image segmentation network, and an image segmentation method, device and equipment. The training method comprises: inputting a first sample image into a first image segmentation network to obtain feature information of the first sample image; obtaining a first mask of a target object and a first sample classification result according to the feature information by using the first image segmentation network; obtaining a second classification result according to the first mask and the feature information by using the first image segmentation network; training the first image segmentation network according to the first mask, the first classification result, the second classification result and label information to obtain a second image segmentation network; screening out pure background samples without the target object by using the first classification result; and supervising whether the target object contains a background image missegmented by using the second classification result, thereby improving segmentation accuracy and robustness of the image segmentation network.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of image processing technology, and more particularly, to a training method of an image segmentation network, and an image segmentation method, device and equipment. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the present disclosure. The description herein is not admitted to be prior art merely by inclusion in this section.

[0003] With the development of multimedia technology, the popularity of mobile devices, and the gradual rise of short video, makeup, and beauty applications, portrait segmentation technology has attracted increasing attention.

[0004] At present, portrait segmentation based on a convolutional neural network (CNN) has become the most common way. However, a portrait segmentation algorithm based on deep learning usually requires considerable computing power. In the context of real-time processing, how to achieve a high-precision segmentation effect with small-capacity computing power is a problem that needs to be researched and solved.

[0005] SUMMARY

[0006] Therefore, embodiments of the present disclosure provide a training scheme of an image segmentation network and an image segmentation scheme.

[0007] In a first aspect of the embodiments of the present disclosure, a training method of an image segmentation network is provided, comprising:

[0008] inputting a first sample image into a first image segmentation network to obtain feature information of the first sample image, wherein the first sample image has annotation information;

[0009] obtaining a first mask of a target object and a first sample classification result from the first image segmentation network according to the feature information, wherein the first sample classification result indicates whether the target object is included in the sample image;

[0010] obtaining a second classification result from the first image segmentation network according to the first mask and the feature information, wherein the second classification result indicates whether the integrity of the target object included in the first sample image in a rectangular bounding box corresponding to the first mask meets a set requirement;

[0011] training the first image segmentation network according to the first mask, the first classification result, the second classification result, and the annotation information, and obtaining a first image segmentation network after training as a second image segmentation network.

[0012] In an embodiment of the present disclosure, the first image segmentation network is utilized to obtain a second classification result according to the first mask and the feature information, including:

[0013] The first mask is down-sampled to obtain a second mask, wherein a resolution of the second mask is the same as a resolution of the feature information;

[0014] The second mask is multiplied with the feature information to obtain foreground feature information;

[0015] The foreground feature information is cropped and scaled to a set scale;

[0016] A vector corresponding to the scaled foreground feature information is obtained;

[0017] The vector is input into a full connection layer in the first image segmentation network to obtain the second classification result.

[0018] In an embodiment of the present disclosure, the first image segmentation network is trained according to the first mask, the first classification result, the second classification result and the annotation information, and the trained first image segmentation network is used as a second image segmentation network, including:

[0019] The first image segmentation network is trained based on a loss function, and the loss function includes:

[0020] A first loss is used to indicate a difference between the first mask and a mask real value determined according to the annotation information;

[0021] A second loss is used to indicate a difference between the first classification result and a first classification real value determined according to the annotation information, wherein the first classification real value is a real value of whether the target object is contained in the first sample image;

[0022] A third loss is used to indicate a difference between the second classification result and a second classification real value determined according to the annotation information, wherein the second classification real value is a real value of whether the integrity of the target object contained in a rectangular bounding box corresponding to the first mask meets a set requirement.

[0023] In an embodiment of the present disclosure, the loss function is obtained by weighted summation of the first loss, the second loss and the third loss.

[0024] In an embodiment of the present disclosure, when an iteration number of training the first image segmentation network is less than a first set number, a weight of the third loss decreases with an increase of the iteration number;

[0025] when the number of iterations for training the first image segmentation network is greater than or equal to a first set number and less than or equal to a second set number, the weight of the first loss increases with the increase of the number of iterations;

[0026] when the number of iterations for training the first image segmentation network is greater than a second set number, the weight of the second loss increases with the increase of the number of iterations.

[0027] In an embodiment of the present disclosure, the weight of the first loss exponentially changes with the number of iterations, the weight of the second loss logarithmically changes with the number of iterations, and the weight of the third loss is determined according to the weight of the first loss and the weight of the second loss.

[0028] In an embodiment of the present disclosure, the method further comprises:

[0029] inputting a second sample image into the second image segmentation network to obtain a third mask of the target object;

[0030] inputting the second sample image into a target detection network to obtain a detection frame of the target object;

[0031] obtaining a positive sample and a negative sample corresponding to the second sample image according to the third mask and the detection frame of the target object, wherein the annotation information of the positive sample indicates that each pixel is a foreground pixel, and the annotation information in the negative sample indicates that each pixel is a non-foreground pixel.

[0032] In an embodiment of the present disclosure, the obtaining of the positive sample and the negative sample corresponding to the second sample image according to the third mask and the detection frame of the target object comprises:

[0033] determining a first pixel in the detection frame of the target object as a positive sample, wherein the first pixel is a pixel whose probability of being determined as a foreground pixel by the second image segmentation network is greater than a first set threshold;

[0034] determining a second pixel outside the detection frame of the target object as a negative sample, wherein the second pixel is a pixel whose probability of being determined as a non-foreground pixel by the second image segmentation network is greater than a second set threshold.

[0035] In an embodiment of the present disclosure, the method further comprises training the second image segmentation network according to the first sample image, the positive sample and the negative sample, and the first image segmentation network obtained after the training is taken as a third image segmentation network.

[0036] In a second aspect of the embodiments of the present disclosure, an image segmentation method is provided, comprising:

[0037] inputting the image to be processed into an image segmentation network to obtain feature information of the image to be processed; and obtaining a first classification result according to the feature information, the first classification result indicating whether the image to be processed contains a target object;

[0038] in response to the first classification result indicating that the image to be processed contains the target object, obtaining a target mask of the target object according to the feature information, and outputting the target mask;

[0039] in response to the first classification result indicating that the image to be processed does not contain the target object, outputting a default image, each pixel value of the default image being a set pixel value.

[0040] In a third aspect of the embodiments of the present disclosure, a training device of an image segmentation network is provided, comprising:

[0041] a first obtaining unit configured to input a first sample image into a first image segmentation network to obtain feature information of the first sample image, wherein the first sample image has label information;

[0042] a second obtaining unit configured to obtain, by using the first image segmentation network, a first mask of a target object and a first sample classification result according to the feature information, the first sample classification result indicating whether the sample image contains the target object;

[0043] a third obtaining unit configured to obtain, by using the first image segmentation network, a second classification result according to the first mask and the feature information, the second classification result indicating whether the integrity of the target object contained in a rectangular bounding box corresponding to the first mask in the first sample image meets a set requirement;

[0044] a first training unit configured to train the first image segmentation network according to the first mask, the first classification result, the second classification result and the label information, and obtain a second image segmentation network by training the first image segmentation network.

[0045] In an embodiment of the present disclosure, the second obtaining unit is specifically configured to:

[0046] perform down-sampling operation on the first mask to obtain a second mask, wherein the resolution of the second mask is the same as the resolution of the feature information;

[0047] multiply the second mask and the feature information to obtain foreground feature information;

[0048] cropping the foreground feature information, and scaling the cropped foreground feature information to a set scale;

[0049] obtaining a vector corresponding to the scaled foreground feature information;

[0050] inputting the vector into a full connection layer in the first image segmentation network to obtain a second classification result.

[0051] In an embodiment of the present disclosure, the first training unit is specifically configured to:

[0052] train the first image segmentation network based on a loss function, the loss function comprising:

[0053] a first loss for indicating a difference between the first mask and a mask real value determined according to the annotation information;

[0054] a second loss for indicating a difference between the first classification result and a first classification real value determined according to the annotation information, wherein the first classification real value is a real value of whether the first sample image contains the target object;

[0055] a third loss for indicating a difference between the second classification result and a second classification real value determined according to the annotation information, wherein the second classification real value is a real value of whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask meets a set requirement.

[0056] In an embodiment of the present disclosure, the first training unit is specifically configured to:

[0057] train the first image segmentation network based on a loss function, the loss function comprising:

[0058] a first loss for indicating a difference between the first mask and a mask real value determined according to the annotation information;

[0059] a second loss for indicating a difference between the first classification result and a first classification real value determined according to the annotation information, wherein the first classification real value is a real value of whether the first sample image contains the target object;

[0060] a third loss for indicating a difference between the second classification result and a second classification real value determined according to the annotation information, wherein the second classification real value is a real value of whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask meets a set requirement.

[0061] In an embodiment of the present disclosure, the loss function is obtained by weighted sum of the first loss, the second loss and the third loss.

[0062] In an embodiment of the present disclosure, when the number of iterations for training the first image segmentation network is less than a first set number, the weight of the third loss decreases with the increase of the number of iterations.

[0063] When the number of iterations for training the first image segmentation network is greater than or equal to the first set number and less than or equal to a second set number, the weight of the first loss increases with the increase of the number of iterations.

[0064] When the number of iterations for training the first image segmentation network is greater than the second set number, the weight of the second loss increases with the increase of the number of iterations.

[0065] In an embodiment of the present disclosure, the weight of the first loss exponentially changes with the number of iterations, the weight of the second loss logarithmically changes with the number of iterations, and the weight of the third loss is determined according to the weight of the first loss and the weight of the second loss.

[0066] In an embodiment of the present disclosure, the device further comprises a sample obtaining unit, configured to:

[0067] input the second sample image into the second image segmentation network to obtain a third mask of the target object;

[0068] input the second sample image into a target detection network to obtain a detection frame of the target object;

[0069] obtain a positive sample and a negative sample corresponding to the second sample image according to the third mask and the detection frame of the target object, wherein the annotation information of the positive sample indicates that each pixel is a foreground pixel, and the annotation information in the negative sample indicates that each pixel is a non-foreground pixel.

[0070] In an embodiment of the present disclosure, when the second training unit is used to obtain a positive sample and a negative sample corresponding to the second sample image according to the third mask and the detection frame of the target object, the second training unit is specifically configured to:

[0071] determine a first pixel in the detection frame of the target object as a positive sample, wherein the first pixel is a pixel whose probability of being determined as a foreground pixel by the second image segmentation network is greater than a first set threshold;

[0072] determine a second pixel outside the bounding box of the target object as a negative sample, wherein the second pixel is a pixel determined by the second image segmentation network as a non-foreground pixel with a probability greater than a second set threshold.

[0073] In an embodiment of the present disclosure, the apparatus further comprises a second training unit configured to:

[0074] train the second image segmentation network according to the first sample image, the positive sample, and the negative sample, and obtain a first image segmentation network after the training as a third image segmentation network.

[0075] In a fourth aspect of the embodiments of the present disclosure, an image segmentation apparatus is provided, comprising:

[0076] a feature extraction unit configured to input a to-be-processed image into an image segmentation network to obtain feature information of the to-be-processed image, and obtain a first classification result according to the feature information, the first classification result indicating whether the to-be-processed image contains a target object;

[0077] a first processing unit configured to, in response to the first classification result indicating that the to-be-processed image contains the target object, obtain a target mask of the target object according to the feature information, and output the target mask;

[0078] a second processing unit configured to, in response to the first classification result indicating that the to-be-processed image does not contain the target object, output a default image, each pixel value of the default image being a set pixel value.

[0079] In a fifth aspect of the embodiments of the present disclosure, a computing device is provided, comprising a processor, and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method according to any of the embodiments of the present disclosure.

[0080] In a sixth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, comprising a computer program, the computer program being executed by a processor to implement the method according to any of the embodiments of the present disclosure.

[0081] According to the training method of the image segmentation network, the image segmentation method, the device and the equipment, the feature information of the first sample image is obtained by using the first image segmentation network, the first mask of the target object and the first classification result indicating whether the first sample image contains the target object are obtained according to the feature information, the second classification result indicating whether the integrity of the target object contained in the rectangular boundary box corresponding to the first mask meets the set requirement is obtained according to the first mask and the feature information, and finally the first image segmentation network is trained according to the first mask, the first classification result, the second classification result and the annotation information. By using the first classification result to filter out pure background samples without the target object, and by using the second classification result to supervise whether the target object contains the background image missegmented, the segmentation accuracy and the robustness of the image segmentation network are improved. BRIEF DESCRIPTION OF DRAWINGS

[0082] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0083] Figure 1 A flowchart schematically showing a training method of an image segmentation network according to an embodiment of the present disclosure is shown;

[0084] Figure 2A An exemplary first sample image is shown;

[0085] Figure 2B An output image of a third task branch with a second classification result of 1 is shown;

[0086] Figure 2C An output image of a third task branch with a second classification result of 0 is shown;

[0087] Figure 3A A system architecture diagram to which the training method of the image segmentation network according to an embodiment of the present disclosure is applied is schematically shown;

[0088] Figure 3B A schematic diagram of the training method of the image segmentation network according to an embodiment of the present disclosure is shown;

[0089] Figure 4A A flowchart schematically showing another training method of an image segmentation network according to an embodiment of the present disclosure is shown;

[0090] Figure 4B A schematic diagram showing how to obtain positive samples and negative samples according to an embodiment of the present disclosure is shown;

[0091] Figure 5A flowchart of an image segmentation method according to an embodiment of the present disclosure is schematically shown;

[0092] Figure 6 A structural diagram of a training apparatus of an image segmentation network according to an embodiment of the present disclosure is schematically shown;

[0093] Figure 7 A structural diagram of a training apparatus of another image segmentation network according to an embodiment of the present disclosure is schematically shown;

[0094] Figure 8 A computer readable storage medium according to an embodiment of the present disclosure is schematically shown;

[0095] Figure 9 A structural diagram of a computing device according to an embodiment of the present disclosure is schematically shown.

[0096] In the drawings, identical or corresponding reference signs indicate identical or corresponding parts. DETAILED DESCRIPTION

[0097] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present disclosure, and not to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0098] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, a device, an apparatus, a method or a computer program product. Therefore, the present disclosure can be embodied in the form of entire hardware, entire software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0099] According to embodiments of the present disclosure, a training method of an image segmentation network, and an image segmentation method, apparatus, computing device and storage medium are proposed. In this document, it should be understood that any number of elements in the drawings is used for illustration only, not for limitation, and any naming is only for distinction, not for any limiting meaning.

[0100] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present disclosure, and not to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0101] Portrait segmentation refers to separating the portrait and the background from an image or a video sequence, which is widely used in photographing, beautifying, special effects and other scenes. With the popularity of smart phones and the improvement of mobile computing performance, more and more applications for mobile portrait processing have emerged, and real-time portrait segmentation technology has become increasingly important. With the increasingly powerful representation ability of deep learning, using CNN for portrait segmentation has gradually become the main way. However, in order to improve the accuracy of portrait segmentation, the number of model parameters is increasing, and the amount of calculation is increasing, which brings great difficulty to real-time processing requirements. On the other hand, portrait images are complex and changeable. The portrait may appear at any position in the image, and may be in any posture, any number of people, any dress or wearing any accessories, which leads to the fact that the existing portrait segmentation technology often cannot completely obtain the target portrait area, resulting in the phenomenon of less segmentation or more segmentation of the portrait part, which affects the segmentation accuracy of the algorithm.

[0102] Therefore, the training scheme of the image segmentation network proposed by the embodiment of the present disclosure aims to improve the accuracy and precision of image segmentation of the image segmentation network without increasing the inference prediction time and the computing power.

[0103] Figure 1 A flowchart of a training method of an image segmentation network according to an embodiment of the present disclosure is schematically shown. The training method of the image segmentation network can be executed by a server, including a local server or a cloud server, etc., and the method can also be realized by a processor calling computer readable instructions stored in a memory. As Figure 1 shown, the method can include the following processing:

[0104] In step 101, a first sample image is input into a first image segmentation network to obtain feature information of the first sample image.

[0105] Wherein, the first image segmentation network is an image segmentation network to be trained, and the first sample image has annotation information. Wherein, the annotation information includes information indicating whether all or part of the pixels belong to the target object.

[0106] In some embodiments, in order to achieve real-time speed, a real-time segmentation model such as Bi SeNetV2 can be used as the first image segmentation network. Those skilled in the art should understand that the training method mentioned in the embodiment of the present disclosure is not limited to the network, and is also applicable to other segmentation models.

[0107] In step 102, using the first image segmentation network, the first mask of the target object and the first sample classification result are obtained according to the feature information, and the first sample classification result indicates whether the first sample image contains the target object.

[0108] In the embodiments of the present disclosure, multi-task prediction can be performed according to the feature information of the first sample image.

[0109] One of the tasks is to output a first mask of a target object. Taking a portrait as an example of the target object to be segmented, the output first mask can be a portrait binary mask. In the embodiments of the present disclosure, the branch performing this task can be referred to as a first task branch.

[0110] Another task that can also be included is to perform binary classification on the first sample image to distinguish whether the first sample image contains a target object. If the first sample image does not contain a target object, the first sample classification result is 1; if the first sample image contains a target object, the first sample classification result is 0. In the embodiments of the present disclosure, the branch performing this task can be referred to as a second task branch.

[0111] By adding the second task branch of performing binary classification on the first sample image, pure background samples without a target object can be preliminarily pre-screened out, and mask missegmentation can be reduced, for example, to avoid missegmentation caused by objects similar in shape and color to the target object in the first sample image without a target object, thereby improving the robustness of the segmentation result.

[0112] In step 103, a second classification result is obtained by using the first image segmentation network according to the first mask and the feature information, the second classification result indicating whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask in the first sample image meets a set requirement.

[0113] Another task that can also be included in the embodiments of the present disclosure is binary classification of the image contained in the rectangular bounding box corresponding to the first mask to distinguish whether the integrity of the target object contained in the rectangular bounding box in the first sample image meets a set requirement. If the integrity meets the set requirement, the second classification result is 1; if the integrity does not meet the set requirement, the second classification result is 0. The set requirement can include that the rectangular bounding box contains a complete target object, that is, contains all the pixels of the target object; the set requirement can also be that the proportion of the target object contained in the rectangular bounding box is higher than a set threshold, such as containing more than 90% of the pixels of the target object; the set requirement can also include that the mask corresponding foreground in the rectangular bounding box does not contain redundant pixels other than the target object, that is, the mask corresponding foreground does not contain background pixels. In the embodiments of the present disclosure, the branch performing this task can be referred to as a third task branch.

[0114] Taking a portrait as an example of the target object to be segmented, the branch can output the second classification result as 1 in the case that the complete portrait is contained in the rectangular portrait frame; output the second classification result as 0 in the case that the complete portrait is not contained in the rectangular portrait frame, for example, the rectangular portrait frame produces a shift or the background is incorrectly segmented.

[0115] For example, wherein, Figures 2A-2C is the first sample image, Figure 2A is the output image with the second classification result being 1, Figure 2B is the output image with the second classification result being 0, and Figure 2C is the output image with the second classification result being 0, and Figure 2C It can be seen that the foreground corresponding to the portrait mask in the rectangular boundary frame contains the background image that is incorrectly segmented.

[0116] By adding a third task branch for performing binary classification on the image contained in the rectangular boundary frame corresponding to the first mask, it can be effectively distinguished whether the completeness of the foreground extracted by the mask predicted by the first task branch meets the set requirements, and it can be supervised whether the segmentation result contains the incorrectly segmented background image or lacks part of the pixels, so as to improve the fineness of the image segmentation network.

[0117] In step 104, the first image segmentation network is trained according to the first mask, the first classification result, the second classification result, and the annotation information, and the first image segmentation network obtained after training is used as the second image segmentation network.

[0118] Since the annotation information of the first sample image indicates whether all or part of the pixels belong to the target object, that is, the annotation information is pixel-level, the true value corresponding to the first mask, the first classification result, and the second classification result can be obtained through the annotation information, so that the first image segmentation network can be trained through the first mask, the first classification result, the second classification result, and the annotation information, and the first image segmentation network completed after training is used as the second image segmentation network.

[0119] In the embodiments of the present disclosure, the feature information of the first sample image is obtained by using the first image segmentation network, and the first mask of the target object and the first classification result indicating whether the target object is contained in the first sample image are obtained according to the feature information, and the second classification result indicating whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask meets the set requirement is obtained according to the first mask and the feature information, and finally the first image segmentation network is trained according to the first mask, the first classification result, the second classification result and the annotation information, the pure background sample without the target object is screened out by using the first classification result, and whether the target object contains the background image missegmented is supervised by using the second classification result, so that the segmentation accuracy and the robustness of the image segmentation network are improved.

[0120] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure will be specifically introduced below.

[0121] Figure 3A The system architecture diagram to which the training method of the image segmentation network according to the embodiments of the present disclosure is applied is schematically shown.

[0122] As shown in Figure 3A , the first sample image 30 with annotation information is input into the first image segmentation network 300. In this example, the target object to be segmented is a portrait. First, the feature extraction network 301 in the first image segmentation network 300 is used to perform feature extraction on the first sample image 30 to obtain the feature information 302. The feature information 302 is, for example, the encoded feature obtained by performing feature encoding on the first sample image 30.

[0123] In the first task branch 310, the resolution recovery network 303 is used to recover the resolution from the feature information 302 to obtain the first mask 31. In this example, the portrait mask is obtained in the first task branch 310.

[0124] In the second task branch 320, the feature information 302 is rearranged into a vector 304, and a fully connected layer with two nodes is used for binary classification to obtain the first classification result 32 indicating whether the target object is contained in the first sample image 30. In this example, the classification result indicating whether the portrait is contained in the first sample image 30 is obtained in the second task branch 320.

[0125] In the third task branch 330, the second classification result 33 is obtained according to the feature information 302 and the first mask 31. Specifically, the first mask 31 is first converted into a second mask with the same resolution as the feature information 302. For example, the first mask 31 can be down-sampled to obtain the second mask, and then the second mask is multiplied with the feature information 302 to obtain foreground feature information. The foreground feature information is cropped and scaled to a specified scale, and a vector 305 corresponding to the scaled foreground feature information is obtained. A fully connected layer with two nodes is used for binary classification to obtain the second classification result 33. In this example, the third task branch 330 obtains a classification result indicating whether the integrity of the person image contained in the rectangular bounding box corresponding to the person image mask meets the specified requirement.

[0126] In some embodiments, the first image segmentation network can be trained using a multi-task fusion loss. As shown in FIG. 3B, the loss function used for training includes a first loss, a second loss, and a third loss. Figure 3B

[0127] The first loss, which can also be referred to as a segmentation loss, is used to indicate the difference between the first mask 31 and a mask ground truth value determined according to the annotation information. In the embodiments of the present disclosure, since the annotation information indicates whether each pixel in the first sample image belongs to the target object, the mask ground truth value corresponding to the target object can be obtained according to the annotation information. Moreover, the segmentation loss can be obtained by using a binary classification softmax for each pixel.

[0128] The second loss, which can also be referred to as a full-image classification loss, is used to indicate the difference between the first classification result 32 and a first classification ground truth value determined according to the annotation information, wherein the first classification ground truth value is a true value indicating whether the first sample image contains the target object. For example, in the case where the annotation information indicates that no pixel belongs to the target object, the true value of the first classification result can be determined as 0; otherwise, in the case where at least one pixel belongs to the target object, the true value of the first classification result can be determined as 1.

[0129] The third loss, which can also be referred to as an integrity classification loss, is used to indicate the difference between the second classification result 33 and a second classification ground truth value determined according to the annotation information, wherein the second classification ground truth value is a true value indicating whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask meets the specified requirement. Since the annotation information indicates whether each pixel in the first sample image belongs to the target object, it can be determined whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask meets the specified requirement, thereby obtaining the true value of the second classification result. ​

[0130] In some embodiments, the loss function is obtained by weighted sum of the first loss, the second loss and the third loss, i.e., the above losses are fused and trained by weighted formation, as shown in formula (1):

[0131] Loss = a1*first loss + a2*second loss + a3*third loss (1)

[0132] wherein a1 is the weight corresponding to the first loss, a2 is the weight corresponding to the second loss, and a3 is the weight corresponding to the third loss.

[0133] Since the problems targeted by the three task branches are different, the embodiments of the present disclosure propose a targeted and phased training strategy.

[0134] In the first phase of training, for example, when the number of iterations of training the first image segmentation network is less than a first set number, the weight a2 of the second loss decreases as the number of iterations increases. This is because at the beginning of training, the goal of the entire image segmentation network output result is to prioritize distinguishing between target objects and non-target objects, and in the case of target objects, the prediction of the other two tasks is performed. Usually, a2 is set to be greater than a1 and a3 at the beginning. However, as the number of training iterations increases, the process of gradually converging the network, the focus is no longer on distinguishing between target objects and non-target objects, and a2 gradually decreases.

[0135] In the second phase of training, for example, when the number of iterations of training the first image segmentation network is greater than or equal to the first set number and less than or equal to a second set number, the weight a1 of the first loss increases as the number of iterations increases. In the second phase, a1 is gradually increased to learn the mask prediction task.

[0136] In the third phase of training, for example, when the number of iterations of training the first image segmentation network is greater than the second set number, the weight a3 of the third loss increases as the number of iterations increases. As the number of iterations increases, it enters the stage of refining the mask prediction result, and a3 can be gradually increased, for example, to reach a balance with a1 or a little more, to constrain the first image segmentation network to further fine-tune the first mask.

[0137] In the embodiments of the present disclosure, by making the weights of each loss present different change trends in different training phases, targeted and phased training is achieved. In some embodiments, the weight a2 of the second loss can change exponentially with the number of iterations. For example, a2 can be attenuated according to formula (2):

[0138] a2 = λ2e-z (2)

[0139] wherein λ2 is a coefficient, which can be adjusted according to actual training; and the parameter z is determined according to the number of iterations. In an example, the current number of iterations i can be mapped to the value of the parameter z according to the value range of the parameter z. For example, in the case where the range of the parameter z is (0, N1] and the total number of iterations is M1, the current number of iterations i can be mapped to the parameter z by formula (3):

[0140]

[0141] In some embodiments, the weight a3 of the third loss can change logarithmically with the number of iterations. For example, a3 can increase according to formula (4):

[0142] a3 = λ3log(z) (4)

[0143] wherein λ3 is a coefficient, which can be adjusted according to actual training; and the parameter z is determined according to the number of iterations. In an example, the current number of iterations i can be mapped to the value of the parameter z according to the value range of the parameter z. For example, in the case where the range of the parameter z is [1, N2) and the total number of iterations is M2, the current number of iterations i can be mapped to the parameter z by formula (5):

[0144]

[0145] In some embodiments, the weight a1 of the first loss can be determined according to the weight a2 of the second loss and the weight a3 of the third loss. For example, a1 can be adjusted according to formula (6):

[0146] a1 = max(1-a2-a3, λ1) (6)

[0147] wherein λ1 is a coefficient for limiting the minimum value of a1, which can be set to 0.01, for example, to ensure that the third loss plays a certain role in the training process.

[0148] In the embodiments of the present disclosure, by making the weights corresponding to different losses present different change trends, stage-by-stage dynamic weight adjustment can be achieved, effectively enabling the image segmentation network to be trained in different stages with pertinence, solving the accuracy requirements of the network in different stages, and maximizing the respective roles of different branch tasks.

[0149] By the training method of the image segmentation network, the first image segmentation network is preliminarily trained using the first sample image, and the obtained second image segmentation network has a certain target object segmentation capability. However, since the target object segmentation image is complex, for example, in the case of segmenting a portrait, the gender, skin color, and photographing posture of the person are different, the length, color, and hairstyle of the hair are different, the dressing form and color of the person are different, and the shape and decoration of the clothing also have a lot of possibilities, for example, the person may wear a hat, carry a handbag, or carry a schoolbag. The background of the portrait image can also be diverse and complex, and the landscape, scenery, and indoor environment can have great differences. However, it is very difficult to obtain a high-precision labeled specimen covering all the above scenarios, and the labeling cost is very high. Therefore, in order to avoid the problem of insufficient samples, the semi-supervised image segmentation network training method is proposed in the embodiments of the present disclosure to further train the second image segmentation network. In the embodiments of the present disclosure, the stage of training using the first sample image described above can be referred to as the first training stage, and the semi-supervised training stage described below can be referred to as the second training stage.

[0150] Figure 4A The flowchart of the second stage training method of the image segmentation network according to the embodiments of the present disclosure is schematically shown. As shown in Figure 4A the second sample image 41 is input to the second image segmentation network 401 which has completed the first stage training, and the second image segmentation network 401 segments the second sample image to obtain a third mask 43 of the target object. The second sample image 41 is a sample image without labeled information.

[0151] At the same time, the second sample image 41 is input to the target detection network 402 to obtain a detection result 44 of the target object, wherein the detection result contains a detection box of the target object. The target detection algorithm used by the target detection network 402 can be, for example, Fast-rcnn, and in the case of detecting a human body, the target detection network is a human body detection network, which outputs a human body detection box.

[0152] Then, according to the third mask 43 and the detection box of the target object contained in the detection result 44, the positive sample corresponding to the second sample image 41 and the negative sample can be obtained, wherein the labeled information of the positive sample indicates that each pixel is a foreground pixel, and the labeled information of the negative sample indicates that each pixel is a non-foreground pixel.

[0153] By processing a large number of second sample images without labeled information through the above steps, a positive sample set and a negative sample set can be obtained, and the second image segmentation network 401 is optimized and trained using the obtained positive sample set and negative sample set.

[0154] In one example, the first sample image 40 can also be input to the second image segmentation network 401, and the network parameters of the second image segmentation network 401 are adjusted according to the difference between the predicted first mask 42 and the mask ground truth indicated by the annotation information, and the second image segmentation network 401 is optimized and trained together with the obtained positive sample set and negative sample set.

[0155] In the embodiments of the present disclosure, the mask of the unlabeled sample image is obtained by using the image segmentation network, and the detection box of the target object of the sample image is obtained by using the target detection network, to obtain the positive sample and the negative sample corresponding to the sample image, reasonably and sufficiently mine the effective information of the unlabeled sample image, optimize the image segmentation network, and increase the generalization ability of the image segmentation network.

[0156] Figure 4B A schematic diagram of obtaining positive samples and negative samples according to the embodiments of the present disclosure is shown. As shown in Figure 4B The first pixel in the detection box of the target object is determined as a positive sample, wherein the first pixel is a pixel whose probability of being a foreground pixel determined by the second image segmentation network is greater than a first set threshold; and the second pixel outside the detection box of the target object is determined as a negative sample, wherein the second pixel is a pixel whose probability of being a non-foreground pixel determined by the second image segmentation network is greater than a second set threshold.

[0157] In one example, all the pixels in the detection box of the target object can be roughly considered as positive samples of the target object, and the label is set to 1; and the label outside the detection box of the target object is 0. Then, among the pixels with label 1, the pixel whose probability of being a foreground pixel determined by the second image segmentation network is greater than a first set threshold, for example, 0.8, is determined as a positive sample; and among the pixels with label 0, the pixel whose probability of being a non-foreground pixel determined by the second image segmentation network is greater than a second set threshold, for example, 0.8, is determined as a negative sample.

[0158] In the embodiments of the present disclosure, by performing preliminary region segmentation according to the detection box of the target object, the credibility and accuracy of the positive samples and negative samples obtained by using the unlabeled samples can be improved.

[0159] The second-stage optimization training method provided by the embodiments of the present disclosure can extract sufficient positive and negative sample sets in various scenes from a large number of unlabeled sample images, and can fully cover various forms of target objects in the images and changes of costumes and scenes. On the basis of limited labeled samples, additional effective information is provided at extremely low cost to enhance the generalization ability and robustness of the image segmentation network. At the same time, all these operations are performed in the training process and do not affect the inference prediction time, and do not increase the additional calculation amount in actual inference, thereby adding effective assistance for real-time image segmentation.

[0160] Figure 5 A flowchart of an image segmentation method according to an embodiment of the present disclosure is schematically shown. The image segmentation method can be performed by an electronic device such as a terminal device or a server. The terminal device can be a fixed terminal or a mobile terminal, such as a mobile phone, a tablet computer, a game console, a desktop computer, an advertising machine, an all-in-one machine, a vehicle-mounted terminal, and the like. The server includes a local server or a cloud server, and the like. The method can also be realized by a processor calling computer readable instructions stored in a memory. As shown in the figure, the method can include the following processing: Figure 5

[0161] The image segmentation network can be an image segmentation network that has completed first-stage training, or an image segmentation network that has completed second-stage optimization training.

[0162] The first classification result is judged. In response to the first classification result indicating that the target object is contained in the image to be processed, a target mask of the target object is obtained according to the feature information, and the target mask is output. In response to the first classification result indicating that the target object is not contained in the image to be processed, a default image is output, and each pixel value in the default image is a set pixel value. That is, in the case where the first classification result indicates that the target object is contained in the image to be processed, the mask of the target object is further predicted according to the feature information, and the predicted target mask is output. In the case where the first classification result indicates that the target object is not contained in the image to be processed, the output is all set pixel values, such as a mask with a value of 0.

[0163] In the embodiments of the present disclosure, by adding the step of filtering the presence or absence of the target object, the direct output of the missegmentation result for the image to be processed containing an object similar to the target object is avoided, and compared with the use of the target detection network alone, the complexity of the process is reduced, and the efficiency of image segmentation is improved. ​

[0164] To implement the training method of the image segmentation network according to any one of the embodiments of the present disclosure, the embodiments of the present disclosure further provide a training device of an image segmentation network. Figure 6 The structure schematic diagram of the training device of the image segmentation network according to an embodiment of the present disclosure is schematically shown, for example, the device can be applied to a computing device. In the following description, the functions of each module of the device will be simply described, and the detailed processing can be combined with the description of the training method of the image segmentation network according to any one of the aforementioned embodiments of the present disclosure. As shown in the figure, the device can include: Figure 6

[0165] The first acquisition unit 601 is configured to input a first sample image into a first image segmentation network to obtain feature information of the first sample image, wherein the first sample image has annotation information.

[0166] The second acquisition unit 602 is configured to use the first image segmentation network to obtain a first mask of a target object and a first sample classification result according to the feature information, wherein the first sample classification result indicates whether the target object is contained in the sample image.

[0167] The third acquisition unit 603 is configured to use the first image segmentation network to obtain a second classification result according to the first mask and the feature information, wherein the second classification result indicates whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask in the first sample image meets a set requirement.

[0168] The first training unit 604 is configured to train the first image segmentation network according to the first mask, the first classification result, the second classification result and the annotation information, and the first image segmentation network obtained after the training is used as a second image segmentation network.

[0169] In an embodiment of the present disclosure, the second acquisition unit is specifically configured to:

[0170] Perform downsampling operation on the first mask to obtain a second mask, wherein the resolution of the second mask is the same as the resolution of the feature information.

[0171] Multiply the second mask and the feature information to obtain foreground feature information.

[0172] Crop the foreground feature information and scale the cropped foreground feature information to a set scale.

[0173] Obtain a vector corresponding to the scaled foreground feature information.

[0174] Input the vector into a fully connected layer in the first image segmentation network to obtain a second classification result.​

[0175] In an embodiment of the present disclosure, the first training unit is specifically configured to:

[0176] train the first image segmentation network based on a loss function, the loss function comprising:

[0177] a first loss for indicating a difference between the first mask and a mask ground truth value determined according to the annotation information;

[0178] a second loss for indicating a difference between the first classification result and a first classification ground truth value determined according to the annotation information, wherein the first classification ground truth value is a ground truth value of whether the target object is contained in the first sample image;

[0179] a third loss for indicating a difference between the second classification result and the second classification ground truth value determined according to the annotation information, wherein the second classification ground truth value is a ground truth value of whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask meets a set requirement.

[0180] In an embodiment of the present disclosure, the first training unit is specifically configured to:

[0181] train the first image segmentation network based on a loss function, the loss function comprising:

[0182] a first loss for indicating a difference between the first mask and a mask ground truth value determined according to the annotation information;

[0183] a second loss for indicating a difference between the first classification result and a first classification ground truth value determined according to the annotation information, wherein the first classification ground truth value is a ground truth value of whether the target object is contained in the first sample image;

[0184] a third loss for indicating a difference between the second classification result and the second classification ground truth value determined according to the annotation information, wherein the second classification ground truth value is a ground truth value of whether the integrity of the target object contained in the rectangular bounding box corresponding to the first mask meets a set requirement.

[0185] In an embodiment of the present disclosure, the loss function is obtained by weighted summation of the first loss, the second loss and the third loss.

[0186] In an embodiment of the present disclosure, when the number of iterations for training the first image segmentation network is less than a first set number, the weight of the third loss decreases with the increase of the number of iterations;

[0187] The weight of the first loss increases with the increase of the iteration number when the iteration number of training the first image segmentation network is greater than or equal to a first set number and less than or equal to a second set number.

[0188] The weight of the second loss increases with the increase of the iteration number when the iteration number of training the first image segmentation network is greater than a second set number.

[0189] In an embodiment of the present disclosure, the weight of the first loss exponentially changes with the iteration number, the weight of the second loss logarithmically changes with the iteration number, and the weight of the third loss is determined according to the weight of the first loss and the weight of the second loss.

[0190] In an embodiment of the present disclosure, the device further comprises a sample obtaining unit configured to:

[0191] inputting a second sample image into the second image segmentation network to obtain a third mask of the target object;

[0192] inputting the second sample image into a target detection network to obtain a detection frame of the target object;

[0193] obtaining a positive sample and a negative sample corresponding to the second sample image according to the third mask and the detection frame of the target object, wherein the annotation information of the positive sample indicates that each pixel is a foreground pixel, and the annotation information of the negative sample indicates that each pixel is a non-foreground pixel.

[0194] In an embodiment of the present disclosure, when the second training unit is configured to obtain a positive sample and a negative sample corresponding to the second sample image according to the third mask and the detection frame of the target object, the second training unit is specifically configured to:

[0195] determining a first pixel in the detection frame of the target object as a positive sample, wherein the first pixel is a pixel determined by the second image segmentation network as having a probability greater than a first set threshold value of being a foreground pixel;

[0196] determining a second pixel outside the detection frame of the target object as a negative sample, wherein the second pixel is a pixel determined by the second image segmentation network as having a probability greater than a second set threshold value of being a non-foreground pixel.

[0197] In an embodiment of the present disclosure, the device further comprises a second training unit configured to:

[0198] training the second image segmentation network according to the first sample image, the positive sample and the negative sample, and obtaining a first image segmentation network after the training as a third image segmentation network.

[0199] In a fourth aspect of the embodiments of the present disclosure, an image segmentation device is provided, comprising:

[0200] a feature extraction unit configured to input a to-be-processed image into an image segmentation network to obtain feature information of the to-be-processed image, and obtain a first classification result according to the feature information, the first classification result indicating whether the to-be-processed image contains a target object;

[0201] a first processing unit configured to, in response to the first classification result indicating that the to-be-processed image contains the target object, obtain a target mask of the target object according to the feature information, and output the target mask;

[0202] a second processing unit configured to, in response to the first classification result indicating that the to-be-processed image does not contain the target object, output a default image, each pixel value of the default image being a set pixel value.

[0203] To implement the image segmentation method of any of the embodiments of the present disclosure, the embodiments of the present disclosure further provide an image segmentation device. Figure 7 An example of a structure of an image segmentation device according to an embodiment of the present disclosure is shown schematically, for example, the device can be applied to a computing device. In the following description, the functions of each module of the device will be simply described, and the detailed processing can be combined with the description of the image segmentation method of any of the embodiments of the present disclosure described above. As shown in the figure, the device can include: Figure 7

[0204] a feature extraction unit 701 configured to input a to-be-processed image into an image segmentation network to obtain feature information of the to-be-processed image, and obtain a first classification result according to the feature information, the first classification result indicating whether the to-be-processed image contains a target object;

[0205] a first processing unit 702 configured to, in response to the first classification result indicating that the to-be-processed image contains the target object, obtain a target mask of the target object according to the feature information, and output the target mask;

[0206] a second processing unit 703 configured to, in response to the first classification result indicating that the to-be-processed image does not contain the target object, output a default image, each pixel value of the default image being a set pixel value.

[0207] ​It should be noted that although the training apparatus of the image segmentation network and the several units / modules or sub-units / modules of the image segmentation apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into several units / modules.

[0208] The embodiments of the present disclosure further provide a computer readable storage medium. As shown in the figure, the storage medium stores a computer program 801, which can execute the training method of the image segmentation network or the image segmentation method of any embodiment of the present disclosure when executed by a processor. Figure 8 The embodiments of the present disclosure further provide a computer readable storage medium. As shown in the figure, the storage medium stores a computer program 801, which can execute the training method of the image segmentation network or the image segmentation method of any embodiment of the present disclosure when executed by a processor.

[0209] The embodiments of the present disclosure further provide a computing device, which can include a memory for storing computer instructions executable on a processor, and the processor for implementing the training method of the image segmentation network or the image segmentation method of any embodiment of the present disclosure when executing the computer instructions.

[0210] Figure 9 An example structure of the computing device is shown in the figure, which can include but is not limited to a processor 91, a memory 92, a bus 93 connecting different system components including the memory 92 and the processor 91. Figure 9

[0211] The memory 92 stores computer instructions executable by the processor 91, so that the processor 91 can execute the training method of the image segmentation network or the image segmentation method of any embodiment of the present disclosure. The memory 92 can include a random access memory unit RAM 921, a cache memory unit 922 and / or a read-only memory unit ROM 923. The memory 92 can further include a program tool 925 having a set of program modules 924, which include but are not limited to an operating system, one or more application programs, other program modules and program data, which can include implementation of a network environment in one or more combinations.

[0212] The bus 93 can include, for example, a data bus, an address bus and a control bus, etc. The computing device 90 can also communicate with an external device 95, such as a keyboard, a Bluetooth device, etc. through an I / O interface 94. The computing device 90 can also communicate with one or more networks, such as a local area network, a wide area network, a public network, etc. through a network adapter 96. As shown in the figure, the computing device 90 can further include a display interface 97, which can be used to display information to a user. Figure 9 ​As shown, the network adapter 96 can also communicate with other modules of the computing device 90 over the bus 93.

[0213] Moreover, although the operations of the method(s) herein can be described in a particular, sequential order, this order is not meant to be a limitation and is not intended to imply that

[0214] While the spirit and principles of the disclosure have been described with reference to several particular embodiments, it is to be understood that the disclosure is not limited to the particular embodiments disclosed and that various modifications in form and detail can be made without departing from the scope of the disclosure. It is further understood that the disclosure encompasses all such modifications and changes and equivalents thereof within the spirit and scope of the appended claims.

Claims

1. A training method for an image segmentation network, characterized in that, include: The first sample image is input into the first image segmentation network to obtain the feature information of the first sample image, wherein the first sample image has annotation information; Using the first image segmentation network, a first mask of the target object and a first classification result are obtained based on the feature information. The first classification result indicates whether the target object is contained in the sample image. Using the first image segmentation network, a second classification result is obtained based on the first mask and the feature information. The second classification result indicates whether the integrity of the target object in the first sample image contained in the rectangular bounding box corresponding to the first mask meets the set requirements. Based on the first mask, the first classification result, the second classification result, and the annotation information, the first image segmentation network is trained, and the trained first image segmentation network serves as the second image segmentation network. This process includes: training the first image segmentation network based on a loss function, wherein the loss function is obtained by fusing a first loss, a second loss, and a third loss. The first loss is used to indicate the difference between the first mask and the actual mask value determined according to the annotation information; The second loss is used to indicate the difference between the first classification result and the first classification true value determined according to the annotation information, wherein the first classification true value is the true value of whether the target object is contained in the first sample image; The third loss is used to indicate the second classification result and the second classification true value determined according to the annotation information, wherein the second classification true value is the true value of whether the integrity of the target object in the first sample image contained in the rectangular bounding box corresponding to the first mask meets the set requirements.

2. The method according to claim 1, characterized in that, The step of using the first image segmentation network to obtain a second classification result based on the first mask and the feature information includes: A downsampling operation is performed on the first mask to obtain a second mask, wherein the resolution of the second mask is the same as the resolution of the feature information; The second mask is multiplied with the feature information to obtain the foreground feature information; The foreground feature information is cropped out, and the cropped foreground feature information is scaled to a set size; Obtain the vector corresponding to the scaled foreground feature information; The vector is input into the fully connected layer of the first image segmentation network to obtain the second classification result.

3. The method according to claim 1, characterized in that, The loss function is obtained by fusing the first loss, the second loss, and the third loss, including: the loss function is obtained by weighted summation of the first loss, the second loss, and the third loss.

4. The method according to claim 3, characterized in that, If the number of iterations for training the first image segmentation network is less than a first predetermined number, the weights of the third loss decrease as the number of iterations increases. When the number of iterations for training the first image segmentation network is greater than or equal to a first set number and less than or equal to a second set number, the weight of the first loss increases with the increase of the number of iterations. If the number of iterations for training the first image segmentation network is greater than the second set number, the weights of the second loss increase with the increase of the number of iterations.

5. The method according to claim 3, characterized in that, The weight of the first loss changes exponentially with the number of iterations, the weight of the second loss changes logarithmically with the number of iterations, and the weight of the third loss is determined based on the weights of the first loss and the second loss.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The second sample image is input into the second image segmentation network to obtain the third mask of the target object; The second sample image is input into the target detection network to obtain the detection box of the target object; Based on the detection bounding box of the third mask and the target object, positive samples and negative samples corresponding to the second sample image are obtained, wherein the annotation information of the positive sample indicates that each pixel is a foreground pixel, and the annotation information of the negative sample indicates that each pixel is a non-foreground pixel.

7. The method according to claim 6, characterized in that, The step of obtaining the positive and negative samples corresponding to the second sample image based on the detection bounding box of the third mask and the target object includes: The first pixel within the detection box of the target object is determined as a positive sample, wherein the first pixel is a pixel whose probability of being a foreground pixel determined by the second image segmentation network is greater than a first set threshold; The second pixel outside the detection box of the target object is determined as a negative sample, wherein the second pixel is a pixel whose probability of being a non-foreground pixel determined by the second image segmentation network is greater than a second set threshold.

8. The method according to claim 6, characterized in that, The method further includes: training the second image segmentation network based on the first sample image, the positive sample, and the negative sample, and using the trained first image segmentation network as the third image segmentation network.

9. An image segmentation method, characterized in that, include: The image to be processed is input into an image segmentation network to obtain the feature information of the image to be processed; A first classification result is obtained based on the feature information, and the first classification result indicates whether the image to be processed contains a target object; the image segmentation network is trained based on the method of any one of claims 1-8; In response to the first classification result indicating that the image to be processed contains the target object, a target mask of the target object is obtained based on the feature information, and the target mask is output. In response to the first classification result indicating that the target object is not contained in the image to be processed, a default image is output, wherein each pixel value in the default image is a set pixel value.

10. A training device for an image segmentation network, characterized in that, include: The first acquisition unit is used to input the first sample image into the first image segmentation network to obtain the feature information of the first sample image, wherein the first sample image has annotation information; The second acquisition unit is used to obtain a first mask of the target object and a first classification result based on the feature information using the first image segmentation network, wherein the first classification result indicates whether the sample image contains the target object. The third acquisition unit is used to obtain a second classification result by using the first image segmentation network based on the first mask and the feature information. The second classification result indicates whether the integrity of the target object in the first sample image contained in the rectangular bounding box corresponding to the first mask meets the set requirements. A first training unit is configured to train the first image segmentation network based on the first mask, the first classification result, the second classification result, and annotation information. The first image segmentation network obtained after training serves as the second image segmentation network. The training unit includes: training the first image segmentation network based on a loss function, wherein the loss function is obtained by fusing a first loss, a second loss, and a third loss. The first loss is used to indicate the difference between the first mask and the actual mask value determined according to the annotation information; The second loss is used to indicate the difference between the first classification result and the first classification true value determined according to the annotation information, wherein the first classification true value is the true value of whether the target object is contained in the first sample image; The third loss is used to indicate the second classification result and the second classification true value determined according to the annotation information, wherein the second classification true value is the true value of whether the integrity of the target object in the first sample image contained in the rectangular bounding box corresponding to the first mask meets the set requirements.

11. The apparatus according to claim 10, characterized in that, The second acquisition unit is specifically used for: A downsampling operation is performed on the first mask to obtain a second mask, wherein the resolution of the second mask is the same as the resolution of the feature information; The second mask is multiplied with the feature information to obtain the foreground feature information; The foreground feature information is cropped out, and the cropped foreground feature information is scaled to a set size; Obtain the vector corresponding to the scaled foreground feature information; The vector is input into the fully connected layer of the first image segmentation network to obtain the second classification result.

12. The apparatus according to claim 10, characterized in that, The loss function is obtained by fusing the first loss, the second loss, and the third loss, including: the loss function is obtained by weighted summation of the first loss, the second loss, and the third loss.

13. The apparatus according to claim 12, characterized in that, If the number of iterations for training the first image segmentation network is less than a first predetermined number, the weights of the third loss decrease as the number of iterations increases. When the number of iterations for training the first image segmentation network is greater than or equal to a first set number and less than or equal to a second set number, the weight of the first loss increases with the increase of the number of iterations. If the number of iterations for training the first image segmentation network is greater than the second set number, the weights of the second loss increase with the increase of the number of iterations.

14. The apparatus according to claim 12, characterized in that, The weight of the first loss changes exponentially with the number of iterations, the weight of the second loss changes logarithmically with the number of iterations, and the weight of the third loss is determined based on the weights of the first loss and the second loss.

15. The apparatus according to any one of claims 10 to 14, characterized in that, The device further includes a sample acquisition unit for: The second sample image is input into the second image segmentation network to obtain the third mask of the target object; The second sample image is input into the target detection network to obtain the detection box of the target object; Based on the detection bounding box of the third mask and the target object, positive samples and negative samples corresponding to the second sample image are obtained, wherein the annotation information of the positive sample indicates that each pixel is a foreground pixel, and the annotation information of the negative sample indicates that each pixel is a non-foreground pixel.

16. The apparatus according to claim 15, characterized in that, The step of obtaining the positive and negative samples corresponding to the second sample image based on the detection bounding box of the third mask and the target object includes: The first pixel within the detection box of the target object is determined as a positive sample, wherein the first pixel is a pixel whose probability of being a foreground pixel determined by the second image segmentation network is greater than a first set threshold; The second pixel outside the detection box of the target object is determined as a negative sample, wherein the second pixel is a pixel whose probability of being a non-foreground pixel determined by the second image segmentation network is greater than a second set threshold.

17. The apparatus according to claim 15, characterized in that, The device further includes a second training unit for: The second image segmentation network is trained based on the first sample image, the positive sample, and the negative sample, and the resulting first image segmentation network is used as the third image segmentation network.

18. An image segmentation apparatus, characterized in that, The device includes: A feature extraction unit is used to input the image to be processed into an image segmentation network to obtain feature information of the image to be processed; and to obtain a first classification result based on the feature information, wherein the first classification result indicates whether the image to be processed contains a target object; the image segmentation network is trained based on the method of any one of claims 1-8. A first processing unit is configured to, in response to the first classification result indicating that the image to be processed contains the target object, obtain a target mask of the target object based on the feature information, and output the target mask; The second processing unit is configured to output a default image in response to the first classification result indicating that the image to be processed does not contain the target object, wherein each pixel value in the default image is a set pixel value.

19. A computing device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as claimed in any one of claims 1 to 8 or claim 9.

20. A computer-readable storage medium, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8 or claim 9.

Images