Semi-supervised semantic segmentation and model training method and device based on weighted contrast network

Abstract

The present application relates to the field of semantic segmentation technology, and specifically to a semi-supervised semantic segmentation and model training method and device based on a weighted contrast network. The training method includes: obtaining a labeled image set containing a target object and a corresponding label set, as well as an unlabeled image set containing a target object but without a label; inputting the labeled image set into the first basic network of the first branch and the second basic network of the second branch respectively, and correspondingly obtaining a first target mask set and a second target mask set; inputting the unlabeled image set into the first basic network, the second basic network, the first enhanced network and the second enhanced network, and correspondingly obtaining a third target mask set, a fourth target mask set, a first enhanced pseudo label set and a second enhanced pseudo label set; adjusting the parameters of the first basic network and the second basic network based on the total difference to complete the training of the semantic segmentation model. This improves the problem of poor performance of the existing technology due to imbalanced sample categories.

Description

Technical Field

[0001] The present application relates to the field of semantic segmentation technology, and in particular to a method and device for semi-supervised semantic segmentation and model training based on a weighted contrast network. Background Art

[0002] With the rapid development of deep learning, computer vision tasks have flourished across various industries. Image semantic segmentation methods based on deep convolutional neural networks are gaining increasing attention, and their application is gradually expanding from routine everyday image semantic segmentation to semantic segmentation in specific application scenarios. The application of deep learning to semantic segmentation has significantly improved image semantic segmentation performance. Training a good segmentation model with a neural network requires a large number of labeled training samples. Semantic segmentation is a pixel-by-pixel classification process, and labeling high-quality pixel-level training samples is a labor-intensive, material, and financially intensive process.

[0003] Furthermore, semantic segmentation tasks face the problem of class imbalance. Because semantic segmentation is a pixel-based classification task, the number of randomly acquired class training samples for different task scenarios can vary significantly. An imbalance in the number of training samples increases the difficulty of model training while reducing both robustness and accuracy. Therefore, a method and apparatus for semi-supervised semantic segmentation and model training based on a weighted contrast network is needed. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a semi-supervised semantic segmentation and model training method and device based on a weighted contrast network, which improves the problem of poor model training performance in the prior art due to sample imbalance.

[0005] To achieve the above-mentioned purpose and other related purposes, the present application provides a semi-supervised semantic segmentation model training method based on a weighted contrast network, the training method comprising: obtaining a labeled image set containing a target object and a corresponding label set, and an unlabeled image set containing a target object but without labels; wherein the number of unlabeled images is much larger than the number of labeled images; inputting the labeled image set into the first basic network of the first branch and the second basic network of the second branch respectively, performing semantic segmentation on the target objects in the labeled image set, and correspondingly obtaining a first target mask set and a second target mask set; wherein the first branch comprises a first basic network and a first enhanced network, the second branch comprises a second basic network and a second enhanced network, and the first enhanced network, the second enhanced network, the first basic network, and the second basic network are The structures are the same but the initial parameters are different; the unlabeled image set is respectively input into the first basic network, the second basic network, the first enhanced network and the second enhanced network, and semantic segmentation is performed on the target objects in the unlabeled image set, and a third target mask set, a fourth target mask set, a first enhanced pseudo label set and a second enhanced pseudo label set are correspondingly obtained; the difference between the first target mask set, the second target mask set and the label set, the difference between the third target mask set and the second enhanced pseudo label set, and the difference between the fourth target mask set and the first enhanced pseudo label set are respectively calculated to obtain a total difference; the parameters of the first basic network and the second basic network are adjusted based on the total difference to complete the training of the semantic segmentation model; wherein the semantic segmentation model includes the trained first basic network and / or the second basic network.

[0006] In one embodiment of the present invention, the labeled image set is input into the first basic network of the first branch and the second basic network of the second branch respectively, and semantic segmentation is performed on the target objects in the labeled image set to obtain the first target mask set and the second target mask set respectively, including: performing a first perturbation process on the labeled image set to obtain a perturbed labeled image set; inputting the perturbed labeled image set into the first basic network, performing semantic segmentation on the target objects in the perturbed labeled image set, and obtaining the first target mask set; inputting the perturbed labeled image set into the second basic network, performing semantic segmentation on the target objects in the perturbed labeled image set, and obtaining the second target mask set.

[0007] In one embodiment of the present invention, the perturbed labeled image set is input into the first basic network, and the target objects in the perturbed labeled image set are semantically segmented to obtain the corresponding first target mask set, including: inputting the perturbed labeled image set into the first basic network, and performing semantic segmentation on the target objects in the perturbed labeled image set to obtain the maximum category confidence of each pixel of all labeled images in the perturbed labeled image set; based on the maximum category confidence of each pixel, respectively calculating the confidence mean of each perturbed labeled image; for each pixel in the perturbed labeled image: determining whether the maximum category confidence of the pixel is greater than or equal to the corresponding confidence mean: if so, using the category number corresponding to the maximum category confidence as the first target mask of the pixel; otherwise, not assigning the first target mask to the pixel; and counting the first target masks in all perturbed labeled images to obtain the first target mask set.

[0008] In one embodiment of the present invention, the unlabeled image set is respectively input into the first enhancement network and the second enhancement network, and semantic segmentation is performed on the target objects in the unlabeled image set, and a first enhanced pseudo label set and a second enhanced pseudo label set are correspondingly obtained, including: performing a second perturbation process on the unlabeled image set to obtain a perturbed unlabeled image set; inputting the perturbed unlabeled image set into the first enhancement network, and performing semantic segmentation on the target objects in the perturbed unlabeled image set, and correspondingly obtaining a first enhanced pseudo label set; inputting the perturbed unlabeled image set into the second enhancement network, and performing semantic segmentation on the target objects in the perturbed unlabeled image set, and correspondingly obtaining a second enhanced pseudo label set.

[0009] In one embodiment of the present invention, the perturbed unlabeled image set is input into a second enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain a corresponding second enhanced pseudo label set, including: inputting the perturbed unlabeled image set into the second enhancement network, performing semantic segmentation on the target objects in the perturbed unlabeled image set, and obtaining the maximum category confidence of each pixel point of all unlabeled images in the perturbed unlabeled image set; based on the maximum category confidence of each pixel point, calculating the confidence mean of each perturbed unlabeled image; for each pixel point in the perturbed unlabeled image: determining whether the maximum category confidence of the pixel point is greater than the confidence mean; if so, using the category number corresponding to the maximum category confidence as the second enhanced pseudo label of the pixel point; otherwise, not assigning the second enhanced pseudo label to the pixel point; and counting the second enhanced pseudo labels in all perturbed unlabeled images to obtain a second enhanced pseudo label set.

[0010] In one embodiment of the present invention, calculating the difference between the third target mask set and the second enhanced pseudo-label set includes: determining the category of each second enhanced pseudo-label in the second enhanced pseudo-label set; calculating the category ratio of each category of second enhanced pseudo-label to all second enhanced pseudo-labels in the second enhanced pseudo-label set; for each category of second enhanced pseudo-label in the second enhanced pseudo-label set: calculating the first semi-supervised difference between the third target mask set and the second enhanced pseudo-label of that category; and weighting the first semi-supervised difference based on the category ratio to obtain a final difference.

[0011] In one embodiment of the present invention, the semantic segmentation model is obtained by weighting a trained first basic network and a trained second basic network.

[0012] In one embodiment of the present invention, a semantic segmentation method is also provided, which includes: obtaining an image to be segmented; inputting the image into a semantic segmentation model to obtain a target mask for segmenting the target object in the image; wherein the semantic segmentation model is trained by any of the above-mentioned semi-supervised semantic segmentation model training methods based on a weighted contrast network; and segmenting the image based on the target mask.

[0013] In one embodiment of the present invention, a semi-supervised semantic segmentation model training system based on a weighted contrast network is also provided, and the training system includes: an image acquisition module for acquiring a labeled image set containing a target object and a corresponding label set, and an unlabeled image set containing a target object but without labels; wherein the number of unlabeled images is much larger than the number of labeled images; a supervised module for inputting the labeled image set into the first basic network of the first branch and the second basic network of the second branch, respectively, performing semantic segmentation on the target objects in the labeled image set, and correspondingly obtaining a first target mask set and a second target mask set; wherein the first branch includes a first basic network and a first enhanced network, and the second branch includes a second basic network and a second enhanced network, and the structures of the first enhanced network, the second enhanced network, the first basic network, and the second basic network are the same but the initial parameters are different. The number is different; an unsupervised module is used to input the unlabeled image set into the first basic network, the second basic network, the first enhanced network and the second enhanced network respectively, perform semantic segmentation on the target objects in the unlabeled image set, and obtain a third target mask set, a fourth target mask set, a first enhanced pseudo label set and a second enhanced pseudo label set accordingly; a loss calculation module is used to respectively calculate the difference between the first target mask set, the second target mask set and the label set, the difference between the third target mask set and the second enhanced pseudo label set, and the difference between the fourth target mask set and the first enhanced pseudo label set to obtain a total difference; a parameter updating module is used to adjust the parameters of the first basic network and the second basic network based on the total difference to complete the training of the semantic segmentation model; wherein, the semantic segmentation model includes the trained first basic network and / or the second basic network.

[0014] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements any of the above-mentioned semi-supervised semantic segmentation model training methods based on weighted contrast networks or the semantic segmentation methods.

[0015] In one embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a computer processor, the computer executes any of the above-mentioned semi-supervised semantic segmentation model training methods based on weighted contrast networks or the above-mentioned semantic segmentation methods.

[0016] As described above, the present invention's method and device for semi-supervised semantic segmentation and model training based on a weighted contrast network has the following beneficial effects: by introducing unlabeled images, which are much larger in number than labeled images, and combining the enhanced network to generate pseudo-labels for training, the reliance on a large amount of pixel-level manual annotation is effectively reduced, significantly reducing the manpower and time costs in the data preparation process. In addition, by generating pseudo-labels through the enhanced network and cross-comparing them with the prediction results of the basic network, the potential semantic information in the unlabeled images can be mined, further enhancing the model's learning ability and generalization ability for complex scenes. Therefore, this method of the present invention improves the problem of poor performance of the prior art due to imbalanced sample categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a semi-supervised semantic segmentation model training method based on a weighted contrast network provided in an embodiment of the present application;

[0019] Figure 2 Schematic diagram of the structure of the semi-supervised semantic segmentation model based on weighted contrast network provided in this application;

[0020] Figure 3 A flowchart of the semantic segmentation method provided in an embodiment of the present application;

[0021] Figure 4 Shown is a structural block diagram of a semi-supervised semantic segmentation model training system based on a weighted contrast network provided by an embodiment of the present invention;

[0022] Figure 5 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0025] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0026] The inventors found that as the semi-supervised image classification algorithm shows super high results, the demand for designing a semi-supervised semantic segmentation algorithm that uses a small amount of labeled training samples and a large number of unlabeled training samples is becoming more and more urgent, and semi-supervised semantic segmentation is also receiving more and more attention. When only a small amount of labeled data is needed to train a good neural network model, it can not only save resources but also be deployed in a timely manner to various application scenarios to obtain greater benefits. However, an imbalance in the number of training samples often leads to an increase in the difficulty of model training and reduces the robustness and accuracy of the model. Therefore, if the model can adaptively adjust the loss and number of samples during training, the segmentation algorithm can be applied to a wider range of tasks. Therefore, the two difficulties in scene image semantic segmentation are the label annotation problem and the training imbalance problem.

[0027] The present application provides a training method for a semi-supervised semantic segmentation model based on a weighted contrast network. The semantic segmentation model consists of two basic networks and their corresponding two enhanced networks. The four network structures are exactly the same, but different initialization methods are used to enhance the diversity and robustness of the model. During the training process, the two basic networks are subjected to standard supervised training using labeled images to make full use of manually annotated information. For unlabeled images, the enhanced network first generates enhanced pseudo-labels and uses them as supervisory signals to cross-guide the training of the basic network of the other branch, thereby mining the potential semantic features in the image. In order to alleviate the problem of uneven category distribution during training, an adaptive loss weighting mechanism is designed to dynamically adjust the loss weight according to the pixel proportion of different categories in the image enhanced pseudo-label to improve the model's learning ability for low-frequency categories. In addition, in the supervised training path, a difficult sample screening mechanism is introduced to select relevant pixel areas from the labeled images to guide the basic network to focus on areas that are difficult to classify for key learning. In the semi-supervised path, only pseudo-label pixels with high confidence in the unlabeled images are retained for training to ensure the reliability of the semi-supervised signal. After each training batch is completed, an exponential sliding average strategy is used to perform momentum updates on the corresponding augmented network based on the current parameters of the base network to maintain the prediction stability and pseudo-label quality of the augmented network. A semantic segmentation model is constructed using the converged base network to achieve accurate segmentation of target objects in the image.

[0028] like Figure 1 As shown in FIG, the training method of the semi-supervised semantic segmentation model based on the weighted contrast network includes the following steps:

[0029] S11. Obtain a set of labeled images containing target objects and a corresponding label set, as well as a set of unlabeled images containing target objects but without labels; wherein the number of unlabeled images is much greater than the number of labeled images.

[0030] To construct a dataset for a semi-supervised semantic segmentation model, multiple sample images containing target objects are acquired. These images are then divided into labeled and unlabeled images based on the presence of manually annotated pixel-level semantic labels, forming a labeled image set and an unlabeled image set. Labeled images have corresponding label masks that characterize the semantic category of each pixel in the labeled image, such as whether the pixel is a vehicle, road, or pedestrian. Unlabeled images, while still containing target objects, lack manually annotated labels. The method for acquiring sample images is not limited and can be adapted based on the actual task needs. For example, for traffic scenes, sample images can be collected using multiple cameras, such as cameras installed on traffic roads. During the sampling process, sample images are acquired by covering multiple typical scene areas where cameras are deployed. Random sampling is performed across images from different stations to ensure the breadth and representativeness of the training samples in terms of spatial and scene distribution, further enhancing the model's adaptability to real-world scenarios.

[0031] Furthermore, considering that manual pixel-level semantic annotation has problems such as high cost and slow speed, in order to effectively reduce the annotation overhead and improve the training efficiency, this application uses a much larger number of unlabeled images than labeled images, where "much larger" means that the number of unlabeled images is 10 times or more different from the number of labeled images. In this way, while significantly reducing the cost of manual annotation, the potential semantic information in the unlabeled images can be fully utilized through the pseudo-label mechanism, thereby enhancing the generalization ability of the semi-supervised semantic segmentation model for complex scenes and improving the segmentation accuracy. For example, the sample images include m labeled images for training, unlabeled images for training, validation images for verification and several test images for testing model performance, where . Use the segmentation annotation tool Marked Label images.

[0032] In terms of data annotation, suppose the training set includes m labeled images, denoted as ,in Represents the i-th labeled image and its corresponding label. Similarly, assuming that the training set includes n unlabeled images, denoted as ,in represents the jth unlabeled image, , that is, the number of unlabeled images is much larger than the number of labeled images. and a collection of n unlabeled images The union of t images constitutes the training set, which is used to train the semantic segmentation model. The validation set consisting of t images is recorded as ,in Represents the kth verification image and the corresponding label in the verification set, .

[0033] S12. Input the labeled image set into the first basic network of the first branch and the second basic network of the second branch respectively, perform semantic segmentation on the target objects in the labeled image set, and obtain the first target mask set and the second target mask set accordingly; wherein, the first branch includes the first basic network and the first enhanced network, the second branch includes the second basic network and the second enhanced network, and the structures of the first enhanced network, the second enhanced network, the first basic network, and the second basic network are the same but the initial parameters are different.

[0034] During the training process, the labeled image set is input into two branches with symmetrical structures respectively: the first branch includes the first basic network and the first enhanced network, and the second branch includes the second basic network and the second enhanced network. The structures of these four networks are the same, but the initialization methods of the enhanced network and the basic network are different. Among them, the initialization methods include but are not limited to uniform distribution initialization, normal distribution initialization, kaiming initialization, etc. Among them, the two basic networks can adopt the same initialization method or different initialization methods. For example, the first basic network uses Xavier uniform distribution initialization, and the second basic network uses Kaiming uniform distribution initialization. After the initialization is completed, the labeled image set is input into the first basic network and the second basic network respectively, and the first target mask set and the second target mask set are obtained accordingly, wherein each first target mask / second target mask is used to represent the semantic category of all pixels in the corresponding labeled image. It can be understood that the basic network and the enhanced network can use any of the current mainstream segmentation networks, including but not limited to , or A variant of series and Those skilled in the art can adaptively select the corresponding network based on actual task requirements, and there is no specific limitation.

[0035] In an optional embodiment of the present application, step S12 includes steps S121 to S123:

[0036] S121 . Perform a first disturbance process on the labeled image set to obtain a disturbed labeled image set.

[0037] In order to improve the robustness of the semantic segmentation model for images, before the labeled image is input into the first basic network, the image needs to be subjected to a first perturbation process, so as to effectively simulate the diverse observation conditions in real life, reduce the risk of model overfitting and improve the robustness of the model, obtain the perturbed labeled image, and form a set of perturbed labeled images. It is understandable that for each image, only one type of first perturbation can be performed, and the image can also be subjected to multiple different types of first perturbations, which are not limited here. Among them, the first perturbation method includes but is not limited to randomly flipping the image horizontally or vertically; adjusting the image saturation and hue; injecting Gaussian noise into the image to simulate sensor interference during image acquisition; and randomly cropping and pasting the image to improve the model's robustness to occlusion and local structural changes.

[0038] It should be noted that, considering that the images used in this application are all RGB images, the first perturbation process is performed on the labeled image set to obtain the perturbed labeled image set, which includes the following process: first, normalizing the labeled image set to obtain a normalized labeled image set; performing the first perturbation process on the normalized labeled image set to obtain a perturbed labeled image set. Specifically, all labeled images in the labeled image set are normalized, that is, ,in Represents the original labeled image, h and w are the height and width of the image. In this embodiment, the width and height of the image are , the number of image channels is 3, The normalized labeled images are then constrained to have a value range of RGB channels between 0 and 1, which facilitates the convergence of semantic segmentation model training. The normalized labeled images are then subjected to the first perturbation process to obtain a set of perturbed labeled images.

[0039] S122: Input the perturbed labeled image set into the first basic network, perform semantic segmentation on the target objects in the perturbed labeled image set, and obtain a first target mask set.

[0040] The perturbed labeled image set is fed into the first base network. For each labeled image, pixel-level semantic segmentation is performed on the target object in the image, generating predicted probabilities for each pixel in each semantic category. For each pixel, softmax normalization is performed to obtain the semantic category with the highest predicted probability. The semantic categories corresponding to all pixels are summed to obtain the first object mask. For all perturbed labeled images in the current training batch, the corresponding first object mask set is obtained.

[0041] In an optional embodiment of the present application, step S122 includes the following process:

[0042] First, the perturbed labeled image set is input into the first basic network, and semantic segmentation is performed on the target objects in the perturbed labeled image set to obtain the maximum category confidence of each pixel point of all labeled images in the perturbed labeled image set.

[0043] For each perturbed labeled image, the following process is performed: the perturbed labeled image is input into the first basic network, and the predicted probability of each pixel in each semantic category is generated. After softmax normalization, the highest predicted probability corresponding to each pixel is obtained and used as the maximum category confidence of the pixel.

[0044] After obtaining the maximum category confidence of each pixel, the confidence mean of each perturbed labeled image is calculated based on the maximum category confidence of each pixel.

[0045] For each perturbed labeled image, calculate the mean of the maximum class confidence scores for all pixels and use this as the image's mean confidence score. This mean confidence score represents the accuracy of the first base network's semantic predictions for the entire image and can be used as a threshold for filtering difficult-to-train samples.

[0046] Then, for each pixel in the perturbed labeled image, determine whether the maximum category confidence of the pixel is greater than the corresponding confidence mean. If so, use the category number corresponding to the maximum category confidence as the first target mask for the pixel. Otherwise, do not assign the first target mask to the pixel.

[0047] For each pixel in the perturbed labeled image, the maximum category confidence of the pixel is compared with the confidence mean of the entire image: if the maximum category confidence of the pixel is higher than or equal to the adaptive confidence mean of the current image, the prediction result of the pixel is considered to have high confidence, and the category number corresponding to its maximum category confidence is used as the first target mask of the pixel; otherwise, it is considered to be a low-confidence area, and to avoid interfering with training, the first target mask is not assigned to the pixel. In this way, pixel areas in the supervised image that are difficult to distinguish or uncertain by the model can be dynamically screened out, thereby constructing a confidence mask map, making the model training more focused on sample areas with learning value, improving the segmentation ability of complex areas such as boundaries, occlusions, and weak contrast, and enhancing the overall robustness and generalization performance of the model.

[0048] Finally, the first target masks in all perturbed labeled images are counted to obtain the first target mask set.

[0049] The first target masks of all perturbed labeled images are arranged according to the input order of the images to form a first target mask set.

[0050] S123: Input the perturbed labeled image set into the second basic network, perform semantic segmentation on the target objects in the perturbed labeled image set, and obtain a second target mask set.

[0051] The perturbed labeled image set is fed into the second base network. For each labeled image, pixel-level semantic segmentation is performed on the target object in the image, generating predicted probabilities for each pixel in each semantic category. For each pixel, softmax normalization is performed to obtain the semantic category with the highest predicted probability. The semantic categories corresponding to all pixels are summed to obtain the second object mask. For all perturbed labeled images in the current training batch, the corresponding second object mask set is obtained.

[0052] It is understandable that the acquisition of the second target mask set can also be achieved through the above-mentioned pixel confidence-based method. That is, after the second basic network completes the semantic segmentation of the perturbed labeled image, the maximum category confidence of each pixel is also calculated, and based on the image-level adaptive confidence mean, only pixels with a value above the mean are retained as valid predictions in the second target mask, thereby constructing the second target mask set. The specific implementation method is the same as the acquisition method of the first target mask set mentioned above and will not be described in detail here.

[0053] S13. Input the unlabeled image set into the first basic network, the second basic network, the first enhanced network and the second enhanced network respectively, perform semantic segmentation on the target objects in the unlabeled image set, and obtain a third target mask set, a fourth target mask set, a first enhanced pseudo label set and a second enhanced pseudo label set accordingly.

[0054] The unlabeled image set is fed into the first base network, the second base network, the first augmented network, and the second augmented network, respectively, to perform pixel-level semantic segmentation of the target objects in the image. The first base network and the second base network output a third target mask set and a fourth target mask set, respectively, to capture the network's prediction results for the unlabeled image. The first augmented network and the second augmented network then output a first augmented pseudo-label set and a second augmented pseudo-label set, which are used to generate high-confidence semi-supervised signals to guide the base network in semi-supervised training.

[0055] In an optional embodiment of the present application, the process of obtaining the first enhanced pseudo label set and the second enhanced pseudo label set is as follows:

[0056] First, the unlabeled image set is subjected to a second perturbation process to obtain a perturbed unlabeled image set.

[0057] A second perturbation is performed on the unlabeled image set to obtain a perturbed unlabeled image set. Considering that the base network can withstand stronger image changes under the guidance of labeled data, the intensity of the first perturbation is greater than the second perturbation, so as to make the base network more robust and generalizable.

[0058] The perturbed unlabeled image set is then input into the first enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain a corresponding first enhanced pseudo-label set.

[0059] The perturbed unlabeled images are fed into the first augmentation network to obtain the predicted probability distribution for each pixel in each semantic category. After normalizing the predicted probabilities using softmax, for each pixel, the semantic category with the highest predicted probability is selected as the predicted category for the currently selected pixel. The predicted categories of all pixels are sorted according to their order in the image to form the first augmented pseudo-label for the image. The first augmented pseudo-label set is generated for the perturbed unlabeled images fed into the first augmentation network in the current training batch.

[0060] Finally, the perturbed unlabeled image set is input into the second enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain the corresponding second enhanced pseudo label set.

[0061] The perturbed unlabeled image set is fed into the second augmentation network to obtain the predicted probability distribution of each pixel across all semantic categories. Each predicted probability is then normalized using softmax. For each pixel, the semantic category with the highest predicted probability is selected as the predicted category for the currently selected pixel. The predicted categories of all pixels are sorted according to their order in the image to form the second augmented pseudo-label for the image. A corresponding second augmented pseudo-label set is generated for the perturbed unlabeled image set fed into the second augmentation network for the current training batch.

[0062] In an optional embodiment of the present application, after the perturbed unlabeled image set is input into the second enhancement network, the process of obtaining the second enhanced pseudo-label set includes the following processing steps:

[0063] First, the perturbed unlabeled image set is input into the second enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain the maximum category confidence of each pixel point of all unlabeled images in the perturbed unlabeled image set.

[0064] The perturbed unlabeled image set is fed into the second augmentation network. For each perturbed unlabeled image, pixel-level semantic segmentation is performed on the target object, generating a predicted probability distribution for each pixel in the unlabeled image across all semantic categories. The predicted probabilities are then normalized using softmax to obtain the normalized category probability for each pixel across all semantic categories. The maximum probability is then extracted as the maximum category confidence for that pixel, which is used to measure the model's prediction reliability at that location.

[0065] Then, based on the maximum category confidence of each pixel, the mean confidence of each perturbed unlabeled image is calculated.

[0066] Since the quality of the pseudo-labels determines the final prediction results of the model during the semi-supervised training process, pseudo-labels with high confidence are undoubtedly added to the training, while pseudo-labels with low confidence will interfere with the training of the model. Therefore, the present invention sets an adaptive threshold to filter out high-quality pseudo-labels to participate in the training and remove incorrect pseudo-labels. Specifically, the following process is performed for each perturbed unlabeled image: the maximum category confidence of all pixels in the unlabeled image is accumulated and divided by the total number of pixels in the image, thereby obtaining the adaptive confidence mean of the unlabeled image as shown in formula (1):

[0067]

[0068] in , h and represents the height and width of the unlabeled image, represents the total amount of semantic categories, Represents the mean confidence value of the r-th enhancement network on the current image, i and j are the height index and width index of the pixel in the unlabeled image, d is the category number to which the current pixel belongs, For the rth enhanced network to analyze unlabeled images The forward output feature map of is the current network parameter of the r-th enhanced network, is the wth unlabeled image in the unlabeled image set.

[0069] Then, for each pixel in the perturbed unlabeled image, determine whether the maximum category confidence of the pixel is greater than or equal to the confidence mean: if so, use the category number corresponding to the maximum category confidence as the second enhanced pseudo label of the pixel; otherwise, do not assign a second enhanced pseudo label to the pixel.

[0070] For each perturbed unlabeled image, the following process is performed for each pixel in the image: determine whether the maximum category confidence of the pixel is greater than the mean confidence of the image. If so, it means that the prediction result of the pixel is more reliable, so the category number corresponding to its maximum category confidence is used as the second enhanced pseudo-label of the pixel. On the contrary, if the maximum category confidence of the pixel is less than the mean confidence of the image, the second enhanced pseudo-label is not assigned to the pixel, that is, the pixel does not participate in the subsequent semi-supervised training, wherein not assigning the second enhanced pseudo-label means setting the second enhanced pseudo-label of the pixel to zero. Specifically, the present application takes into account that the average prediction probability will become higher and higher during the network training process. In order to improve the training accuracy, only when the prediction probability value of a single pixel is greater than the average prediction probability is considered to be a high-confidence pseudo-label, and the rth enhanced pseudo-label of the pixel is obtained by marking the high-confidence pixel position. , as shown in formula (2):

[0071]

[0072] in, , , is the rth enhanced pseudo label of pixel (i, j) in the unlabeled image.

[0073] Finally, the second enhanced pseudo labels in all perturbed unlabeled images are counted to obtain the second enhanced pseudo label set.

[0074] After all the perturbed unlabeled images are processed according to the above process, the second enhanced pseudo labels obtained by confidence judgment in all images in the perturbed unlabeled image set are integrated to construct a second enhanced pseudo label set.

[0075] For example, the two enhanced networks obtain pseudo labels in the same way, using Represents the feature map output by the r-th enhanced network, and obtains the r-th enhanced pseudo label The calculation of is shown in formula (3):

[0076]

[0077] (3)

[0078] in , The r-th enhanced pseudo label map generated by the r-th enhanced network, It is The unlabeled image after the enhanced network is perturbed by the wth input Output feature map, function Corresponding to The channel index number corresponding to the maximum value in the channel direction, that is, the category to which it belongs, is the rth enhanced pseudo label corresponding to the pixel in the i-th row and j-th column of the unlabeled image, and there are a total of categories, , To apply softmax to the network prediction output on the category channel dimension, the probability of each pixel is normalized across all categories.

[0079] S14. Calculate the difference between the first target mask set, the second target mask set and the label set, the difference between the third target mask set and the second enhanced pseudo label set, and the difference between the fourth target mask set and the first enhanced pseudo label set to obtain a total difference.

[0080] For labeled images, a first target mask set generated by the first base network is compared pixel by pixel with the corresponding manually annotated label set to obtain a first supervised difference. Similarly, a second target mask set generated by the second base network is compared pixel by pixel with the corresponding manually annotated label set to obtain a second supervised difference. The first supervised difference and the second supervised difference can be calculated using a cross-entropy loss or a weighted cross-entropy loss. Preferably, in an optional embodiment of the present application, the first supervised difference and the second supervised difference are cross-entropy losses.

[0081] In order to enhance the robustness of training, an adaptive threshold is set for labeled images to filter out difficult-to-train samples. First, the adaptive supervision confidence threshold for labeled supervised training is calculated as shown in formula (4):

[0082] (4)

[0083] in, is the adaptive supervision confidence threshold calculated by the qth basic network on the sth labeled image, which is used to determine which pixels belong to difficult training samples. , For the qth basic network to the perturbed labeled images The prediction result of , the prediction score of the dth category at pixel point (i, j), h and Indicates the height and width of the labeled image. For the labeled image set labeled images. According to the above adaptive supervision confidence threshold, the qth target mask corresponding to the difficult training sample in the labeled image set can be calculated as shown in formulas (5) and (6):

[0084] (5)

[0085] (6)

[0086] in, The qth target mask map generated for the qth base network is used to mark which pixels in the current labeled image belong to difficult training samples. is the confidence mean of the labeled image at the output of the qth basic network (the calculation method is the same as the confidence mean of the unlabeled image), is the qth target mask value representing the pixel at the i-th row and j-th column of the s-th labeled image.

[0087] The qth supervised difference of the final labeled image is obtained by using the label mask corresponding to the labeled difficult training sample as formula (7):

[0088] (7)

[0089] in, is the qth supervision difference, is a set of labeled images, For the labeled image set labeled images, is the weighted pixel-level supervision difference calculated by the q-th base network on the s-th labeled image.

[0090] For unlabeled images, the first enhanced pseudo-label set generated by the first enhanced network and the fourth target mask set generated by the second base network are compared pixel by pixel to obtain the first semi-supervised difference. Similarly, the second enhanced pseudo-label set generated by the second enhanced network and the third target mask set generated by the first base network are compared pixel by pixel to obtain the second semi-supervised difference. The first semi-supervised difference and the second semi-supervised difference are used to guide the learning direction of the base network on unsupervised samples. The semi-supervised difference also uses an adaptive threshold to generate a pseudo-label mask. Specifically, the label mask is used to obtain the semi-supervised difference that is finally used in model training, as shown in formula (8):

[0091] (8)

[0092] in, is the rth semi-supervised difference, is an unlabeled image set, is the sth unlabeled image in the unlabeled image set, h and w are the height and width of the unlabeled image respectively, is the weighted difference of the r-th basic network, is the rth enhanced pseudo label. Finally, the total loss of model training is ,in The coefficient representing the semi-supervised difference is then back-propagated to update the parameters of the two base networks.

[0093] To ensure the rationality and class balance of pseudo-label participation in training, this embodiment further performs a fine-grained analysis of pseudo-label differences and introduces a class weighting mechanism to improve the segmentation accuracy and overall training stability of the model in minority classes. Specifically, in an optional embodiment of the present invention, calculating the difference between the third target mask set and the second enhanced pseudo-label set includes the following process:

[0094] First, the category of each second enhanced pseudo label in the second enhanced pseudo label set is determined.

[0095] For each pseudo-label image in the second enhanced pseudo-label set, the category numbers of all pixels with assigned pseudo-labels are determined. Through this process, the distribution of each category in the pseudo-label image in the second enhanced pseudo-label set can be obtained, providing a basis for category weighting in subsequent difference calculations.

[0096] Then, the proportion of each type of second enhanced pseudo label in all second enhanced pseudo labels in the second enhanced pseudo label set is calculated.

[0097] Traverse each pseudo-label image in the second enhanced pseudo-label set and count all pixels that have been assigned a class label. Count them separately by class number to obtain the number of pixels corresponding to each semantic class. Calculate the ratio of the number of pixels in each class to the total number of labeled pixels to obtain the proportion of that class in the second enhanced pseudo-label set.

[0098] Then, for each type of second enhanced pseudo label in the second enhanced pseudo label set, a first semi-supervised difference between the third target mask set and the type of second enhanced pseudo label is calculated.

[0099] All pixel positions corresponding to the current semantic category are extracted from the second enhanced pseudo-label set to construct a pseudo-label mask for the category. The pixel prediction results corresponding to the above positions in the segmentation results of the corresponding image are then extracted from the third target mask set and compared pixel by pixel with the pseudo-label of the category. For each category, a standard metric function such as cross entropy loss or weighted cross entropy loss is used to calculate the difference between the third target mask and the second enhanced pseudo-label corresponding to the pixel point to obtain the first semi-supervised difference. This difference is used to measure the learning effect of the basic network in the area of ​​this type of pseudo-label, and will be weighted according to the category ratio in the subsequent difference fusion process to improve the model's segmentation ability for low-frequency categories and the overall training balance.

[0100] For example, the pseudo labels generated by the first enhancement network are With the output of the second base network Compare and calculate the second semi-supervised difference of the second basic network. Similarly, the second enhanced network generates With the output of the first base network Compare and calculate the first semi-supervised difference of the first basic network, as shown in formula (9):

[0101] (9)

[0102] in, is the initial rth semi-supervised difference of the entire unlabeled image, The r-th enhanced pseudo label map generated by the r-th enhanced network, is the output of the qth basic network, The sth unlabeled image after the second perturbation of the qth base network The output, are the parameters of the qth basic network, ,and , C is the total amount of categories, operator Indicates the multiplication of elements at corresponding positions in the matrix, , is the initial rth semi-supervised difference of the unlabeled image at pixel (i, j), is the rth enhanced pseudo label corresponding to the pixel in the i-th row and j-th column of the unlabeled image, The sth unlabeled image after the second perturbation of the qth base network The output of , is the predicted value of the cth category. The first enhanced pseudo label generated by the first enhanced network guides the training of the second basic network, and the second enhanced pseudo label generated by the second enhanced network guides the training of the first enhanced network, forming a cross-contrast training with each other.

[0103] Finally, the first half of the supervised difference is weighted based on the category ratio to obtain the final difference.

[0104] To further alleviate the impact of sample category imbalance on training, a dynamic weighting strategy is introduced in the loss function, that is, in each batch: according to the number of pixels of each category in the pseudo-label image, its proportion in the entire image is calculated. , using it as an inverse weight and acting on the original cross entropy loss to obtain the weighted difference of the r-th basic network , as shown in formula (10) (11):

[0105] (10)

[0106] = (11)

[0107] in, Indicates the category ratio of the qth enhanced pseudo label to all qth enhanced pseudo labels for each pixel in the current image. ,and , is the weighted difference of the r-th basic network at pixel (i, j), is the initial rth semi-supervised difference of the entire unlabeled image, is the number of pixels belonging to category d in all r-th enhanced pseudo labels in the image, is the ratio of the qth enhanced pseudo-label at pixel (i, j) to all qth enhanced pseudo-labels. This method is adaptively updated in each round of training, effectively avoiding the problem of dominant categories dominating the training.

[0108] For labeled images, the same method can be used to adjust the supervision loss weights, and the loss is shown in formula (12):

[0109] (12)

[0110] in, is the total difference of the q-th base network in the s-th labeled image, For the The true category labels of the labeled images, For the qth basic network to the The output of labeled images, is the difference of the qth basic network at the pixel (i, j) of the labeled image. Since the proportion of category pixels in each training image is different, in each The network can adaptively adjust the loss weight coefficient, increasing the loss weight when there are few training samples of a certain type, and reducing the loss weight when there are many, effectively alleviating the performance problems caused by the imbalance of training samples. , is the weighted pixel-level supervision difference calculated by the q-th base network on the labeled image, Indicates the category ratio of each pixel in the current labeled image to the category in the entire image. for is the weighted pixel-level supervised difference calculated by the qth base network on the pixel (i, j) of the labeled image. Then the qth supervised difference can be calculated as shown in formula (13):

[0111] (13)

[0112] in, is the qth supervised difference of the sth labeled image, is a set of labeled images, is the sth labeled image in the labeled image set, h and w are the height and width of the labeled image respectively, is the weighted pixel-level supervision difference calculated by the q-th base network on the s-th labeled image, The qth object mask generated by the qth base network for the sth labeled image.

[0113] S15. Adjust parameters of the first basic network and the second basic network based on the total difference to complete the training of the semantic segmentation model; wherein the semantic segmentation model includes the trained first basic network and / or the second basic network.

[0114] The main function of the enhanced network is to dynamically generate pseudo labels for unlabeled images. Each iteration updates the base network by mixing several labeled and unlabeled images. Using an exponential moving average strategy, the base network parameter values ​​are used to update the corresponding enhanced network parameter values. In the semantic segmentation model, the base network parameters are updated through backpropagation of the loss gradient. The enhanced network does not participate in backpropagation. Using an exponential moving average strategy, the base network parameter momentum is used to update the corresponding enhanced network parameters. The specific steps are as follows:

[0115] The parameters of the enhanced network at the current iteration are the weighted sum of the basic network parameters at the current iteration and the enhanced network parameters at the previous iteration. The enhanced network parameter update formula is: ;in Indicates the current The first iteration Parameters of enhanced network a, Indicates the current The first iteration Parameters of the base network b, To control the step size of momentum update. According to the pre-designed training strategy, Afterwards, by using a small number of labeled images and a large number of unlabeled images, the model can learn more information hidden behind the scene images to improve the final detection accuracy and obtain more accurate segmentation results.

[0116] Furthermore, in an optional embodiment of the present application, the semantic segmentation model is obtained by weighting the trained first basic network and the trained second basic network.

[0117] As a training strategy, the parameter optimizer can be set to , the initial learning rate is set to , the momentum is set to 0.9, and the weight decay factor is ; In the specific implementation of the training strategy, set The size is 5, which contains 2 unlabeled images and 3 labeled images, and set 100 , in the 20th Every time after that After the end, calculate the accuracy on the validation set and retain the model. The accuracy is calculated and compared with the accuracy of the previous model. If the accuracy of the latter model exceeds that of the previous model, the previous model is replaced. The intersection-over-union (IoU) evaluation criterion measures the effectiveness of the model by measuring the ratio of the intersection and union of the ground-truth region under test and the region predicted by the model. It is understood that the parameters in the above training strategy can be adaptively adjusted based on the actual task needs and model accuracy requirements, and are not limited here.

[0118] In summary, if Figure 2 As shown in the figure, the normalized input image undergoes the first and second perturbations, respectively, and is then fed into the base network and the augmented network. The augmented network processes the second perturbation image to generate a corresponding feature map. Similarly, the base network performs semantic segmentation on the first perturbation image to obtain a corresponding feature map. The second supervised loss is then calculated using the first augmented pseudo-label and the feature map output by the second base network, and the first supervised loss is calculated using the second augmented pseudo-label and the feature map output by the first base network. The final total loss updates the two base networks, and the corresponding augmented network is then updated accordingly.

[0119] Therefore, this application uses semantic segmentation, semi-supervised training, data perturbation, exponential average moving strategy, loss weighting and contrast cross-training to achieve real-time semantic segmentation of images, improve the semantic segmentation accuracy of scene images, and solve the problem of high overhead of manual labeling of semantic segmentation tasks and the problem of imbalance of scene image training samples. Based on the comparative training of two basic networks and two enhanced networks, the enhanced network generates pseudo labels for unlabeled images. The pseudo labels of one enhanced network and the unlabeled image feature map output by the other basic network are used to calculate the semi-supervised loss, and the labeled images and manual labels are used to calculate the full-supervised loss. The category loss value is dynamically adjusted by the number of each category of the training samples to alleviate the problem of imbalance of the training samples. Pseudo labels with high confidence and labeled images that are difficult to distinguish are screened out for gradient backpropagation. The basic network parameters are updated according to the gradient and then the parameters of the enhanced network are updated by momentum, and finally a more accurate segmentation map is obtained.

[0120] like Figure 3 As shown, the present application also provides a semantic segmentation method, which includes:

[0121] S31, obtaining an image to be segmented;

[0122] S32. Inputting the image into a semantic segmentation model to obtain the target mask for segmenting the target object in the image; wherein the semantic segmentation model is trained using any of the above-mentioned semi-supervised semantic segmentation model training methods based on a weighted contrast network;

[0123] S33. Segment the image based on the target mask.

[0124] When an image needs to be segmented, the image to be segmented is input into the semantic segmentation model, the semantic category of each pixel in the image is predicted, and the target mask is obtained based on the prediction result. The target mask is then used to segment the target object in the image.

[0125] like Figure 4 As shown, the semi-supervised semantic segmentation model training system 400 based on the weighted contrast network includes: an image acquisition module 410, a supervised module 420, an unsupervised module 430, a loss calculation module 440 and a parameter update module 450. The above-mentioned image acquisition module 410 is used to obtain a set of labeled images containing target objects and a corresponding label set, as well as a set of unlabeled images containing target objects but without labels; wherein the number of unlabeled images is much larger than the number of labeled images. The supervised module 420 is used to input the labeled image set into the first basic network of the first branch and the second basic network of the second branch respectively, perform semantic segmentation on the target objects in the labeled image set, and obtain the first target mask set and the second target mask set accordingly; wherein the first branch includes a first basic network and a first enhanced network, and the second branch includes a second basic network and a second enhanced network, and the first enhanced network, the second enhanced network, the first basic network, and the second basic network have the same structure but different initial parameters. The unsupervised module 430 is used to input the unlabeled image set into the first basic network, the second basic network, the first enhanced network and the second enhanced network respectively, and perform semantic segmentation on the target objects in the unlabeled image set, and obtain the third target mask set, the fourth target mask set, the first enhanced pseudo label set and the second enhanced pseudo label set respectively. The loss calculation module 440 is used to respectively calculate the difference between the first target mask set, the second target mask set and the label set, the difference between the third target mask set and the second enhanced pseudo label set, and the difference between the fourth target mask set and the first enhanced pseudo label set to obtain the total difference. The parameter update module 450 is used to adjust the parameters of the first basic network and the second basic network based on the total difference to complete the training of the semantic segmentation model; wherein, the semantic segmentation model includes the trained first basic network and / or second basic network.

[0126] Regarding the specific limitations of the semi-supervised semantic segmentation model training system based on the weighted contrast network, please refer to the limitations of the semi-supervised semantic segmentation model training method based on the weighted contrast network above, which will not be repeated here. The various modules in the above-mentioned semi-supervised semantic segmentation model training system based on the weighted contrast network can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0127] It should be noted that, in order to highlight the innovative part of this application, this embodiment does not introduce modules that are not closely related to solving the technical problems raised by this application, but this does not mean that there are no other modules in this embodiment.

[0128] like Figure 5 As shown, the electronic device 5 may include a memory 51, a processor 52 and a bus, and may also include a computer program stored in the memory 51 and executable on the processor 52, such as a training program for a semi-supervised semantic segmentation model based on a weighted contrast network.

[0129] The memory 51 includes at least one type of readable storage medium, including flash memory, a removable hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 51 may be an internal storage unit of the electronic device 5, such as a removable hard disk of the electronic device 5. In other embodiments, the memory 51 may also be an external storage device of the electronic device 5, such as a plug-in removable hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 51 may include both an internal storage unit of the electronic device 5 and an external storage device. The memory 51 can be used not only to store application software installed in the electronic device 5 and various data, such as the code for training a semi-supervised semantic segmentation model based on a weighted contrastive network, but also to temporarily store data that has been output or is about to be output.

[0130] In some embodiments, the processor 52 may be comprised of an integrated circuit, such as a single packaged integrated circuit or multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 52 is the control core (control unit) of the electronic device 5. It utilizes various interfaces and circuits to connect the various components of the electronic device 5. It executes programs or modules stored in the memory 51 (e.g., a program for training a semi-supervised semantic segmentation model based on a weighted contrast network) and accesses data stored in the memory 51 to perform various functions and process data.

[0131] The processor 52 executes the operating system and various installed applications of the electronic device 5. The processor 52 executes the applications to implement the steps in the above-mentioned weighted contrast network-based semi-supervised semantic segmentation model training method.

[0132] Exemplarily, the computer program may be divided into one or more modules, one or more of which are stored in the memory 51 and executed by the processor 52 to complete the present application. One or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 5. For example, the computer program may be divided into an image acquisition module 410, a supervised module 420, an unsupervised module 430, a loss calculation module 440, and a parameter update module 450.

[0133] The above-mentioned integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, computer device, or network device, etc.) or a processor to perform part of the functions of the training method of the semi-supervised semantic segmentation model in various embodiments of the present application.

[0134] In summary, the present application discloses a method and device for semi-supervised semantic segmentation and model training based on a weighted contrast network, designs a weighted contrast semi-supervised network segmentation model, and uses consistency regularization and pseudo-label semi-supervised training to obtain a semantic segmentation model with good results in the case of a small number of labeled scene images and a large number of unlabeled scene images. At the same time, the average exponential moving strategy is used to allow the parameters of the basic network to momentum update the parameters of the enhanced network, which in turn enhances the pseudo-labels with high confidence that the network can generate to compare and guide the basic network training; at the same time, in order to avoid the problem of sample imbalance during training, the semantic segmentation model can automatically adjust the weight coefficient to stabilize the training. In addition, the present application uses pseudo-labels with high confidence and difficult-to-distinguish training samples to further improve the robustness of the model, and the present application can be embedded in the camera of the monitoring system to perform real-time image semantic segmentation. Through the configuration of the monitoring camera, the image semantic segmentation inference algorithm can be flexibly adjusted to achieve higher semantic segmentation accuracy. Therefore, the present application effectively overcomes the various shortcomings of the existing technology and has high industrial utilization value.

[0135] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A semi-supervised semantic segmentation model training method based on weighted contrast network, characterized in that: The training method comprises: Obtain a set of labeled images containing target objects and their corresponding label sets, as well as a set of unlabeled images containing target objects but without labels; the number of unlabeled images is much larger than the number of labeled images; Inputting the labeled image set into the first base network of the first branch and the second base network of the second branch, respectively, performing semantic segmentation on the target objects in the labeled image set, and correspondingly obtaining a first target mask set and a second target mask set; wherein the first branch includes a first base network and a first enhancement network, and the second branch includes a second base network and a second enhancement network, and the first enhancement network, the second enhancement network, the first base network, and the second base network have the same structure but different initial parameters; Inputting the unlabeled image set into the first base network, the second base network, the first enhancement network, and the second enhancement network respectively, performing semantic segmentation on the target objects in the unlabeled image set, and correspondingly obtaining a third target mask set, a fourth target mask set, a first enhancement pseudo label set, and a second enhancement pseudo label set; Calculating the difference between the first target mask set, the second target mask set and the label set, the difference between the third target mask set and the second enhanced pseudo label set, and the difference between the fourth target mask set and the first enhanced pseudo label set, respectively, to obtain a total difference; Adjusting parameters of the first basic network and the second basic network based on the total difference to complete the training of the semantic segmentation model; wherein the semantic segmentation model includes the trained first basic network and / or the second basic network; Calculating the difference between the third target mask set and the second enhanced pseudo label set includes: Determining a category of each second enhanced pseudo label in the second enhanced pseudo label set; Calculate the proportion of each type of second enhanced pseudo label in all second enhanced pseudo labels in the second enhanced pseudo label set: For each type of second enhanced pseudo label in the second enhanced pseudo label set: calculating a first semi-supervised difference between the third target mask set and the type of second enhanced pseudo label; The first half of the supervised difference is weighted based on the category ratio to obtain the final difference.

2. The method for training a semi-supervised semantic segmentation model based on a weighted contrastive network according to claim 1, wherein: The step of inputting the labeled image set into the first basic network of the first branch and the second basic network of the second branch, performing semantic segmentation on the target objects in the labeled image set, and correspondingly obtaining a first target mask set and a second target mask set includes: Performing a first perturbation process on the labeled image set to obtain a perturbed labeled image set; Inputting the perturbed labeled image set into the first basic network, performing semantic segmentation on the target objects in the perturbed labeled image set to obtain a first target mask set; The perturbed labeled image set is input into the second basic network, and semantic segmentation is performed on the target objects in the perturbed labeled image set to obtain a second target mask set.

3. The method for training a semi-supervised semantic segmentation model based on a weighted contrastive network according to claim 2, wherein: The step of inputting the perturbed labeled image set into the first basic network and performing semantic segmentation on the target objects in the perturbed labeled image set to obtain a first target mask set includes: Input the perturbed labeled image set into the first basic network, perform semantic segmentation on the target object in the perturbed labeled image set, and obtain the maximum category confidence of each pixel point of all labeled images in the perturbed labeled image set; Based on the maximum category confidence of each pixel, calculate the mean confidence of each perturbed labeled image; For each pixel in the perturbed labeled image: Determine whether the maximum category confidence of the pixel is greater than or equal to the corresponding confidence mean: If yes, the category number corresponding to the maximum category confidence is used as the first target mask of the pixel; Otherwise, the first target mask is not assigned to the pixel; The first target masks in all perturbed labeled images are counted to obtain the first target mask set.

4. The method for training a semi-supervised semantic segmentation model based on a weighted contrastive network according to claim 1, wherein: The unlabeled image set is input into the first enhancement network and the second enhancement network respectively, and semantic segmentation is performed on the target objects in the unlabeled image set to obtain a first enhanced pseudo label set and a second enhanced pseudo label set, including: performing a second perturbation process on the unlabeled image set to obtain a perturbed unlabeled image set; The perturbed unlabeled image set is input into the first enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain a first enhanced pseudo label set. The perturbed unlabeled image set is input into the second enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain a corresponding second enhanced pseudo label set.

5. The method for training a semi-supervised semantic segmentation model based on a weighted contrastive network according to claim 4, wherein: The perturbed unlabeled image set is input into the second enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain a corresponding second enhanced pseudo label set, including: The perturbed unlabeled image set is input into the second enhancement network, and semantic segmentation is performed on the target objects in the perturbed unlabeled image set to obtain the maximum category confidence of each pixel point of all unlabeled images in the perturbed unlabeled image set; Based on the maximum category confidence of each pixel, calculate the mean confidence of each perturbed unlabeled image; For each pixel in the perturbed unlabeled image: Determine whether the maximum category confidence of the pixel is greater than the mean confidence: If yes, the category number corresponding to the maximum category confidence is used as the second enhanced pseudo label of the pixel; Otherwise, the second enhanced pseudo label is not assigned to the pixel; The second enhanced pseudo labels in all perturbed unlabeled images are counted to obtain a second enhanced pseudo label set.

6. The method for training a semi-supervised semantic segmentation model based on a weighted contrastive network according to claim 1, wherein: The semantic segmentation model is obtained by weighting the trained first basic network and the trained second basic network.

7. A semantic segmentation method, characterized in that: The semantic segmentation method comprises: Obtain the image to be segmented; Inputting the image into a semantic segmentation model to obtain a target mask for segmenting the target object in the image; wherein the semantic segmentation model is trained by the semi-supervised semantic segmentation model training method based on a weighted contrast network according to any one of claims 1 to 6; The image is segmented based on the object mask.

8. A semi-supervised semantic segmentation model training system based on weighted contrastive network, characterized in that: The training system comprises: An image acquisition module is used to acquire a set of labeled images containing target objects and a corresponding set of labels, as well as a set of unlabeled images containing target objects but without labels; wherein the number of unlabeled images is much greater than the number of labeled images; A supervised module is configured to input the labeled image set into a first base network of a first branch and a second base network of a second branch, respectively, to perform semantic segmentation on the target objects in the labeled image set, and to obtain a first target mask set and a second target mask set accordingly; wherein the first branch includes a first base network and a first enhancement network, and the second branch includes a second base network and a second enhancement network, and the first enhancement network, the second enhancement network, the first base network, and the second base network have the same structure but different initial parameters; an unsupervised module, configured to input the unlabeled image set into the first base network, the second base network, the first enhancement network, and the second enhancement network, respectively, to perform semantic segmentation on the target objects in the unlabeled image set, and correspondingly obtain a third target mask set, a fourth target mask set, a first enhancement pseudo label set, and a second enhancement pseudo label set; a loss calculation module, configured to respectively calculate the difference between the first target mask set, the second target mask set and the label set, the difference between the third target mask set and the second enhanced pseudo label set, and the difference between the fourth target mask set and the first enhanced pseudo label set, to obtain a total difference; A parameter updating module, configured to adjust parameters of the first base network and the second base network based on the total difference to complete the training of the semantic segmentation model; wherein the semantic segmentation model includes the trained first base network and / or the trained second base network; Calculating the difference between the third target mask set and the second enhanced pseudo label set includes: Determining a category of each second enhanced pseudo label in the second enhanced pseudo label set; Calculate the proportion of each type of second enhanced pseudo label in all second enhanced pseudo labels in the second enhanced pseudo label set: For each type of second enhanced pseudo label in the second enhanced pseudo label set: calculating a first semi-supervised difference between the third target mask set and the type of second enhanced pseudo label; The first half of the supervised difference is weighted based on the category ratio to obtain the final difference.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the semi-supervised semantic segmentation model training method based on a weighted contrast network as described in any one of claims 1 to 6 or the semantic segmentation method as described in claim 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the semi-supervised semantic segmentation model training method based on a weighted contrast network as described in any one of claims 1 to 6 or the semantic segmentation method as described in claim 7.

Images