Occlusion-aware seamless segmentation method and system for pseudo-label generation and passive domain scene adaptation

Abstract

The present invention discloses an occlusion-aware seamless segmentation method and system for pseudo-label generation and passive domain scene adaptation. The method further improves the labeling accuracy of pseudo-labels by threshold screening and data volume comparison for conventional instance-level pseudo-labels. At the same time, the generated pseudo-labels are used to design a weighted loss guided by the uncertainty region of the instance-level prediction branch in the occlusion-aware seamless segmentation task, thereby improving the accuracy of the segmentation model. Combined with the screening of low-quality pseudo-labels, a non-modal guided instance mixing strategy is proposed, thereby further increasing the number of samples available for training. The problem of poor segmentation effect of the final trained model caused by the small number of samples of individual categories is thereby improved, and the passive domain scene adaptation effect is finally improved.

Description

Technical Field

[0001] The present invention belongs to the field of segmentation, and in particular relates to an occlusion-aware seamless segmentation method and system for pseudo-label generation and passive domain scene adaptation. Background Art

[0002] Semantic segmentation is a task that classifies each pixel in an image, aiming to assign a category label to each pixel in the scene. Common semantic segmentation methods include FCN and DeepLab series, which achieve pixel-level classification through convolutional neural networks (CNN) and are widely used in various scene understanding tasks, such as autonomous driving and medical image processing. Instance segmentation is a further development based on semantic segmentation. It not only classifies each pixel in the image, but also distinguishes different objects of the same category. This means that instance segmentation not only needs to know which category the object belongs to, but also needs to distinguish the specific instance of each object. The classic instance segmentation method is Mask R-CNN, which combines object detection and segmentation algorithms to achieve efficient instance-level segmentation. Panoptic segmentation combines semantic segmentation and instance segmentation together, classifying each pixel in the image (semantic segmentation) and distinguishing specific object instances (instance segmentation). Panoptic segmentation provides a more comprehensive scene understanding, and treats "things" (such as cars, people) and "background" (such as roads, buildings) in a unified manner. Panoptic-DeepLab is a representative method for this type of task. It provides a simple and powerful panoramic segmentation framework by integrating semantic and instance information. Amodal segmentation is a more advanced segmentation task that aims to not only segment the visible part of the image, but also predict the occluded object area. This method allows the restoration of parts of the image that are occluded by other objects, making the scene understanding more complete, such as Figure 3 As shown. Typical non-modal segmentation methods include ORCNN, which combines shape and contour priors to infer the complete contour of the object, and can accurately segment the object even when it is partially occluded. However, all of the above methods are performed for special segmentation tasks, and the occlusion-aware seamless segmentation task unifies the advantages of the above methods, allowing the model to simultaneously complete the semantic segmentation of the background class and the non-modal instance segmentation of the object class. More importantly, the occlusion-aware seamless segmentation task requires the model to output all the segmentation results at one time through the input image. Further, in the segmentation task, panoramic image segmentation is a segmentation task for 360-degree panoramic images. Panoramic images can cover a wider viewing angle, making this task more complicated. Therefore, the occlusion-aware seamless segmentation task further requires the combination of labeled pinhole image datasets (called source domain) and unlabeled panoramic image datasets (target domain) for training, so that the segmentation model can complete semantic segmentation and non-modal instance segmentation of the object class at one time on the panoramic image.

[0003] However, the existing passive domain scenes of occlusion-aware seamless segmentation tasks cannot be adapted, and there is an urgent need to propose a new method to solve this technical problem. Summary of the invention

[0004] The present invention focuses on solving the problem of completing the occlusion-aware seamless segmentation task under the task requirement of passive domain scene adaptation, and proposes a pseudo-label generation and passive domain scene adaptive occlusion-aware seamless segmentation method and system, which generate high-quality pseudo-labels in the absence of real labels in the target domain. Through the generation of pseudo-labels, only the model trained in the source domain and the unlabeled target domain data can be used for training to obtain a high-precision segmentation model. This method can better protect privacy and fits in some situations where, due to data privacy, suppliers can only provide trained models to avoid leakage of source domain training data.

[0005] The technical solution of the present invention is as follows:

[0006] On the one hand, a method for generating pseudo labels for data in a target domain includes:

[0007] 1) Use the segmentation model in the source domain to predict the image data in the target domain, and obtain the initial semantic prediction, instance prediction, and non-modal instance prediction of each image;

[0008] The instance prediction and the amodal instance prediction include a target mask, a current target mask score, and a current target mask category;

[0009] 2) From the initial instance prediction and amodal instance prediction, obtain the number of corresponding data for each object category whose current target mask score exceeds the set threshold;

[0010] Get the current target mask score of each object category in the instance prediction in turn, which is greater than the set instance threshold The number of instance prediction data N1, and the current target mask score of each object category in the amodal instance prediction is greater than the amodal instance threshold The number of non-modal instance prediction data N2;

[0011] Instance Threshold and non-modal instance threshold The value range is [0,1], which is determined based on experience and experiments;

[0012] 3) From the initial instance predictions and amodal instance predictions, sort all current target mask scores of each object category in descending order, and set the selection percentage threshold of each category sorting sequence;

[0013] The current target mask scores of each object category in the statistical instance prediction are the top αins %N3 is the number of instance prediction data;

[0014] Count the top current target mask scores for each object category in the amodal instance prediction α ains %Number of non-modal instance prediction data N4;

[0015] 4) The data corresponding to max(N1, N3) is set as the instance definite label, and the rest of the data is set as the instance uncertain region, and the instance definite label and the instance uncertain region are used as instance pseudo labels; the data corresponding to max(N2, N4) is set as the non-modal instance definite label, and the rest of the data is set as the non-modal instance uncertain region, and the corresponding non-modal instance definite label and the non-modal instance uncertain region are used as non-modal instance pseudo labels; based on the initial semantic prediction, the semantic label corresponding to the maximum prediction value of each semantic category is obtained, and then the semantic pseudo label is obtained after screening using the prediction probability threshold of the semantic category label.

[0016] In order to prevent the model from easily predicting some background areas as instances, which would lead to poor model training results, the instance-level pseudo labels predicted in the previous step are further restricted;

[0017] For the target mask area or uncertain area with a determined label in the instance-level pseudo-label, if part of the area does not belong to the object class of the semantic pseudo-label, the part of the area is set to be modified as the background label. The semantic pseudo-label means that the category of each pixel position is the object class or the background class;

[0018] Furthermore, the target mask of the determined label in each instance and non-modal instance in the target domain image and the object class prediction mask of the uncertain region and the target domain image are ANDed to obtain the instance and non-modal instance pseudo labels of the filtered target domain image;

[0019] Among them, the object class prediction mask of the target domain image is obtained according to the following formula:

[0020]

[0021] in, Display pictures At pixel position The object class prediction mask at , T represents the object class target category in all semantic categories, Display pictures At pixel position The semantic category labels at the location include object categories and background categories. , Display pictures At pixel position The semantic category index value corresponding to the maximum predicted value of the semantic category c at , Display pictures At pixel position The predicted value of the semantic category c at .

[0022] In a second aspect, a passive domain scene adaptive occlusion-aware seamless segmentation method is provided, comprising:

[0023] Step 1: Generate pseudo labels for image data in the target domain using one of the above-mentioned pseudo label generation methods for the target domain, including pseudo labels in three forms: semantic, instance, and non-modal instance;

[0024] Step 2: Use the images in the target domain and the corresponding pseudo labels to form training samples, and use the training samples to train the segmentation model of the source domain;

[0025] During the training process, for the current iteration training At this moment, the training sample is the training picture And the corresponding semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels: , the segmentation model outputs semantic branch predictions , instance branch prediction , non-modal instance branch prediction ; Use the sum of the prediction losses of the semantic branch, instance branch, and non-modal instance branch for back propagation, and after updating the segmentation model parameters, continue iterative training until the set number of iterations is reached, and obtain the occlusion-aware seamless segmentation model;

[0026] Step 3: Use the occlusion-aware seamless segmentation model to segment the image in the target domain.

[0027] Furthermore, during the model training process, when calculating the prediction loss of the instance classification branch, it is necessary to use both the certain label and the uncertain region in the instance-level pseudo-label;

[0028] For instance-level branches, the loss calculation of the instance branch is the same as that of the amodal instance branch.

[0029] The instance branch prediction loss calculation process is as follows:

[0030] Current training iteration At this moment, for the input image in the current training process , the corresponding instance pseudo label generated It is divided into a binary instance label mask and a binary instance uncertainty mask, where the binary instance label mask Refers to the instance-determined label in the instance pseudo-label The target mask of the binary instance is uncertain It refers to the instance uncertainty area in the instance pseudo label The target mask;

[0031] Display pictures in sequence The first, second, ..., Instances determine the label, Corresponding The target mask;

[0032] Then, the training images Example prediction The target mask in , and the binary instance label mask is used to calculate the binary classification loss, and then the binary classification loss is multiplied pixel by pixel with the binary instance uncertainty mask to obtain the prediction loss of the final instance branch :

[0033]

[0034] Among them, BCE represents the binary classification loss, Represents training pictures The instance pseudo-label of Binarized instance label masks;

[0035] Under this operation, the prediction of the uncertain region will not participate in the final loss calculation and back propagation. It is a weighted loss guided by the uncertainty region for the instance-level prediction branch in the occlusion-aware seamless segmentation task.

[0036] Non-modal instance branch:

[0037] Current training iteration At this moment, for the input image in the current training process , the corresponding non-modal instance pseudo label generated It is divided into a binary non-modal instance label mask and a binary non-modal instance uncertainty mask, where the binary non-modal instance label mask Refers to the instance-determined label in the non-modal instance pseudo-label The target mask of the binary non-modal instance uncertainty mask Refers to the uncertainty area of ​​the non-modal instance in the non-modal instance pseudo-label The target mask;

[0038] Display pictures in sequence The first, second, ..., Instances determine the label, Corresponding The target mask;

[0039] The segmentation model is based on the current input training picture Obtained non-modal instance branch prediction The target mask in , and the binary classification loss is calculated with the binary non-modal instance label mask, and then the binary classification loss is multiplied pixel by pixel with the binary non-modal instance uncertainty mask to obtain the prediction loss of the final non-modal instance branch :

[0040]

[0041] in, Represents training pictures The non-modal instance pseudo-label of Binarized amodal instance label masks.

[0042] Furthermore, semantic prediction With semantic pseudo-labels Constructing semantic branch loss :

[0043]

[0044] Among them, CE represents the cross entropy loss.

[0045] Furthermore, a non-modal guided target mixing strategy is adopted to increase the number of targets in each image in the training samples and increase the diversity of the training samples. The specific steps are as follows:

[0046] 1) Generate high-quality hard labels:

[0047] First, all images in the target domain are predicted by the non-modal instance prediction obtained by the segmentation model in the source domain. The current target mask score obtained is greater than the set hard label threshold. and all current target masks before scoring % of the non-modal instance prediction data, select the corresponding non-modal instance prediction with the larger data amount; then, among the selected non-modal instance predictions, determine whether the target mask category is consistent with the category in the corresponding semantic label. If not, remove the non-modal instance prediction corresponding to the target mask category; use the final non-modal instance prediction as a high-quality hard label;

[0048] 2) Screen out high-quality unobstructed hard labels:

[0049] When the target mask predicted by each non-modal instance in the high-quality hard label does not overlap with the target mask area of ​​the high-quality hard label predicted by other non-modal instances in the current image, the high-quality hard label predicted by the non-modal instance is retained, otherwise, the high-quality hard label predicted by the non-modal instance is discarded; the high-quality hard labels predicted by all retained non-modal instances are used as high-quality unobstructed hard labels;

[0050] 3) Target random mixing:

[0051] All high-quality unoccluded hard labels are regarded as a high-quality unoccluded target repository. In each iterative training of the segmentation model, R high-quality unoccluded hard labels predicted by non-modal instances are randomly selected from the high-quality unoccluded target repository as targets to be mixed. The target image data of the corresponding image position is obtained according to the target mask in the high-quality unoccluded hard label of each target to be mixed, and it is pasted to the image of the current target domain. At the same time, according to the pasted position and category, the semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels of the image of the current target domain are changed accordingly.

[0052] In this way, the sample data available for training can be effectively increased, that is, the number of recognizable targets in each picture is increased, and ultimately the passive domain scene adaptation effect is improved, making the occlusion-aware seamless segmentation model more effective.

[0053] In a third aspect, a passive domain scene adaptive occlusion-aware seamless segmentation system is provided, comprising:

[0054] Pseudo-label generation module: a pseudo-label generation method for target domain data is used to generate pseudo-labels for image data in the target domain, including pseudo-labels in the form of semantic, instance, and non-modal instance;

[0055] Training module: adopting the above-mentioned passive domain scene adaptive occlusion-aware seamless segmentation method, using the pictures in the target domain and the corresponding pseudo labels to form training samples, and using the training samples to train the segmentation model of the source domain;

[0056] During the training process, for the current iteration training At this moment, the training sample is the training picture And the corresponding semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels: , the segmentation model outputs semantic branch predictions , instance branch prediction , non-modal instance branch prediction ; Use the sum of the prediction losses of the semantic branch, instance branch, and non-modal instance branch for back propagation, and after updating the segmentation model parameters, continue iterative training until the set number of iterations is reached, and obtain the occlusion-aware seamless segmentation model;

[0057] Segmentation module: Use the occlusion-aware seamless segmentation model to segment the image in the target domain.

[0058] In a fourth aspect, an electronic terminal includes:

[0059] one or more processors;

[0060] a memory storing one or more computer programs;

[0061] The processor calls the computer program to implement:

[0062] The steps of the above-mentioned method for generating pseudo-labels of data in the target domain or the above-mentioned method for adaptive occlusion-aware seamless segmentation of passive domain scenes.

[0063] In a fifth aspect, a computer-readable storage medium stores a computer program, wherein the computer program is called by a processor to implement:

[0064] The steps of the above-mentioned method for generating pseudo-labels of data in the target domain or the above-mentioned method for adaptive occlusion-aware seamless segmentation of passive domain scenes.

[0065] Beneficial Effects

[0066] Compared with the existing method, the advantages of the present invention are:

[0067] 1. The technical solution of the present invention designs a method for generating pseudo-labels for target domain data suitable for seamless segmentation tasks. Directly using model predictions as labels will result in many erroneous predictions as training samples, resulting in poor model adaptation effect. Simply using threshold screening to eliminate low-quality predictions will result in a small number of training samples for some categories. The pseudo-label generation method proposed in the technical solution of the present invention can, on the one hand, generate high-quality pseudo-labels as training samples, and on the other hand, ensure that different categories can have a certain number of training samples, thereby ensuring the training effect of the model.

[0068] 2. The technical solution of the present invention further designs a weighted loss guided by the uncertain region for the instance-level prediction branch in the occlusion-aware seamless segmentation task for pseudo-labels. This type of loss will prevent the prediction of the uncertain region from participating in the final loss calculation and back propagation, thereby avoiding some target regions that are correctly predicted as targets by the model but are incorrectly restricted to unlabeled due to labels, resulting in poor model training results.

[0069] 3. The technical solution of the present invention proposes a non-modal guided target mixing strategy, which effectively increases the number of targets available for training by screening out high-quality unobstructed targets as targets for mixing, thereby improving the model's prediction effect on the target. Furthermore, it also solves the problem that some low-quality predictions are eliminated during the pseudo-label generation process, resulting in a reduction in trainable targets and affecting the model training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a segmentation schematic diagram of the segmentation model in the technical solution of the present invention;

[0071] Figure 2 A schematic diagram of the process of determining labels and uncertain regions and object class prediction masks and operations in the technical solution of the present invention;

[0072] Figure 3 This is a schematic diagram of instance prediction and non-modal instance prediction mentioned in the present invention, wherein (a) is the input original image, (b) is the instance label of (a), and (c) is the non-modal instance label of (a). DETAILED DESCRIPTION

[0073] Example 1

[0074] A method for generating pseudo-labels for data in a target domain, comprising:

[0075] 1) Use the segmentation model in the source domain to predict the image data in the target domain, and obtain the initial semantic prediction, instance prediction, and non-modal instance prediction of each image;

[0076] The instance prediction and the amodal instance prediction include a target mask, a current target mask score, and a current target mask category;

[0077] 2) From the initial instance prediction and amodal instance prediction, obtain the number of corresponding data for each object category whose current target mask score exceeds the set threshold;

[0078] Get the current target mask score of each object category in the instance prediction in turn, which is greater than the set instance threshold The number of instance prediction data N1, and the current target mask score of each object category in the amodal instance prediction is greater than the amodal instance threshold The number of non-modal instance prediction data N2;

[0079] Instance Threshold and non-modal instance threshold The value range is [0,1], and based on experience and experiments, the values ​​in this example are 0.5 and 0.3 respectively;

[0080] 3) From the initial instance predictions and amodal instance predictions, sort all current target mask scores of each object category in descending order, and set the selection percentage threshold of each category sorting sequence;

[0081] The current target mask scores of each object category in the statistical instance prediction are the top α ins %N3 is the number of instance prediction data;

[0082] Count the top current target mask scores for each object category in the amodal instance prediction α ains %Number of non-modal instance prediction data N4;

[0083] In this example α ins %and α ains The values ​​of % are 20% and 50% respectively;

[0084] 4) The data corresponding to max(N1, N3) is set as the instance definite label, and the rest of the data is set as the instance uncertain region, and the instance definite label and the instance uncertain region are used as instance pseudo labels; the data corresponding to max(N2, N4) is set as the non-modal instance definite label, and the rest of the data is set as the non-modal instance uncertain region, and the corresponding non-modal instance definite label and the non-modal instance uncertain region are used as non-modal instance pseudo labels; based on the initial semantic prediction, the semantic label corresponding to the maximum prediction value of each semantic category is obtained, and then the semantic pseudo label is obtained after screening using the prediction probability threshold of the semantic category label.

[0085] In order to prevent the model from easily predicting some background areas as instances, which would lead to poor model training results, the instance-level pseudo labels predicted in the previous step are further restricted;

[0086] For the target mask area or uncertain area with a determined label in the instance-level pseudo-label, if part of the area does not belong to the object class of the semantic pseudo-label, the part of the area is set to be modified as the background label. The semantic pseudo-label means that the category of each pixel position is the object class or the background class;

[0087] For the acquisition of instance pseudo labels, an example is as follows: first obtain the prediction of the car category in the instance prediction, and first execute the first method: count the number of predictions of car targets greater than the set instance threshold of 0.3 to be 1000; then execute the second method: arrange the predictions belonging to the car category from large to small according to the mask score in each prediction, and then count the number of predictions of the top 70% of car targets with mask scores to be 1200; at this time, since the number of predictions in the second method is greater than the first, the corresponding predictions obtained in the second method are used as the instance determination label of the car category, and the remaining predictions of the second method are set as the instance uncertainty area. Then, perform the same operation on other categories of instance labels one by one. For non-modal instance labels, it is also the same method.

[0088] The process of obtaining semantic pseudo-labels is a separate process. The process of obtaining instance pseudo-labels and amodal pseudo-labels is similar, but different from the process of obtaining semantic pseudo-labels. The three pseudo-labels of semantics, instance, and amodal instance are obtained to construct training samples. In the passive domain task, the only training samples obtained are unlabeled pictures, so pseudo-labels need to be obtained for training.

[0089] The process of obtaining semantic pseudo labels is as follows;

[0090] Based on the semantic prediction probabilities of each pixel in the unlabeled target domain image, the semantic pseudo-label of the unlabeled target image is obtained according to the prediction probability threshold of the semantic category label;

[0091]

[0092] in, Display pictures At pixel position The object class prediction mask at , T represents the object class target category in all semantic categories, Display pictures At pixel position The semantic category labels at the location include object categories and background categories. , Display pictures At pixel position The semantic category index value corresponding to the maximum predicted value of the semantic category c at , Display pictures At pixel position The predicted value of the semantic category c at ;

[0093] For example, according to the position of each pixel Generate semantic pseudo labels based on the probability of corresponding semantic categories ;

[0094] The process of obtaining instance pseudo labels is as follows:

[0095] First, get all instance prediction sets for all images , Indicates Pictures Middle Instance prediction of a target, Indicates an instance, Indicates The number of targets in a picture, N represents the total number of pictures in the target domain;

[0096] Second, filter the target instance predictions;

[0097] 1) According to the set instance scoring threshold Filter all instance prediction sets and retain predictions with mask scores greater than , and obtain a set of instance predictions based on the score threshold filter , the set has a total of N1 targets;

[0098] 2) Sort all instance prediction sets by predicted mask scores from large to small, and then calculate the instance percentage threshold according to the set instance percentage threshold. Filter out the top predictions in all the sorted instance prediction sets % prediction, obtain the instance prediction set based on the percentage threshold screening , the set has a total of N3 targets;

[0099] 3) Take the filtered instance set corresponding to max(N1,N3) as the instance determination label set , and set other predictions as the instance uncertainty region set ;

[0100] 4) For each image, obtain the instance training sample of each image ,in Indicates the current image Determine the number of tags in express belongs to the current picture The definite label of the jth instance in Current image in All target masks are added together to obtain the current image The instance uncertainty region ;

[0101] The process of obtaining non-modal instance pseudo-labels is similar to that of instance pseudo-labels, and will not be further introduced here.

[0102] like Figure 2 As shown, the target mask of the determined label in each instance and non-modal instance in the target domain image and the object class prediction mask of the uncertain region and the target domain image are operated to obtain the instance and non-modal instance pseudo labels of the filtered target domain image;

[0103] Among them, the object class prediction mask of the target domain image is obtained according to the following formula:

[0104]

[0105] in, Display pictures At pixel position The object class prediction mask at , T represents the object class target category in all semantic categories, Display pictures At pixel position The semantic category labels at the location include object categories and background categories. , Display pictures At pixel position The semantic category index value corresponding to the maximum predicted value of the semantic category c at , Display pictures At pixel position The predicted value of the semantic category c at .

[0106] For each image, the instance and non-modal instance pseudo labels , the target mask and uncertainty area of ​​each target of the pseudo label are compared with the object class prediction mask of the corresponding image Perform AND operations to obtain the final instance pseudo label of each image and non-modal instance pseudo-labels , Display pictures in sequence The first, second, ..., Determine the label for each instance; Indicates the current image The instance uncertainty region;

[0107] Display pictures in sequence The 1st, 2nd, ..., A non-modal instance determines the label; Indicates the current image The uncertainty region of the non-modal instance;

[0108] For example, the semantic categories are (1, 2, 3, 4, 5), where (1, 2) is the background class prediction category and (3, 4, 5) is the object class prediction category.

[0109] Example 2

[0110] like Figure 1 As shown, a passive domain scene adaptive occlusion-aware seamless segmentation method comprises:

[0111] Step 1: Generate pseudo labels for image data in the target domain using the above-mentioned pseudo label generation method for the target domain, including pseudo labels in the form of semantic, instance, and non-modal instance;

[0112] Step 2: Use the images in the target domain and the corresponding pseudo labels to form training samples, and use the training samples to train the segmentation model of the source domain;

[0113] During the training process, for the current iteration training At this moment, the training sample is the training picture And the corresponding semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels: , the segmentation model outputs semantic branch predictions , instance branch prediction , non-modal instance branch prediction ; Use the sum of the prediction losses of the semantic branch, instance branch, and non-modal instance branch for back propagation, and after updating the segmentation model parameters, continue iterative training until the set number of iterations is reached, and obtain the occlusion-aware seamless segmentation model;

[0114] Step 3: Use the occlusion-aware seamless segmentation model to segment the image in the target domain.

[0115] During model training, when calculating the prediction loss of the instance classification branch, it is necessary to use both the certain labels and the uncertain regions in the instance-level pseudo labels.

[0116] For instance-level branches, the loss calculation of the instance branch is the same as that of the amodal instance branch.

[0117] The instance branch prediction loss calculation process is as follows:

[0118] Current training iteration At this moment, for the input image in the current training process , the corresponding instance pseudo label generated It is divided into a binary instance label mask and a binary instance uncertainty mask, where the binary instance label mask Refers to the instance-determined label in the instance pseudo-label The target mask of the binary instance is uncertain It refers to the instance uncertainty area in the instance pseudo label The target mask;

[0119] Display pictures in sequence The first, second, ..., Instances determine the label, Corresponding The target mask;

[0120] Then, the training images Example prediction The target mask in , and the binary instance label mask is used to calculate the binary classification loss, and then the binary classification loss is multiplied pixel by pixel with the binary instance uncertainty mask to obtain the prediction loss of the final instance branch :

[0121]

[0122] Among them, BCE represents the binary classification loss, Represents training pictures The instance pseudo-label of Binarized instance label masks;

[0123] Under this operation, the prediction of the uncertain region will not participate in the final loss calculation and back-propagation. This is a weighted loss guided by the uncertainty region for the instance-level prediction branch in the occlusion-aware seamless segmentation task.

[0124] Non-modal instance branch:

[0125] Current training iteration At this moment, for the input image in the current training process , the corresponding non-modal instance pseudo label generated It is divided into a binary non-modal instance label mask and a binary non-modal instance uncertainty mask, where the binary non-modal instance label mask Refers to the instance-determined label in the non-modal instance pseudo-label The target mask of the binary non-modal instance uncertainty mask Refers to the uncertainty area of ​​the non-modal instance in the non-modal instance pseudo-label The target mask;

[0126] Display pictures in sequence The first, second, ..., Instances determine the label, Corresponding The target mask;

[0127] The segmentation model is based on the current input training picture Obtained non-modal instance branch prediction The target mask in , and the binary classification loss is calculated with the binary non-modal instance label mask, and then the binary classification loss is multiplied pixel by pixel with the binary non-modal instance uncertainty mask to obtain the prediction loss of the final non-modal instance branch :

[0128]

[0129] in, Represents training pictures The non-modal instance pseudo-label of Binarized amodal instance label masks.

[0130] Semantic Prediction With semantic pseudo-labels Constructing semantic branch loss :

[0131]

[0132] Among them, CE represents the cross entropy loss.

[0133] Adopt a non-modal guided target mixing strategy to increase the number of targets in each image in the training sample and increase the diversity of the training samples. The specific steps are as follows:

[0134] 1) Generate high-quality hard labels:

[0135] First, all images in the target domain are predicted by the non-modal instance prediction obtained by the segmentation model in the source domain. The current target mask score obtained is greater than the set hard label threshold. and all current target masks before scoring % of the non-modal instance prediction data, select the corresponding non-modal instance prediction with the larger data amount; then, among the selected non-modal instance predictions, determine whether the target mask category is consistent with the category in the corresponding semantic label. If not, remove the non-modal instance prediction corresponding to the target mask category; use the final non-modal instance prediction as a high-quality hard label;

[0136] 2) Screen out high-quality unobstructed hard labels:

[0137] When the target mask predicted by each non-modal instance in the high-quality hard label does not overlap with the target mask area of ​​the high-quality hard label predicted by other non-modal instances in the current image, the high-quality hard label predicted by the non-modal instance is retained, otherwise, the high-quality hard label predicted by the non-modal instance is discarded; the high-quality hard labels predicted by all retained non-modal instances are used as high-quality unobstructed hard labels;

[0138] 3) Target random mixing:

[0139] All high-quality unoccluded hard labels are regarded as a high-quality unoccluded target repository. In each iterative training of the segmentation model, R high-quality unoccluded hard labels predicted by non-modal instances are randomly selected from the high-quality unoccluded target repository as targets to be mixed. The target image data of the corresponding image position is obtained according to the target mask in the high-quality unoccluded hard label of each target to be mixed, and it is pasted to the image of the current target domain. At the same time, according to the pasted position and category, the semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels of the image of the current target domain are changed accordingly.

[0140] The details are as follows: At the current iteration k in the training process, the training sample is the image And the corresponding pseudo labels First, from the high-quality unobstructed object repository Randomly select R non-modal instances predicted by high-quality unobstructed hard labels as the mixed targets , for each selected target to be mixed , obtain the pixel value of the target area of ​​the corresponding image according to its target mask, and paste it to the current training sample image according to the target mask position . Based on the current randomly selected R mixed targets, the current training sample image After pasting, get the mixed picture of the current iteration . And according to the current randomly selected R mixed targets, the current training sample image Semantic, instance, and amodal instance pseudo-labels of Make corresponding modifications to obtain the mixed pseudo-label at the current iteration time Specifically, for mixed semantic pseudo-labels The semantic pseudo label of the current training sample image According to the position of the target mask of the target to be mixed, it is replaced with the corresponding mask category; for the mixed instance pseudo label , the instance pseudo label of the current training sample image The target mask and the position in the uncertain area that is the same as the target mask of the target to be mixed are set as the background class to match the label of the target being occluded. At the same time, the target to be mixed is directly added as the newly added instance pseudo label. The final mixed instance pseudo label is expressed as ,in and Respectively represent the certain label and uncertain region in the mixed instance pseudo label; for the mixed non-modal instance pseudo label , the non-modal instance pseudo label of the current training sample image The target mask does not need to be changed. The position in the uncertain area that is the same as the target mask of the target sample to be mixed is set as the background class, and the target sample to be mixed is directly added as the newly added non-modal instance pseudo label. The final mixed non-modal instance pseudo label is expressed as ,in and They represent the certain labels and uncertain regions in the mixed amodal instance pseudo labels, respectively.

[0141] Finally, through this non-modal guided instance mixing strategy, training samples for the current batch K training are generated: mixed pictures , mixed semantic pseudo-labels , mixed instance pseudo labels , mixed non-modal instance pseudo-labels In this way, the sample data available for training can be effectively increased, that is, the number of identifiable targets in each picture is increased, and finally the passive domain scene adaptation effect is improved, so that the occlusion-aware seamless segmentation model has a good effect.

[0142] Embodiment 3:

[0143] This embodiment also provides a passive domain scene adaptive occlusion-aware seamless segmentation system, including:

[0144] Pseudo-label generation module: a pseudo-label generation method for target domain data is used to generate pseudo-labels for image data in the target domain, including pseudo-labels in the form of semantic, instance, and non-modal instance;

[0145] Training module: adopting the above-mentioned passive domain scene adaptive occlusion-aware seamless segmentation method, using the pictures in the target domain and the corresponding pseudo labels to form training samples, and using the training samples to train the segmentation model of the source domain;

[0146] During the training process, for the current iteration training At this moment, the training sample is the training picture And the corresponding semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels: , the segmentation model outputs semantic branch predictions , instance branch prediction , non-modal instance branch prediction ; Use the sum of the prediction losses of the semantic branch, instance branch, and non-modal instance branch for back propagation, and after updating the segmentation model parameters, continue iterative training until the set number of iterations is reached, and obtain the occlusion-aware seamless segmentation model;

[0147] Segmentation module: Use the occlusion-aware seamless segmentation model to segment the image in the target domain.

[0148] It should be understood that the implementation process of each module can refer to the content of the aforementioned method. The division of the above functional modules is only a division of logical functions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. At the same time, the above integrated units can be implemented in the form of hardware or software functional units.

[0149] Embodiment 4:

[0150] This embodiment provides an electronic terminal, including: one or more processors; a memory storing one or more computer programs;

[0151] Among them, the processor calls the computer program to implement: the steps of the above-mentioned method for generating data pseudo-labels for the target domain or the above-mentioned method for adaptive occlusion-aware seamless segmentation of passive domain scenes.

[0152] For the specific implementation process of each step, please refer to the description of the above method.

[0153] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.

[0154] Embodiment 5:

[0155] This embodiment provides a computer-readable storage medium storing a computer program, wherein the computer program is called by a processor to implement:

[0156] The steps of the above-mentioned method for generating pseudo-labels of data in the target domain or the above-mentioned method for adaptive occlusion-aware seamless segmentation of passive domain scenes.

[0157] Execute the above steps specifically.

[0158] For the specific implementation process of each step, please refer to the description of the above method.

[0159] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the controller described in any of the foregoing embodiments, such as a hard disk or memory of the controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Further, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0160] Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0161] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A method for generating pseudo-labels for target domain data, characterized in that: include: 1) Use the segmentation model in the source domain to predict the image data in the target domain, and obtain the initial semantic prediction, instance prediction, and non-modal instance prediction of each image; The instance prediction and the amodal instance prediction include a target mask, a current target mask score, and a current target mask category; 2) From the initial instance prediction and amodal instance prediction, obtain the number of corresponding data for each object category whose current target mask score exceeds the set threshold; Get the current target mask score of each object category in the instance prediction in turn, which is greater than the set instance threshold The number of instance prediction data N1, and the current target mask score of each object category in the amodal instance prediction is greater than the amodal instance threshold The number of non-modal instance prediction data N2; 3) From the initial instance predictions and amodal instance predictions, sort all current target mask scores of each object category in descending order, and set the selection percentage threshold of each category sorting sequence; The current target mask scores of each object category in the statistical instance prediction are the top The number of instance prediction data N3; Count the top current target mask scores for each object category in the amodal instance prediction The number of non-modal instance prediction data is N4; 4) The data corresponding to max(N1, N3) is set as the instance definite label, and the rest of the data is set as the instance uncertain region, and the instance definite label and the instance uncertain region are used as instance pseudo labels; the data corresponding to max(N2, N4) is set as the non-modal instance definite label, and the rest of the data is set as the non-modal instance uncertain region, and the corresponding non-modal instance definite label and the non-modal instance uncertain region are used as non-modal instance pseudo labels; based on the initial semantic prediction, the semantic label corresponding to the maximum prediction value of each semantic category is obtained, and then the semantic pseudo label is obtained after screening using the prediction probability threshold of the semantic category label.

2. The method according to claim 1, characterized in that Perform AND operations on the target mask of the determined label in each instance and non-modal instance in the target domain image and the object class prediction mask of the uncertain region and the target domain image to obtain the instance and non-modal instance pseudo labels of the filtered target domain image; Among them, the object class prediction mask of the target domain image is obtained according to the following formula: in, Display pictures At pixel position The object class prediction mask at , T represents the object class target category in all semantic categories, Display pictures At pixel position The semantic category labels at the location include object categories and background categories. , Display pictures At pixel position The semantic category index value corresponding to the maximum predicted value of the semantic category c at , Display pictures At pixel position The predicted value of the semantic category c at .

3. A passive domain scene adaptive occlusion-aware seamless segmentation method, characterized in that: include: Step 1: Generate pseudo labels for image data in the target domain using the method of claim 1 or 2, including pseudo labels in the form of semantic, instance, and non-modal instance; Step 2: Use the images in the target domain and the corresponding pseudo labels to form training samples, and use the training samples to train the segmentation model of the source domain; During the training process, for the current iteration training At this moment, the training sample is the training picture And the corresponding semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels: , the segmentation model outputs semantic branch predictions , instance branch prediction , non-modal instance branch prediction ; Use the sum of the prediction losses of the semantic branch, instance branch, and non-modal instance branch for back propagation, and after updating the segmentation model parameters, continue iterative training until the set number of iterations is reached, and obtain the occlusion-aware seamless segmentation model; Step 3: Use the occlusion-aware seamless segmentation model to segment the image in the target domain.

4. The method according to claim 3, characterized in that During model training, when calculating the prediction loss of the instance classification branch, it is necessary to use both the certain label and the uncertain region in the instance-level pseudo-label. The instance branch prediction loss calculation process is as follows: Current training iteration At this moment, for the input image in the current training process , the corresponding instance pseudo label generated It is divided into a binary instance label mask and a binary instance uncertainty mask, where the binary instance label mask Refers to the instance-determined label in the instance pseudo-label The target mask of the binary instance is uncertain Refers to the instance uncertainty area in the instance pseudo label The target mask; Display pictures in sequence The first, second, ..., Instances determine the label, Corresponding The target mask; Then, the training images Example prediction The target mask in , and the binary instance label mask is used to calculate the binary classification loss, and then the binary classification loss is multiplied pixel by pixel with the binary instance uncertainty mask to obtain the prediction loss of the final instance branch : Among them, BCE represents the binary classification loss, Represents training pictures The instance pseudo-label of Binarized instance label masks; Non-modal instance branch: Current training iteration At this moment, for the input image in the current training process , the corresponding non-modal instance pseudo label generated It is divided into a binary non-modal instance label mask and a binary non-modal instance uncertainty mask, where the binary non-modal instance label mask Refers to the instance-determined label in the non-modal instance pseudo-label The target mask of the binary non-modal instance uncertainty mask Refers to the uncertainty area of ​​the non-modal instance in the non-modal instance pseudo-label The target mask; Display pictures in sequence The first, second, ..., Instances determine the label, Corresponding The target mask; The segmentation model is based on the current input training picture Obtained non-modal instance branch prediction The target mask in , and the binary classification loss is calculated with the binary non-modal instance label mask, and then the binary classification loss is multiplied pixel by pixel with the binary non-modal instance uncertainty mask to obtain the prediction loss of the final non-modal instance branch : in, Represents training pictures The non-modal instance pseudo-label of Binarized amodal instance label masks.

5. The method according to claim 3, characterized in that: Semantic Prediction With semantic pseudo-labels Constructing semantic branch loss : Among them, CE represents the cross entropy loss.

6. The method according to any one of claims 3 to 5, characterized in that: Adopt a non-modal guided target mixing strategy to increase the number of targets in each image in the training sample and increase the diversity of the training samples. The specific steps are as follows: 1) Generate high-quality hard labels: First, all images in the target domain are predicted by the non-modal instance prediction obtained by the segmentation model in the source domain. The current target mask score obtained is greater than the set hard label threshold. and all current target masks before scoring % of the non-modal instance prediction data, select the corresponding non-modal instance prediction with a larger data amount; then, among the selected non-modal instance predictions, determine whether the target mask category is consistent with the category in the corresponding semantic label. If not, remove the non-modal instance prediction corresponding to the target mask category; The final obtained non-modal instance prediction is used as a high-quality hard label; 2) Screen out high-quality unobstructed hard labels: When the target mask predicted by each non-modal instance in the high-quality hard label does not overlap with the target mask area of ​​the high-quality hard label predicted by other non-modal instances in the current image, the high-quality hard label predicted by the non-modal instance is retained, otherwise, the high-quality hard label predicted by the non-modal instance is discarded; the high-quality hard labels predicted by all retained non-modal instances are used as high-quality unobstructed hard labels; 3) Target random mixing: All high-quality unoccluded hard labels are regarded as a high-quality unoccluded target repository. In each iterative training of the segmentation model, N high-quality unoccluded hard labels predicted by non-modal instances are randomly selected from the high-quality unoccluded target repository as targets to be mixed. The target image data of the corresponding image position is obtained according to the target mask in the high-quality unoccluded hard label of each target to be mixed, and it is pasted to the image of the current target domain. At the same time, according to the pasted position and category, the semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels of the image of the current target domain are changed accordingly.

7. A passive domain scene adaptive occlusion-aware seamless segmentation system, characterized in that: include: Pseudo-label generation module: using the method of claim 1 or 2 to generate pseudo-labels for image data in the target domain, including pseudo-labels in the form of semantics, instances, and non-modal instances; Training module: adopting the method described in any one of claims 3 to 6, using pictures in the target domain and corresponding pseudo labels to form training samples, and using the training samples to train the segmentation model of the source domain; During the training process, for the current iteration training At this moment, the training sample is the training picture And the corresponding semantic pseudo-labels, instance pseudo-labels, and non-modal instance pseudo-labels: , the segmentation model outputs semantic branch predictions , instance branch prediction , non-modal instance branch prediction ; The sum of the prediction losses of the semantic branch, instance branch, and non-modal instance branch is used for back propagation, and after updating the segmentation model parameters, iterative training is continued until the set number of iterations is reached, and an occlusion-aware seamless segmentation model is obtained. Segmentation module: Use the occlusion-aware seamless segmentation model to segment the image in the target domain.

8. An electronic terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: The method of any one of claims 1 to 2 or the steps of the method of any one of claims 3 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The method of any one of claims 1 to 2 or the steps of the method of any one of claims 3 to 6.

Images