Open-world semantic segmentation method and device based on region-aware metric learning

Abstract

The present invention provides an open-world semantic segmentation method and apparatus based on region-aware metric learning. The method comprises: performing abnormal region segmentation on a target image to generate unknown regions and region-aware features corresponding to the unknown regions; dividing the unknown region to generate multiple unknown subregions and region-aware features corresponding to the unknown subregions; and determining the category corresponding to the unknown subregion based on the region-aware features corresponding to the unknown subregions and the target region-aware features corresponding to a first target category; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image. The open-world semantic segmentation method based on region-aware metric learning of the present invention further segments the unknown region based on the MCA module to generate unknown subregions for incremental few-shot learning, thereby improving the model's recognition performance for out-of-distribution objects, thereby increasing the precision and accuracy of the segmentation results, and thus improving the final segmentation effect.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and in particular to an open-world semantic segmentation method and device based on region-aware metric learning. Background Art

[0002] Most current computer vision applications expect to be able to handle unknown categories, requiring deep learning models to be able to handle out-of-distribution (OOD) data. These models, which can adapt to new categories without forgetting old ones, are called "open-world" models. However, open-world image segmentation techniques are lacking in the relevant technologies. Summary of the Invention

[0003] The present invention provides an open-world semantic segmentation method and device based on region-aware metric learning, which is used to address the defect of the existing technology in lacking open-world image segmentation technology and realize image segmentation in an open-world environment.

[0004] The present invention provides an open-world semantic segmentation method based on region-aware metric learning, comprising:

[0005] Segmenting the target image into abnormal regions to generate unknown regions and region perception features corresponding to the unknown regions;

[0006] Dividing the unknown region to generate a plurality of unknown sub-regions and region perception features corresponding to the unknown sub-regions;

[0007] Based on the region perception features corresponding to the unknown subregion and the target region perception features corresponding to the first target category, the category corresponding to the unknown subregion is determined; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image.

[0008] According to an open-world semantic segmentation method based on region-aware metric learning provided by the present invention, segmenting the target image into abnormal regions, generating unknown regions and region-aware features corresponding to the unknown regions, includes:

[0009] Performing edge prediction on the target image to generate an edge prediction image;

[0010] Post-processing the edge prediction image to generate multiple candidate regions;

[0011] Performing abnormal segmentation on the multiple candidate regions to generate region-aware features corresponding to the candidate regions and region-aware abnormality probabilities corresponding to the candidate regions;

[0012] Generating uncertainty intensity corresponding to pixels in the candidate area based on the perceived abnormality probability of the area;

[0013] When the uncertainty intensity exceeds a first target threshold, the candidate area corresponding to the uncertainty intensity is determined to be the unknown area.

[0014] According to an open-world semantic segmentation method based on region-aware metric learning provided by the present invention, performing abnormal segmentation on the multiple candidate regions, generating region-aware features corresponding to the candidate regions, and region-aware abnormality probabilities corresponding to the candidate regions, includes:

[0015] Input the multiple candidate regions into a RAML module, and obtain region-aware features corresponding to the candidate regions output by the RAML module;

[0016] Using Circle loss constraints, based on the region-aware features corresponding to the candidate region and the region-aware features corresponding to the second target category, a region-aware anomaly probability corresponding to the candidate region is generated;

[0017] The RAML module is obtained after training based on sample images with region-aware feature labels; the second target category is a known category among multiple feature categories corresponding to the target image.

[0018] According to an open-world semantic segmentation method based on region-aware metric learning provided by the present invention, segmenting the unknown region to generate multiple unknown sub-regions and region-aware features corresponding to the unknown sub-regions includes:

[0019] Inputting the unknown region into a plurality of meta-channels in the MCA module, and obtaining the plurality of unknown sub-regions output by the plurality of meta-channels;

[0020] The MCA module is trained based on the target loss function.

[0021] According to an open-world semantic segmentation method based on region-aware metric learning provided by the present invention, determining the category corresponding to the unknown subregion based on the region-aware features corresponding to the unknown subregion and the target region-aware features corresponding to the first target category includes:

[0022] When the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than a second target threshold, and the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than the cosine similarity between the unknown sub-region and the region-perceptual features corresponding to other unknown categories in the target image except the first target category, it is determined that the category corresponding to the unknown sub-region is the first target category.

[0023] According to an open-world semantic segmentation method based on region-aware metric learning provided by the present invention, before performing abnormal region segmentation on a target image and generating unknown regions and region-aware features corresponding to the unknown regions, the method further includes:

[0024] Inputting the target image into a feature extractor of a closed set segmentation module, and obtaining region-aware features corresponding to a plurality of feature images output by the feature extractor;

[0025] Inputting the region-aware features corresponding to the multiple feature images into the label predictor of the closed set segmentation module, and obtaining a second target category corresponding to the region-aware features corresponding to the target feature image in the multiple feature images output by the label predictor;

[0026] The closed-set segmentation module is trained using a sample image as a sample and a semantic segmentation category corresponding to the sample image as a sample label;

[0027] The second target category corresponding to the target feature image is a trained category.

[0028] The present invention also provides an open-world semantic segmentation device based on region-aware metric learning, comprising:

[0029] A first processing module is used to segment the target image into abnormal regions, generate unknown regions and region perception features corresponding to the unknown regions;

[0030] A second processing module is configured to segment the unknown region to generate a plurality of unknown sub-regions and region perception features corresponding to the unknown sub-regions;

[0031] The third processing module is used to determine the category corresponding to the unknown sub-region based on the region perception features corresponding to the unknown sub-region and the target region perception features corresponding to the first target category; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, an open-world semantic segmentation method based on region-aware metric learning as described above is implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the open-world semantic segmentation methods based on region-aware metric learning as described above.

[0034] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described open-world semantic segmentation methods based on region-aware metric learning.

[0035] The open-world semantic segmentation method and device based on region-aware metric learning provided by the present invention adopt a classic uncertainty-based method to extract candidate regions for abnormal segmentation to generate unknown regions, so as to ensure the integrity of each segmented region; then, based on the MCA module, the unknown regions are further segmented to generate high-quality unknown sub-regions for incremental few-shot learning, thereby improving the model's prediction performance for out-of-distribution objects, thereby improving the precision and accuracy of the segmentation results, and thus improving the final segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is one of the flow charts of the open-world semantic segmentation method based on region-aware metric learning provided by the present invention;

[0038] Figure 2 This is the second flow chart of the open-world semantic segmentation method based on region-aware metric learning provided by the present invention;

[0039] Figure 3 This is one of the schematic diagrams of the effect of the open world semantic segmentation method based on region-aware metric learning provided by the present invention;

[0040] Figure 4 This is the second schematic diagram of the effect of the open-world semantic segmentation method based on region-aware metric learning provided by the present invention;

[0041] Figure 5Schematic diagram of the structure of the open world semantic segmentation device based on region-aware metric learning provided by the present invention;

[0042] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] Related technologies, there are mainly the following open world image segmentation technologies.

[0045] 1) Uncertainty-based methods: The baseline for uncertainty-based methods uses the negative of the maximum SoftMax probability of known categories as the uncertainty probability score (MSP). However, in large-scale datasets, known categories may have high similarity, which affects the distribution after SoftMax normalization and thus weakens the performance of MSP.

[0046] 2) Methods based on pixel-level feature embedding can incorrectly segment objects into fragments and lead to a large number of fine-grained segmentation errors. Moreover, this method forcibly assigns a one-hot vector to each class as a fixed metric center embedding expression, ignoring the natural distribution between different categories and inconsistent with the prior distribution of visual perception.

[0047] Both of the above methods have the problem of poor segmentation effect.

[0048] The following combination Figures 1 to 4 The present invention describes an open-world semantic segmentation method based on region-aware metric learning.

[0049] The execution entity of the open-world semantic segmentation method based on region-aware metric learning of the present invention can be an open-world semantic segmentation device based on region-aware metric learning, or a server, or a user terminal, including but not limited to a mobile phone, a tablet computer, and a PC.

[0050] like Figure 1 As shown, the open-world semantic segmentation method based on region-aware metric learning includes: step 110, step 120 and step 130.

[0051] Step 110: Segment the target image into abnormal regions to generate unknown regions and region perception features corresponding to the unknown regions;

[0052] In this step, the target image is the image to be segmented.

[0053] The target image may include one or more foreground features, and the categories of the foreground features may be the same or different.

[0054] The target image may be an image or video frame retrieved from a database, or may be an image or video frame pulled from a network, or may be an image or video frame collected by a sensor, and the present invention is not limited thereto.

[0055] The unknown region is the region corresponding to the foreground features whose categories are not identified.

[0056] The region-aware features corresponding to the unknown region are features used to characterize foreground features in the unknown region and can be expressed as a low-dimensional vector.

[0057] Abnormal region segmentation (Region-ware Metric Learning, RAML) is used to identify abnormal segmentation regions in the target image.

[0058] For example Figure 2 The target image shown includes foreground objects such as road surface, bus, car, tree, building and electric pole. Among them, road surface, tree, building and electric pole are identifiable categories; bus and car are unidentifiable categories; therefore, the area where the bus and car are located is the unknown area.

[0059] Continue to refer Figure 2 , Figure 2 The process shown by the dotted line corresponds to the process of this step. For the target image X, the method based on uncertainty region separation (URS) is used to perform abnormal segmentation to generate unknown regions and the corresponding regional perception features of the unknown regions ( Figure 2 elliptical area in the ).

[0060] In some embodiments, step 110 may include:

[0061] Perform edge prediction on the target image to generate an edge prediction image;

[0062] Post-process the edge prediction image to generate multiple candidate regions;

[0063] Perform abnormal segmentation on multiple candidate regions, generate region-aware features corresponding to the candidate regions, and generate region-aware abnormality probabilities corresponding to the candidate regions;

[0064] Generate the uncertainty intensity corresponding to the pixels in the candidate area based on the region-aware anomaly probability;

[0065] When the uncertainty intensity exceeds the first target threshold, the candidate region corresponding to the uncertainty intensity is determined to be an unknown region.

[0066] In this embodiment, the candidate region is a region in the target image where abnormal segmentation may occur.

[0067] The unknown region is the region containing foreground features of unknown categories.

[0068] The regional perception anomaly probability corresponding to the candidate region and the uncertainty intensity corresponding to the pixels in the candidate region are both used to characterize the possibility that the candidate region includes foreground features of unknown categories.

[0069] The target image is edge predicted to identify the edge contours of each foreground feature in the target image, and the target image is segmented based on the edge prediction results to generate an edge prediction image.

[0070] Continue to refer Figure 2 ,In the actual implementation process, the uncertainty based OOD object detection method MSP ,can be adopted as the region separation module to generate an edge ,prediction image, and its high uncertainty response around the object ,edge can be used as a guide for region separation.

[0071] In some other embodiments, a Sobel filter may be introduced on the target image to generate an edge prediction image, thereby further enhancing the delineation of fine-grained edges.

[0072] Specifically, through the formula:

[0073]

[0074] Generate an edge prediction image, where E is the edge prediction image, X is the target image, and U is the unnormalized logit (assessment model). is the indicator function, and α and β are hyperparameters that control marginal prediction.

[0075] After generating the edge prediction image E, the post-processing submodule is used to process the edge prediction image E including hole filling and connected component algorithms to generate a set of multiple candidate regions.

[0076] And R i ∈{0,1} H×W

[0077] Among them, R irepresents the i-th region, 1≤i≤T and i is an integer, T is the total number of candidate regions, H and W represent the length and width of the target image respectively.

[0078] It should be noted that the set of candidate regions generated in this step The unknown areas may or may not be included.

[0079] After obtaining the set of candidate regions, for each candidate region R in the set of candidate regions i Perform classification and generate candidate regions R i The corresponding region perception features and candidate regions R i The corresponding regional perception anomaly probability.

[0080] Among them, the candidate region R i The corresponding region perception anomaly probability is used to characterize the probability that an unknown area exists in the candidate area. The greater the anomaly probability, the greater the probability that a location area exists in the candidate area.

[0081] In some embodiments, performing anomaly segmentation on multiple candidate regions and generating region-aware features corresponding to the candidate regions and region-aware anomaly probabilities corresponding to the candidate regions may include:

[0082] Input multiple candidate regions into the RAML module and obtain the region perception features corresponding to the candidate regions output by the RAML module;

[0083] Generate a region-aware anomaly probability corresponding to the candidate region based on the region-aware features corresponding to the candidate region and the region-aware features corresponding to the second target category;

[0084] The RAML module is obtained after training based on sample images with region-aware feature labels; the second target category is a known category among multiple feature categories corresponding to the target image.

[0085] In this embodiment, the RAML module (Region-aware Metric Learning Module) is a region-aware metric learning module used to For each candidate region R i , can be expressed as follows:

[0086]

[0087] Generate region-aware features, where f object is the candidate region R i Corresponding regional perception features; F j,kis the feature vector of pixel (j,k), (j,k) is the coordinate value of the pixel, and D(·) consists of two fully connected layers and is used to control the embedding dimension.

[0088] Then, f object By comparing with all prototypes of known categories through metric learning constrained by Circle loss, Circle loss is used to expand the inter-class distance and reduce the intra-class distance of data samples, thereby improving the performance of the RAML module.

[0089] Specifically, through the formula:

[0090]

[0091] Generate the region perception anomaly probability corresponding to the candidate region, where is the candidate region R i The corresponding regional perception abnormality probability; f object is the candidate region R i Corresponding regional perception features; F is a feature image (F may be generated in advance based on the target image, which will be described in subsequent embodiments and will not be elaborated here); f l is the region-aware feature corresponding to the lth known category (i.e., the region-aware feature corresponding to the second target category), f l It can be used to generate semantic segmentation predictions for the target image in advance; N is used to represent the total number of known categories.

[0092] After generating the region-aware features corresponding to the candidate regions and the region-aware anomaly probabilities corresponding to the candidate regions, based on the region-aware anomaly probabilities, the formula is used:

[0093]

[0094] The uncertainty intensity corresponding to the pixels in the candidate area can be generated, where Q j,k is the candidate region R i The uncertainty intensity corresponding to the pixel (j, k) in the candidate region R i ; F is the feature map, is the probability of regional perception anomaly; is the first output of pixel (j,k) in the unnormalized U.

[0095] In some embodiments, the uncertainty intensity corresponding to each pixel can also be normalized so that 0≤Q j,k≤1; then comparing the normalized uncertainty intensity with the first target threshold to determine whether the candidate area where the pixel corresponding to the normalized uncertainty intensity is located is an unknown area.

[0096] The first target threshold may be user-defined, such as set to 0.5.

[0097] It can be understood that when the uncertainty intensity after normalization exceeds the first target threshold, the candidate area corresponding to the uncertainty intensity is determined to be an unknown area; when the uncertainty intensity after normalization does not exceed the first target threshold, the candidate area corresponding to the uncertainty intensity is determined not to be an unknown area.

[0098] According to the open-world semantic segmentation method based on region-aware metric learning provided by an embodiment of the present invention, by replacing a one-hot setting of a fixed center embedding with Circle loss as the target of metric learning, it not only maintains a good and natural inter-class distance, but also makes the intra-class distribution more concentrated. Such feature space division is more conducive to the segmentation of OOD data.

[0099] Step 120: Segment the unknown region to generate multiple unknown sub-regions and region-aware features corresponding to the unknown sub-regions;

[0100] In this step, when it is determined that the target image includes an unknown region, the unknown region can be further segmented to divide the unknown region into multiple unknown sub-regions, and regional perception features corresponding to each unknown sub-region are generated.

[0101] Continue to refer Figure 2 , Figure 2 The process shown by the solid line corresponds to the process of this step.

[0102] In the actual implementation process, a meta-channel-based Region Separation (MCA) method can be used to perform incremental small-sample learning to segment unknown regions into foreground features with new categories.

[0103] MCA first over-segments the unknown region into several meta-channels, and regions belonging to different meta-channels are aggregated to form a segmentation of the object, which is then evaluated by the region-aware metric learning module RAML.

[0104] For example, after obtaining the unknown area through step 110, the unknown area is marked to generate an abnormal image. And the abnormal image As the input of the MCA module, the unknown region is further segmented by the MCA module, and metric learning is used to segment the OOD objects to subdivide the foreground features in the unknown region into features of different new categories.

[0105] The specific implementation method of this step is described below.

[0106] In some embodiments, step 120 may include:

[0107] Input the unknown region into multiple meta-channels in the MCA module, and obtain multiple unknown sub-regions output by the multiple meta-channels;

[0108] Among them, the MCA module is trained based on the target loss function.

[0109] In this embodiment, an MCA module may be provided to detect abnormal images. Create a subregion in the unknown region in .

[0110] The abnormal image is a small image containing unknown areas in the target image.

[0111] It should be noted that the MCA module includes the SoftMax activation function C, where:

[0112] C∈[0,1] (N+K)×H×W

[0113] Among them, the first N channels are the segmentation results of all categories within the distribution, and the last K (K>M) channels are meta-channels. The K original channels are used to over-segment the unknown area. M is the number of unknown categories in all foreground features in the target image, M≥0 and M and N are both integers.

[0114] The target loss function is an integrated function generated based on all the loss functions related to MCA.

[0115] For example, the objective loss function can be determined by the following formula.

[0116]

[0117] in, is the target loss function, The segmentation loss used to characterize all in-distribution categories, is the Dice Loss loss function, Used to avoid unknown sub-regions (candidates for OOD objects) from being clustered in a few specific channels, Used to reconstruct the entire image using the output of all channels.

[0118] Specifically:

[0119]

[0120] in, Characterize dice loss; (C i ,C j ) represents the i-th and j-th channels of the segmentation output, and (N+K) is the total number of channels.

[0121]

[0122] in, represents the pixel (j, k) output of the i-th channel, and η is a hyperparameter that controls the separation; when the unknown subregion is scattered on the output channel according to Jenson's inequality, Reached minimum value.

[0123]

[0124] Among them, ⊙ is the element-by-element multiplication operator, It is a matrix with all 1s, H×W is used to represent the length and width of the target image.

[0125] During the research and development process, the inventors discovered that in related technologies, MCA tends to segment objects based on local semantic information, so that an unknown object may be segmented into multiple channels and lose its integrity (for example, the windows and wheels of a car may be divided into different channels).

[0126] In the present invention, unknown sub-regions are aggregated from certain meta-channels according to the few-shot (here L-shot) labeled image to generate a set of candidate regions after further segmentation. The final RAML module for incremental few-shot learning, where:

[0127]

[0128] in, is the set of candidate regions generated after further segmenting the unknown region in the target image, and T' is the total number of candidate regions generated after further segmenting the target image.

[0129] Based on the optimized candidate region R' i , the optimized regional perception features f corresponding to each candidate region can be generated object .

[0130] Table 1 shows the comparative relationship between the results of abnormal segmentation generated by the RAML module proposed in the present invention and the results of abnormal segmentation performed by other methods. The RAML module proposed in the present invention can produce higher response values ​​and better integrity in the area of ​​abnormal objects, significantly reducing false negatives.

[0131] Table 1

[0132]

[0133] Figure 3 The example compares the abnormality segmentation method proposed in the present invention with other methods, where (a) is the target image; (b) is the abnormal area annotation; (c) is the edge prediction result generated in step 110; (d) is the result of other related technologies; and (e) is the result of the abnormality segmentation method proposed in the present invention. It can be seen that the present invention can produce higher response values ​​and better integrity for the area of ​​abnormal objects, significantly reducing false negatives.

[0134] Step 130 : Determine the category corresponding to the unknown sub-region based on the region-perception feature corresponding to the unknown sub-region and the target region-perception feature corresponding to the first target category.

[0135] In this step, the first target category is any category among the unknown categories (ie, untrained categories) among the categories corresponding to all foreground features of the target image.

[0136] The target region perception feature is a pre-generated region perception feature corresponding to the foreground feature of the first target category.

[0137] During the incremental learning process, the image corresponding to the foreground features of the unknown category in the target image is first obtained and manually marked to generate a labeled image.

[0138] It is understandable that each unknown category may correspond to at least one labeled image.

[0139] For example, the prototype of the i-th unknown category of a new labeled image can be defined as:

[0140]

[0141] Among them, c i is the target area perception feature corresponding to the prototype of the i-th unknown category; Represent the feature embedding of the i-th unknown class in the j-th annotated image; L is the total number of annotated images corresponding to the i-th unknown class (for example, L can be 1 or 5, etc.); 1≤i≤M.

[0142] After generating the region-aware features corresponding to the unknown sub-region and the region-aware features corresponding to the first target category, the unknown sub-region can be divided into the region corresponding to the first target category by calculating the similarity between the two.

[0143] The following takes cosine similarity as an example to illustrate how to implement this step.

[0144] In some embodiments, step 130 may include:

[0145] When the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than the second target threshold, and the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than the cosine similarity between the region-perceptual feature corresponding to the unknown region and the region-perceptual features corresponding to other position categories in the target image except the first target category, the category corresponding to the unknown sub-region is determined to be the first target category.

[0146] In this embodiment, the formula:

[0147]

[0148] Determine the cosine similarity; where, Used to characterize the cosine similarity between the unknown sub-region and the prototype of the i-th unknown category; f object is the region-aware feature corresponding to the unknown sub-region, c i is the region-aware feature corresponding to the prototype of the i-th unknown category, and M is the total number of unknown categories in the target image.

[0149] After generating the cosine similarity, first screen out a category with the largest cosine similarity to the first target category, and obtain the cosine similarity value corresponding to the category with the largest cosine similarity; then determine whether the cosine similarity value corresponding to the category with the largest cosine similarity is greater than the second target threshold. If the cosine similarity value corresponding to the category with the largest cosine similarity is greater than the second target threshold, the category corresponding to the unknown sub-region is determined as the first target category.

[0150] That is, in the unknown sub-region R' i The corresponding regional perception feature and the regional perception feature c corresponding to the i-th unknown category (i.e., the first target category) i The cosine similarity between the target region perception features is greater than the second target threshold, and the unknown sub-region R' i The corresponding region-aware feature and the region-aware feature c corresponding to the i-th unknown category iThe cosine similarity between them is greater than the unknown sub-region R' i The corresponding region-aware features and the region-aware features c corresponding to all unknown categories except the i-th unknown category i , in the case of the cosine similarity between them, the category corresponding to the unknown sub-region is determined to be the i-th unknown category.

[0151] The second target threshold can be user-defined or can be determined as a hyperparameter for controlling classification. The value of the second target threshold can be set to 0.5. Of course, in other embodiments, the value of the second target threshold can also be set to other values, which are not limited by the present invention. For example, the formula:

[0152]

[0153] Classify the unknown sub-region; where, is the cosine similarity between the unknown subregion and the prototype of the i-th unknown category, is the cosine similarity between the unknown subregion and the prototypes of other unknown categories, θ novel is a hyperparameter that controls the classification (i.e., the second target threshold).

[0154] It is understandable that only in the cosine similarity When the above two criteria are met, the candidate region R' i Can be classified as the i-th new class C out，i .

[0155] During the research and development process, the inventors conducted tests using CityScapes data as an example and compared the method proposed in the present invention with other methods, obtaining the comparison results shown in Table 2.

[0156] It should be noted that in the experiments, car, truck, and bus are the three OOD classes that do not participate in the training phase, while the other 16 classes are regarded as in-distribution classes.

[0157] Table 2

[0158]

[0159] Table 2: Incremental few-shot learning results on Cityscapes for 16+1setting(OOD class is car)and 16+3 setting(OOD classes are car, truck, bus).The unknown classes are in blue.Finetune(FT)is the baseline with catastrophicforgetting.

[0160] Table 2 shows the incremental few-shot learning results on Cityscapes using the 16+1 setting (the OOD class is car) and the 16+3 setting (the OOD classes are car, truck, and bus). The shaded areas represent unknown classes, and Finetune (FT) is the baseline for catastrophic forgetting. This shows that the proposed method achieves excellent results.

[0161] Figure 4 The figure illustrates the comparison between the method proposed in the present invention and other methods, where (a) is the target image; (b) is the label image; (c) is the closed set output; (d) is the anomaly segmentation output; (e) is the MCA output; (f) is the output result of other related technologies; and (g) is the final output result of the present invention.

[0162] according to Figure 4 , the method proposed in this invention shows a significant ability to maintain the integrity of these object results. In addition, the feature embeddings generated by the RAML proposed in this invention maintain reasonable inter-class distances, and their intra-class distribution is also more concentrated. This feature distribution can promote the model to obtain a robust decision boundary.

[0163] According to the open-world semantic segmentation method based on region-aware metric learning provided by an embodiment of the present invention, a classic uncertainty-based method is adopted to extract candidate regions for abnormal segmentation to generate unknown regions to ensure the integrity of each segmented region; then the unknown regions are further segmented based on the MCA module to generate high-quality unknown sub-regions for incremental few-shot learning, thereby improving the model's prediction performance for out-of-distribution objects, thereby improving the precision and accuracy of the segmentation results, and thus improving the final segmentation effect.

[0164] In some embodiments, before step 110, the method may further include:

[0165] Input the target image into the feature extractor of the closed set segmentation module, and obtain the region-aware features corresponding to the multiple feature images output by the feature extractor;

[0166] Inputting the region-aware features corresponding to the plurality of feature images into a label predictor of a closed set segmentation module, and obtaining a second target category corresponding to the target feature image from the region-aware features corresponding to the plurality of feature images output by the label predictor;

[0167] Among them, the closed set segmentation module (or semantic segmentation network) is trained with sample images as samples and the semantic segmentation categories corresponding to the sample images as sample labels;

[0168] The second target category corresponding to the target feature image is a trained category.

[0169] In this embodiment, the closed set segmentation module (or semantic segmentation network) includes: feature extractor and label predictor in:

[0170]

[0171] The output end of the feature extractor is connected to the input end of the label predictor. The feature extractor is used to extract foreground features in the target image and input the foreground features into the label predictor. The label predictor is used to generate region-aware features corresponding to each foreground feature, predict the category corresponding to each foreground feature, and output the predicted category.

[0172] The target feature image is a feature image corresponding to features of known categories in the multiple feature images, and each target feature image corresponds to a second target category.

[0173] The second target category is a known category among the categories corresponding to all foreground features in the target image, that is, a trained category, that is, a category that can be predicted by the label predictor.

[0174] For example, the second target category can be expressed as:

[0175] C in ={C in,1 ,C in,2 ,…C in,N}

[0176] Among them, C in There are N categories within the distribution, and all N categories are annotated in the training dataset.

[0177] The first target category can be expressed as:

[0178] C out ={C out,1 ,C out,2 ,…C out,M}

[0179] Among them, Cout are M new classes that are not included in the training dataset.

[0180] During training, for closed set segmentation, based on minimizing the loss Train the module, where For guidance Generate pixel-wise segmentations for in-distribution categories.

[0181]

[0182] in, is the segmentation loss for all in-distribution categories; represents the multi-class cross entropy loss, X is the sample image, and Y is the sample label corresponding to the sample image; H×W is used to represent the length and width of the sample image.

[0183] After the training is completed, a trained feature extractor can be obtained and a trained label predictor

[0184] In practical applications, by inputting the target image into the trained closed set segmentation module, multiple feature images F and unnormalized logit U output by the closed set segmentation module can be obtained, where;

[0185]

[0186]

[0187] Where X is the target image, H×W is the length and width of the target image; By removing The SoftMax layer is obtained.

[0188] The feature image F and the non-normalized logit U generated in this embodiment will be used in steps 110 to 130 above.

[0189] According to the open-world semantic segmentation method based on region-aware metric learning provided by an embodiment of the present invention, open-world semantic segmentation is achieved by proposing an overall model including a backbone module for closed set segmentation, an anomaly segmentation module for depicting unknown regions of OOD data, and an incremental few-shot learning module for splitting unknown regions into objects with new categories. This achieves effective segmentation of new categories in images with small samples, significantly improving the image segmentation effect.

[0190] The open-world semantic segmentation device based on region-aware metric learning provided by the present invention is described below. The open-world semantic segmentation device based on region-aware metric learning described below and the open-world semantic segmentation method based on region-aware metric learning described above can be referenced to each other.

[0191] like Figure 5 As shown, the open world semantic segmentation device based on region-aware metric learning includes: a first processing module 510, a second processing module 520 and a third processing module 530.

[0192] The first processing module 510 is used to segment the target image into abnormal regions and generate unknown regions and region perception features corresponding to the unknown regions;

[0193] The second processing module 520 is used to divide the unknown region into multiple unknown sub-regions and generate region perception features corresponding to the unknown sub-regions;

[0194] The third processing module 530 is used to determine the category corresponding to the unknown sub-region based on the region perception feature corresponding to the unknown sub-region and the target region perception feature corresponding to the first target category; wherein the first target category is an unknown category among multiple feature categories corresponding to the target image.

[0195] According to the open-world semantic segmentation device based on region-aware metric learning provided by an embodiment of the present invention, a classic uncertainty-based method is adopted to extract candidate regions for abnormal segmentation to generate unknown regions to ensure the integrity of each segmented region; then the unknown regions are further segmented based on the MCA module to generate high-quality unknown sub-regions for incremental few-shot learning, thereby improving the model's prediction performance for out-of-distribution objects, thereby improving the precision and accuracy of the segmentation results, and thus improving the final segmentation effect.

[0196] In some embodiments, the first processing module 510 may also be configured to:

[0197] Perform edge prediction on the target image to generate an edge prediction image;

[0198] Post-process the edge prediction image to generate multiple candidate regions;

[0199] Perform abnormal segmentation on multiple candidate regions, generate region-aware features corresponding to the candidate regions, and generate region-aware abnormality probabilities corresponding to the candidate regions;

[0200] Generate the uncertainty intensity corresponding to the pixels in the candidate area based on the region-aware anomaly probability;

[0201] When the uncertainty intensity exceeds the first target threshold, the candidate region corresponding to the uncertainty intensity is determined to be an unknown region.

[0202] In some embodiments, the first processing module 510 may also be configured to:

[0203] Input multiple candidate regions into the RAML module and obtain the region perception features corresponding to the candidate regions output by the RAML module;

[0204] Using the Circle loss constraint, the region-aware anomaly probability corresponding to the candidate region is generated based on the region-aware features corresponding to the candidate region and the region-aware features corresponding to the second target category.

[0205] The RAML module is obtained after training based on sample images with region-aware feature labels; the second target category is a known category among multiple feature categories corresponding to the target image.

[0206] In some embodiments, the second processing module 520 may also be configured to:

[0207] Input the unknown region into multiple meta-channels in the MCA module, and obtain multiple unknown sub-regions output by the multiple meta-channels;

[0208] Among them, the MCA module is trained based on the target loss function.

[0209] In some embodiments, the third processing module 530 may also be configured to:

[0210] When the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than the second target threshold, and the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than the cosine similarity between the unknown sub-region and the region-perceptual features corresponding to other unknown categories in the target image except the first target category, the category corresponding to the unknown sub-region is determined to be the first target category.

[0211] In some embodiments, the apparatus may further include a fourth processing module configured to:

[0212] Before performing abnormal region segmentation on the target image and generating unknown regions and region-aware features corresponding to the unknown regions, the target image is input into a feature extractor of a closed set segmentation module to obtain region-aware features corresponding to a plurality of feature images output by the feature extractor;

[0213] Inputting the region-aware features corresponding to the plurality of feature images into a label predictor of a closed set segmentation module, and obtaining a second target category corresponding to the region-aware features corresponding to the target feature image in the plurality of feature images output by the label predictor;

[0214] Among them, the closed set segmentation module is trained with sample images as samples and the semantic segmentation categories corresponding to the sample images as sample labels;

[0215] The second target category corresponding to the target feature image is a trained category.

[0216] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute an open-world semantic segmentation method based on region-aware metric learning, the method comprising: performing abnormal region segmentation on the target image to generate unknown regions and region-aware features corresponding to the unknown regions; dividing the unknown region to generate multiple unknown sub-regions and region-aware features corresponding to the unknown sub-regions; determining the category corresponding to the unknown sub-region based on the region-aware features corresponding to the unknown sub-regions and the target region-aware features corresponding to the first target category; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image.

[0217] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0218] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the open-world semantic segmentation method based on region-aware metric learning provided by the above methods, and the method includes: performing abnormal region segmentation on the target image to generate unknown regions and region-aware features corresponding to the unknown regions; dividing the unknown region to generate multiple unknown sub-regions and region-aware features corresponding to the unknown sub-regions; determining the category corresponding to the unknown sub-region based on the region-aware features corresponding to the unknown sub-region and the target region-aware features corresponding to the first target category; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image.

[0219] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned open-world semantic segmentation method based on region-aware metric learning, the method comprising: performing abnormal region segmentation on the target image to generate unknown regions and region-aware features corresponding to the unknown regions; dividing the unknown region to generate multiple unknown sub-regions and region-aware features corresponding to the unknown sub-regions; determining the category corresponding to the unknown sub-region based on the region-aware features corresponding to the unknown sub-region and the target region-aware features corresponding to the first target category; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image.

[0220] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0221] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A region-aware metric learning-based open-world semantic segmentation method, characterized by: include: Segmenting the target image into abnormal regions to generate unknown regions and region perception features corresponding to the unknown regions; Dividing the unknown region to generate a plurality of unknown sub-regions and region perception features corresponding to the unknown sub-regions; Determining a category corresponding to the unknown subregion based on the region perception feature corresponding to the unknown subregion and the target region perception feature corresponding to a first target category; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image; The step of segmenting the target image into abnormal regions and generating unknown regions and region perception features corresponding to the unknown regions includes: Performing edge prediction on the target image to generate an edge prediction image; Post-processing the edge prediction image to generate multiple candidate regions; Performing abnormal segmentation on the multiple candidate regions to generate region-aware features corresponding to the candidate regions and region-aware abnormality probabilities corresponding to the candidate regions; Generating uncertainty intensity corresponding to pixels in the candidate area based on the perceived abnormality probability of the area; When the uncertainty intensity exceeds a first target threshold, determining the candidate area corresponding to the uncertainty intensity as the unknown area; The performing abnormal segmentation on the multiple candidate regions to generate region-aware features corresponding to the candidate regions and region-aware abnormality probabilities corresponding to the candidate regions includes: Inputting the plurality of candidate regions into a RAML module, and obtaining region-aware features corresponding to the candidate regions output by the RAML module; the RAML model is a region-aware metric learning module for classifying a set of candidate regions; Using Circle loss constraints, based on the region-aware features corresponding to the candidate region and the region-aware features corresponding to the second target category, a region-aware anomaly probability corresponding to the candidate region is generated; The RAML module is obtained after training based on sample images with region-aware feature labels; the second target category is a known category among multiple feature categories corresponding to the target image; The segmenting of the unknown region to generate a plurality of unknown sub-regions and region perception features corresponding to the unknown sub-regions includes: The unknown region is input into multiple meta-channels in the MCA module to obtain the multiple unknown sub-regions output by the multiple meta-channels; wherein the MCA module is used to further segment the unknown region to generate high-quality unknown sub-regions for incremental few-shot learning; wherein the MCA module is trained based on the target loss function.

2. The open-world semantic segmentation method based on region-aware metric learning according to claim 1, characterized in that The determining the category corresponding to the unknown sub-region based on the region perception feature corresponding to the unknown sub-region and the target region perception feature corresponding to the first target category includes: When the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than a second target threshold, and the cosine similarity between the region-perceptual feature corresponding to the unknown sub-region and the target region-perceptual feature corresponding to the first target category is greater than the cosine similarity between the unknown sub-region and the region-perceptual features corresponding to other unknown categories in the target image except the first target category, it is determined that the category corresponding to the unknown sub-region is the first target category.

3. The open-world semantic segmentation method based on region-aware metric learning according to claim 1, characterized in that Before segmenting the target image into abnormal regions and generating unknown regions and region perception features corresponding to the unknown regions, the method further includes: Inputting the target image into a feature extractor of a closed set segmentation module, and obtaining region-aware features corresponding to a plurality of feature images output by the feature extractor; Inputting the region-aware features corresponding to the multiple feature images into the label predictor of the closed set segmentation module, and obtaining a second target category corresponding to the region-aware features corresponding to the target feature image in the multiple feature images output by the label predictor; The closed set segmentation module is trained by taking a sample image as a sample and a semantic segmentation category corresponding to the sample image as a sample label; The second target category corresponding to the target feature image is a trained category.

4. An open-world semantic segmentation device based on region-aware metric learning, characterized in that include: A first processing module is used to segment the target image into abnormal regions, generate unknown regions and region perception features corresponding to the unknown regions; A second processing module is configured to segment the unknown region to generate a plurality of unknown sub-regions and region perception features corresponding to the unknown sub-regions; a third processing module, configured to determine a category corresponding to the unknown subregion based on the region perception feature corresponding to the unknown subregion and the target region perception feature corresponding to a first target category; wherein the first target category is an unknown category among the multiple feature categories corresponding to the target image; Wherein, the device is also used for: Performing edge prediction on the target image to generate an edge prediction image; Post-processing the edge prediction image to generate multiple candidate regions; Performing abnormal segmentation on the multiple candidate regions to generate region-aware features corresponding to the candidate regions and region-aware abnormality probabilities corresponding to the candidate regions; Generating uncertainty intensity corresponding to pixels in the candidate area based on the perceived abnormality probability of the area; When the uncertainty intensity exceeds a first target threshold, determining the candidate area corresponding to the uncertainty intensity as the unknown area; The device is also used for: Inputting the plurality of candidate regions into a RAML module, and obtaining region-aware features corresponding to the candidate regions output by the RAML module; the RAML model is a region-aware metric learning module for classifying a set of candidate regions; Using Circle loss constraints, based on the region-aware features corresponding to the candidate region and the region-aware features corresponding to the second target category, a region-aware anomaly probability corresponding to the candidate region is generated; The RAML module is obtained after training based on sample images with region-aware feature labels; the second target category is a known category among multiple feature categories corresponding to the target image; The device is also used to: input the unknown area into multiple meta-channels in the MCA module, and obtain the multiple unknown sub-regions output by the multiple meta-channels; wherein, the MCA module is used to further segment the unknown area to generate high-quality unknown sub-regions for incremental few-shot learning; wherein, the MCA module is trained based on the target loss function.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the open world semantic segmentation method based on region-aware metric learning as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the open-world semantic segmentation method based on region-aware metric learning is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the open-world semantic segmentation method based on region-aware metric learning is implemented.

Images