Construction method of unsupervised semantic segmentation model, image semantic segmentation method and equipment

Abstract

The invention discloses a construction method of an unsupervised semantic segmentation model, and an image semantic segmentation method and device, and belongs to the technical field of image processing. Based on a prior principle that similar samples appear in groups, potential similar features of image block features of each image block in the whole image are searched by utilizing mutual attention weights between every two image blocks in intra-batch images obtained by a ViT network; segmenting head features, output by a segmenting head, of all image blocks of the in-batch images are aggregated on the basis of mutual attention weights between every two processed image blocks, segmenting head aggregation features of all the image blocks of the in-batch images are obtained and further input into a projection head, and corresponding projection aggregation features are obtained; through a feature aggregation mode, the feature expression ability of a segmentation head and a projection head is improved, sufficient expression is given to strong semantics appearing in a large probability, meanwhile, sufficient expression is given to weak semantics appearing in a small probability, and then image semantic segmentation is finely and accurately achieved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and more specifically, relates to a method for constructing an unsupervised semantic segmentation model, an image semantic segmentation method and a device. Background Art

[0002] Unsupervised semantic segmentation aims to discover and locate semantically meaningful categories in an image corpus without any label annotations, thereby saving a lot of manpower and financial resources. At present, there have been many works trying to use some prior knowledge to train semantic segmentation models without any supervision, such as superpixel segmentation, edge detection, and salient object detection. However, such methods rely on the underlying features of pixels such as color. Even though the salient object detection method can use the intermediate semantic priors of the image, it still relies heavily on the object-centric scene. Real scenes often have complex semantic structures, which makes the above methods unable to produce good representations. Recently, due to the successful application of self-supervised representation models in the field of vision, using the high-level semantic priors brought by them to train semantic segmentation models has become a hot direction for unsupervised semantic segmentation.

[0003] ViT (Vision Transformer) divides an image into multiple image blocks. In the field of unsupervised semantic segmentation, each image block is called an anchor. Position information is added to each anchor point, and an additional global tag is added before the image anchor point and position information are applied to the self-attention layer to match the global information. In addition, since the self-attention mechanism involves the comparison of image block features, it is natural to construct a weighted matrix with semantics based on the attention score on the anchor point, which successfully introduces the self-attention mechanism into the visual field. Based on the feature acquisition and expression capabilities of ViT, scholars have proposed many improvements to ViT for supervised learning, unsupervised learning, and dense prediction. Due to the unique characteristics of the ViT model, many self-supervised frameworks have been studied around the ViT model. In addition, the high-quality semantic representation brought by the self-supervised framework has greatly promoted the development of unsupervised semantic segmentation methods. These unsupervised semantic segmentation methods use the high-level semantic priors generated by the self-supervised model DINO, and further group the pixel-level semantic representations into different semantic groups through clustering, and achieve a good result. It can be foreseen that the priors brought by the self-supervised ViT model are bound to provide a good idea for the next unsupervised semantic segmentation method.

[0004] Although some existing unsupervised semantic segmentation methods based on self-supervised ViT dense features have produced good results, some of these methods simply cluster these dense features using various clustering schemes, relying too much on the semantic quality of the dense features generated by self-supervised ViT, ignoring the unreliability of the dense features, and easily misjudge pseudo-negative samples as pseudo-positive samples during training. In order to get rid of the possibility of misjudgment to the greatest extent, other methods rely on high thresholds and only select very reliable pseudo-positive samples for clustering training, thereby ignoring small categories with a small number of samples, making it difficult for the model to discover potential categories and accurately identify potential small category pixels, and thus unable to accurately achieve image semantic segmentation. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a method for constructing an unsupervised semantic segmentation model, an image semantic segmentation method and a device, so as to solve the technical problem that the prior art cannot accurately identify potential small category pixels and thus cannot achieve image semantic segmentation in a precise and accurate manner.

[0006] In order to achieve the above objectives, in a first aspect, the present invention provides a method for constructing an unsupervised semantic segmentation model, comprising:

[0007] For each intra-batch image T in a batch of pre-acquired images:

[0008] The ViT network is used to divide T into R rows and R columns of image blocks and perform feature extraction to obtain the features of each image block of T and the mutual attention weights between each image block; the features of each image block of T are processed by the segmentation head and the projection head in turn to obtain the projection features of each image block; R is a positive integer;

[0009] The segmentation head features of each image block of T obtained after the segmentation head form the first segmentation head feature matrix S1 of size R×R; construct the corresponding R×R weight matrix A for the kth image block of T k , A k The lth element of is the mutual attention weight between the kth image block and the lth image block of T; k After normalizing each element in, calculate the average value a k , and A k Less than a k Set all elements of to 0; k = 1, 2, ... R × R,; l = (i-1) × R + j; i = 1, 2, ..., R; j = 1, 2, ..., R; and compare S1 and A k The segmentation head aggregate features of the kth image block of T are obtained by adding the elements after the dot multiplication operation; the segmentation head aggregate features of each image block of T are input into the projection head to obtain the projection aggregate features of each image block of T;

[0010] A training loss function is constructed, and based on the training loss function, the segmentation head and the projection head are trained at the same time; the training loss function includes: a first clustering loss function whose goal is to maximize the similarity between the projection features of the image blocks in the batch and the projection features of the pseudo positive samples of the image blocks, and minimize the similarity between the projection features of the pseudo negative samples of the image blocks, and a second clustering loss function whose goal is to maximize the similarity between the projection aggregate features of the image blocks in the batch and the projection aggregate features of the pseudo positive samples of the image blocks, and minimize the similarity between the projection aggregate features of the pseudo negative samples of the image blocks;

[0011] After training, an unsupervised semantic segmentation model consisting of a cascaded ViT network, a segmentation head, a projection head, and a classification module is constructed;

[0012] The classification module is used to calculate the similarity between the projection feature of each image block of the image to be segmented and the projection feature of each pseudo-positive sample in the application stage, and mark the image block as the category number carried by the pseudo-positive sample with the greatest similarity, thereby obtaining the semantic segmentation result of the image to be segmented;

[0013] The pseudo positive and negative samples of any image block b are determined in the following manner: all image blocks in the image block set whose similarity with image block b is greater than a preset positive sample threshold and except image block b are taken as pseudo positive samples of image block b; all image blocks in the image block set whose similarity with image block b is less than or equal to the preset positive sample threshold are taken as pseudo negative samples of image block b; the image block set includes: every image block in every image in a batch.

[0014] Further preferably, the method for constructing the unsupervised semantic segmentation model further includes:

[0015] For each image T in a batch:

[0016] The ViT network is used to transform the enhanced image T of T a and neighboring images T n Divide into R rows and R columns of image blocks and perform feature extraction to obtain T a and T n The image block features of n It is obtained by calculating the similarity between T and each image in the pre-collected image set, and randomly selecting an image from the top M images with the highest similarity as T. n ; M ≥ 1;

[0017] T n The features of each image block are processed by the segmentation head to obtain T n The segmentation head features of each image block are used to form a second segmentation head feature matrix S2 of size R×R;

[0018] Calculate the lth image block feature of T and T a The similarity between the l-th image block features is used as the corresponding screening threshold; l = (i-1) × R + j; i = 1, 2, ..., R; j = 1, 2, ..., R;

[0019] T n The k-th image block constructs the corresponding R×R weight matrix B k , B k The lth element of is T n The similarity between the kth image block feature of T and the lth image block feature of B k All elements in B that are smaller than the corresponding screening threshold are set to 0; k = 1, 2, ..., R × R; S2 and B k Add the elements after the dot multiplication operation to get T n The segmentation head aggregates features of the k-th image block; T n The segmentation head aggregates the features of each image block and inputs them into the projection head to obtain T n Projection aggregation features of each image patch;

[0020] The training loss function also includes: a third clustering loss function whose goal is to maximize the similarity between the projected aggregate features of the neighboring image blocks of the images in the batch and the projected aggregate features of the pseudo positive samples of the image blocks, and minimize the similarity between the projected aggregate features of the pseudo negative samples of the image blocks.

[0021] Further preferably, the similarity between T and a certain image C in the pre-collected image set is calculated in the following manner:

[0022] The average value of each image block feature of T is taken as the scene feature of T;

[0023] The average value of each image block feature of C is used as the scene feature of C; wherein each image block feature of C is obtained through the ViT network;

[0024] The similarity between the scene features of T and the scene features of image C is calculated as the similarity between T and C.

[0025] More preferably, T a is the image after brightness transformation of T.

[0026] Further preferably, the method for constructing the unsupervised semantic segmentation model further includes:

[0027] For each image T in the batch: use the ViT network to transform the enhanced image T of T a Divide into R rows and R columns of image blocks and perform feature extraction to obtain T aThe features of each image block; T a The features of each image block are processed by the segmentation head and the projection head in turn to obtain T a Projection features of each image block;

[0028] The training loss function also includes: an alignment loss function that aims to maximize the similarity between the projection features of the kth image block in the batch image and the projection features of the kth image block in its enhanced image; k = 1, 2, ..., R×R.

[0029] Further preferably, the pseudo positive and negative samples of any image block b are determined by:

[0030] Obtain the maximum value of the similarity between the image block feature F of the image block b and each reference feature in the positive sample reference pool as the corresponding positive sample threshold Th; F is obtained through the ViT network;

[0031] Calculate the similarity between F and each image block feature except F in the anchor feature set within the batch, and take the image blocks corresponding to the image block features with similarity greater than Th as pseudo positive samples of b, and take the image blocks corresponding to the image block features with similarity less than or equal to Th as pseudo negative samples of b;

[0032] The intra-batch anchor feature set includes: features of each image block of each intra-batch image;

[0033] The positive sample reference pool is obtained in the following way: each sample image in the pre-collected sample pool is divided into multiple image blocks by using the ViT network and features are extracted. After the features of each image block are obtained, an image block feature is randomly selected as the reference feature of the sample image; a positive sample reference pool including the reference features of all sample images is constructed.

[0034] Further preferably, during the training process, the positive sample reference pool is re-acquired every preset number of training iterations; after the training is completed, the positive sample reference pool is fixed.

[0035] In a second aspect, the present invention provides an image semantic segmentation method for use in an application phase, comprising:

[0036] Input the image to be segmented into the unsupervised semantic segmentation model to obtain the corresponding semantic segmentation result;

[0037] Among them, the unsupervised semantic segmentation model is constructed using the unsupervised semantic segmentation model construction method provided by the first aspect of the present invention.

[0038] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the method provided in the first aspect and / or the second aspect of the present invention when executing the computer program.

[0039] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the first aspect and / or the second aspect of the present invention is performed.

[0040] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0041] 1. The present invention provides a method for constructing an unsupervised semantic segmentation model. Based on the a priori principle that similar samples appear in groups, the mutual attention weights between two image blocks in a batch image are obtained by using the ViT network, and the weight values ​​less than the weight mean are set to 0 to characterize the degree of correlation between image blocks, and then find the potential similar features of the image features of each image block in the whole image; the segmentation head features of each image block in the batch image output by the segmentation head are aggregated based on the processed mutual attention weights between two image blocks to obtain the segmentation head aggregated features of each image block in the batch image, and further input them into the projection head to obtain the corresponding projection aggregated features; the expressive power of the segmentation head and projection head features is improved by feature aggregation, while giving full expression to the strong semantics that appear with a high probability, it also gives sufficient expression to the weak semantics that appear with a low probability, can accurately identify potential small category pixels, and then realize image semantic segmentation precisely and accurately.

[0042] 2. Furthermore, the method for constructing an unsupervised semantic segmentation model provided by the present invention takes into account that the same image has the same scene information, and the semantic similarity of image block features of the same category in the same image will be higher than the semantic similarity of the same category in other images. This difference increases the probability of misjudgment when using image block features to select pseudo-positive samples in other images. Therefore, when searching for pseudo-positive samples across images, the present invention only searches for pseudo-positive samples from images with similar scenes to the image, introduces neighboring images of images in the batch based on the prior rule that pixel samples of the same category have similar scenes, and calculates the similarity between the image block features in the neighboring images and the image block features in the batch, and in order to reduce the image block features of the same image blocks between different images The deviation of semantic similarity is improved, and the generalization of similar feature screening is improved. The similarity between the image block features of the image in the batch and the image block features of the corresponding image block of the enhanced image is calculated as the corresponding screening threshold, and the similarity between the image block features of the neighboring images that are less than the corresponding screening threshold and the image block features of the image in the batch is set to 0; the segmentation head features of each image block of the neighboring image output by the segmentation head are aggregated based on the similarity between the image block features of the processed neighboring image and the image block features of the image in the batch, and the segmentation head aggregated features of each image block of the neighboring image are obtained, and further input into the projection head to obtain the corresponding projection aggregated features, which can further improve the expressiveness of the segmentation head and projection head features, thereby further improving the quality of image semantic segmentation.

[0043] 3. Furthermore, the method for constructing an unsupervised semantic segmentation model provided by the present invention, when obtaining neighboring images of images in a batch, takes the mean of the feature vectors of each image block of the entire image in the batch as the scene feature vector of the image, and by calculating the similarity between the scene feature vectors of other images in the image set, it is possible to find multiple most similar neighboring images for each image in the batch.

[0044] 4. Furthermore, the method for constructing an unsupervised semantic segmentation model provided by the present invention constructs a loss function with the goal of maximizing the similarity between the projection features of each image block in each image in the batch and the projection features of the corresponding image block in the enhanced image, and trains the segmentation head and the projection head; through this operation, the semantic consistency between the projection features of the images in the batch and the projection features of their enhanced images is aligned, the relationship between the images in the batch and their enhanced images is shortened, the semantic quality of the features is improved, and the model performance is further improved.

[0045] 5. Furthermore, the construction method of the unsupervised semantic segmentation model provided by the present invention, since the positive sample reference pool is relatively large and the image block features in the same image have a certain correlation, in order to reduce the complexity of calculating the threshold and ensure the randomness of the sample pool selection, a random image block feature is selected from each image block feature obtained by the ViT network for each sample image as the reference feature of the sample image, and added to the positive sample reference pool. When obtaining pseudo positive and negative samples of the image block, the maximum value of the similarity between the image block feature of the image block and each reference feature in the positive sample reference pool is used as the corresponding positive sample threshold; the positive sample threshold is the threshold value that is most similar to the image block feature and a random feature, and the image block feature greater than the threshold is used as the positive sample feature with the same semantic feature as the image block feature, and the image block feature less than the threshold is used as a random negative sample feature that is unrelated to the image block feature. Based on this, the pseudo positive and negative samples of the image block can be accurately obtained.

[0046] 6. Furthermore, the method for constructing an unsupervised semantic segmentation model provided by the present invention can further improve the generalization ability of the model by re-acquiring a positive sample reference pool every preset number of training iterations during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 An overall flow chart of a method for constructing an unsupervised semantic segmentation model provided by an embodiment of the present invention;

[0048] Figure 2 An overall framework diagram of an unsupervised semantic segmentation model provided by an embodiment of the present invention

[0049] Figure 3 A schematic diagram of a process for obtaining pseudo positive and negative samples corresponding to each image block in a batch of images provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of a process for acquiring projected aggregate features of each image block in a batch of images based on mutual attention weights provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of a process for acquiring projection aggregation features of each image block of a neighboring image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] In order to achieve the above objectives, in a first aspect, the present invention provides a method for constructing an unsupervised semantic segmentation model, comprising:

[0054] For each intra-batch image T in a batch of pre-acquired images:

[0055] The ViT network is used to divide T into R rows and R columns of image blocks and perform feature extraction to obtain the features of each image block of T and the mutual attention weights between each image block; the features of each image block of T are processed by the segmentation head and the projection head in turn to obtain the projection features of each image block; R is a positive integer;

[0056] The segmentation head features of each image block of T obtained after the segmentation head form the first segmentation head feature matrix S1 of size R×R; construct the corresponding R×R weight matrix A for the kth image block of T k , A k The lth element of is the mutual attention weight between the kth image block and the lth image block of T; k After normalizing each element in, calculate the average value a k , and A k Less than a k Set all elements of A to 0; k = 1, 2, ... R × R,; l = (i-1) × R + j; i = 1, 2, ..., R; j = 1, 2, ..., R; k The segmentation head aggregate features of the kth image block of T are obtained by adding the elements after the dot multiplication operation; the segmentation head aggregate features of each image block of T are input into the projection head to obtain the projection aggregate features of each image block of T;

[0057] A training loss function is constructed, and based on the training loss function, the segmentation head and the projection head are trained at the same time; the training loss function includes: a first clustering loss function whose goal is to maximize the similarity between the projection features of the image blocks in the batch and the projection features of the pseudo positive samples of the image blocks, and minimize the similarity between the projection features of the pseudo negative samples of the image blocks, and a second clustering loss function whose goal is to maximize the similarity between the projection aggregate features of the image blocks in the batch and the projection aggregate features of the pseudo positive samples of the image blocks, and minimize the similarity between the projection aggregate features of the pseudo negative samples of the image blocks;

[0058] After training, an unsupervised semantic segmentation model consisting of a cascaded ViT network, a segmentation head, a projection head, and a classification module is constructed;

[0059] Among them, the classification module is used to calculate the similarity between the projection features of each image block of the image to be segmented and the projection features of each pseudo-positive sample in the application stage, and mark the image block as the category number carried by the pseudo-positive sample with the greatest similarity, so as to obtain the semantic segmentation result of the image to be segmented.

[0060] It should be noted that the ViT network used in the present invention is a pre-trained ViT network.

[0061] It should be noted that when performing similarity calculation, the Euclidean distance calculation formula, the cosine similarity calculation formula, the Manhattan distance calculation formula, etc. can be used, and there is no limitation here.

[0062] It should be noted that general clustering loss functions are applicable here. In addition, contrast loss functions can also be used for description, which is not limited here.

[0063] It should be noted that ViT divides the image into multiple image blocks. In the field of unsupervised semantic segmentation, each image block is called an anchor point. The anchor point feature mentioned below is the image block feature.

[0064] During the research process, the present invention discovered the prior rule that samples of the same category appear in groups in the same image. Specifically, in the field of semantic segmentation, the definition of objects is fine-grained to the pixel level or block level, and these fine-grained objects often appear in groups to form a whole object. Therefore, the anchor features of a certain category are often also objects that appear in groups to form a whole, or multiple objects appear in a picture at the same time. Therefore, for the search for pseudo-positive samples of small category samples, the most likely place to find them is in the original picture. The anchor features of the same category have a higher semantic similarity, and the mutual attention weights between the anchor features are used to judge the degree of correlation between blocks. Therefore, based on the prior principle that samples of the same category appear in groups, the present invention uses the mutual attention weights of the ViT network to find the potential similar features of each anchor feature in the entire image, and improves the expression ability of the segmentation head and projection head features by feature aggregation. While giving full expression to the strong semantics that appear with a high probability, it gives sufficient expression to the weak semantics that appear with a low probability, and can accurately identify potential small category pixels, thereby accurately and accurately realizing the technical problem of image semantic segmentation.

[0065] Furthermore, during the research process, the present invention also found that pixel samples of the same category have a priori rules of similar scenes. Since the same image has the same scene information (such as environment, lighting, color temperature, etc.), the semantic similarity of the same category anchor features in the same image will be higher than the semantic similarity of the same category in other images. This difference increases the probability of misjudgment when using anchor features to select pseudo-positive samples in other images. Therefore, when searching for pseudo-positive samples across images, pseudo-positive samples are only searched from images with similar scenes to the image. Based on the above analysis, preferably, in an optional implementation, the method for constructing the above unsupervised semantic segmentation model also includes:

[0066] For each image T in a batch:

[0067] The ViT network is used to transform the enhanced image T of T a and neighboring images T n Divide into R rows and R columns of image blocks and perform feature extraction to obtain T a and T n The image block features of n It is obtained by calculating the similarity between T and each image in the pre-collected image set, and randomly selecting an image from the top M images with the highest similarity as T. n ; M ≥ 1;

[0068] T n The features of each image block are processed by the segmentation head to obtain T n The segmentation head features of each image block are used to form a second segmentation head feature matrix S2 of size R×R;

[0069] Calculate the lth image block feature of T and T a The similarity between the l-th image block features is used as the corresponding screening threshold; l = (i-1) × R + j; i = 1, 2, ..., R; j = 1, 2, ..., R;

[0070] T n The k-th image block constructs the corresponding R×R weight matrix B k , B k The lth element of is T n The similarity between the kth image block feature of T and the lth image block feature of B k All elements in B that are smaller than the corresponding screening threshold are set to 0; k = 1, 2, ..., R × R; S2 and B k Add the elements after the dot multiplication operation to get T n The segmentation head aggregates features of the k-th image block; T n The segmentation head aggregates the features of each image block and inputs them into the projection head to obtain T nProjection aggregation features of each image patch;

[0071] The training loss function also includes: a third clustering loss function whose goal is to maximize the similarity between the projected aggregate features of the neighboring image blocks of the images in the batch and the projected aggregate features of the pseudo positive samples of the image blocks, and minimize the similarity between the projected aggregate features of the pseudo negative samples of the image blocks.

[0072] It should be noted that since the anchor features of the images within the batch and the dense features of the neighboring images are not passed through the ViT network in the same batch, it is impossible to use the mutual attention weights between the image blocks to judge the similarity between the anchor features of the original image and the anchor features of the neighboring images as in the intra-batch image aggregation. Based on this, the present embodiment uses the similarity between the anchor features to judge the semantic similarity. At the same time, in order to improve the generalization of the screening, the similarity between the anchor features of the images within the batch and the anchor features corresponding to the enhanced images is used as the corresponding similarity screening threshold in the neighboring images. When the similarity between the anchor features of a certain image block of the neighboring image and the anchor features of a certain image block of the batch image is less than the corresponding screening threshold, the similarity is set to 0, thereby constructing the similarity weight matrix of the aggregation of the segmentation head features of the neighboring images, and guiding the segmentation head features of each image block of the neighboring images to achieve aggregation.

[0073] It should be noted that there are many ways to measure the similarity between T and a certain image C in the pre-collected image set, such as directly measuring the similarity between image pixels or measuring the similarity between image features. When calculating the similarity, the Euclidean distance calculation formula, the cosine similarity calculation formula, the Manhattan distance calculation formula, etc. can be used, which are not limited here.

[0074] Preferably, in an optional implementation, the similarity between image features is measured as the similarity between images, and considering that if the features of the entire image in the batch are matched with the features of the entire image in the pre-collected image set, the computational complexity is high, therefore, the mean of the anchor point features of each image block of the entire image is used as the scene feature of the image, and the similarity between the scene features is measured as the similarity between images to reduce the computational complexity. Specifically, the similarity between T and a certain image C in the pre-collected image set is calculated in the following way:

[0075] The average value of each image block feature of T is taken as the scene feature of T;

[0076] The average value of each image block feature of C is used as the scene feature of C; wherein each image block feature of C is obtained through the ViT network;

[0077] The similarity between the scene features of T and the scene features of image C is calculated as the similarity between T and C.

[0078] It should be noted that the enhanced image T of T a There are many ways to obtain T, which can be obtained by brightness conversion, color saturation conversion, chromaticity adjustment, etc. Preferably, in an optional implementation, T a is the image after brightness transformation of T.

[0079] In an optional implementation manner, the method for constructing the unsupervised semantic segmentation model further includes:

[0080] For each image T in the batch: use the ViT network to transform the enhanced image T of T a Divide into R rows and R columns of image blocks and perform feature extraction to obtain T a The features of each image block; T a The features of each image block are processed by the segmentation head and the projection head in turn to obtain T a Projection features of each image block;

[0081] The training loss function also includes: an alignment loss function that aims to maximize the similarity between the projection features of the kth image block in the batch image and the projection features of the kth image block in its enhanced image; k = 1, 2, ..., R×R.

[0082] Through the above operations, the semantic consistency between the projected features of the images in the batch and the projected features of their enhanced images is aligned, the relationship between the images in the batch and their enhanced images is shortened, the semantic quality of the features is improved, and the model performance is further improved.

[0083] It should be noted that the pseudo positive and negative samples of any image block b in the present invention are determined in the following manner: all image blocks in the image block set whose similarity with image block b is greater than a preset positive sample threshold and except for image block b are taken as pseudo positive samples of image block b; all image blocks in the image block set whose similarity with image block b is less than or equal to a preset positive sample threshold are taken as pseudo negative samples of image block b; the image block set includes: each image block of each image in a batch. Wherein, when image block b is an image block of an image in a batch, it belongs to the image block set; when image block b is not an image block of an image in a batch, it does not belong to the image block set.

[0084] In an optional implementation, when measuring the similarity between an image block b and an image block in an image block set, the similarity between the image block features of the two is used for measurement; specifically, the pseudo positive sample of image block b is an image block corresponding to an image block feature in the intra-batch anchor feature set whose similarity with the image block feature of b is greater than a preset positive sample threshold and except for the image block feature of b, and the pseudo negative sample is an image block corresponding to an image block feature in the intra-batch anchor feature set whose similarity with the image block feature of b is less than or equal to the preset positive sample threshold; wherein the intra-batch anchor feature set includes: each image block feature of each image in the batch.

[0085] Among them, when the image block b is an image block of an image within the batch, its image block feature belongs to the intra-batch anchor feature set; when the image block b is not an image block of an image within the batch, its image block feature does not belong to the intra-batch anchor feature set.

[0086] Preferably, in another optional implementation manner, the pseudo positive and negative samples of any image block b are determined by:

[0087] Obtain the maximum value of the similarity between the image block feature F of the image block b and each reference feature in the positive sample reference pool as the corresponding positive sample threshold Th; F is obtained through the ViT network;

[0088] Calculate the similarity between F and each image block feature except F in the anchor feature set within the batch, and take the image blocks corresponding to the image block features with similarity greater than Th as pseudo positive samples of b, and take the image blocks corresponding to the image block features with similarity less than or equal to Th as pseudo negative samples of b;

[0089] The intra-batch anchor feature set includes: features of each image block of each intra-batch image;

[0090] The positive sample reference pool is obtained in the following way: each sample image in the pre-collected sample pool is divided into multiple image blocks by using the ViT network and features are extracted. After the features of each image block are obtained, an image block feature is randomly selected as the reference feature of the sample image; a positive sample reference pool including the reference features of all sample images is constructed.

[0091] When the image block b is an image block of an image within the batch, its image block features belong to the intra-batch anchor feature set; when the image block b is not an image block of an image within the batch, its image block features do not belong to the intra-batch anchor feature set.

[0092] It should be noted that, since the positive sample reference pool is relatively large and there is a certain correlation between the anchor features in the same image, in order to reduce the complexity of calculating the threshold and ensure the randomness of the sample pool selection, an anchor feature is randomly selected from each anchor feature obtained by the ViT network for each sample image as the reference feature of the sample image and added to the positive sample reference pool. The positive sample threshold can be regarded as the threshold that is most similar to an anchor feature and a random feature. Anchor features greater than this threshold are regarded as positive sample features with the same semantic features as this anchor feature, and anchor features less than this threshold are regarded as random negative sample features that are unrelated to this anchor feature.

[0093] Preferably, during the training process, the positive sample reference pool is re-acquired every preset number of training iterations; after the training is completed, the positive sample reference pool is fixed. In an optional implementation, the preset number of training iterations is 100.

[0094] In order to further illustrate the method for constructing an unsupervised semantic segmentation model provided by the present invention, a specific embodiment is described in detail below:

[0095] In this embodiment, a training set is collected in advance, a part of images are selected from the training set to form a sample pool, and a batch of images are selected from the training set, and each image in the batch of images is recorded as an image within the batch; wherein each image block of the image within the batch carries a corresponding category number. Generally, the number of images in the sample pool is much larger than the number of images in the selected batch of images.

[0096] like Figure 1 The figure shows the overall framework diagram of the present embodiment, which mainly includes the generation of global pseudo positive and negative samples, and the clustering calculation of the pseudo positive and negative samples and three features. Specifically, the framework input includes the original image (image within the batch) and the enhanced image obtained after brightness transformation. The two images share a backbone network. The backbone network includes the ViT network pre-trained by the self-supervised model DINO, as well as a segmentation head and a projection head. The ViT network divides the input images into R rows and R columns of image blocks, and extracts the features of each image block, and obtains a total of N = R × R anchor features (image block features). In this embodiment, R is 28. The anchor features of each image block output by the ViT network of the sample pool image and the image within the batch enter the positive sample generation module to obtain a global pseudo positive and negative sample set.

[0097] The mutual attention weights obtained by the ViT network for the images within the batch are subjected to feature aggregation with the output results of the segmentation head. The feature aggregation results are used to calculate the clustering loss in the clustering module together with the pseudo positive and negative sample sets. Similarly, the anchor features of each image block of the images within the batch are output by the segmentation head and the projection head, and the clustering loss is calculated in the clustering module together with the pseudo positive and negative samples. The projection features output by the backbone network of the original image and the enhanced image are used to calculate the alignment loss through the semantic alignment module. This embodiment uses unsupervised learning and clustering algorithms to perform semantic segmentation on the original image, so as to improve the classification effect of potential pseudo positive sample pixels.

[0098] like Figure 2 As shown, this embodiment includes the following parts: generation of enhanced images, selection and update of sample pools, generation of pseudo positive and negative samples, calculation of intra-batch image feature clustering loss, intra-image feature aggregation based on mutual attention weights, intra-image clustering loss calculation, generation of neighbor images, neighbor image feature aggregation, clustering loss calculation between neighbor images, calculation of semantic alignment loss between original images and enhanced images, model training, and performance evaluation on CocoStuff and Cityscapes datasets.

[0099] 1. Generation of positive sample reference pool and calculation of positive sample threshold

[0100] The ViT backbone trained with the self-supervised model DINO is used as the feature extractor. In order to find positive samples with high confidence globally, a positive sample reference pool is constructed. This reference pool will be used as the standard for the anchor features of each image block in the subsequent batch to filter out pseudo positive samples from this batch. The process of constructing the positive sample reference pool is as follows:

[0101] 2048 images are randomly selected from the training set as sample images to form a sample pool; each sample image is passed through the ViT network to extract N anchor features (i.e., dense features), where N is the number of blocks each image is divided into; N = R × R, in this embodiment, R is 28, that is, the ViT network divides the input images into image blocks of 28 rows and 28 columns. An anchor feature is randomly selected from the N anchor features as the reference feature of this image, and the reference features of all sample pool images constitute a positive sample reference pool, which is updated every 100 generations of training to ensure randomness.

[0102] For each image block obtained after the image in the batch passes through the ViT network, the cosine similarity between each anchor point feature and each reference feature in the positive sample reference pool is calculated respectively, and the maximum value of the obtained similarities is taken as the positive sample threshold Th corresponding to the image block.

[0103] 2. High reliability false positive and negative sample screening and clustering calculation within the batch

[0104] After obtaining the positive sample threshold of the anchor feature of each image block in the batch image, the cosine similarity calculation is performed on each anchor feature in the batch image and all other anchor features in the batch, and all anchor features greater than the positive sample feature threshold are used as pseudo positive samples of the anchor feature, and the remaining features are used as negative samples of the anchor feature. Specifically, the schematic diagram of the pseudo positive and negative sample acquisition process corresponding to each image block in the batch image is shown in the figure below. Figure 3 shown.

[0105] The anchor point features of each image block in the batch image are input into the segmentation head to obtain the segmentation head features, and then the segmentation head features are passed through the projection head to obtain the projection features of each image block; a first clustering loss function is constructed between the projection features of each image block in the batch image and the projection features of the corresponding pseudo positive and negative samples. The clustering loss function aims to maximize the similarity between the projection features of each image block and the projection features of each pseudo positive sample of the image block, and minimize the similarity with the projection features of each pseudo negative sample of the image block.

[0106] Among them, the first clustering loss function is defined as:

[0107]

[0108] z k represents the projection feature of the kth image block in the batch; P represents the set of pseudo positive sample indexes corresponding to the kth image block in the batch, |P| represents the total number of indexes in P; N represents the set of negative sample indexes corresponding to the kth image block in the batch; z p represents the projection feature of the pth pseudo positive sample corresponding to the kth image block in the batch; z n represents the projection feature of the nth pseudo negative sample corresponding to the kth image block in the batch; τ represents the temperature coefficient.

[0109] 3. Acquisition of aggregated features and clustering calculation in the original image

[0110] The process of obtaining the projected aggregate features of each image block in the batch based on the mutual attention weight is as follows: Figure 4 As shown in the figure. Based on the prior rule that pixel samples of the same category appear in groups in the same image, pseudo positive samples with similar semantics are mined from the images in the batch as much as possible and combined into an aggregated pseudo positive sample vector. The ViT network provides good prior knowledge. The anchor features of the same category have high semantic similarity. The mutual attention weight is used to judge the degree of correlation between blocks.

[0111] The mutual attention score weights between each pair of image blocks in the batch image are obtained after the batch image passes through the ViT network. For each image block in the batch image, a weight matrix of size R×R is constructed, and the lth element of the weight matrix is ​​the mutual attention weight between the image block of T and the lth image block; each weight in the weight matrix is ​​normalized by the Softmax layer and the average value is calculated; all elements in the weight matrix that are less than the corresponding average value are set to 0; l=(i-1)×R+j; i=1,2,…,R; j=1,2,…,R;

[0112] The segmentation head features of each image block obtained after the anchor point features of each image block in the batch image are processed by the segmentation head constitute a first segmentation head feature matrix of size R×R, which is added after point multiplication with the weight matrix of each image block in the batch image to obtain the segmentation head aggregated features of the image block in the batch image, and further input into the projection head to obtain the corresponding projection aggregated features; construct a second clustering loss function between the projection aggregated features of each image block in the batch image and the projection aggregated features of the corresponding pseudo, positive and negative samples, and the clustering loss function aims to maximize the similarity between the projection aggregated features of each image block and the projection aggregated features of each pseudo positive sample of the image block, and minimize the similarity with the projection aggregated features of each pseudo negative sample of the image block.

[0113] Among them, the second clustering loss function is defined as:

[0114]

[0115] z k ' represents the graph projection aggregation feature of the kth image block in the batch; P' represents the set of pseudo positive sample indexes corresponding to the i-th image block in the batch, |P'| represents the total number of indexes in P'; N' represents the set of negative sample indexes corresponding to the kth image block in the batch; z p ' represents the projected aggregate feature of the pth pseudo positive sample corresponding to the kth image block in the batch; z n ' represents the projected aggregate feature of the nth pseudo negative sample corresponding to the kth image block in the batch; τ represents the temperature coefficient.

[0116] 4. Acquisition of aggregated features within neighboring images and compact clustering calculation

[0117] The generation process of the projected aggregation features of each image block of the neighboring image is as follows: Figure 5 As shown in the figure. Based on the prior rule that similar samples often appear in similar scenes, we try to mine feature blocks with similar semantics to the anchor blocks in the original image in the neighboring images, and aggregate the segmentation head features of the neighboring images according to the similarity weight to obtain the segmentation head feature aggregation vector. The feature aggregation vector of the neighboring images is obtained in two stages.

[0118] In the first stage, the self-supervised ViT and KNN algorithms are used to achieve the matching of neighboring images. The anchor features of each image block obtained after the original image (in-batch image) passes through the ViT network are averaged, and the average value of the anchor features is used as the scene feature of the original image. The KNN algorithm is used to calculate the similarity between the scene features of the original image and the scene features of each image in the pre-collected image set, and the 30 images most similar to the original image are obtained, and one image is randomly selected from them as the neighboring image of the original image.

[0119] In the second stage, the original image is transformed for brightness to obtain an enhanced image of the original image. In order to reduce the deviation of semantic similarity of similar anchor features between different images and improve the generalization of similar feature screening, the cosine similarity between the anchor features of the original image and the corresponding anchor features in the enhanced image is calculated as the similarity screening threshold between the anchor features of the original image and the anchor features of the neighboring images.

[0120] For each image block of the neighboring image, a weight matrix of size R×R is constructed, and the lth element of the weight matrix is ​​the similarity between the anchor feature of the image block and the anchor feature of the lth image block of the original image; all elements in the weight matrix that are less than the corresponding screening threshold are set to 0; l=(i-1)×R+,; i=1,2,…,R;j=1,2,…,R;

[0121] The segmentation head features of each image block obtained after the anchor point features of each image block of the neighboring image are processed by the segmentation head constitute a second segmentation head feature matrix of size R×R, which is added after point multiplication with the weight matrix of each image block of the neighboring image to obtain the segmentation head aggregation features of the image block of the neighboring image, and further input into the projection head to obtain the corresponding projection aggregation features; construct a third clustering loss function between the projection aggregation features of each image block of the neighboring image and the projection aggregation features of the corresponding pseudo positive and negative samples, and the loss function aims to maximize the similarity between the projection aggregation features of each image block of the neighboring image and the projection aggregation features of each pseudo positive sample corresponding to the image block, and minimize the similarity with the projection aggregation features of each pseudo negative sample corresponding to the image block.

[0122] Among them, the third clustering loss function is defined as:

[0123]

[0124] z k ” represents the graph projection aggregation feature of the kth image block of the neighboring image; P” represents the set of pseudo positive sample indexes corresponding to the i-th image block of the neighboring image, |P”| represents the total number of indexes in P”; N” represents the set of negative sample indexes corresponding to the k-th image block of the neighboring image; z p” represents the projected aggregate feature of the pth pseudo positive sample corresponding to the kth image block of the neighboring image; z n ” represents the projected aggregate feature of the nth pseudo negative sample corresponding to the kth image block of the neighboring image; τ represents the temperature coefficient.

[0125] It should be noted that the method for obtaining the pseudo positive and negative samples of each image block of the neighboring image is the same as the method for obtaining the pseudo positive and negative samples of each image block of the batch image, which will not be described in detail here.

[0126] 5. Self-supervised semantic alignment between original and enhanced images

[0127] A loss function is constructed with the goal of maximizing the similarity between the projection features of the kth image block in the batch image and the projection features of the kth image block in its enhanced image as the alignment loss function; where k = 1, 2, ..., R × R;

[0128] Based on the loss functions obtained in the above process, the segmentation head and the projection head are trained simultaneously;

[0129] After training, an unsupervised semantic segmentation model consisting of a series of ViT networks, segmentation heads, projection heads, and classification modules is constructed;

[0130] Among them, the classification module is used to calculate the similarity between the projection features and each corresponding pseudo-positive sample for each image block of the image to be segmented in the application stage, and mark the image block as the category number carried by the pseudo-positive sample with the largest similarity, thereby obtaining the semantic segmentation result of the image to be segmented.

[0131] It should be noted that the method for obtaining the pseudo positive samples of each image block of the image to be segmented is the same as the method for obtaining the pseudo positive samples of each image block in the batch image, which will not be described in detail here.

[0132] In summary, the present invention can use some image-based prior knowledge to expand the pseudo-positive samples on the anchor features generated by the self-supervised ViT network. Under the premise of not reducing the quality of large-category pseudo-positive samples, the pseudo-positive samples of small categories are increased by clustering the large-category pseudo-positive sample features and the converged features, thereby improving the performance in identifying potential small-category pixels, and can solve the problem of limited recognition of potential pixel categories by the existing self-supervised ViT. At the same time, the present invention uses the attention scores between the anchor features in the image carried by the anchor features output by the ViT network, as well as the similarity between neighboring images, to improve the expressive power of the segmentation head and projection head features by feature aggregation, while giving sufficient expression to the strong semantics that appear with a high probability, and giving sufficient expression to the weak semantics that appear with a low probability, thereby improving the semantic segmentation accuracy of semantic areas with smaller size and lower frequency in unsupervised semantic segmentation.

[0133] In a second aspect, the present invention provides an image semantic segmentation method for use in an application phase, comprising:

[0134] Input the image to be segmented into the unsupervised semantic segmentation model to obtain the corresponding semantic segmentation result;

[0135] Among them, the unsupervised semantic segmentation model is constructed using the unsupervised semantic segmentation model construction method provided by the first aspect of the present invention.

[0136] The related technical solution is the same as the method for constructing the unsupervised semantic segmentation model provided in one aspect of the present invention, and will not be described in detail here.

[0137] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the method provided in the first aspect and / or the second aspect of the present invention when executing the computer program.

[0138] The related technical solutions are the same as the method for constructing an unsupervised semantic segmentation model provided in one aspect of the present invention and the image semantic segmentation method provided in the second aspect, which will not be described in detail here.

[0139] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the first aspect and / or the second aspect of the present invention is performed.

[0140] The related technical solutions are the same as the method for constructing an unsupervised semantic segmentation model provided in one aspect of the present invention and the image semantic segmentation method provided in the second aspect, which will not be described in detail here.

[0141] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing an unsupervised semantic segmentation model, characterized in that: include: For each intra-batch image T in a batch of pre-acquired images: The ViT network is used to divide T into R rows and R columns of image blocks and perform feature extraction to obtain the features of each image block of T and the mutual attention weights between each two image blocks; The features of each image block of T are processed by the segmentation head and the projection head in sequence to obtain the projection features of each image block; R is a positive integer; The segmentation head features of each image block of T obtained after the segmentation head are used to form a first segmentation head feature matrix S1 of size R×R; a corresponding weight matrix A of size R×R is constructed for the kth image block of T k , A k The lth element of is the mutual attention weight between the kth image block and the lth image block of T; k After normalizing the elements in, calculate the average value a k , and A k Less than a k Set all elements of A to 0; k = 1, 2, ... R × R,; l = (i-1) × R + j; i = 1, 2, ..., R; j = 1, 2, ..., R; and compare S1 and A k The segmentation head aggregate features of the kth image block of T are obtained by adding the elements after the dot multiplication operation; the segmentation head aggregate features of each image block of T are input into the projection head to obtain the projection aggregate features of each image block of T; Constructing a training loss function, and based on the training loss function, simultaneously training the segmentation head and the projection head; the training loss function includes: a first clustering loss function with the goal of maximizing the similarity between the projection features of the image blocks in the batch and the projection features of the pseudo positive samples of the image blocks, and minimizing the similarity between the projection features of the pseudo negative samples of the image blocks, and a second clustering loss function with the goal of maximizing the similarity between the projection aggregate features of the image blocks in the batch and the projection aggregate features of the pseudo positive samples of the image blocks, and minimizing the similarity between the projection aggregate features of the pseudo negative samples of the image blocks; After the training is completed, an unsupervised semantic segmentation model including the cascaded ViT network, the segmentation head, the projection head and the classification module is constructed; The classification module is used to calculate the similarity between the projection feature of each image block of the image to be segmented and the projection feature of each pseudo positive sample in the application stage, and mark the image block as the category number carried by the pseudo positive sample with the greatest similarity, so as to obtain the semantic segmentation result of the image to be segmented; The pseudo positive and negative samples of any image block b are determined in the following manner: image blocks in an image block set whose similarity with image block b is greater than a preset positive sample threshold and image blocks other than image block b are all taken as pseudo positive samples of image block b; image blocks in the image block set whose similarity with image block b is less than or equal to a preset positive sample threshold are all taken as pseudo negative samples of image block b; the image block set includes: each image block of each image in a batch.

2. The method for constructing an unsupervised semantic segmentation model according to claim 1, characterized in that: Also includes: For each image T in a batch: The ViT network is used to transform the enhanced image T of T a and neighboring images T n Divide into R rows and R columns of image blocks and perform feature extraction to obtain T a and T n The image block features of n It is obtained by calculating the similarity between T and each image in the pre-collected image set, and randomly selecting an image from the top M images with the highest similarity as T. n ; M ≥ 1; T n The features of each image block of T are processed by the segmentation head, and T n The segmentation head features of each image block are used to form a second segmentation head feature matrix S2 of size R×R; Calculate the lth image block feature of T and T a The similarity between the l-th image block features is used as the corresponding screening threshold; l = (i-1) × R + j; i = 1, 2, ..., R; j = 1, 2, ..., R; T n The k-th image block constructs the corresponding R×R weight matrix B k , B k The lth element of is T n The similarity between the kth image block feature of T and the lth image block feature of B k All elements in B that are smaller than the corresponding screening threshold are set to 0; k = 1, 2, ..., R × R; S2 and B k Add the elements after the dot multiplication operation to get T n The segmentation head aggregates features of the k-th image block; T n The segmentation head aggregated features of each image block are input into the projection head to obtain T n Projection aggregation features of each image patch; The training loss function also includes: a third clustering loss function whose goal is to maximize the similarity between the projected aggregate features of the neighboring image blocks of the images in the batch and the projected aggregate features of the pseudo positive samples of the image blocks, and minimize the similarity between the projected aggregate features of the pseudo negative samples of the image blocks.

3. The method for constructing an unsupervised semantic segmentation model according to claim 2, characterized in that: The similarity between T and an image C in the pre-collected image set is calculated as follows: The average value of each image block feature of T is taken as the scene feature of T; The average value of each image block feature of C is used as the scene feature of C; wherein each image block feature of C is obtained through the ViT network; The similarity between the scene features of T and the scene features of image C is calculated as the similarity between T and C.

4. The method for constructing an unsupervised semantic segmentation model according to claim 2, characterized in that: T a is the image after brightness transformation of T.

5. The method for constructing an unsupervised semantic segmentation model according to any one of claims 1 to 4, characterized in that: Also includes: For each image T in the batch: use the ViT network to enhance the image T of T a Divide into R rows and R columns of image blocks and perform feature extraction to obtain T a The features of each image block; T a The features of each image block are processed by the segmentation head and the projection head in turn to obtain T a Projection features of each image block; The training loss function also includes: an alignment loss function that aims to maximize the similarity between the projection features of the kth image block in the batch image and the projection features of the kth image block in its enhanced image; k=1, 2, ..., R×R.

6. The method for constructing an unsupervised semantic segmentation model according to any one of claims 1 to 4, characterized in that: The pseudo positive and negative samples of any image block b are determined in the following way: Obtaining the maximum value of the similarity between the image block feature F of the image block b and each reference feature in the positive sample reference pool as the corresponding positive sample threshold Th; F is obtained through the ViT network; Calculate the similarity between F and each image block feature except F in the anchor feature set within the batch, and take the image blocks corresponding to the image block features with similarity greater than Th as pseudo positive samples of b, and take the image blocks corresponding to the image block features with similarity less than or equal to Th as pseudo negative samples of b; The intra-batch anchor feature set includes: features of each image block of each intra-batch image; The positive sample reference pool is obtained in the following manner: each sample image in the pre-collected sample pool is divided into multiple image blocks using the ViT network and feature extraction is performed. After the features of each image block are obtained, an image block feature is randomly selected as the reference feature of the sample image; and the positive sample reference pool including the reference features of all sample images is constructed.

7. The method for constructing an unsupervised semantic segmentation model according to claim 6, characterized in that: During the training process, the positive sample reference pool is re-acquired every preset number of training iterations; After the training is completed, the positive sample reference pool is fixed.

8. A method for image semantic segmentation, characterized in that: For the application phase, including: Input the image to be segmented into the unsupervised semantic segmentation model to obtain the corresponding semantic segmentation result; Wherein, the unsupervised semantic segmentation model is constructed using the unsupervised semantic segmentation model construction method described in any one of claims 1-7.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is performed.

Images