A small sample semantic segmentation method

Abstract

The application discloses a small sample semantic segmentation method, comprising the following steps: extracting features of support set pictures through a basic backbone feature extraction network to obtain feature layers at various levels; multiplying the feature layers of the support set pictures with support set real masks to segment objects in the support set pictures; generating basic discrete cosine transform (DCT) vectors, and expanding the generated vectors to the same size as the feature layers to obtain total frequency domain vectors; multiplying corresponding feature vectors and frequency domain vectors, and then dividing by object pixel sums in the pictures to finally obtain multiple prototypes; wherein the lowest frequency domain prototype can represent global understanding of the object, and other frequency domain prototypes can represent local cognition of the object. The small sample semantic segmentation method of the application extracts object information in pictures in the frequency domain, and describes the objects in the pictures in detail from two aspects of global and local.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a few-sample semantic segmentation method. Background Technology

[0002] In recent years, with the continuous development of deep learning, a large number of excellent semantic segmentation models have emerged. Semantic segmentation classifies all pixels in an image, ultimately forming a classification mask for the image. Semantic segmentation is trained on large labeled datasets. During testing, semantic segmentation is also performed on objects of the same category. For example, if the training set contains cats and dogs, the test set will often also contain cats and dogs. These semantic segmentation models are widely used in autonomous driving, face detection, and other fields. However, traditional semantic segmentation is a fully supervised deep learning process, requiring convergent training under a large amount of supervised data to develop a high-performance model.

[0003] Unlike traditional semantic segmentation, few-shot semantic segmentation aims to segment new classes in a query set of images using one or more labeled support set images. For example, the model is first trained on several labeled images of dogs. After training, the model learns from several labeled images of cats, and then segments a cat in another image—this is the model's testing phase. Therefore, few-shot semantic segmentation addresses the problem of extremely scarce and difficult-to-obtain data in some domains, such as medical images, images of unusual parts in industry, aerial images, and images of microorganisms.

[0004] Most few-shot semantic segmentation methods follow the prototype network approach. These methods generally employ global average pooling to abstract object information in an image into a prototype, and then use the object information extracted from the prototype to guide the segmentation of objects in other images.

[0005] Existing technologies extract information about objects in support set images using global average pooling. However, this method extracts incomplete information, often only including the main features of the object while neglecting local features. For example, given a photo of a cat, global average pooling can only segment the main body, such as the cat's body, but struggles to segment local parts like the ears, legs, and tail. Other existing technologies extract local information about objects in images through local clustering, but this method lacks a global understanding of the object. For instance, given a photo of a cat, local clustering only segments local features like the ears, legs, and tail by clustering local parts of the cat. However, this method lacks an understanding of the cat's shape, fur texture, and overall body characteristics. Summary of the Invention

[0006] In view of the aforementioned shortcomings of existing technologies, the technical problem to be solved by this invention is that existing few-shot semantic segmentation methods, following prototype networks, generally employ global average pooling to extract information about objects in support set images. This method typically only includes the main features of the objects but ignores their local features; however, other local clustering methods lack a global understanding of the objects when extracting local information. Therefore, this invention provides a few-shot semantic segmentation method that extracts object information from images in the frequency domain, providing a detailed description of objects in the image from both global and local perspectives, reducing dependence on data volume, and achieving object segmentation with limited data.

[0007] To achieve the above objectives, this invention provides a few-sample semantic segmentation method, comprising the following steps:

[0008] Feature extraction is performed on the support set images using a basic backbone feature extraction network to obtain feature layers at various levels;

[0009] Multiply the feature layer of the support set image with the support set ground mask to segment the objects in the support set image;

[0010] Generate the basic Discrete Cosine Transform (DCT) vector and expand the generated vector to the same size as the feature layer to obtain the total frequency domain vector;

[0011] Multiply the corresponding feature vector and frequency domain vector, and then divide by the sum of the object pixels in the image to obtain multiple prototypes.

[0012] The lowest frequency prototype can represent the global perception of an object, while the other frequency prototypes can represent the local perception of an object. The lowest frequency prototype originates from DCT(0,0) and can be represented by P0, while the other frequency prototypes originate from DCT components other than the 0-frequency component and can be represented by P0. i (i∈{1, 2, ..., N-1}) represents this. For a more detailed explanation, see Equations 10 and 11 below.

[0013] Furthermore, feature extraction is performed on the support set images through a basic backbone feature extraction network to obtain feature layers at various levels. Specifically, this includes extracting features from the support set images... Send to basic backbone network Obtain support set features

[0014]

[0015] Furthermore, the basic backbone feature extraction network is set to be one of any backbone network.

[0016] Furthermore, the feature layer of the support set image is multiplied with the support set ground mask to segment objects in the support set image. Specifically, this includes multiplying the obtained support set features F... s With the corresponding mask Multiplying them yields the features of the support set objects. These features are then copied K×K times (K=7 in this model), where the mask M needs to be transformed into the sum of features F. s Same size;

[0017] M′=Resize(M), (2)

[0018] F′ s =F s ·M′, (3)

[0019] F″ s =Duplicate(F′) s (4)

[0020] Where Resize(·) is the scaling function. This is the mask after scaling. `Duplicate(·)` is the copy function. This is the feature vector after copying.

[0021] Furthermore, a basic Discrete Cosine Transform (DCT) vector is generated, and the generated vector is expanded to the same size as the feature layer to obtain the total frequency domain vector. Specifically, this includes calculating the basic Discrete Cosine Transform vector. And the basic discrete cosine transform vector is extended to be F′ s The same size yields the final total frequency domain vector.

[0022] B = {B o B1, ..., B N -1}, (5)

[0023] N = K·K,

[0024] B t =Expand(DCT(u,v)), (6)

[0025] t∈{0, 1, ..., N-1},

[0026]

[0027]

[0028] in, is the t-th frequency domain component vector of the total frequency domain vector, representing the value at that frequency; Expand(·) is the channel expansion function; This is the floor function. These are the length and width values ​​after slicing according to K; u and v are the corresponding frequency sequence values;

[0029]

[0030]

[0031] Where α(u) and α(v) are the coefficients of the basic discrete cosine transform DCT(u,v), which can be calculated by Equation 7; i and j represent the spatial positions of the frequency domain subvectors, corresponding to the spatial positions of the feature map.

[0032] Furthermore, the basic discrete cosine transform vector can be set to one or more.

[0033] Furthermore, the corresponding feature vector and frequency domain vector are multiplied together, and then divided by the sum of the object pixels in the image, ultimately yielding multiple prototypes. Specifically, this includes multiplying the aforementioned support set object features F″. s Multiply the total frequency domain vector B by the dot product, then divide by the object value in the mask to obtain multiple frequency domain prototypes.

[0034] P = {P0, P1, ..., P} N-1}, (10)

[0035]

[0036] in, From the corresponding frequency domain vector B t The generated prototype.

[0037] Furthermore, the generated frequency domain prototype can be single or multiple.

[0038] Technical effect

[0039] This invention provides a few-sample semantic segmentation method that avoids the loss of object information and achieves a comprehensive understanding of objects in images. It utilizes DCT frequency components to extract global features from low-frequency components and local detail features from high-frequency components, thus mining global features of objects from a global perspective and local features from a local perspective. This reduces the dependence on data volume, enabling object segmentation with limited data. Furthermore, it makes the semantic segmentation model more intelligent, allowing it to segment objects of a completely new category in another image after learning from several images of new object categories.

[0040] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a preferred embodiment of the present invention for a small-sample semantic segmentation method;

[0042] Figure 2 This is a schematic diagram of the segmentation result of a few-sample semantic segmentation method according to a preferred embodiment of the present invention;

[0043] Figure 3 This is a general framework model diagram of a few-sample semantic segmentation method according to a preferred embodiment of the present invention. Detailed Implementation

[0044] To make the technical problems, solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] In the following description, specific details, such as particular internal procedures and techniques, are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will appreciate that the invention may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of the invention with unnecessary detail.

[0046] like Figure 1 and 3 As shown, this embodiment of the invention provides a few-sample semantic segmentation method, including the following steps:

[0047] Feature extraction is performed on the support set images using a basic backbone feature extraction network to obtain feature layers at various levels;

[0048] Multiply the feature layer of the support set image with the support set ground mask to segment the objects in the support set image;

[0049] Generate the basic Discrete Cosine Transform (DCT) vector and expand the generated vector to the same size as the feature layer to obtain the total frequency domain vector;

[0050] Multiply the corresponding feature vector and frequency domain vector, and then divide by the sum of the object pixels in the image to obtain multiple prototypes.

[0051] The lowest frequency prototype can represent the global perception of an object, while the other frequency prototypes can represent the local perception of an object. The lowest frequency prototype originates from DCT(0,0) and can be represented by P0, while the other frequency prototypes originate from DCT components other than the 0-frequency component and can be represented by P0. i (i∈{1, 2, ..., N-1}) represents this. For a more detailed explanation, see Formulas 10 and 11 above.

[0052] Assuming the final frequency domain prototype is generated, the above steps are not sequential. For example, the object feature extraction in step two and the generation of the basic Discrete Cosine Transform (DCT) vector in step three can be performed in parallel, without any order. Given the final frequency domain prototype, these steps are not sequential and can be performed in parallel, improving computational efficiency when resources allow.

[0053] Specifically, feature extraction is performed on the support set images using a basic backbone feature extraction network to obtain feature layers at various levels. This includes extracting features from the support set images. Send to basic backbone network Obtain support set features

[0054]

[0055] In this embodiment, the basic backbone feature extraction network is set to any backbone network such as ResNet or VGG.

[0056] The feature layer of the support set image is multiplied with the ground truth mask of the support set to segment objects in the support set image. Specifically, this includes multiplying the obtained support set features F... s With the corresponding mask Multiplying them yields the features of the support set objects. These features are then copied K×K times (K=7 in this model), where the mask M needs to be transformed into the sum of features F. s Same size;

[0057] M′=Resize(M), (2)

[0058] F′ s =F s ·M′, (3)

[0059] F″ s =Duplicate(F′) s (4)

[0060] Where Resize(·) is the scaling function. This is the mask after scaling. `Duplicate(·)` is the copy function. This is the feature vector after copying.

[0061] Generate the basic Discrete Cosine Transform (DCT) vector and expand the generated vector to the same size as the feature layer to obtain the total frequency domain vector. Specifically, this includes calculating the basic Discrete Cosine Transform vector. The basic discrete cosine transform vector is then expanded to the same size as Fs′ to obtain the final total frequency domain vector.

[0062] B = {B0, B1, ..., B} N -1}, (5)

[0063] N = K·K,

[0064] B t =Expand(DCT(u,v)), (6)

[0065] t∈{0, 1, ..., N-1},

[0066]

[0067]

[0068] in, is the t-th frequency domain component vector of the total frequency domain vector, representing the value at that frequency; Expand(·) is the channel expansion function; This is the floor function. These are the length and width values ​​after slicing according to K; u and v are the corresponding frequency sequence values;

[0069]

[0070]

[0071] Where α(u) and α(v) are the coefficients of the basic discrete cosine transform DCT(u,v), which can be calculated by Equation 7; i and j represent the spatial positions of the frequency domain subvectors, corresponding to the spatial positions of the feature map.

[0072] The basic discrete cosine transform vector can be set to one or more.

[0073] Multiply the corresponding feature vector and frequency domain vector, then divide by the sum of object pixels in the image to obtain multiple prototypes. Specifically, this includes combining the aforementioned support set object features F″. s Multiply the total frequency domain vector B by the dot product, then divide by the object value in the mask to obtain multiple frequency domain prototypes.

[0074] P = {P0, P1, ..., P} N-1}, (10)

[0075]

[0076] in, From the corresponding frequency domain vector B t The generated prototype.

[0077] Formula 10 contains the prototypes generated at all frequencies, where the prototype generated at the lowest frequency can be represented by P0. The prototype subscripts 0 to N-1 represent increases in frequency, respectively. The values ​​of u and v above represent the frequencies from 0 to N-1. and N is K*K. For example, P0 represents the prototype generated in the case of DCT(0,0), u=0, v=0. P1 represents the prototype generated in the case of DCT(0,0), u=0, v=0. u = 0, The prototypes generated under these circumstances, and so on.

[0078] The generated frequency domain prototypes can be single or multiple. Due to varying real-world scenarios, information extraction from objects can be performed using different frequency domain sizes. Therefore, this method can generate single or multiple frequency domain prototypes. When there is only one frequency domain prototype, the method degenerates into a global average pooling method. In this experiment, 49 prototypes were obtained, corresponding to a 7x7 frequency domain size. The size of the frequency domain scale affects the number of frequency domain prototypes generated. Generally, a larger frequency domain scale results in more frequency domain prototypes, providing a richer description of the objects in the image, which is more beneficial for handling complex and diverse objects. For simple objects, such as a human hand, which typically only has five fingers and smooth skin texture, single or several frequency domain prototypes can be used. In summary, the generated frequency domain prototypes can be single or multiple, and the frequency domain size should be chosen based on the object being processed. Figure 3 The diagram shows the overall framework model of a few-shot semantic segmentation method. The FPGM module in the diagram represents the method of this invention. The overall framework model diagram further illustrates that this invention is a plug-and-play module that can help few-shot semantic segmentation models capture object information more comprehensively.

[0079] like Figure 2 As shown, we use Figure 2 Taking the airplane in the first row as an example, this method will be explained in detail. Figure 3As shown, we first feed the aircraft images from the support set and query set into a pre-trained feature extractor to obtain the corresponding support set and query set features. Next, we feed the intermediate-level support set features into our proposed method (FPGM) to generate the complete frequency domain prototype, and then feed it along with the high-level and intermediate-level features into PAMM to obtain the corresponding attention map, query set pseudomask, and query set pseudomask index. Then, we feed the aforementioned query set pseudomask, query set pseudomask index, and intermediate-level support set features into FPSM to obtain the final, more effective prototype. Finally, we feed these more effective prototypes, query set features, and attention maps into FEM to obtain the final segmentation result. We can see that the aircraft segmentation result for the first query set is very close to the actual result, completely segmenting the tail, wings, and engine nozzles of the aircraft, rather than just the main body (fuselage), greatly improving the segmentation quality and completeness.

[0080] Table 1 compares the overall framework model of this invention with other advanced models in PASCAL-5. i Comparison of experimental results on the database. Table 1 shows that the overall model constructed using this method reaches the current state-of-the-art level and outperforms other methods in the evaluation metric mIoU (mean intersection-over-union ratio). Specifically, based on the VGG-16 backbone network, this model outperforms other models by 1.5% and 2.3% in 1-shot and 5-shot settings, respectively. Based on the ResNet-50 backbone network, this model is on par with other state-of-the-art methods in both 1-shot and 5-shot settings, respectively. Based on the ResNet-101 backbone network, this model outperforms other models by 1.3% and 1.3% in both 1-shot and 5-shot settings, respectively.

[0081]

[0082]

[0083] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A few-sample semantic segmentation method, characterized in that, Includes the following steps: Feature extraction is performed on the support set images using a basic backbone feature extraction network to obtain feature layers at various levels. Specifically, this includes extracting features from the support set images... Send to basic backbone network , obtain support set features , ； The basic backbone feature extraction network is set to one of any backbone network; The feature layer of the support set image is multiplied with the ground truth mask of the support set to segment objects in the support set image. Specifically, this includes multiplying the obtained support set features... Mask after scaling Multiplying them yields the features of the support set objects. And copy these features In this model The mask It needs to be transformed and features first Same size; ， in For scaling transformation function, This is the mask after scaling. For copy function, The feature vector after copying; Generate the basic Discrete Cosine Transform (DCT) vector and expand the generated vector to the same size as the feature layer to obtain the total frequency domain vector; Multiply the corresponding feature vector and frequency domain vector, and then divide by the sum of the object pixels in the image to obtain multiple prototypes. The lowest frequency prototype represents the global understanding of an object, while other frequency prototypes can represent the local understanding of an object.

2. The few-sample semantic segmentation method as described in claim 1, characterized in that, Generate the basic Discrete Cosine Transform (DCT) vector and expand the generated vector to the same size as the feature layer to obtain the total frequency domain vector. Specifically, this includes calculating the basic Discrete Cosine Transform vector. And the basic discrete cosine transform vector is extended to be and The same size yields the final total frequency domain vector. ; ， in, The first frequency domain vector is the first frequency domain vector. Each frequency domain vector represents the value at that frequency; This is the channel expansion function; This is the floor function; According to The length and width values ​​after slicing; For the corresponding frequency sequence values; ， in, and Based on Discrete Cosine Transform The coefficient is calculated using formula 7; and It is represented as the spatial location of the frequency domain vector, which corresponds to the spatial location of the feature map.

3. The few-sample semantic segmentation method as described in claim 2, characterized in that, The basic discrete cosine transform vector can be set to one or more.

4. The few-sample semantic segmentation method as described in claim 2, characterized in that, Multiply the corresponding feature vector and frequency domain vector, then divide by the sum of object pixels in the image to obtain multiple prototypes. Specifically, this includes combining the aforementioned support set object features. Sum of total frequency domain vectors Dot product, then divide by the object value in the mask to obtain multiple frequency domain prototypes. .

5. The few-sample semantic segmentation method as described in claim 1, characterized in that, The generated frequency domain prototypes can be single or multiple.

Images