An unsupervised image semantic segmentation method based on attention mechanism

Abstract

An unsupervised image semantic segmentation method based on an attention mechanism removes some redundant background information from an RGB image through an attention module; uses an unsupervised image semantic segmentation network to extract image semantic information, and gives the same label to pixels belonging to the same category in the image to achieve image semantic information extraction. The present invention can be used to solve the problems of waste of computing power and decreased segmentation accuracy due to the presence of a large amount of honor background information in unsupervised image semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to an unsupervised image semantic segmentation method based on an attention mechanism. Background Art

[0002] As the amount of image data has grown explosively, the workload required for image processing has increased accordingly. In order to reduce the workload as much as possible, researchers are constantly developing more automated and accurate image processing algorithms, with the goal of making the machine's sensors close to the recognition capabilities of the human eye, and making autonomous analysis and judgments based on what they "see", thus reducing people's burden.

[0003] As one of the important characteristics of the human visual system, the visual attention mechanism enables humans to quickly select a few salient objects from complex scenes to pay attention to. It is an important means for humans to process a large amount of external information with limited resources. In view of its efficient data screening ability, the introduction of the visual attention mechanism into the field of computer information processing, especially the field of image processing that requires calculation of massive data, has very important theoretical value and practical significance. The attention mechanism originated from the research of experts on human attention. Since the rapid development of deep learning, it has been widely used in many fields of deep learning and has become a core technology in natural language processing, statistical learning, image detection and other fields. The structural model based on the attention mechanism can not only record the positional relationship between information, but also measure the importance of different information features according to the weight of information. By determining whether the information features are relevant or irrelevant, dynamic weight parameters are established to strengthen key information and weaken useless information, thereby improving the efficiency of deep learning algorithms and improving some defects of traditional deep learning.

[0004] Image semantic segmentation refers to the image segmentation technology that classifies each pixel in the image according to the semantic content expressed by it. Image semantic segmentation can greatly compress the image storage volume while retaining only the semantic content of the image. It is an important application of semantic communication and one of the most important basic technologies in the field of visual intelligence. The semantic segmentation effect is related to the ability of the intelligent system to understand its application scenario, so it has great application value in important fields such as unmanned driving, robot cognition and navigation, security monitoring and drone landing systems. However, the above tasks usually require a large amount of labeled data that matches the scenario under consideration to obtain reliable performance. It is very expensive, time-consuming and error-prone to collect and annotate a large number of data sets for each new task and field. In addition, in many cases, due to various reasons, it may not be possible to obtain sufficient training data, and a large amount of data in other fields and tasks is related to the task under consideration to some extent. These considerations are particularly true for semantic segmentation, in which case the learning architecture requires a large amount of manually annotated data, which is very expensive to obtain. In order to solve the problem of difficult training data annotation, designing more flexible and scalable unsupervised semantic segmentation methods will receive more and more attention, and unsupervised semantic segmentation is the trend of future development.

[0005] Although semantic segmentation technology has achieved amazing results, due to its short development time and high complexity, there are still many problems that need to be solved if it is to be fully applied in real life, such as insufficient segmentation accuracy and low algorithm efficiency. Therefore, the research and improvement of semantic segmentation algorithms are of great significance to the development of computer vision technology. Summary of the invention

[0006] In order to overcome the problem that traditional unsupervised image semantic segmentation technology requires a lot of computing power, which causes a certain degree of computing power waste for images with a large amount of redundant background information, the present invention provides an unsupervised image semantic segmentation method based on an attention mechanism, wherein a spatial attention mechanism is used to reduce redundant background information of the image, retain key areas of the image, and an unsupervised image semantic segmentation model is used as a segmentation model to segment the image, thereby effectively improving the efficiency and accuracy of existing unsupervised image semantic segmentation.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] An unsupervised image semantic segmentation method based on an attention mechanism comprises the following steps:

[0009] S1: Get the affine transformation matrix θ. First, θ is initialized as the identity transformation matrix. The parameters of θ are continuously corrected through the loss function, and finally the desired affine transformation matrix is ​​obtained.

[0010] S2: After the RGB image U is input, the coordinate point position of the input image U corresponding to the coordinate point of the feature map V is calculated according to the affine transformation matrix obtained in the previous stage. The calculation method is as follows:

[0011]

[0012] in, represents the position of the pixel, s represents the input feature image coordinate point, t represents the output feature map coordinate point, A θ is the affine transformation matrix obtained in the S1 stage;

[0013] S3: Calculate the grayscale value of a specific pixel in the output feature map by interpolation. The calculation method is as follows:

[0014]

[0015] Where W and H represent the width and height of the input image, V i c is the position in channel c The gray value of pixel i, is the gray value of the cth channel point (n, m) on the input feature map;

[0016] S4: Use the feature extraction module to extract deep features {x n};

[0017] S5: The one-dimensional (1D) convolutional layer computes the feature response vector {r n};

[0018] S6: Feature response vector {r n}Use batch normalization function (BatchNormalization) on each axis of pixel category space to get {r′ n}, so that {r′ n} has zero mean and unit variance;

[0019] S7: Use the argmax function to select {r′ n} to determine the category label of each pixel {c n};

[0020] S8: Calculate the loss function and perform back propagation to update the parameters, where the loss function consists of feature similarity loss and spatial continuity loss, and μ represents the weight loss function that balances the two loss functions, which is defined as follows:

[0021] L=L sim ({r′ n , c n})+μLcon ({r′ n}) (3)

[0022] The feature similarity loss function is as follows:

[0023]

[0024] in,

[0025]

[0026] Where N is the total number of pixels in the input image, and the response map {r n =W c x n} is obtained by applying a linear classifier, where {W c ∈R q×p}, then the response map is normalized to {r′ n};

[0027] The spatial continuity loss function is defined as follows:

[0028]

[0029] Where r′ ξ，η Represents the response graph {r′ n}The pixel value at (ξ, η);

[0030] By applying a spatial continuity loss, excessive pixel labels due to complex patterns or textures are removed.

[0031] Furthermore, in step S1, in the two-dimensional image, the affine transformation matrix θ is a 2×3 matrix.

[0032] Furthermore, in step S2, the coordinate mapping relationship is that the target image is mapped to the input image. The reason is that the coordinate mapping needs to collect pixels from different coordinates of the original image to the target image. Each sampling needs to traverse the coordinates of the target image, and the coordinates of the collected original image are not fixed. Therefore, the coordinates of each position of the transformed output feature map can be obtained at the corresponding coordinate point on the input feature map.

[0033] Furthermore, in step S3, when or When it is greater than 1, the corresponding max() item will be 0, so only (x i ,y i ) The grayscale values ​​of the four points around it determine the grayscale of the target pixel, and when and The smaller it is, the greater the influence (i.e. the closer it is to the point (n, m)), and the greater the weight.

[0034] Furthermore, in step S8, the goal behind the feature similarity loss function is to enhance the similarity of similar features. Once the image pixels are clustered according to their features, the feature vectors within the same category should be similar to each other, while the feature vectors of different categories should be different from each other. By minimizing this loss function, the network weights are updated to facilitate the extraction of more effective features for classification.

[0035] The beneficial effects of the present invention are: using the spatial attention mechanism to reduce redundant background information of the image, retaining the key areas of the image, and using the unsupervised image semantic segmentation model as a segmentation model to segment the image, thereby effectively improving the efficiency and accuracy of the existing unsupervised image semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of an unsupervised image semantic segmentation method based on an attention mechanism of the present invention.

[0037] Figure 2 It is another flowchart of an unsupervised image semantic segmentation method based on the attention mechanism of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0039] Reference Figure 1 and Figure 2 , an unsupervised image semantic segmentation method based on attention mechanism, comprising the following steps:

[0040] S1: Get the affine transformation matrix θ. First, θ is initialized as the identity transformation matrix. The parameters of θ are continuously corrected through the loss function, and finally the desired affine transformation matrix is ​​obtained. In a two-dimensional image, the affine transformation matrix θ is a 2×3 matrix.

[0041] S2: After the RGB image U is input, the coordinate point position of the input image U corresponding to the coordinate point of the feature map V is calculated according to the affine transformation matrix obtained in the previous stage. The calculation method is as follows:

[0042]

[0043] in, represents the position of the pixel, s represents the input feature image coordinate point, t represents the output feature map coordinate point, A θis the affine transformation matrix obtained in the first stage. Through this step, the corresponding coordinate points of each position of the transformed output feature map on the input feature map are obtained;

[0044] S3: Use interpolation to calculate the grayscale value of a specific pixel in the output feature map. The calculation method is as follows:

[0045]

[0046] Where W and H represent the width and height of the input image, V i c is the position in channel c The gray value of pixel i, is the gray value of the cth channel point (n, m) on the input feature map. or When it is greater than 1, the corresponding max() item will be 0, that is, only (x i ,y i ) The grayscale values ​​of the four points around it determine the grayscale of the target pixel, and when and The smaller it is, the greater the influence (i.e. the closer it is to the point (n, m)), and the greater the weight;

[0047] S4: Use the feature extraction module to extract deep features {x n};

[0048] S5: The one-dimensional (1D) convolutional layer calculates the response vector {r n};

[0049] S6: The response vector is normalized using the batch normalization function on each axis of the category space to obtain {r′ n}, so that {r′ n} has zero mean and unit variance;

[0050] S7: By using the argmax function, select n} to obtain the category label of each pixel {c n};

[0051] S8: Calculate the loss function and perform back propagation to update the parameters. The loss function consists of feature similarity loss and spatial continuity loss. μ represents the weight loss function that balances the two loss functions. The specific definition is as follows:

[0052] L=L sim ({r′ n , c n})+μL con ({r′ n}) (3)

[0053] The feature similarity loss function is as follows:

[0054]

[0055] in,

[0056]

[0057] Where c n is the category label, which is determined by assigning the category ID to the response vector using the argmax function. N is the total number of pixels in the input image. The response map {r n =W c x n} is obtained by applying a linear classifier, where {W c ∈R q ×p}, then the response map is normalized to {r′ n}, so that {r′ n}With zero mean and unit variance, the goal behind this loss function is to enhance the similarity of similar features. Once the image pixels are clustered according to their features, the feature vectors within the same category should be similar to each other, while the feature vectors of different categories should be different from each other. By minimizing this loss function, the network weights are updated to promote the extraction of more effective features for classification;

[0058] The spatial continuity loss function is defined as follows:

[0059]

[0060] Where r′ ξ，η Represents the response graph {r′ n}The pixel value at (ξ, η);

[0061] By applying spatial continuity loss, excessive pixel labels due to other reasons such as complex patterns or textures can be removed to a certain extent.

[0062] In order to backpropagate the loss function, the relevant gradients are defined as follows:

[0063]

[0064]

[0065] Accordingly, Similar to the above formula.

[0066] The method provided in this embodiment can optimize the problem of excessive redundant background area leading to waste of computing power in traditional unsupervised image semantic segmentation methods, and realizes the process of using the spatial attention mechanism algorithm as an image de-redundancy tool and using the unsupervised image semantic segmentation algorithm to generate semantic images. This can be used to solve the problem of excessive redundant background information in the image encountered in unsupervised image semantic segmentation leading to waste of computing power and decreased accuracy of the unsupervised segmentation algorithm.

[0067] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. An unsupervised image semantic segmentation method based on attention mechanism, characterized in that: The method comprises the following steps: S1: Get the affine transformation matrix θ. First, θ is initialized as the identity transformation matrix. The parameters of θ are continuously corrected through the loss function, and finally the desired affine transformation matrix is ​​obtained. S2: After the RGB image U is input, the coordinate point position of the input image U corresponding to the coordinate point of the feature map V is calculated according to the affine transformation matrix obtained in the previous stage. The calculation method is as follows: in, represents the position of the pixel, s represents the input feature image coordinate point, t represents the output feature map coordinate point, A θ is the affine transformation obtained in stage S1; S3: Calculate the grayscale value of a specific pixel in the output feature map by interpolation. The calculation method is as follows: Where W and H represent the width and height of the input image, V i c is the position in channel c The gray value of pixel i, is the gray value of the cth channel point (n, m) on the input feature map; S4: Use the feature extraction module to extract deep features {x n }; S5: The one-dimensional convolutional layer calculates the feature response vector {r n }; S6: Feature response vector {r n }Use batch normalization function on each axis of pixel category space to get {r′ n }, so that {r′ n } has zero mean and unit variance; S7: Use the argmax function to select {r′ n } to determine the category label of each pixel {c n }; S8: Calculate the loss function and perform back propagation to update the parameters, where the loss function consists of feature similarity loss and spatial continuity loss, and μ represents the weight loss function that balances the two loss functions, which is defined as follows: L=L sim ({r′ n ,c n })+μL con ({r′ n }) (3) The feature similarity loss function is as follows: in, Where N is the total number of pixels in the input image, and the response map {r n =W c x n } is obtained by applying a linear classifier, where {W c ∈R q×p }, then the response map is normalized to {r′ n }; The spatial continuity loss function is defined as follows: Where r ′ ξ,η Represents the response graph {r′ n }The pixel value at (ξ,η); By applying a spatial continuity loss, excessive pixel labels due to complex patterns or textures are removed.

2. The unsupervised image semantic segmentation method based on the attention mechanism as claimed in claim 1, characterized in that: In step S1, in a two-dimensional image, the affine transformation matrix θ is a 2×3 matrix.

3. The unsupervised image semantic segmentation method based on the attention mechanism as described in claim 1 or 2, characterized in that: In step S2, the coordinate mapping relationship is that the target image is mapped to the input image. The reason is that the coordinate mapping needs to collect pixels from different coordinates of the original image to the target image. Each sampling needs to traverse the coordinates of the target image, and the coordinates of the collected original image are not fixed. Therefore, the coordinates of each position of the transformed output feature map can be obtained at the corresponding coordinate point on the input feature map.

4. The unsupervised image semantic segmentation method based on the attention mechanism according to claim 1 or 2, characterized in that: In step S3, when or When it is greater than 1, the corresponding max() item will be 0, so only (x i ,y i ) The grayscale values ​​of the four points around it determine the grayscale of the target pixel, and when and The smaller it is, the greater the influence, that is, the closer to the point (n, m), the greater the weight.

5. The unsupervised image semantic segmentation method based on the attention mechanism as claimed in claim 1 or 2, characterized in that: In step S8, the goal behind the feature similarity loss function is to enhance the similarity of similar features. Once the image pixels are clustered according to their features, the feature vectors within the same category should be similar to each other, while the feature vectors of different categories should be different from each other. By minimizing this loss function, the network weights are updated to promote the extraction of more effective features for classification.

Images