Weakly supervised semantic segmentation method and apparatus based on attention mask

Abstract

The present disclosure relates to a weakly supervised semantic segmentation method and apparatus based on an attention mask. The method comprises: inputting a sample image into an attention encoder, so as to obtain a global class token feature, an image classification result and two semantic segmentation results; then inputting the sample image into the attention encoder again, so as to generate an attention matrix and to randomly generate a target mask matrix; compensating for the attention matrix by means of the target mask matrix, so as to obtain a compensated attention matrix; on the basis of the compensated attention matrix, generating each local class token feature; and distinguishing positive and negative characteristics of the local class token features. Model losses not only include losses in terms of image classification and image semantic segmentation, but also comprises losses caused by distinguishing positive and negative characteristics of local class token features and performing comparative learning together with global class token features, and semantic segmentation of a model is supervised by means of introducing a plurality of losses, thereby improving the accuracy of semantic segmentation.

Description

A weakly supervised semantic segmentation method and device based on attention mask Technical Field

[0001] The present disclosure relates to the field of semantic segmentation technology, and in particular to a weakly supervised semantic segmentation method and device based on attention mask. Background Art

[0002] Current image semantic segmentation methods rely on fully supervised training with pixel-level annotations. This requires densely labeling every pixel in an image, significantly increasing both manpower and time costs. Weakly supervised semantic segmentation, on the other hand, only requires training data labeled with the object categories in the image. This advantage in both manpower and time costs has led to widespread interest in weakly supervised semantic segmentation.

[0003] Weakly Supervised Semantic Segmentation (WSSS) uses bounding box annotations, scribble annotations, point annotations, or image-level category annotations to predict the category label for each pixel in an image. This paper focuses on a class of methods that trains a neural network based on image-level labels (e.g., the categories of objects appearing in an image) to implement semantic segmentation models.

[0004] Most weakly supervised semantic segmentation methods based on image-level classification annotations are based on Class Activation Maps (CAMs). CAMs are a technique based on deep classification networks that generate feature maps with the same number of channels as the total number of classes, showing the approximate location of objects in each class.

[0005] Currently, how to improve the accuracy of weakly supervised semantic segmentation is an urgent problem to be solved.

[0006] Summary of the Invention

[0007] The present invention provides a method and apparatus for weakly supervised semantic segmentation based on attention mask.

[0008] According to a first aspect of an embodiment of the present disclosure, a weakly supervised semantic segmentation method based on attention mask is provided, comprising:

[0009] Obtaining a sample image and an image classification label corresponding to the sample image;

[0010] Input the sample image into the attention encoder;

[0011] Obtaining patch token features and global category token features through the attention encoder, generating an image classification result and a class activation map for the sample image through the patch token features, obtaining a first semantic segmentation result for the sample image through the class activation map, and decoding the patch token features to obtain a second semantic segmentation result;

[0012] generating an attention matrix by the attention encoder, randomly generating a target mask matrix, compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix, and generating a local category token feature according to the compensated attention matrix;

[0013] Determine, according to the auxiliary class activation map and the target mask matrix, an average activation value of some activation values ​​in the class activation map that are not affected by the mask in the target mask matrix; if the average activation value is greater than a preset threshold, use the local class token feature as a positive local class token feature; if the average activation value is less than or equal to the preset threshold, use the local class token feature as a negative local class token feature;

[0014] The attention encoder is trained with the optimization goal of minimizing the overall loss, where the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

[0015] Optionally, inputting the sample image into an attention encoder specifically includes:

[0016] Segmenting the sample image into a plurality of sub-images to determine a plurality of initial patch token features corresponding to the plurality of sub-images, and determining an initial category token feature corresponding to the sample image;

[0017] The multiple initial patch token features are concatenated with the initial category token features and input into the attention encoder.

[0018] Optionally, the randomly generating a target mask matrix specifically includes:

[0019] determining a size of the attention matrix;

[0020] Downsampling the size by a preset multiple to obtain a sampling size, and randomly generating an initial mask matrix having a size equal to the sampling size;

[0021] The initial mask matrix is ​​upsampled by the preset multiple to obtain the target mask matrix.

[0022] Optionally, the attention matrix includes: a query matrix, a key matrix, and a value matrix;

[0023] The compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix specifically includes:

[0024] The compensated attention matrix is ​​determined by the following formula: Z (l) =A′ (l) V (l)

[0025] Among them, A' is the compensated attention matrix, M is the target mask matrix, Q is the query matrix, K is the key matrix, V is the value matrix, Z is the output matrix, and l is the lth self-attention encoding layer of the attention encoder.

[0026] Optionally, the attention encoder includes a plurality of self-attention encoding layers; and the method further includes:

[0027] Obtain an output result of a target self-attention encoding layer among the multiple self-attention encoding layers.

[0028] Optionally, the output of the target self-attention encoding layer includes intermediate patch token features, and the method further comprises:

[0029] determining an auxiliary classification result based on the intermediate patch token features;

[0030] The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss also includes: the difference between the auxiliary classification result and the image classification label.

[0031] Optionally, the output result of the target self-attention encoding layer includes the auxiliary class activation map, and the method further includes:

[0032] Determine a first affinity matrix based on the foreground-background segmentation result determined by the auxiliary class activation map;

[0033] determining a second affinity matrix based on the patch token features;

[0034] The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss further includes: a difference between the first affinity matrix and the second affinity matrix.

[0035] Optionally, generating an attention matrix by the attention encoder, randomly generating a target mask matrix, compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix, and generating local category token features according to the compensated attention matrix, specifically includes:

[0036] A plurality of target mask matrices are randomly generated. For each target mask matrix, the sample image is input into the attention encoder to generate an attention matrix corresponding to the sample image. The generated attention matrix is ​​compensated by the target mask matrix to obtain a corresponding compensated attention matrix. Based on the corresponding compensated attention matrix, a local category token feature corresponding to the target mask matrix is ​​generated; and the plurality of local category token features are used as the local category token features.

[0037] Optionally, the method further includes:

[0038] Use the trained attention encoder to perform image semantic segmentation on the image to be recognized.

[0039] According to a second aspect of an embodiment of the present disclosure, a weakly supervised semantic segmentation apparatus based on attention mask is provided, comprising:

[0040] An acquisition module, configured to acquire a sample image and an image classification label corresponding to the sample image;

[0041] An input module is configured to input the sample image into an attention encoder, obtain patch token features and global category token features through the attention encoder, generate an image classification result and a class activation map for the sample image through the patch token features, obtain a first semantic segmentation result for the sample image through the class activation map, and decode the patch token features to obtain a second semantic segmentation result;

[0042] a mask compensation module, configured to generate an attention matrix using the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix using the target mask matrix to obtain a compensated attention matrix, and generate local category token features based on the compensated attention matrix;

[0043] a judgment module, configured to determine, based on the auxiliary class activation map and the target mask matrix, an average activation value of some activation values ​​in the class activation map that are not affected by the mask in the target mask matrix; if the average activation value is greater than a preset threshold, the local class token feature is used as a positive local class token feature; if the average activation value is less than or equal to the preset threshold, the local class token feature is used as a negative local class token feature;

[0044] A training module is used to train the attention encoder with the optimization goal of minimizing the overall loss, where the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

[0045] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned weakly supervised semantic segmentation method based on attention mask.

[0046] According to the fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the processor implements the above-mentioned weakly supervised semantic segmentation method based on attention mask when executing the computer program.

[0047] At least one of the above technical solutions adopted in the present disclosure can achieve the following beneficial effects:

[0048] As can be seen from the above-mentioned weakly supervised semantic segmentation method based on attention mask, this method can obtain a sample image and the image classification label corresponding to the sample image, input the sample image into the attention encoder, obtain the patch token feature and the global class token feature through the attention encoder, and generate the image classification result and class activation map of the sample image through the patch token feature. The first semantic segmentation result of the sample image is obtained through the class activation map, and the patch token feature is decoded to obtain the second semantic segmentation result. The attention encoder generates an attention matrix, randomly generates a target mask matrix, and compensates the attention matrix with the target mask matrix to obtain a compensated attention matrix. Based on the compensated attention matrix, a local class token feature is generated. Based on the auxiliary class activation map and the target mask matrix, the average activation value of the activation values ​​in the class activation map that are not affected by the mask in the target mask matrix is ​​determined. If the average activation value is greater than a preset threshold, the local class token feature is regarded as a positive local class token feature. If the average activation value is less than or equal to the preset threshold, the local class token feature is regarded as a negative local class token feature. The attention encoder is trained with the optimization goal of minimizing the overall loss. The overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature, so as to perform image semantic segmentation on the image to be recognized through the trained attention encoder.

[0049] From the above content, it can be seen that this method can be divided into two extraction methods when extracting features through the attention encoder. The first method does not use a mask mechanism and directly obtains the image classification result, as well as two semantic segmentation results obtained by the class activation map and the patch token feature respectively. The second method uses a mask mechanism to supplement the attention mechanism through the mask matrix to obtain local category token features. Then, the idea of ​​contrastive learning can be combined to distinguish the positive and negative local category token features, so as to participate in model training. Therefore, model training not only includes classification loss and segmentation loss, but also includes contrastive learning loss related to local category token features. By introducing multiple losses, the loss of image semantic segmentation can be supervised more accurately, thereby improving the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0051] FIG1 is a flow chart of a weakly supervised semantic segmentation method based on attention mask provided by an embodiment of the present disclosure;

[0052] FIG2 is a schematic diagram of a masking strategy provided by an embodiment of the present disclosure;

[0053] FIG3 is a schematic diagram of generating a target mask matrix according to an embodiment of the present disclosure;

[0054] FIG4 is a schematic diagram of a complete process of model training provided by an embodiment of the present disclosure;

[0055] FIG5 is a schematic diagram of a weakly supervised semantic segmentation device based on attention mask provided by an embodiment of the present disclosure;

[0056] FIG6 is a schematic diagram of an electronic device corresponding to FIG1 provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0058] The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0059] The present disclosure provides a training method for a weakly supervised semantic segmentation model based on an attention mask. The training method for the model can be executed by an electronic device such as a server. The present disclosure uses the server executing the training method for the model as an example to illustrate.

[0060] Figure 1 is a flow chart of a weakly supervised semantic segmentation method based on attention mask provided by an embodiment of the present disclosure, which specifically includes the following steps.

[0061] S100: Obtain a sample image and an image classification label corresponding to the sample image.

[0062] S102: Input the sample image into the attention encoder.

[0063] S104: Obtain patch token features and global category token features through the attention encoder, and generate an image classification result and a class activation map for the sample image through the patch token features, obtain a first semantic segmentation result for the sample image through the class activation map, and decode the patch token features to obtain a second semantic segmentation result.

[0064] S106: Generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate each local category token feature according to the compensated attention matrix.

[0065] In weakly supervised semantic segmentation, the label corresponding to the training sample refers to the category of the target object contained in the image, rather than marking the target object category of each pixel in the image as in conventional semantic segmentation.

[0066] Based on this, the server can obtain a sample image and an image classification label corresponding to the sample image, where the image classification label is used to indicate a category corresponding to an object in the image.

[0067] The sample image can be input into an attention encoder, which obtains patch token features and global class token features through the attention encoder. The patch token features are used to generate an image classification result and a class activation map for the sample image. The class activation map is used to obtain a first semantic segmentation result for the sample image, and the patch token features are decoded to obtain a second semantic segmentation result. The attention encoder can be the attention encoder in the Transformer model, which uses a self-attention mechanism and includes multiple self-attention layers.

[0068] In addition, the sample image can be input into the attention encoder to generate an attention matrix and a randomly generated target mask matrix, and the attention matrix is ​​compensated by the target mask matrix to obtain the compensated attention matrix, and the local category token feature is generated according to the compensated attention matrix.

[0069] The difference between the two processes mentioned above is that one is to input the sample image into the attention encoder without using the target mask matrix (which can be called the first process), and the other is to input the sample image into the attention encoder and use the target mask matrix (which can be called the second process). The role of the target mask matrix is ​​to use the mask in the attention mechanism to focus on extracting local information in the image. This is because the attention mechanism in the Transformer model has the sparse characteristic. The attention encoder will extract the associations between each feature, but this extraction may extract unnecessary associations. Therefore, this method can cover some of the associations through the mask (target mask matrix) and obtain the final category token features without considering these associations.

[0070] There are different ways to input the sample image into the attention encoder. For example, the sample image can be input in stages, such as first inputting the sample image into the attention encoder that does not use the target mask matrix, and then inputting the sample image into the attention encoder that uses the target mask matrix. For another example, different branches of the attention encoder can be set, one branch is the first sub-encoder that does not include the target mask matrix, and the other branch is the second sub-encoder that includes the target mask matrix. The sample image can be input into the first sub-encoder and the second sub-encoder at the same time.

[0071] It should be noted that in some examples, a target mask matrix can be generated once, and the sample image can be input into the attention encoder. The attention matrix can be compensated by the target mask matrix generated this time, thereby obtaining a local category token feature.

[0072] In some examples, the target mask matrix can be randomly generated multiple times, each time with a different target mask matrix. The sample image can be input into the attention encoder multiple times, and each time the attention matrix is ​​compensated by a target mask matrix, so that multiple local category token features can be obtained. For example, multiple target mask matrices can be randomly generated, and the sample image can be input into the attention encoder to obtain an attention matrix. For each target mask matrix, the attention matrix is ​​compensated by the target mask matrix to obtain the corresponding compensated attention matrix. Based on the compensated attention matrix, the local category token features corresponding to the target mask matrix are generated. The local category token features finally extracted contain multiple local category tokens.

[0073] The above two processes are described in detail below.

[0074] It should be noted that the sample image can be divided into several sub-images to determine multiple initial patch token features corresponding to the several sub-images, as well as to determine the initial category token features corresponding to the sample image (the initial category token features can be randomly initialized), and the multiple initial patch token features and the initial category token features are input into the attention encoder (the above two processes are the same when the sample image is input into the attention encoder). In some examples, the multiple initial patch token features and the initial category token features are input into the attention encoder by concatenating multiple initial patch token features of dimension N*D and initial category token features of dimension 1*D along the first dimension to obtain a token of dimension (N+1)*D, which is then input into the attention encoder, where D represents the dimension of the token embedding.

[0075] The attention encoder can perform attention weighting on the initial patch token features and the initial category token features through the attention mechanism, thereby obtaining weighted patch token features and category token features. The patch token features and global category token features obtained in the first process of the above two processes are weighted. In the second process, there is no need to finally obtain the patch token features (the patch token features exist in the intermediate process). Therefore, only the features of each local category token are obtained, and the features of each local category token are weighted.

[0076] For example, For the input image, the visual Transformer encoder can split I into N = W′×H′ non-overlapping patches, which can be represented as (i.e., sub-image), where P is the patch size, H is the height of the input image, and W is the width of the input image. The block (patch) I' is then tiled and projected to obtain N patch tokens (i.e., Patch Token), which can be represented as Where D represents the dimension of the token embedding. The token embedding is concatenated with the associated learnable class token (i.e., class token) and fed into the standard Transformer encoder. In some examples, multiple patch tokens are concatenated with a single learnable class token. In addition, a positional embedding (i.e., positional embedding) is added on top of the patch token. The patch token mentioned here is the initial patch token feature, and the learnable class token is the initial class token feature.

[0077] The attention encoder can contain multiple self-attention encoding layers, i.e., Transformer layers.

[0078] Specifically, in the Transformer layer l (in the Transformer model, layer l can be a self-attention encoding layer), a linear learnable transformation is first applied to map the token sequence to the query matrix Bond Matrix Sum Matrix Here, the token sequence refers to the sequence mentioned above after adding the position embedding on top of the patch token and concatenating the patch token with the category token. In some examples, this token sequence can be mapped through a simple linear layer.

[0079] Here, D k represents the dimensions of Q and K, and D v is the dimension of V. This is done by computing the product of the query matrix and the key matrix and then dividing by To implement the self-attention mechanism. This process obtains a continuous attention matrix Contains the global relationship between pairs in each layer. Output matrix Z (l) As V (l) The weighted mixture is generated, and the weights come from the attention matrix after the softmax operation (normalization operation) The above relationship can be expressed by the following formulas (1) and (2): Z (l) =A′ (l) V (l) (2)

[0080] Assume that the patch token output sequence of each Transformer layer is reshaped into a feature map Where D is the feature dimension and H′×W′ is the spatial dimension. Then applying pooling and convolution layers can generate image classification results (classification logits). At the same time, through the feature map and the parameters of the corresponding classifier The matrix multiplication between them can generate the class activation map of the lth layer It can be expressed by the following formula (3): (l) =W (l)` Z (l) (3)

[0081] It should be noted that the above process is the attention weighting process of one layer in the attention encoder.

[0082] The class activation map obtained from the last layer of the Transformer encoder is passed through a pixel refinement module to generate a pseudo segmentation label (the first semantic segmentation result). In some examples, if the Transformer encoder has a total of l layers, then the class activation map obtained here is the class activation map output by the lth layer of the Transformer encoder. The pixel refinement module can improve the class activation map based on the pixel information in the input image and the spatial information of the neighborhood, thereby ensuring that adjacent pixels with similar image appearance have the same semantics. The patch tokens output from the last layer of the Transformer encoder are reshaped into a feature map and passed into the decoder to generate a predicted segmentation label (the second semantic segmentation result). The decoder can be composed of two 3×3 convolutional layers (with an expansion rate of 5) and a 1×1 linear layer.

[0083] The above process does not add a mask to the attention mechanism. The process of adding a mask to the attention mechanism is described below, as shown in Figures 2 and 3. Figure 2 is a schematic diagram of a masking strategy provided by an embodiment of the present disclosure. Figure 3 is a schematic diagram of generating a target mask matrix provided by an embodiment of the present disclosure.

[0084] In the Transformer layer l, an attention mask M is introduced (l)∈{0,1} N×N (for the target mask matrix of layer l) as the switch of the attention matrix, thus obtaining a regularized attention matrix Output matrix Z (l) As V (l) The weighted mixture is generated, and the weights come from the regularized attention matrix after the softmax operation (normalization operation). The output matrix Z at this time (l) Including local category tokens. The above association can be expressed by the following formulas (4) and (5):

[0085] The details of the masking strategy are shown in Figures 2 and 3. Specifically, masking is achieved by randomly dropping “columns” in the attention matrix. Dropping “columns” in the attention matrix can also be said to drop keys, where the key here refers to the key matrix Note that the keys to be masked are independently extracted from the Bernoulli distribution, and the mask ratio p∈(0,1), which means that the sampling of the elements in the key follows the Bernoulli distribution. The Bernoulli distribution here is the binomial distribution in mathematics. The mask can be obtained by independently extracting the keys to be masked at different downsampling resolutions, and then upsampling by the same multiple. Downsampling is also called decimation. For a sample value sequence, a few sample values ​​are sampled once, so that the new sequence obtained is a downsampling of the original sequence. Upsampling is the inverse process of downsampling, also known as upsampling or interpolation. In this embodiment, the size of the attention matrix can be determined first, and then downsampled by a preset multiple according to the size to obtain the sampling size, and an initial mask matrix of the same size as the sampling size can be randomly generated, and then the initial mask matrix is ​​upsampled by a preset multiple to obtain the target mask matrix. This strategy can form a continuous area for the keys not covered by the mask at different resolutions. The mask generated at this time is in the token dimension, and then expanded to the shape of the attention matrix to form M∈{0,1} N×N .

[0086] S108: Determine, based on the auxiliary class activation map and the target mask matrix, the average activation value of some activation values ​​in the class activation map that are not affected by the mask in the target mask matrix; if the average activation value is greater than a preset threshold, use the local class token feature as a positive local class token feature; if the average activation value is less than or equal to the preset threshold, use the local class token feature as a negative local class token feature.

[0087] S110: The attention encoder is trained with minimizing the overall loss as the optimization goal, where the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

[0088] In some examples, the overall loss is the sum of the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

[0089] In some examples, the trained attention encoder can further perform image semantic segmentation on the image to be recognized.

[0090] After determining the local category token features, training can be performed using the local category token features and the global category token features in a manner similar to contrastive learning, that is, the loss for training the attention encoder includes the loss associated with the local category token features.

[0091] Specifically, the average activation value of the activation values ​​of the part of the class activation map that is not affected by the mask in the target mask matrix can be determined based on the auxiliary class activation map and the target mask matrix. If the average activation value is greater than a preset threshold, the local class token feature can be used as a positive local class token feature. Otherwise, the local class token feature can be used as a negative local class token feature. In some examples, the mask generated by the auxiliary class activation map and the mask generation strategy is used to calculate the average activation value of the area of ​​the obtained class activation map that is not masked by the mask, and to determine the positive / negative label. The purpose of the attention mask is to cut off the connection between the areas generated by the tokens masked by the mask when performing the attention operation.

[0092] Based on this, the loss associated with the local category token feature can be: the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature. Among them, the smaller the difference between the positive local category token feature and the global category token feature, the better, while the larger the difference between the negative local category token feature and the global category token feature, the better.

[0093] The following first introduces other losses for model training, and then explains how to determine the positive local category token features and the negative local category token features. The complete process of model training provided by this method can be shown in Figure 4. Figure 4 is a schematic diagram of the complete process of model training provided by an embodiment of the present disclosure.

[0094] As can be seen from Figure 4, the losses involved in model training include classification loss and segmentation loss. Furthermore, it can also include auxiliary classification loss and affinity loss.

[0095] The classification loss is the difference between the image classification result and the image classification label, which can be calculated by the following formula (6). The patch token features can be input into the pooling and classification layers to calculate the class probability vector p cls , and then use multi-label soft margin loss as the classification function. Classification loss L cls The calculation method is expressed as follows:

[0096] Where C is the total number of categories and y is the true label at the image level. Similarly, the auxiliary classification loss can be calculated using the above formula.

[0097] The auxiliary classification loss can be determined by the output of the intermediate layer of the model. Specifically, the output of the target self-attention encoding layer in the self-attention encoding layer of the attention encoder can be obtained. For example, the intermediate patch token features output by a certain intermediate self-attention encoding layer (for example, when the attention encoder has 12 self-attention encoding layers, the target self-attention encoding layer can be the 10th layer) can be obtained. Then, the auxiliary classification result is determined by the output result, and the attention encoder is trained with the optimization goal of minimizing the difference between the auxiliary classification result and the image classification label. That is, the auxiliary classification loss is the difference between the auxiliary classification result and the image classification label.

[0098] The segmentation loss is the difference between the first semantic segmentation result and the second semantic segmentation result, which can be calculated by cross entropy.

[0099] When determining the affinity loss, the output result of the target self-attention encoding layer can be obtained, and the first affinity matrix can be determined based on the foreground-background segmentation result determined by the output result, and the second affinity matrix can be determined based on the patch token features. Finally, the attention encoder is trained with the optimization goal of minimizing the difference between the first affinity matrix and the second affinity matrix.

[0100] Specifically, the first affinity matrix is ​​obtained through the intermediate self-attention encoding layer. This layer can be used to determine the auxiliary class activation map, which is then used to obtain the foreground-background segmentation result, thereby determining the first affinity matrix. The second affinity matrix is ​​obtained directly through the attention encoder. The patch token features obtained by the final self-attention encoding layer can be used as the second affinity matrix. The auxiliary class activation map can be processed into pseudo-affinity labels to serve as supervision for affinity learning.

[0101] It is difficult to accurately distinguish foreground and background from the activation values ​​of the class activation map, because pixels with medium confidence in the class activation map are not suitable to be labeled as annotated objects or background. Therefore, in order to generate reliable affinity labels, two thresholds β are introduced. fg and β bg Satisfying 0<β bg <β fg <1, the auxiliary class activation map is divided into foreground, background and uncertain area.

[0102] Reliable segmentation labels As follows, it can be expressed by formula (7):

[0103] The background class is labeled as 0, and the uncertainty region is labeled as 255. The first affinity matrix is ​​then constructed based on this segmentation label. Specifically, any two pixels in the reliable segmentation label Y' are selected to form a pixel pair. If the pixel pairs sampled from the segmentation label have the same semantics (for example, both pixels in the pair are from the foreground or both are from the background), the affinity is determined to be positive; otherwise, their affinity is considered negative. When pixels are sampled from the uncertainty region, the affinity is ignored.

[0104] The first affinity matrix determined above is then used as supervision to promote the representation of patch tokens from the last layer of the Transformer encoder. In addition, the cosine similarity can be used to measure the predicted affinity between two final patch tokens. Therefore, the affinity loss L can be calculated by the following formula (8): aff :

[0105] Where T (L) =Γ D×H′×W′→D×N (Z (L) ),Y=Γ H′×W′→N (Y′). Γ(·) is the reshape operator, cos(·,·) represents the cosine function, N + / N -Calculate the number of positive / negative samples. This objective function directly encourages final patch token features with positive relationships to be more similar, and otherwise more discriminative. According to the chain rule, it also helps learn token representations from early Transformer layers.

[0106] After introducing the above segmentation label Y′, we explain how to determine the positive local category token features and the negative local category token features to determine Y′ t The way to split the label Y ′ Similar, but not identical.

[0107] Specifically, let M t =Γ N→H′×W′ (M i: ) is a key mask derived from the corresponding attention mask, where i is an arbitrary row index and Γ(·) represents a reshape operator. Here we use 1 to represent masked tokens and 0 to represent unmasked tokens. Intuitively, if the average activation value of the remaining tokens is high, they are likely to belong to the semantic object. Therefore, we distinguish between positivity and negativity by the following formula (9).

[0108] Where II(·) is the indicator function, ⊙ represents the Hadamard product, and Y t Y′ is the discretized token-level label derived from the activation value of the auxiliary class activation map, μ is associated with the threshold for judging positivity and negativity, and the following formula (10) gives Y′ t The calculation method is given two thresholds β bg and β fg .

[0109] The InfoNCE loss function can be used for contrastive learning (i.e., the InfoNCE loss function can be used to calculate the loss associated with the local category token features). The global and local category token features are linearly projected into a representation space suitable for contrastive learning by the global and local projectors, respectively. The projector consists of 3 linear layers and an L2 normalization layer. Let q be the projected global category token feature, k + / k - is the projected positive / negative local category token feature. The training goal is to minimize / maximize the distance between the global category token feature and the positive / negative local category token feature, the contrast loss L mcc It can be expressed by the following formula (11):

[0110] Among them, N + k + The number of samples, τ is the temperature factor, and ∈ is a small value to ensure numerical stability. The parameters of the global projector are updated using a moving average strategy, i.e., θ g←mθ g +(1-m)θ l , where m is the momentum factor, θ g and θ l The parameters of the global projector and the local projector are respectively. This slowly evolving update of the two projector parameters ensures training stability and enforces representation consistency. The global projector and the local projector can be two fully connected layers.

[0111] It should be noted that the above explanation is basically for the model training process. After the training is completed, the attention encoder can be used to perform semantic segmentation on images with unknown semantic segmentation results. Specifically, the image to be identified can be input into the trained attention encoder to obtain the patch token features of the image to be identified, and then the semantic segmentation result can be obtained through the decoder.

[0112] From the above, we can see that the overall loss can be the sum of classification loss, segmentation loss, loss related to local category token features, auxiliary classification loss, and affinity loss. Taking minimizing the overall loss as the optimization goal, the attention encoder can be trained with the minimum sum of classification loss, segmentation loss, loss related to local category token features, auxiliary classification loss, and affinity loss.

[0113] From the above content, it can be seen that this method can be divided into two extraction methods when extracting features through the attention encoder. The first method does not use a mask mechanism and directly obtains the image classification result, as well as two semantic segmentation results obtained by the class activation map and the patch token feature respectively. The second method uses a mask mechanism to supplement the attention mechanism through the mask matrix to obtain local category token features. Then, the idea of ​​contrastive learning can be combined to distinguish the positive and negative local category token features, so as to participate in model training. Therefore, model training not only includes classification loss and segmentation loss, but also includes contrastive learning loss related to local category token features. Furthermore, model training can also include auxiliary classification loss and affinity loss. By introducing multiple losses (introducing losses obtained through intermediate outputs of multiple attention encoders or various types of outputs), the loss of image semantic segmentation can be supervised more accurately, thereby improving the accuracy of semantic segmentation.

[0114] It should be noted that, for the sake of ease of description, the execution subject of this method is described as a server. The execution subject can be a desktop computer, a server, a large service platform, etc., and is not limited here.

[0115] The above provides a weakly supervised semantic segmentation method based on attention mask for one or more embodiments of the present disclosure. Based on the same idea, the present disclosure also provides a weakly supervised semantic segmentation device based on attention mask, as shown in Figure 5.

[0116] FIG5 is a schematic diagram of a weakly supervised semantic segmentation apparatus based on attention mask provided by an embodiment of the present disclosure, including:

[0117] An acquisition module 501 is configured to acquire a sample image and an image classification label corresponding to the sample image;

[0118] An input module 502 is configured to input the sample image into an attention encoder, obtain patch token features and global class token features through the attention encoder, generate an image classification result and a class activation map for the sample image through the patch token features, obtain a first semantic segmentation result for the sample image through the class activation map, and decode the patch token features to obtain a second semantic segmentation result;

[0119] A mask compensation module 503 is configured to generate an attention matrix using the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix using the target mask matrix to obtain a compensated attention matrix, and generate token features for each local category based on the compensated attention matrix;

[0120] A judgment module 504 is configured to determine, for each local class token feature, an average activation value of a portion of activation values ​​in the class activation map that is not affected by the mask in the target mask matrix based on the auxiliary class activation map and the target mask matrix; if the average activation value is greater than a preset threshold, the local class token feature is regarded as a positive local class token feature; if the average activation value is less than or equal to the preset threshold, the local class token feature is regarded as a negative local class token feature;

[0121] A training module 505 is used to train the attention encoder with minimizing the overall loss as the optimization goal, where the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

[0122] Optionally, the input module 502 is specifically used to divide the sample image into several sub-images to determine the initial patch token features corresponding to the several sub-images and multiple initial category token features corresponding to the sample image; splice the multiple initial patch token features with the initial category token features and input them into the attention encoder.

[0123] Optionally, the mask compensation module 503 is specifically used to determine the size of the attention matrix; downsample the size by a preset multiple to obtain a sampling size, and randomly generate an initial mask matrix of the sampling size; upsample the initial mask matrix by the preset multiple to obtain the target mask matrix.

[0124] Optionally, the attention matrix includes: a query matrix, a key matrix, and a value matrix;

[0125] The mask compensation module 503 is specifically configured to determine the compensated attention matrix using the following formula: Z (l) =A′ (l) V (l)

[0126] Among them, A' is the compensated attention matrix, M is the target mask matrix, Q is the query matrix, K is the key matrix, V is the value matrix, Z is the output matrix, and l is the lth self-attention encoding layer of the attention encoder.

[0127] Optionally, the attention encoder comprises a plurality of self-attention encoding layers;

[0128] The training module 505 is also used to obtain the output result of the target self-attention encoding layer among the multiple self-attention encoding layers.

[0129] Optionally, the output result of the target self-attention encoding layer includes intermediate patch token features, and the training module 505 is also used to determine the auxiliary classification result through the intermediate patch token features; the attention encoder is trained with minimizing the overall loss as the optimization goal, wherein the overall loss also includes: the difference between the auxiliary classification result and the image classification label.

[0130] Optionally, the attention encoder includes a plurality of self-attention encoding layers;

[0131] Optionally, the output result of the target self-attention encoding layer includes the auxiliary class activation map, and the training module 505 is further used to determine a first affinity matrix based on the foreground-background segmentation result determined by the auxiliary class activation map; determine a second affinity matrix based on the patch token features; and train the attention encoder with minimizing the overall loss as the optimization goal, wherein the overall loss also includes: the difference between the first affinity matrix and the second affinity matrix.

[0132] Optionally, the mask compensation module 503 is specifically used to randomly generate multiple target mask matrices; for each target mask matrix, the sample image is input into the attention encoder to generate an attention matrix corresponding to the sample image, the attention matrix generated this time is compensated by the target mask matrix to obtain a corresponding compensated attention matrix, and the local category token features corresponding to the target mask matrix are generated according to the corresponding compensated attention matrix; and the multiple local category token features are used as the local category token features.

[0133] Optionally, the training module 505 is further configured to perform image semantic segmentation on the image to be recognized using the trained attention encoder.

[0134] The present disclosure also provides a computer-readable storage medium storing a computer program, which can be used to execute the above-mentioned weakly supervised semantic segmentation method based on attention mask.

[0135] This disclosure also provides a schematic block diagram of an electronic device, shown in Figure 6. As shown in Figure 6, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for its operations. The processor reads the corresponding computer program from the non-volatile storage into the internal memory and then runs it to implement a weakly supervised semantic segmentation method based on attention masks.

[0136] Of course, in addition to software implementation, this disclosure does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0137] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0138] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0139] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0140] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0141] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0143] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0145] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0146] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0147] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0149] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0151] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0152] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. A weakly supervised semantic segmentation method based on attention mask, characterized in that: include: Obtaining a sample image and an image classification label corresponding to the sample image; Input the sample image into the attention encoder; Obtaining patch token features and global category token features through the attention encoder, and generating an image classification result and a class activation map for the sample image through the patch token features, obtaining a first semantic segmentation result for the sample image through the class activation map, and decoding the patch token features to obtain a second semantic segmentation result; Generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate a local category token feature according to the compensated attention matrix; According to the auxiliary class activation map and the target mask matrix, determine the average activation value of some activation values ​​in the class activation map that are not affected by the mask in the target mask matrix, if the average activation value is greater than a preset threshold, use the local class token feature as a positive local class token feature, if the average activation value is less than or equal to the preset threshold, use the local class token feature as a negative local class token feature; The attention encoder is trained with minimizing the overall loss as the optimization goal, and the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

2. The method according to claim 1, characterized in that The inputting the sample image into the attention encoder specifically includes: Segmenting the sample image into a plurality of sub-images to determine a plurality of initial patch token features corresponding to the plurality of sub-images, and determining an initial category token feature corresponding to the sample image; The multiple initial patch token features are concatenated with the initial category token features and input into the attention encoder.

3. The method according to claim 1 or 2, characterized in that The randomly generated target mask matrix specifically includes: determining a size of the attention matrix; Downsampling the size by a preset multiple to obtain a sampling size, and randomly generating an initial mask matrix having a size equal to the sampling size; The initial mask matrix is ​​upsampled by the preset multiple to obtain the target mask matrix.

4. The method according to any one of claims 1 to 3, characterized in that The attention matrix includes: a query matrix, a key matrix and a value matrix; The compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix specifically includes: The compensated attention matrix is ​​determined by the following formula: From (l) =A′ (l) In (l) Among them, A' is the compensated attention matrix, M is the target mask matrix, Q is the query matrix, K is the key matrix, V is the value matrix, Z is the output matrix, and l is the lth self-attention encoding layer of the attention encoder.

5. The method according to any one of claims 1 to 4, characterized in that: The attention encoder comprises a plurality of self-attention encoding layers; the method further comprises: Obtain an output result of a target self-attention coding layer among the multiple self-attention coding layers.

6. The method according to claim 5, characterized in that The output of the target self-attention encoding layer includes intermediate patch token features, and the method further includes: Determining auxiliary classification results through the intermediate patch token features; The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss also includes: the difference between the auxiliary classification result and the image classification label.

7. The method according to claim 5 or 6, characterized in that The output result of the target self-attention encoding layer includes the auxiliary class activation map, and the method further includes: Determine a first affinity matrix based on the foreground-background segmentation result determined by the auxiliary class activation map; Determining a second affinity matrix according to the patch token characteristics; The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss also includes: a difference between the first affinity matrix and the second affinity matrix.

8. The method according to any one of claims 1 to 7, characterized in that The step of generating the attention matrix by the attention encoder, randomly generating a target mask matrix, and compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix, and generating a local category token feature according to the compensated attention matrix specifically includes: Randomly generate multiple target mask matrices; For each target mask matrix, Input the sample image into the attention encoder to generate an attention matrix corresponding to the sample image, The attention matrix generated this time is compensated by the target mask matrix to obtain the corresponding compensated attention matrix. Generate local category token features corresponding to the target mask matrix according to the corresponding compensated attention matrix; A plurality of local category token features are used as the local category token features.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Use the trained attention encoder to perform image semantic segmentation on the image to be recognized.

10. A weakly supervised semantic segmentation device based on attention mask, characterized in that: include: An acquisition module, used to acquire a sample image and an image classification label corresponding to the sample image; An input module, used to input the sample image into an attention encoder, obtain patch token features and global category token features through the attention encoder, generate an image classification result and a class activation map for the sample image through the patch token features, obtain a first semantic segmentation result for the sample image through the class activation map, and decode the patch token features to obtain a second semantic segmentation result; A mask compensation module, used to generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate each local category token feature according to the compensated attention matrix; A judgment module, used for determining, according to the auxiliary class activation map and the target mask matrix, an average activation value of some activation values ​​in the class activation map that are not affected by the mask in the target mask matrix, if the average activation value is greater than a preset threshold, using the local class token feature as a positive local class token feature, if the average activation value is less than or equal to the preset threshold, using the local class token feature as a negative local class token feature; A training module is used to train the attention encoder with minimizing the overall loss as the optimization goal, wherein the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images