Multi-scale fusion weak supervision semantic segmentation model and method based on edge prior

Abstract

The invention provides a multi-scale fusion weak supervision semantic segmentation model and method based on edge prior guidance, and belongs to the field of computer vision. The model adopts a weight-sharing twin network and comprises a feature extraction module, a pseudo-tag generation module, a self-corresponding distillation module and a segmentation module, an edge information extraction module and an edge information fusion module are added on the basis of a Segform encoder, and edge priori and multi-scale feature deep fusion generated by the encoder can improve pseudo-tag edge blurring, so that the precision of the pseudo-tag is improved. Meanwhile, the pseudo label is extracted by using a self-corresponding distillation method under the condition that external supervision is not introduced. The local consistency of the pseudo label is enhanced by calculating the pixel level change, the generation quality of the pseudo label is improved, and the segmentation precision of the image is further improved. The method provided by the invention needs less annotation data and is lower in annotation cost.

Description

Technical Field

[0001] The invention proposes a multi-scale fusion weakly supervised semantic segmentation model and method based on edge prior, belonging to the field of computer vision. Background Art

[0002] In the field of Computer Vision (CV), semantic segmentation is a crucial task, whose goal is to assign a category label to each pixel in an image, so as to achieve accurate understanding and analysis of the image content. Traditional semantic segmentation methods usually rely on a large amount of pixel-level annotation data, which requires manual labeling pixel by pixel, which is an extremely time-consuming, laborious and costly task. For a high-resolution image, annotators may need to spend hours or even longer to accurately mark the boundaries and categories of each object. Or adjacent objects in the image are identified as the same category, or the background around the object is identified as an object, resulting in low accuracy of semantic segmentation.

[0003] With the rapid development of deep learning technology, deep neural networks have achieved remarkable results in semantic segmentation tasks. However, the training of deep models often requires a large amount of annotated data to learn rich features and patterns to obtain accurate segmentation performance. This reliance on a large amount of accurately annotated data faces many challenges in practical applications. On the one hand, it is unrealistic to obtain large-scale pixel-level annotated data in many fields, especially medical images, satellite images, etc. The annotation of these images requires professional knowledge and experience, and the annotation process is complicated. On the other hand, even if these annotated data can be obtained, the annotation cost will bring a huge economic burden to research and application.

[0004] In order to overcome these problems, weakly supervised semantic segmentation (WSSS) technology came into being. Weakly supervised semantic segmentation aims to use weaker forms of supervision information to train semantic segmentation models to reduce the reliance on a large amount of pixel-level labeled data. This weak supervision information can be in many forms, such as image-level annotation (only annotating the categories contained in the image without providing category information for each pixel), bounding box annotation (annotating the approximate location and bounding box of the object), scribble annotation (the user simply outlines the general outline of the object), etc. Compared with pixel-level annotation, these weakly supervised information are cheaper to obtain and easier to implement. Summary of the invention

[0005] Purpose of the invention: In view of the shortcomings of the prior art, a multi-scale fusion weakly supervised semantic segmentation model and method based on edge prior guidance is provided. In view of the problem that the current method lacks sufficient extraction of comprehensive semantic information, resulting in low quality of end-to-end WSSS pseudo-labels, an edge information extraction module is added on the basis of the Segformer encoder. The deep fusion of edge prior and encoder multi-scale features can improve the edge blur of pseudo-labels. At the same time, the self-corresponding distillation method is used to refine pseudo-labels without introducing external supervision. In addition, this invention enhances the local consistency of pseudo-labels by calculating pixel-level changes. Finally, this invention constructs an effective end-to-end Segformer-based WSSS framework, which improves the quality of pseudo-label generation and thus improves segmentation accuracy.

[0006] In order to achieve the above technical objectives, the present invention discloses a multi-scale fusion weakly supervised semantic segmentation model based on edge prior guidance, which is used in industrial scene image defect segmentation or natural image segmentation. As a semantic segmentation task, industrial scene image defect segmentation or natural image segmentation lacks pixel-level annotation or has high annotation acquisition cost. A pre-trained Segformer encoder is used to build an end-to-end multi-scale fusion weakly supervised semantic segmentation model. The multi-scale fusion weakly supervised semantic segmentation model is a weight-sharing twin network structure, which includes two sub-networks with the same structure and shared weights, and each sub-network includes a feature extraction module, a pseudo-label generation module, a self-corresponding distillation module and a segmentation module;

[0007] The two twin sub-networks of the multi-scale fusion weakly supervised semantic segmentation model are a main network and a sub-network that share weights. The main network and the sub-network have the same structure and are trained simultaneously, so that image features can be extracted more effectively in semantic segmentation.

[0008] The two twin sub-networks receive the work site images or natural images of the same object in the same batch as input. The input image of the secondary network is the input image of the main network scaled by affine transformation. The two twin sub-networks extract features from the input images and learn similar feature representation methods.

[0009] The input image is passed through the feature extraction module to generate a four-layer feature map. The feature map of the specified layer is selected and input into the pseudo-label generation module to generate a pseudo-label to be used as the supervision label of the segmentation module. The four-layer feature map is passed through the segmentation module to output the segmentation result. The segmentation result of the segmentation module and the class activation map of the pseudo-label generation module are input into the corresponding distillation module to optimize the class activation map in a self-learning manner.

[0010] The feature extraction module weights are shared. The feature extraction module includes an edge information extraction module, a Segformer encoder, and an edge information fusion module. The input image is processed by the edge information extraction module and the Segformer encoder respectively, and then fed into the edge information fusion module to generate a four-layer feature map.

[0011] The pseudo-label generation module includes a classification loss module, an initial pseudo-label generator, a change-aware refinement module, and an equivariant regularization module. The fourth-layer features in the four-layer feature map are respectively fed into the classification loss module and the initial pseudo-label generator. The class activation map CAM is generated by the initial pseudo-label generator, and the CAM is used as the initial pseudo-label. Then, the CAM is fed into the change-aware refinement module to generate a refined pseudo-label; the CAM is subjected to affine transformation and fed into the equivariant regularization module.

[0012] The segmentation module includes a decoder and a segmentation loss module. The change-aware refinement module outputs a pseudo-label and feeds it into the segmentation loss module. All four layers of features of the four-layer feature map are fed into the decoder to generate a segmentation result. After the decoder generates the segmentation result, it is fed into the segmentation loss module to calculate the segmentation loss between the pseudo-label and the generated pseudo-label.

[0013] The self-corresponding distillation module includes a class activation feature correspondence module, a segmentation feature correspondence module and a self-corresponding distillation loss module. After the image segmentation results obtained by the decoders in the two twin sub-networks are aggregated, the segmentation feature correspondence module is used to obtain the segmentation feature correspondence. The class activation maps obtained by the initial pseudo-label generators in the two twin sub-networks are aggregated and then the class activation feature correspondence is obtained through the class activation feature correspondence module. The segmentation feature correspondence through the self-corresponding distillation loss module can extract useful semantic information from the class activation feature correspondence, improve the class activation map in a self-learning manner, thereby improving the quality of pseudo-label generation, and finally accurately segmenting the subject in the input image from other content.

[0014] A multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance, the specific steps are as follows:

[0015] S1. The PASCAL VOC 2012 dataset is used as the training set, and a batch of images with different scales are selected from it. After cropping, the images are uniformly input to the same size. The images are then input into the feature extraction module of this method to obtain the edge information and multi-scale features of the images.

[0016] After affine transformation of the same batch of images, they are input into the secondary network of the weight-shared twin network, which includes the weight-shared Segformer encoder and decoder;

[0017] S2. In the main network of the twin network, the image is passed through the edge information extraction module to obtain the original edge prior information of the image. At the same time, the original image is passed through the Segformer encoder of the main network to obtain the original multi-scale features. The edge prior obtained by the edge information extraction module in the main network and the original multi-scale features are passed through the edge information fusion module to obtain the fused multi-scale features. At the same time, the original image is input into the feature extraction module of the secondary network after affine transformation, and the four-layer fused multi-scale features after affine transformation are obtained through the same steps as the main network.

[0018] S3, in the main network, the four-layer fused multi-scale features obtained by S2 are input into the pseudo-label generation module and first pass through the classification loss module to obtain the class activation map. The initial class activation map generated by the pseudo-label generator is used as the initial pseudo-label. After the initial pseudo-label is subjected to image denoising by the change-aware refinement module, the initial pseudo-label can overcome the problems of inconsistent local content recognition and rough image boundaries. The four-layer fused multi-scale features obtained in the secondary network after affine transformation are passed through the pseudo-label generation module to obtain the affine class activation map, and no longer pass through the change-aware refinement module. Since the input of the secondary network undergoes affine transformation, in order to maintain the consistency of the output of the multi-scale fusion weakly supervised semantic segmentation model, an equivariant regularization module is introduced. The class activation map is affine transformed and input into the equivariant regularization module together with the affine class activation map, thereby helping the multi-scale fusion weakly supervised semantic segmentation model to learn more generalized, stable and interpretable feature representations.

[0019] S4. Input the original multi-scale features in the main network and the original multi-scale features after affine transformation in the twin network into the main network decoder and the sub-network decoder respectively, and obtain the segmentation result and the affine segmentation result respectively. For industrial scene images, the defective parts of the parts can be segmented, and for natural images, the main body in the image can be segmented. The segmentation result and the improved pseudo-label calculate the segmentation loss and gradient backpropagate to correct the weights of the decoder and encoder;

[0020] S5, inputting the class activation map and the affine class activation map into the class activation feature correspondence module to obtain the class activation feature correspondence, and at the same time inputting the segmentation result and the affine segmentation result into the segmentation feature correspondence module to obtain the segmentation feature correspondence; inputting the class activation feature correspondence and the segmentation feature correspondence into the self-corresponding distillation loss module to calculate the loss between the class activation feature correspondence and the segmentation feature correspondence, thereby improving the quality of the segmentation result; wherein the segmentation feature correspondence can extract useful semantic information from the class activation feature correspondence, and the correlation between the two can be determined by the value of the segmentation result obtained by the main and auxiliary networks. The higher the value, the more useful the semantic information is, and the semantic information is mapped to the class activation map, so as to improve the class activation map in a self-learning manner;

[0021] S6. Through S1-S5, when the training loss of the scale-fusion weakly supervised semantic segmentation model gradually stabilizes and the segmentation accuracy on the validation set does not improve significantly, the training is considered complete and the number of iterations is designed according to actual needs;

[0022] S7. Use the scale fusion weakly supervised semantic segmentation model to perform multi-scale fusion weakly supervised semantic segmentation on the input image. At this time, the multi-scale fusion weakly supervised semantic segmentation model retains the model weights when the training is completed. The image to be segmented is passed into the encoder module of the main network in the scale fusion weakly supervised semantic segmentation model to generate multi-scale features. At the same time, the image to be segmented is input into the edge extraction module to generate edge information of the image to be segmented. The multi-scale features and edge information of the image to be segmented are passed through the edge information fusion module to generate multi-scale fusion information. Finally, the multi-scale fusion information is input into the decoder to generate the segmentation result of the main content of the image.

[0023] Furthermore, the working process of the feature extraction module is as follows: the edge information of the original input image is extracted using the edge detection operator Prewitt operator. The extraction formula of the Prewitt operator is as follows:

[0024]

[0025] For each pixel (x, y) in image I, the result after processing by the Prewitt operator is:

[0026]

[0027] Among them, G x *I(x,y) represents the image I and the horizontal convolution kernel G x The convolution result at pixel (x, y), G y *I(x,y) represents the image I and the vertical convolution kernel G y The convolution result at the pixel (x, y) indicates the edge strength of the image at the pixel (x, y). By solving and calculating each pixel of the entire image, the edge information f of the image can be obtained. e ;

[0028] The original image is input into the Segformer encoder to obtain multi-scale features: First, the input image I of size H×W×C is divided into blocks and initially embedded, and then stored in P×P image blocks. The total number Each image block is mapped to a D-dimensional feature vector through linear transformation, that is, Then the decoder is input, which is composed of a multi-head self-attention module and a Mix-FFN module in series. Finally, the encoder can output four layers of multi-scale features f i , the output is rearranged into a two-dimensional feature map with a size of

[0029] Edge information e With multi-scale features f i After entering the edge information fusion module, element-by-element multiplication is performed between the two, and a skip connection and a 3×3 convolution are added to obtain the fused features:

[0030]

[0031] Where D is downsampling, F conv is a 3×3 convolution, is element-wise multiplication, It is element-wise addition.

[0032] Furthermore, the pseudo-label generation module works as follows: The pseudo-label generation module selects fusion features The fourth layer characteristics As the input of the pseudo-label generation module, denoted as f k (x, y), so that the features at a small scale can focus on the local details and texture of the image. After the classification loss module, the class activation map is obtained and the class activation map is used as the initial pseudo label. The category score of the image category c in the training set is calculated using the following formula:

[0033]

[0034] where f k (x, y) represents the activation value of the kth channel (x, y) on the fourth layer of the fusion feature, Σ x,y f k (x,y) represents the result of global average pooling of the kth channel, Yes x,y f k The importance of (x, y) for category c, i.e., the weight;

[0035] Class activation mapping is implemented by weighting the contribution of each channel of the feature map to the classification score of a specific class. Formally, the class activation map m of class c is c for:

[0036]

[0037] Use class activation maps as initial pseudo labels;

[0038] In the secondary network, the affine-transformed fused multi-scale features are passed through the pseudo-label generation module to obtain the affine class activation map, which serves as the input of the subsequent modules;

[0039] The initial pseudo-label input change-aware refinement module performs denoising and refinement to overcome local inconsistencies in the class activation map by image denoising; specifically, for positions (i, j) and (k, l) in image I, the pixel-level change for::

[0040]

[0041] Among them, the first difference (x i,j-1,kl -x i,j,kl ) 2 The square of the difference between the pixel at position (i, j) and the pixel at its upper adjacent position (i, j-1) is used to measure the degree of change between adjacent pixels in the vertical direction. Similarly, the second difference is used to measure the degree of change between adjacent pixels in the horizontal direction.

[0042] The initial pseudo-label is generated by the change-aware refinement module and sent to the segmentation loss module in the segmentation module. The segmentation result obtained by the segmentation module is compared with the pseudo-label to calculate the segmentation loss.

[0043] Since affine changes are used between twin networks, the WSSS model can learn more generalizable, stable and interpretable feature representations. The equivariant regularization loss is introduced to construct equivariant constraints to maintain consistency between twin networks:

[0044]

[0045] where A(·) represents the affine transform, m1 is the class activation map, m2 is the affine class activation map, E(·) is the encoder, and I represents the input image.

[0046] Furthermore, the segmentation module is implemented as follows: the decoder is a lightweight decoder that integrates only MLP layers; the multi-layer features F extracted by the encoder i First, the channel dimension is unified through the multi-layer perceptron MLP layer, then the features are upsampled to 1 / 4 and concatenated together, and then the concatenated features F are fused. Finally, another MLP layer uses the fused features to predict the resolution The segmentation mask M, where N cls is the number of categories;

[0047] The decoder formula is as follows:

[0048]

[0049] M=Linear(C,N cls )(F),

[0050] Among them, Linear(C in ,Cout )(·) is a linear layer, C in and C out They are the input and output vector dimensions respectively; the fused multi-scale features and the fused multi-scale features after affine transformation are input into the decoder and the segmentation result and affine segmentation result are output;

[0051] The segmentation results and pseudo labels are calculated through the segmentation loss module:

[0052]

[0053] Where N is the total number of pixels in the image, y i,c is an indicator variable. If the true category of pixel i is c, then y i,c =1, otherwise 0, is the probability that the model predicts that pixel i belongs to category c;

[0054] Use gradient backpropagation to correct the weights of the decoder and encoder.

[0055] Furthermore, the self-correspondence distillation module works as follows: the class activation map and the affine class activation map are input into the class activation feature correspondence module to obtain the class activation feature correspondence. Specifically, given an image And affine transformation A(·), use the feature map extracted by the encoder to generate the class activation map: Let the class activation map m1, use the process: It represents the process of generating the class activation map m1 from the feature map. D represents the dimension, which here represents RGB, so the value is 3. C is the number of channels of the image. The corresponding class activation feature is:

[0056]

[0057] in, Represents the class activation feature correspondence, m1 and m2 represent the class activation maps generated by the main network and the twin network respectively;

[0058] Similarly, the segmentation feature is expressed as:

[0059]

[0060] in, Represents the segmentation feature correspondence, s1 and s2 represent the segmentation results generated by the main network and the twin network respectively;

[0061] When there is a significant correlation between two segmentation results, the segmentation features correspond to The larger the value, the higher the correlation between the two segmentation results; Segmentation feature correspondence and The class activation feature correspondence is simply element-wise multiplied to achieve the self-corresponding distillation loss

[0062]

[0063] where max(·,0) represents zero clamping.

[0064] A computer device comprises a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute a multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance.

[0065] Beneficial effects: This method introduces a change-aware refinement module by drawing on the idea of ​​image denoising. In view of the problem of blurred edges and local inconsistency of pseudo-labels in weakly supervised semantic segmentation; the edge extraction module and edge information fusion module of the present invention optimize the quality of pseudo-labels by introducing edge information, while the edge refinement perception module uses pixel-level changes to overcome the problem of local inconsistency of pseudo-labels, further optimizes and updates the initial pseudo-labels, and finally improves the accuracy of weakly supervised semantic segmentation. Compared with the original weakly supervised semantic segmentation model, the present invention effectively improves the generation quality of pseudo-labels and provides a new solution for weakly supervised semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the multi-scale fusion weakly supervised semantic segmentation model based on edge prior guidance of the present invention;

[0067] Figure 2 Schematic diagram of the structure of the Segformer encoder and decoder used in the embodiment of the present invention; DETAILED DESCRIPTION

[0068] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:

[0069] like Figure 1As shown, the present invention discloses a multi-scale fusion weakly supervised semantic segmentation model based on edge prior guidance, including a feature extraction module, a pseudo-label generation module, a self-corresponding distillation module and a segmentation module. At the same time, in order to better obtain image semantic information, this model adopts a weight-shared twin network, and the twin network is composed of two twin sub-networks with the same structure and shared weights. The two twin sub-networks receive two different inputs respectively, and then extract features from the inputs. Due to the shared weights, the two twin sub-networks will learn similar feature representations. The PASCAL VOC 2012 dataset is used as a training and test set, and the SBD dataset is used to enhance it. Only image-level labels are used for annotation, and mIoU is used as an evaluation indicator. The feature extraction module includes an edge information extraction module and a Segformer encoder, wherein the edge information extraction module uses the Prewitt operator to obtain edge priors, and the Segformer encoder outputs multi-scale features. After the edge information fusion module, different scale features all obtain edge information. At the same time, the original image is input into the weight-shared twin network after affine transformation, and the twin network includes a weight-shared encoder and a decoder. Based on the encoding features fused with edge priors, the classification loss is calculated, and the class activation map is generated as the initial pseudo label after passing through the initial pseudo label generator. After passing through the change-aware refinement module, the initial pseudo label is refined and denoised to generate a pseudo label. After the original class activation map undergoes affine transformation, the equivariant regularization loss is calculated with the class activation map in the twin network through the equivariant regularization module, so that the twin networks maintain consistency, prompting the model to learn more universal features and improve the accuracy and stability of subsequent segmentation. The multi-scale fusion feature input is only integrated by the MLP layer to output the segmentation result. The twin network branch outputs the segmentation result in the same way. The segmentation result and the pseudo label calculate the segmentation loss and then back-propagate to update the model parameters. The class activation maps of the two branches of the twin network pass through the class activation feature correspondence module to generate class activation feature correspondence, and the segmentation result passes through the segmentation feature correspondence module to generate segmentation feature correspondence. The two are input from the corresponding distillation loss module to calculate and improve the original class activation map. After the weakly supervised semantic segmentation model training is completed, it enters the inference stage. At this time, the segmentation model does not change the model weights. The image to be segmented is passed into the trained segmentation model into the encoder module to generate multi-scale features, and at the same time, it is input into the edge extraction module to generate edge information. The two pass through the edge information fusion module to generate multi-scale fusion information, and finally the multi-scale fusion information is input into the decoder to generate the segmentation result.

[0070] The multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance includes the following steps:

[0071] S1. The PASCAL VOC 2012 dataset is selected as the training set. The pre-trained Segformer is used to build an end-to-end WSSS model. The image is input into the feature extraction module to obtain edge information and multi-scale features. At the same time, the image is affine transformed and then input into the weight-shared twin network. The twin network includes a weight-shared encoder and decoder.

[0072] The image edge information obtained by the edge information extraction module in S2 and S1 is fused with the multi-scale coding feature by the edge information fusion module to obtain the fusion feature;

[0073] S3, after calculating the classification loss using the fusion features in the multi-scale fusion features, the pseudo-label generator is used to generate the initial class activation map as the initial pseudo-label, and the initial pseudo-label is sent to the change-aware refinement module to generate the denoised pseudo-label;

[0074] S4, the initial class activation map after affine transformation and the class activation map generated by the twin sub-network are sent to the equivariant regularization module to calculate the loss, so as to help the model learn more generalizable, stable and interpretable feature representations;

[0075] S5, input the fused multi-scale features into the decoder consisting of only the MLP layer to obtain the segmentation result, and the segmentation result and the denoised pseudo-label obtained in S3 are used to calculate the segmentation loss and gradient backpropagation to correct the weights of the decoder and encoder in the model;

[0076] S6. Use the self-correspondence distillation module to calculate the self-correspondence distillation loss for the class activation feature correspondence and the segmentation feature correspondence. The segmentation feature correspondence can extract useful semantic information from the class activation feature correspondence to improve the class activation map in a self-learning manner.

[0077] S7. When the training loss of the multi-scale fusion weakly supervised semantic segmentation model gradually stabilizes and the segmentation accuracy on the validation set does not improve significantly, the training can be stopped and the inference stage can be entered. The image is input into the decoder after the feature extraction module, and the segmentation result is finally output. The segmentation effect is evaluated using mIoU. Furthermore, for the Segformer encoder part, the parameters are initialized using the ImageNet-1K pre-trained initial values. The original images in the PASCALVOC 2012 dataset are subjected to simple data augmentation strategies such as random rescaling, random horizontal flipping, and random cropping, and the crop size is set to 512×512.

[0078] like Figure 2As shown, for the Segformer encoder part, the input image size is H×W×C, where H and W are the height and width of the image, respectively, and C is the number of channels of the image. In the overlapping patch embedding module, the image is divided into multiple overlapping image blocks, each of which is usually a square or rectangular area of ​​a fixed size. There is usually a certain overlap between these image blocks to ensure that important image information is not lost. The encoder generates multi-level and multi-scale features when given an input image. Specifically, for an input image of size H×W×3, a hierarchical feature map F is obtained after performing an overlapping patch merging operation. i , whose resolution is where i∈{1,2,3,4}, and C i+1 Greater than C i .

[0079] The Efficient Self-Atten module uses a reduction ratio R to reduce the length of the original self-attention sequence, thereby reducing the computational complexity from O(N 2 ) is reduced to

[0080]

[0081] where K is the sequence to be parsimonious, is the value that reshapes K into a shape of Sequence; Linear(C in ,C out )(·) refers to the addition of a C in dimensional tensor as input and generate a C out dimensional tensor as the output of the linear layer. Therefore, the dimension of the new K is Suppose from stage 1 to stage 4, R is set to [64, 16, 4, 1].

[0082] Using Patch Merging, you can move features from Zoom out to However, the original Patch Merging is used for non-overlapping images or patches, and the local continuity around the image blocks cannot be guaranteed. Since the image is divided into multiple overlapping image blocks in the overlapping patch embedding module, the local continuity around these image blocks can be preserved. In the overlapping patch merging module, setting K=7, S=4, P=3 and K=3, S=2, P=1 produces features of the same size as the non-overlapping process. This operation can be implemented in Pytorch as "nn.Conv2D".

[0083] Since the test data may have a resolution mismatch with the training data, position encoding needs to be inserted, which will also lead to a decrease in accuracy. For this reason, the Mix-FFN module is introduced, and 3×3 convolution is used in the feedforward network (FFN) to consider the impact of zero padding on the leaked position information:

[0084] X out =MLP(GELU(Conv 3×3 (MLP(X in ))))+X in

[0085] Furthermore, it is proposed to use the edge information extraction module to obtain the edge information of the original image. In this module, the Prewitt operator is used to extract the edge information:

[0086]

[0087] For each pixel (x, y) in image I, the result after processing by the Prewitt operator is:

[0088]

[0089] Among them, G x *I(x,y) represents the image I and the horizontal convolution kernel G x The convolution result at pixel (x, y), G y *I(x,y) represents the image I and the vertical convolution kernel G y The convolution result at pixel (x, y). This result represents the edge strength of the image at pixel (x, y). By performing such calculations on each pixel of the entire image, the edge detection result of the image can be obtained.

[0090] Furthermore, in order to enable the multi-scale features generated by the encoder to effectively obtain edge prior information, the edge information f e With multi-scale features f i After entering the edge information fusion module, element-by-element multiplication is performed between the two, and a skip connection and a 3×3 convolution are added to obtain the fused features:

[0091]

[0092] Where D is downsampling, F conv is a 3×3 convolution, is element-wise multiplication, It is element-wise addition.

[0093] Furthermore, the fourth-layer features generated by the encoder are used to calculate the classification loss, and the multi-label soft margin loss is used as the classification loss for network training:

[0094]

[0095] Where l is the true class label and C is the total number of categories.

[0096] Class Activation Map (CAM) enables the classifier to learn target location. Given a feature map And category c, global average pooling and fully connected layers are used to calculate the category score:

[0097]

[0098] where f k (x, y) represents the activation value of the kth channel (x, y) on the fourth layer of the fusion feature, ∑ x,y f k (x,y) represents the result of global average pooling of the kth channel, Yes x,y f k The importance (i.e. weight) of (x,y) for category c.

[0099] Class activation mapping is implemented by weighting the contribution of each channel of the feature map to the classification score of a specific class. Formally, the class activation map m of class c is c for:

[0100]

[0101] Class activation mapping is used as the initial pseudo-label. However, the initial pseudo-label obtained by CAM is usually locally inconsistent and has rough boundaries, so a change-aware refinement module is added to introduce the idea of ​​image denoising, which can be specifically divided into the following steps:

[0102] Step 1: For positions (i, j) and (k, l) in image I, calculate the image pixel-level change:

[0103]

[0104] Step 2: Calculate the local information term k rgb :

[0105]

[0106] where σ ij represents the standard deviation and α represents the smoothing weight.

[0107] Step 3: In order to enhance the local consistency of pseudo labels, for pixels with large changes in the image, calculate the verification kernel k ij,kl To avoid some sudden changes in values:

[0108]

[0109] Where N(i,j) is the neighborhood set of (i,j), obtained using dilated convolution.

[0110] Step 4: Update pixel labels (CAM) using an iterative update strategy

[0111]

[0112] Furthermore, the training model uses a twin network. After the image undergoes affine transformation, it is sent to a weight-sharing branch network, and equivariant constraints are used to maintain consistency between the twin networks. For this purpose, the following equilateral regularization loss is designed:

[0113]

[0114] where A(·) represents the affine transformation, m1 is the class activation map, m2 is the affine class activation map, and E(·) is the encoder.

[0115] Further, if Figure 2 As shown in the figure, the decoder part integrates a lightweight decoder consisting of only MLP layers, which reduces the computational requirements. The decoder outputs the segmentation results in the following steps:

[0116] Step 1: Multi-layer features F from the encoder i The channel dimension is unified through the MLP layer:

[0117]

[0118] Step 2: Features are upsampled to 1 / 4 and concatenated together:

[0119]

[0120] Step 3: Use the MLP layer to fuse the concatenated features F:

[0121]

[0122] Step 4: Another MLP layer uses the fused features to predict the resolution The segmentation mask M, where N cls is the number of categories:

[0123] M=Linear(C,N cls )(F),

[0124] Where M is the prediction mask, Linear(C in ,C out )(·) is a linear layer, C inand C out are the input and output vector dimensions respectively.

[0125] Furthermore, the predicted mask M and pseudo-label are used to calculate the segmentation loss:

[0126] General cross entropy loss:

[0127]

[0128] Where p is the true distribution and q is the predicted distribution.

[0129] For background loss (bg loss ):

[0130]

[0131] True distribution p bg The pixel position is 1 (i.e. the true category is background) at the background pixel position and 0 at other positions. The predicted distribution p bg is the probability distribution that the model predicts that each pixel belongs to different categories.

[0132] For the foreground loss (fg loss ):

[0133]

[0134] The final segmentation loss (seg loss ):

[0135] seg loss =(bg loss +f gloss )×0.5

[0136] Furthermore, in order to improve the class activation map of the original image, a self-corresponding distillation module is designed, which does not require additional labels or external supervision and can help the network obtain comprehensive image semantic information. And affine transformation A(·), use the feature map extracted by the encoder to generate the class activation map. Taking the class activation map m1 as an example, use E: Represents the process of generating a class activation map m1 from a feature map.

[0137]

[0138] Then the CAM feature correspondence can be further expressed as:

[0139]

[0140] This formula represents the cosine similarity between the position (h1, w1) of feature m1 and the position (h2, w2) of m2.

[0141] Similarly, the segmentation feature corresponding module can be expressed as:

[0142]

[0143] Inspired by self-supervised feature learning, this invention considers aligning the segmentation feature correspondence with the network's own class activation feature correspondence. The self-correspondence distillation loss module pushes the corresponding class activations together to enhance semantic relevance. When there is a significant correlation between two segmentation predictions, Segmentation feature correspondence and The class activation feature correspondences are simply element-wise multiplied to implement the self-corresponding distillation loss:

[0144]

[0145] where max(·,0) represents zero clamping.

[0146] Furthermore, during the entire training phase, the AdamW optimizer is used, and its optimization algorithm is divided into the following steps:

[0147] Step 1: First, the gradient of the model parameters is calculated through the back-propagation algorithm.

[0148] Step 2: Update the first-order moment estimate m t and the second-order moment estimate v t , the first-order moment estimate is the exponentially weighted average of the gradient, and the second-order moment estimate is the exponentially weighted average of the square of the gradient:

[0149] m t =β1m t-1 +(1-β1)g t ,

[0150] where g t is the gradient at the current moment, and β1 is the decay rate of the first-order moment estimate.

[0151]

[0152] where β2 is the decay rate of the second-order moment estimate.

[0153] Step 3: Calculate the bias-corrected first-order moment estimates and second-order moment estimates:

[0154]

[0155] Step 4: Calculate the update Δθ t :

[0156]

[0157] Where η is the learning rate and ∈ is a small constant to prevent the denominator from being zero.

[0158] Step 5: Before updating the model parameters, apply weight decay:

[0159]

[0160] Furthermore, in the inference stage, the original image outputs the segmentation result after outputting the trained feature extraction network and segmentation network, and mIOU is used to evaluate the segmentation effect:

[0161]

[0162] Where k represents the number of categories (including background categories), p ij represents the number of pixels that belong to category i but are predicted to be category j, p ii It indicates the number of pixels that belong to category i and are predicted to be category i. The value of mIOU is between 0 and 1. The larger the value, the higher the overlap between the segmentation result and the true value, and the better the model performance.

[0163] The present invention provides a multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance, including a feature extraction module, a pseudo-label generation module, a self-corresponding distillation module and a segmentation module. The invention uses the PASCAL VOC 2012 data set as a training and test set, and uses the SBD data set to enhance it. The invention only uses image-level labels for annotation, and uses mIoU as an evaluation indicator. The feature extraction module includes an edge information extraction module and a Segformer encoder, wherein the edge information extraction module uses the Prewitt operator to obtain edge priors, and the Segformer encoder outputs multi-scale features. After passing through the edge information fusion module, features of different scales all obtain edge information. At the same time, the original image is input into a weight-sharing twin network after affine transformation, and the twin network includes a weight-sharing encoder and a decoder. Based on the encoding features fused with edge priors, the classification loss is calculated, and a class activation map is generated as an initial pseudo-label after passing through an initial pseudo-label generator, and then the initial pseudo-label is refined and denoised after passing through a change-aware refinement module to generate a pseudo-label. After the original class activation map is affine-transformed, the class activation map in the twin network is used to calculate the equivariant regularization loss through the equivariant regularization module, which can narrow the gap between weak supervision and full supervision, prompting the model to learn more universal features and improve the accuracy and stability of subsequent segmentation. The multi-scale fusion feature input is only integrated by the decoder of the MLP layer and then outputs the segmentation result. The twin network branch uses the same method to output the segmentation result. The segmentation result and the pseudo label calculate the segmentation loss and then back-propagate to update the model parameters. The class activation maps of the two branches of the twin network are generated by the class activation feature correspondence module after passing through the class activation feature correspondence module. The segmentation result is generated by the segmentation feature correspondence module. The two are input from the corresponding distillation loss module to calculate and improve the original class activation map. When the weakly supervised semantic segmentation model is trained, it enters the inference stage. At this time, the segmentation model does not change the model weights. The image to be segmented is passed into the trained segmentation model and passed into the encoder module to generate multi-scale features. At the same time, it is input into the edge extraction module to generate edge information. The two pass through the edge information fusion module to generate multi-scale fusion information. Finally, the multi-scale fusion information is input into the decoder to generate the segmentation result.

[0164] Example:

[0165] A multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance. The training steps are as follows:

[0166] Step 1: The original images in the PASCAL VOC 2012 dataset are subjected to simple data augmentation strategies such as random rescaling, random horizontal flipping, and random cropping, and the crop size is set to 512×512.

[0167] Step 2: Use the ImageNet-1K pre-trained initial values ​​to initialize the encoder parameters, input the image into the feature extraction module to obtain edge information and multi-scale features, and after performing affine transformation on the image, input it into the weight-shared twin network, which includes a weight-shared encoder and decoder.

[0168] Step 3: The edge information is fused with the multi-scale features to obtain the multi-scale fusion features. These features are input into the decoder to output the final segmentation result.

[0169] Step 4: Use the fourth-layer fusion features in the encoder to calculate the classification loss and then obtain the CAM as the initial pseudo-label through the initial pseudo-label generator, and then denoise and refine the initial pseudo-label to obtain the pseudo-label.

[0170] Step 5: After performing affine transformation on the CAM, the input is input into the equivariant regularization module to calculate the equivariant regularization loss between the CAM generated by the twin network.

[0171] Step 6: The CAM and segmentation results generated by the two branches of the twin network are passed through the self-corresponding distillation module to calculate the self-corresponding distillation loss.

[0172] Step 7: Calculate the segmentation loss between the segmentation result and the pseudo label.

[0173] The PyTorch framework is used to implement the invented method. For the entire training phase, the AdamW optimizer is used and the initial learning rate is set to 6×10 -5 , the weight decay factor is 0.01. The total training loss is:

[0174]

[0175] where λ1, λ2, and λ3 are set to 0.1, 0.01, and 1, respectively.

[0176] In order to illustrate the effectiveness of this method, this method is compared with the current mainstream semantic segmentation model, and an ablation experiment is performed on the proposed module, using mIoU as the evaluation indicator.

[0177] Table 1: Comparison of semantic segmentation on PASCAL VOC 2012 dataset:

[0178]

[0179]

[0180] It can be observed that the VOC 2012 dataset outperforms all other end-to-end supervised methods, and the performance is comparable to that of multi-stage supervised methods, and even reaches 85.5% of Segformer in the fully supervised method. This verifies the effectiveness of this method.

[0181] Table 2: Ablation test results of this method on the PASCAL VOC 2012 dataset:

[0182]

[0183] As shown in Table 2, the Transformer-based baseline model achieves 47.2% mIoU. Compared with the baseline, the change-aware refinement module and the self-corresponding distillation module of this invention further improve the mIoU. The edge information extraction module added by this invention further improves the mIoU. After the equilateral regularization loss, this method finally achieves 64.9% mIoU.

Claims

1. A multi-scale fusion weakly supervised semantic segmentation model based on edge prior, characterized in that: Used in industrial scene image defect segmentation or natural image segmentation. As a semantic segmentation task, due to the lack of pixel-level annotation or high cost of obtaining annotations, the pre-trained Segformer encoder is used to build an end-to-end multi-scale fusion weakly supervised semantic segmentation model. The multi-scale fusion weakly supervised semantic segmentation model is a weight-sharing twin network structure, which includes two sub-networks with the same structure and shared weights. Each sub-network includes a feature extraction module, a pseudo-label generation module, a self-corresponding distillation module and a segmentation module; The two twin sub-networks of the multi-scale fusion weakly supervised semantic segmentation model are a main network and a sub-network that share weights. The main network and the sub-network have the same structure and are trained simultaneously, so that image features can be extracted more effectively in semantic segmentation. The two twin sub-networks receive the work site images or natural images of the same object in the same batch as input. The input image of the secondary network is the input image of the main network scaled by affine transformation. The two twin sub-networks extract features from the input images and learn similar feature representation methods. The input image is passed through the feature extraction module to generate a four-layer feature map. The feature map of the specified layer is selected and input into the pseudo-label generation module to generate a pseudo-label to be used as the supervision label of the segmentation module. The four-layer feature map is passed through the segmentation module to output the segmentation result. The segmentation result of the segmentation module and the class activation map of the pseudo-label generation module are input into the corresponding distillation module to optimize the class activation map in a self-learning manner. The feature extraction module weights are shared. The feature extraction module includes an edge information extraction module, a Segformer encoder, and an edge information fusion module. The input image is processed by the edge information extraction module and the Segformer encoder respectively, and then fed into the edge information fusion module to generate a four-layer feature map. The pseudo-label generation module includes a classification loss module, an initial pseudo-label generator, a change-aware refinement module, and an equivariant regularization module. The fourth-layer features in the four-layer feature map are respectively fed into the classification loss module and the initial pseudo-label generator. The class activation map CAM is generated by the initial pseudo-label generator, and the CAM is used as the initial pseudo-label. Then, the CAM is fed into the change-aware refinement module to generate a refined pseudo-label; the CAM is subjected to affine transformation and fed into the equivariant regularization module. The segmentation module includes a decoder and a segmentation loss module. The change-aware refinement module outputs a pseudo-label and feeds it into the segmentation loss module. All four layers of features of the four-layer feature map are fed into the decoder to generate a segmentation result. After the decoder generates the segmentation result, it is fed into the segmentation loss module to calculate the segmentation loss between the pseudo-label and the generated pseudo-label. The self-corresponding distillation module includes a class activation feature correspondence module, a segmentation feature correspondence module and a self-corresponding distillation loss module. After the image segmentation results obtained by the decoders in the two twin sub-networks are aggregated, the segmentation feature correspondence module is used to obtain the segmentation feature correspondence. The class activation maps obtained by the initial pseudo-label generators in the two twin sub-networks are aggregated and then the class activation feature correspondence is obtained through the class activation feature correspondence module. The segmentation feature correspondence through the self-corresponding distillation loss module can extract useful semantic information from the class activation feature correspondence, improve the class activation map in a self-learning manner, thereby improving the quality of pseudo-label generation, and finally accurately segmenting the subject in the input image from other content.

2. A multi-scale fusion weakly supervised semantic segmentation method based on edge prior using the multi-scale fusion weakly supervised semantic segmentation model based on edge prior according to claim 1, characterized in that: The specific steps are as follows: S1. The PASCAL VOC 2012 dataset is used as the training set, and a batch of images with different scales are selected from it. After cropping, the images are uniformly input to the same size. The images are then input into the feature extraction module of this method to obtain the edge information and multi-scale features of the images. After affine transformation of the same batch of images, they are input into the secondary network of the weight-shared twin network, which includes the weight-shared Segformer encoder and decoder; S2. In the main network of the twin network, the image is passed through the edge information extraction module to obtain the original edge prior information of the image. At the same time, the original image is passed through the Segformer encoder of the main network to obtain the original multi-scale features. The edge prior obtained by the edge information extraction module in the main network and the original multi-scale features are passed through the edge information fusion module to obtain the fused multi-scale features. At the same time, the original image is input into the feature extraction module of the secondary network after affine transformation, and the four-layer fused multi-scale features after affine transformation are obtained through the same steps as the main network. S3, in the main network, the four-layer fused multi-scale features obtained by S2 are input into the pseudo-label generation module and first pass through the classification loss module to obtain the class activation map. The initial class activation map generated by the pseudo-label generator is used as the initial pseudo-label. After the initial pseudo-label is subjected to image denoising by the change-aware refinement module, the initial pseudo-label can overcome the problems of inconsistent local content recognition and rough image boundaries. The four-layer fused multi-scale features obtained in the secondary network after affine transformation are passed through the pseudo-label generation module to obtain the affine class activation map, and no longer pass through the change-aware refinement module. Since the input of the secondary network undergoes affine transformation, in order to maintain the consistency of the output of the multi-scale fusion weakly supervised semantic segmentation model, an equivariant regularization module is introduced. The class activation map is affine transformed and input into the equivariant regularization module together with the affine class activation map, thereby helping the multi-scale fusion weakly supervised semantic segmentation model to learn more generalized, stable and interpretable feature representations. S4. Input the original multi-scale features in the main network and the original multi-scale features after affine transformation in the twin network into the main network decoder and the sub-network decoder respectively, and obtain the segmentation result and the affine segmentation result respectively. For industrial scene images, the defective parts of the parts can be segmented, and for natural images, the main body in the image can be segmented. The segmentation result and the improved pseudo-label calculate the segmentation loss and gradient backpropagate to correct the weights of the decoder and encoder; S5, inputting the class activation map and the affine class activation map into the class activation feature correspondence module to obtain the class activation feature correspondence, and at the same time inputting the segmentation result and the affine segmentation result into the segmentation feature correspondence module to obtain the segmentation feature correspondence; inputting the class activation feature correspondence and the segmentation feature correspondence into the self-corresponding distillation loss module to calculate the loss between the class activation feature correspondence and the segmentation feature correspondence, thereby improving the quality of the segmentation result; wherein the segmentation feature correspondence can extract useful semantic information from the class activation feature correspondence, and the correlation between the two can be determined by the value of the segmentation result obtained by the main and auxiliary networks. The higher the value, the more useful the semantic information is, and the semantic information is mapped to the class activation map, so as to improve the class activation map in a self-learning manner; S6. Through S1-S5, when the training loss of the scale-fusion weakly supervised semantic segmentation model gradually stabilizes and the segmentation accuracy on the validation set does not improve significantly, the training is considered complete and the number of iterations is designed according to actual needs; S7. Use the scale fusion weakly supervised semantic segmentation model to perform multi-scale fusion weakly supervised semantic segmentation on the input image. At this time, the multi-scale fusion weakly supervised semantic segmentation model retains the model weights when the training is completed. The image to be segmented is passed into the encoder module of the main network in the scale fusion weakly supervised semantic segmentation model to generate multi-scale features. At the same time, the image to be segmented is input into the edge extraction module to generate edge information of the image to be segmented. The multi-scale features and edge information of the image to be segmented are passed through the edge information fusion module to generate multi-scale fusion information. Finally, the multi-scale fusion information is input into the decoder to generate the segmentation result of the main content of the image.

3. The multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance according to claim 2 is characterized in that: The working process of the feature extraction module is as follows: The edge detection operator Prewitt operator is used to extract the edge information of the original input image. The extraction formula of the Prewitt operator is as follows: For each pixel (x, y) in image I, the result after processing by the Prewitt operator is: Among them, G x *I(x,y) represents the image I and the horizontal convolution kernel G x The convolution result at pixel (x, y), G y *I(x,y) represents the image I and the vertical convolution kernel G y The convolution result at the pixel (x, y) indicates the edge strength of the image at the pixel (x, y). By solving and calculating each pixel of the entire image, the edge information f of the image can be obtained. e ; The original image is input into the Segformer encoder to obtain multi-scale features: First, the input image I of size H×W×C is divided into blocks and initially embedded, and then stored in P×P image blocks. The total number Each image block is mapped to a D-dimensional feature vector through linear transformation, that is, Then the decoder is input, which is composed of a multi-head self-attention module and a Mix-FFN module in series. Finally, the encoder can output four layers of multi-scale features f i , the output is rearranged into a two-dimensional feature map with a size of Edge information e With multi-scale features f i After entering the edge information fusion module, element-by-element multiplication is performed between the two, and a skip connection and a 3×3 convolution are added to obtain the fused features: Where D is downsampling, F conv is a 3×3 convolution, is element-wise multiplication, It is element-wise addition.

4. The multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance according to claim 3 is characterized in that: The working process of the pseudo-label generation module is as follows: The pseudo label generation module selects fusion features The fourth layer characteristics As the input of the pseudo-label generation module, denoted as f k (x, y), so that the features at a small scale can focus on the local details and texture of the image. After the classification loss module, the class activation map is obtained and the class activation map is used as the initial pseudo label. The category score of the image category c in the training set is calculated using the following formula: where f k (x, y) represents the activation value of the kth channel (x, y) on the fourth layer of the fusion feature, Σ x,y f k (x,y) represents the result of global average pooling of the kth channel, Yes x,y f k The importance of (x, y) for category c, i.e., the weight; Class activation mapping is implemented by weighting the contribution of each channel of the feature map to the classification score of a specific class. Formally, the class activation map m of class c is c for: Use class activation maps as initial pseudo labels; In the secondary network, the affine-transformed fused multi-scale features are passed through the pseudo-label generation module to obtain the affine class activation map, which serves as the input of the subsequent modules; The initial pseudo-label input change-aware refinement module performs denoising and refinement to overcome local inconsistencies in the class activation map by image denoising; specifically, for positions (i, j) and (k, l) in image I, the pixel-level change for:: Among them, the first difference (x i,j-1,kl -x i,j,kl ) 2 The square of the difference between the pixel at position (i, j) and the pixel at its upper adjacent position (i, j-1) is used to measure the degree of change between adjacent pixels in the vertical direction. Similarly, the second difference is used to measure the degree of change between adjacent pixels in the horizontal direction. The initial pseudo-label is generated by the change-aware refinement module and sent to the segmentation loss module in the segmentation module. The segmentation result obtained by the segmentation module is compared with the pseudo-label to calculate the segmentation loss. Since affine changes are used between twin networks, the WSSS model can learn more generalizable, stable and interpretable feature representations. The equivariant regularization loss is introduced to construct equivariant constraints to maintain consistency between twin networks: where A(·) represents the affine transform, m1 is the class activation map, m2 is the affine class activation map, E(·) is the encoder, and I represents the input image.

5. The multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance according to claim 2 is characterized in that: The implementation process of the segmentation module is as follows: The decoder is a lightweight decoder that integrates only MLP layers; the multi-layer features F extracted by the encoder i First, the channel dimension is unified through the multi-layer perceptron MLP layer, then the features are upsampled to 1 / 4 and concatenated together, and then the concatenated features F are fused. Finally, another MLP layer uses the fused features to predict the resolution The segmentation mask M, where N cls is the number of categories; The decoder formula is as follows: M=Linear(C,N cls )(F), Among them, Linear(C in ,C out )(·) is a linear layer, C in and C out They are the input and output vector dimensions respectively; the fused multi-scale features and the fused multi-scale features after affine transformation are input into the decoder and the segmentation result and affine segmentation result are output; The segmentation results and pseudo labels are calculated through the segmentation loss module: Where N is the total number of pixels in the image, y i,c is an indicator variable. If the true category of pixel i is c, then y i,c =1, otherwise 0, is the probability that the model predicts that pixel i belongs to category c; Use gradient backpropagation to correct the weights of the decoder and encoder.

6. The multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance according to claim 2 is characterized in that: The working process of the corresponding distillation module is as follows: Class activation mapping and affine class activation mapping input class activation feature correspondence module to obtain class activation feature correspondence. Specifically, given an image And affine transformation A(·), use the feature map extracted by the encoder to generate the class activation map: Let the class activation map m1, use the process: It represents the process of generating the class activation map m1 from the feature map. D represents the dimension, which here represents RGB, so the value is 3. C is the number of channels of the image. The corresponding class activation feature is: in, Represents the class activation feature correspondence, m1 and m2 represent the class activation maps generated by the main network and the twin network respectively; Similarly, the segmentation feature is expressed as: in, Represents the segmentation feature correspondence, s1 and s2 represent the segmentation results generated by the main network and the twin network respectively; When there is a significant correlation between two segmentation results, the segmentation features correspond to The larger the value, the higher the correlation between the two segmentation results; Segmentation feature correspondence and The class activation feature correspondence is simply element-wise multiplied to achieve the self-corresponding distillation loss where max(·,0) represents zero clamping.

7. A computer device, characterized in that: It includes a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the multi-scale fusion weakly supervised semantic segmentation method based on edge prior guidance as described in any one of claims 2-6.

Images