Weak supervision image segmentation model and method considering global information, and computer system

Abstract

The model includes: an input layer configured to accept an image; the pre-processing module is configured to pre-process the image, and the pre-processing content comprises the steps of increasing the number of channels of the image and standardizing the original low-level features of the image; the convolution modulator with the attention mechanism is configured to perform multiple convolution operations on the preprocessed image, and add high-level features obtained after the last convolution operation and original low-level features input into the convolution modulator during the first convolution by using a residual link to obtain fusion features; the pooling layer is configured to perform pooling operation on the fusion features extracted by the convolution modulator; and the full connection layer is configured to map and output the pooled fusion feature to generate a segmentation result. According to the method, the image segmentation precision and efficiency are improved by combining the advantages of the Transform and the CNN.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and relates to weakly supervised image segmentation taking into account global information, and global feature extraction based on deep learning, and specifically to a weakly supervised image segmentation model, method and computer system taking into account global information. Background Art

[0002] Image segmentation is a key technology in the field of computer vision, which aims to divide an image into multiple regions with similar properties or functions. With the rapid development of deep learning technology, image segmentation methods have made significant breakthroughs. By learning a large amount of image data, deep learning models can automatically learn the feature representation of images, thereby achieving accurate image segmentation. Image segmentation has shown broad application prospects in many fields. For example, in medical image analysis, image segmentation can help doctors diagnose diseases more accurately; in the field of autonomous driving, image segmentation can help vehicles better understand the surrounding environment; in satellite image analysis, image segmentation can help researchers better understand the structure and changes of the earth's surface. However, image segmentation methods based on deep learning also face some challenges. For example, how to design a more effective network structure to improve segmentation accuracy and efficiency? How to overcome the differences between different objects? How to balance the complexity of the model and the demand for computing resources, etc.

[0003] CNN is a popular network currently used for image segmentation. When CNN segments images, it can improve the accuracy of image segmentation by increasing the network depth and designing different expansion rates to increase the receptive field, but it is often limited by its specific convolution kernel size and cannot integrate global information. In the Transformer model, the self-attention mechanism enables the model to learn the dependencies between different positions, thereby achieving more accurate image segmentation. Transformer can better combine global information and improve the problem that CNN cannot integrate global information, but the huge amount of computation brought by Transforms when performing high-resolution image segmentation is unacceptable. Summary of the invention

[0004] The present invention proposes a weakly supervised image segmentation model, method and computer system that take into account global information, which improves the accuracy and efficiency of image segmentation by combining the advantages of Transformer and CNN.

[0005] First, a weakly supervised image segmentation model that takes global information into account is proposed, including: an input layer, which is configured to accept an image; a preprocessing module, which is configured to preprocess the image, and the preprocessing content includes: increasing the number of channels of the image and standardizing the original low-level features of the image; a convolutional modulator with an attention mechanism, which is configured to perform multiple convolution operations on the preprocessed image, and use residual links to add the high-level features obtained after the last convolution operation to the original low-level features input to the convolutional modulator during the first convolution to obtain a fused feature; a pooling layer, which is configured to perform a pooling operation on the fused features extracted by the convolutional modulator; and a fully connected layer, which is configured to output the pooled fused feature map to generate a segmentation result.

[0006] Secondly, a weakly supervised image segmentation method that takes into account global information is proposed, including: accepting an image; preprocessing the image, the preprocessing content includes: increasing the number of channels of the image and standardizing the original low-level features of the image; using a convolutional modulator with an attention mechanism to perform multiple convolution operations on the preprocessed image, and using residual links to add the high-level features obtained after the last convolution operation to the original low-level features input to the convolutional modulator during the first convolution to obtain fused features; using a pooling layer to pool the fused features extracted by the convolutional modulator; and using a fully connected layer to output the pooled fused feature map to generate a segmentation result.

[0007] In some examples, the convolutional modulator achieves self-attention by performing Hadamard products on the features extracted by convolution with the original low-level features.

[0008] In some examples, the convolution modulator includes two branches, the first branch includes layer normalization and deformable convolution, and the second branch includes layer normalization. The convolution modulator is configured as follows: the layer normalization of the first branch normalizes or standardizes the original low-level features, and the deformable convolution performs a deformable convolution operation on the normalized or standardized original low-level features; the layer normalization of the second branch normalizes or standardizes the original low-level features; the features extracted after the deformable convolution operation of the first branch are multiplied by the Hadamard product with the original low-level features after normalization or standardization of the second branch to obtain high-level features; finally, a layer normalization is used to normalize or standardize the high-level features.

[0009] In some examples, the deformable convolution is configured to take into account global information of the image.

[0010] In a third aspect, a computer system is proposed, comprising: a processor; a memory comprising one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the weakly supervised image segmentation method that takes into account global information.

[0011] In a fourth aspect, a computer-readable storage medium is proposed for storing non-temporary computer-readable instructions, which, when executed by a computer, can implement the weakly supervised image segmentation method that takes into account global information. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is an architecture diagram of a weakly supervised image segmentation model taking into account global information provided by one embodiment of the present invention.

[0013] Figure 2 It is a labeled sample of true color image data and remote sensing data provided by an embodiment of the present invention.

[0014] Figure 3 Yes Figure 2 Convert the images and labels into the format required for deep learning.

[0015] Figure 4 This is a comparison of segmentation results provided by an embodiment of the present invention (the leftmost column is a true color image, the middle column is the segmentation result of the present invention, and the rightmost column is the segmentation result based on the residual network). DETAILED DESCRIPTION

[0016] The present invention improves CNN into a network including a self-attention mechanism (called: a weakly supervised image segmentation model / lightweight image segmentation model / model / network that takes into account global information), so as to combine the advantages of Transformer and CNN and improve image segmentation accuracy and efficiency. The specific convolution modulator of the model of the present invention has a self-attention mechanism and a larger convolution kernel, which can take into account the global features of the image. In addition, the model of the present invention is lightweight compared to the transform model and the conventional ResNet, runs faster than Transfomer, is more robust than ResNet, and has high application value. The model of the present invention is applied to land use / cover mapping, which is helpful to explore the spatiotemporal variation characteristics of land objects.

[0017] First combine Figure 1 Introduces a weakly supervised image segmentation model that takes into account global information. The model consists of an input layer, a preprocessing module, a convolutional modulator with an attention mechanism, a pooling layer, and a fully connected layer.

[0018] The input layer is configured to accept images.

[0019] The preprocessing module is configured to preprocess the image, including increasing the number of channels of the image and normalizing the original low-level features of the image. Features include the physical contours, edges, colors, textures, etc. of the image, as well as abstract features related to the category. Increasing the number of channels means increasing the number of features. The more features, the more conducive to computer learning, thereby teaching the computer to recognize features. The increase in the number of channels is achieved by increasing the number of convolution kernels. For example, the number of channels can be increased to 64.

[0020] Standardize the features to reduce the impact of discrete features on the overall:

[0021]

[0022] Where x is the matrix of the image, E[x] is the mean of the minimum batch, and Var[x] is the variance of the minimum batch. However, forcing the data distribution in this way will cause some features to be unable to be learned. Therefore, the present invention introduces γ and β, and allows the model to adaptively learn these two parameters and ∈ through back propagation to obtain the output of the BN layer.

[0023] As mentioned above, the features learned by CNN are physical contours, edges, colors, textures, and abstract features related to categories that are recognized in images. Standardization can unify the data of all features into a standard normal distribution, reducing the impact of features with large variance (discrete features) on the overall data distribution, so that the model pays more attention to common features, reduces the learning of discrete features, and improves generalization.

[0024] The convolutional modulator with attention mechanism (ConvMod) is configured to perform convolution operations on the preprocessed image to extract high-level features. The convolutional modulator implements self-attention using a unique Hadamard product. Figure 1 , the convolution modulator contains two branches, the first branch contains layer normalization (Layer Normalization, referred to as LN) and deformable convolution (Dconv), and the second branch contains layer normalization (Layer Normalization, referred to as LN). The LN of the first branch normalizes or standardizes the original low-level features, and then Dconv performs deformable convolution operations on the original low-level features after LN; the LN of the second branch normalizes or standardizes the original low-level features; the features extracted after the deformable convolution operation of the first branch are multiplied by the Hadamard product with the original low-level features after the LN of the second branch to obtain high-level features; finally, an LN is used to normalize or standardize the high-level features. Note that the standardization in the convolution modulator is different from the normalization of ordinary convolution. Experiments have found that using ordinary BN in the convolution modulator will reduce accuracy.

[0025] The present invention directly uses k×k depth convolution to generate weights and re-weighs the values ​​through Hadamard product:

[0026] Z=A⊙V,

[0027] A=DConv k×k (W1X)

[0028] V=W2X

[0029] Where W1 and W2 are the weight matrices of two linear layers (two branches), and ⊙ represents the Hadamard product. k×k Represents a depthwise convolution with a kernel size of k×k. The convolution modulation operation associates each spatial position (h, w) with all pixels in a square area centered at k×k, where h represents the height of the image and w represents the width of the image. Information interaction between channels can be achieved through linear layers. The output of each spatial position is the weighted sum of all pixels in the square area.

[0030] The convolution modulator has a large convolution kernel, which can take into account the global information of the image. For example, the size of Dconv is (11, 11). The convolution modulator performs multiple convolution operations on the preprocessed image to increase the amount of information and improve the accuracy of segmentation. The preferred number of convolution operations in this embodiment is 2. The high-level features obtained after the last convolution operation are added to the low-level features corresponding to the input to the convolution modulator during the first convolution using residual links to obtain fused features. The following is the fusion formula:

[0031]

[0032] Where y is the output, represents the features after convolution, x represents the original features; W s is the parameter, x passes through W s The output shape after processing is the same as that after convolution, and addition operations can be performed.

[0033] The pooling layer is configured to perform a pooling operation on the fused features obtained by the convolutional modulator, such as a global average pooling operation. Global average pooling is a pooling operation that aggregates the information of the entire feature map. The global average pooling layer does not set the pooling window size, but directly performs average pooling on the entire feature map. By averaging the feature values ​​of each channel, a pooled feature value corresponding to each channel is generated. The resulting pooled feature vector can be regarded as a global information representation of the entire feature map.

[0034] The fully connected layer is configured to reshape the pooled fusion features and map the output to generate the segmentation result.

[0035] The following is an embodiment of a weakly supervised image segmentation method that takes global information into account. This method is coupled with the aforementioned model, so only a brief description of the method is given below.

[0036] Step 1: Get the image.

[0037] Step 2: preprocess the image, including increasing the number of image channels and standardizing the image features.

[0038] In step 3, a convolutional modulator with an attention mechanism is used to perform multiple convolution operations on the preprocessed image, and a residual link is used to add the high-level features obtained after the last convolution operation to the low-level features input to the convolutional modulator during the first convolution to obtain a fused feature.

[0039] Step 5: Perform a pooling operation on the fused features obtained by the convolutional modulator, such as a global average pooling operation.

[0040] Step 6: Reshape the pooled fusion features and map the output to generate the segmentation result.

[0041] The structure and principle of the model are explained in detail above.

[0042] The present invention also provides an embodiment of a computer. The computer includes a processor and a memory. The memory is used to store non-temporary computer-readable instructions (e.g., one or more computer program modules). The processor is used to run the non-temporary computer-readable instructions, and when the non-temporary computer-readable instructions are run by the processor, one or more steps in the weakly supervised image segmentation method taking into account global information described above can be executed. The memory and the processor can be interconnected through a bus system and / or other forms of connection mechanisms.

[0043] For example, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor may be a general-purpose processor or a special-purpose processor, and may control other components in the computer to perform desired functions.

[0044] For example, the memory may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor may run one or more computer program modules to implement various functions of the computer.

[0045] The present invention also provides an embodiment of a computer-readable storage medium, which is used to store non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, one or more steps in the above-mentioned weakly supervised image segmentation method taking into account global information can be implemented. That is, when the weakly supervised image segmentation method and model taking into account global information provided in the embodiment of the present application are implemented in the form of software and sold or used as an independent product, they can be stored in a computer-readable storage medium. For relevant instructions on the storage medium, please refer to the corresponding description of the memory in the computer above, which will not be repeated here.

[0046] Two types of data were selected, one of which is Figure 2 The RGB three-channel true color image shown in a has a size of 480×640; the other is Figure 2 The Landsat satellite image shown in c has 7 band information and a size of 375×301.

[0047] Using ENVI software Figure 2 a. Add labels to the images shown in 2c. The results are as follows Figure 2 b, 2d. Convert images and labels into the format required for deep learning, such as Figure 3 a, 3b, and then input them into the deep learning model for 10 rounds of model training and evaluation to verify the robustness of the model.

[0048] In order to evaluate the land cover classification results, the confusion matrix is ​​used to count the prediction results and actual results of each classification, and the overall accuracy (OA) and Kappa coefficient are used as evaluation indicators. The calculation formulas of OA and Kappa coefficient are as follows:

[0049]

[0050] Where: x ii is the number of samples correctly classified; N is the total number of samples; po The sum of the number of correctly classified samples in each category divided by the total number of samples is the overall classification accuracy (OA); p e The Kappa coefficient is the sum of the true number of each category multiplied by the predicted number of that category divided by the square of the total number of all categories. The Kappa coefficient falls between 0 and 1 and can be divided into five groups to represent different levels of consistency. As shown in Table 4.

[0051] Table 1 Kappa coefficient level classification

[0052]

[0053] The average Kappa coefficient of 10 experiments on true color images is 0.969, and OA is 0.975; under the remote sensing data set, the Kappa coefficient is 92.46, and OA is 90.68; without increasing the network depth, the model and method of the present invention achieve good segmentation accuracy.

[0054] Figure 4 This is a comparison of segmentation results provided by an embodiment of the present invention (the leftmost column is a true color image, the middle column is the segmentation result of the present invention, and the rightmost column is the segmentation result based on the residual network).

[0055] The model of the present invention is compared with the residual network, and the results are shown in Figure 4 . Figure 4 a. Figure 4 d is the original true color image, Figure 4 b. Figure 4 e is to use the model of the present invention to Figure 4 a. Figure 4 d is the result after segmentation. Figure 4 c. Figure 4 f is the use of ordinary network (residual network) to Figure 4 a. Figure 4 d is the result after segmentation. Figure 4 In a, 4b, and 4c, blue represents the blue sky background, red represents the tower, and green represents the trees. Figure 4 In d, 4e, and 4f, green represents forest land, dark blue represents water bodies, light blue represents bare land, and red represents built-up land.

[0056] like Figure 4The segmentation result of the tower shown in c shows some wrong segmentation in the ordinary convolution, which divides part of the tower into blue sky and green plants. The convolution modulator based on residual link in the model of the present invention can take into account global information well and reduce the noise in large objects because of its special self-attention mechanism and large convolution kernel. The convolution modulator has a good ability to take into account global information, so it can better extract long-distance dependencies between objects. For example, when performing land use / cover mapping, it can extract dirt roads in mountainous areas, such as Figure 4 As shown in the red rounded rectangle in e. Figure 4 e. In the yellow rectangle, the model of the present invention correctly segments the cultivated land and other information; while the common network shows a large number of water areas, which is incorrect. Figure 4 f. Continue to refer Figure 4 e. Figure 4 f, in the blue rectangular box, the model of the present invention correctly segments the bare land, while the common network identifies it as building land. In addition, the segmentation network based on the residual convolutional modulator in the present invention has a smoother segmentation of the river.

[0057] The above verifies that the weakly supervised image segmentation model and method of the present invention that takes into account global information can effectively improve the image segmentation accuracy, especially when performing land use / cover mapping, the output result is more consistent with the real surface. The model of the present invention can take into account global information and extract long-distance dependencies between objects, which is particularly evident when segmenting large objects and slender objects, and the model still maintains a good segmentation effect in a small sample environment.

[0058] Note that when using the model and method of the present invention for land use / cover mapping, a coordinate system needs to be added to the result so that it can be used for geographic analysis. The results show that the model can achieve good segmentation accuracy without increasing the network depth.

Claims

1. A weakly supervised image segmentation model taking into account global information, characterized in that: include: An input layer, which is configured to accept an image; A preprocessing module is configured to preprocess the image, and the preprocessing content includes: increasing the number of channels of the image and standardizing the original low-level features of the image; A convolutional modulator with an attention mechanism is configured to perform multiple convolution operations on the preprocessed image and use residual links to add the high-level features obtained after the last convolution operation to the original low-level features input to the convolutional modulator during the first convolution to obtain fused features; A pooling layer, configured to perform a pooling operation on the fused features extracted by the convolutional modulator; The fully connected layer is configured to output the pooled fused feature map to generate the segmentation result.

2. The weakly supervised image segmentation model taking into account global information according to claim 1, characterized in that The convolutional modulator achieves self-attention by performing Hadamard product between the features extracted by convolution and the original low-level features.

3. The weakly supervised image segmentation model taking into account global information according to claim 2, characterized in that: The convolution modulator includes two branches, the first branch includes layer normalization and deformable convolution, and the second branch includes layer normalization. The convolution modulator is configured as follows: the layer normalization of the first branch normalizes or standardizes the original low-level features, and the deformable convolution performs a deformable convolution operation on the normalized or standardized original low-level features; the layer normalization of the second branch normalizes or standardizes the original low-level features; the features extracted after the deformable convolution operation of the first branch are multiplied by the Hadamard product with the original low-level features normalized or standardized by the second branch to obtain high-level features; finally, a layer normalization is used to normalize or standardize the high-level features.

4. The weakly supervised image segmentation model taking into account global information according to claim 2, characterized in that The deformable convolution is configured to take into account the global information of the image.

5. A weakly supervised image segmentation method taking into account global information, characterized in that: include: Accept the image; Preprocess the image, including: increasing the number of image channels and standardizing the original low-level features of the image; The convolution modulator with attention mechanism is used to perform multiple convolution operations on the preprocessed image, and the high-level features obtained after the last convolution operation are added to the original low-level features input to the convolution modulator during the first convolution using residual links to obtain fused features. Use the pooling layer to perform pooling operation on the fused features extracted by the convolutional modulator; The fully connected layer is used to output the pooled fusion feature map to generate the segmentation result.

6. The weakly supervised image segmentation method taking into account global information according to claim 5, characterized in that: The convolutional modulator achieves self-attention by performing Hadamard product between the features extracted by convolution and the original low-level features.

7. The weakly supervised image segmentation method taking into account global information according to claim 6, characterized in that: The convolution modulator includes two branches. The first branch includes layer normalization and deformable convolution, and the second branch includes layer normalization. The layer normalization of the first branch normalizes or standardizes the original low-level features, and the deformable convolution performs a deformable convolution operation on the normalized or standardized original low-level features; the layer normalization of the second branch normalizes or standardizes the original low-level features; the features extracted after the deformable convolution operation of the first branch are multiplied by the Hadamard product with the original low-level features after normalization or standardization of the second branch to obtain high-level features; finally, a layer normalization is used to normalize or standardize the high-level features.

8. The weakly supervised image segmentation method taking into account global information according to claim 7, characterized in that: Deformable convolution can take into account the global information of the image.

9. A computer system, characterized in that: include: processor; a memory including one or more computer program modules; Wherein, the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the weakly supervised image segmentation method taking into account global information as described in any one of claims 5-8.

10. A computer-readable storage medium for storing non-transitory computer-readable instructions, characterized in that: When the non-transitory computer-readable instructions are executed by a computer, the weakly supervised image segmentation method taking into account global information as described in any one of claims 5 to 8 can be implemented.

Images