Weakly supervised image segmentation model, method and computer system considering global information

Abstract

The model includes: an input layer configured to receive an image; a preprocessing module configured to preprocess the image, including increasing the number of image channels and normalizing the original low-level features of the image; a convolutional modulator with an attention mechanism configured to perform multiple convolution operations on the preprocessed image and, using residual links, add the high-level features obtained after the last convolution operation to the original low-level features input to the convolutional modulator during the first convolution to obtain fused features; a pooling layer configured to perform a pooling operation on the fused features extracted by the convolutional modulator; and a fully connected layer configured to output the pooled fused feature map to generate a segmentation result. The present invention improves image segmentation accuracy and efficiency by combining the advantages of Transformer and CNN.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and relates to weakly supervised image segmentation taking into account global information and global feature extraction based on deep learning, and specifically to a weakly supervised image segmentation model, method and computer system taking into account global information. Background Art

[0002] Image segmentation is a key technology in computer vision, aiming to divide an image into regions with similar properties or functions. With the rapid development of deep learning technology, image segmentation methods have achieved significant breakthroughs. By absorbing large amounts of image data, deep learning models can automatically learn image feature representations, thereby achieving accurate image segmentation. Image segmentation has shown broad application prospects in various fields. For example, in medical image analysis, image segmentation can help doctors diagnose diseases more accurately; in autonomous driving, image segmentation can help vehicles better understand their surroundings; and in satellite imagery analysis, image segmentation can help researchers better understand the structure and changes of the Earth's surface. However, deep learning-based image segmentation methods also face challenges. For example, how to design more effective network structures to improve segmentation accuracy and efficiency? How to overcome the variability between different objects? How to balance model complexity and computing resource requirements?

[0003] CNNs are currently a popular network for image segmentation. When CNNs segment images, increasing network depth and designing different dilation rates to improve receptive field can improve segmentation accuracy. However, these methods are often limited by their specific convolution kernel size, which prevents them from integrating global information. In the Transformer model, the self-attention mechanism enables the model to learn the dependencies between different locations, thereby achieving more accurate image segmentation. Transformers can better integrate global information, improving CNN's inability to integrate global information. However, the enormous computational effort required by Transformers for high-resolution image segmentation is unacceptable. Summary of the Invention

[0004] The present invention proposes a weakly supervised image segmentation model, method and computer system that takes global information into account, which improves the accuracy and efficiency of image segmentation by combining the advantages of Transformer and CNN.

[0005] First, a weakly supervised image segmentation model that takes global information into account is proposed, including: an input layer, which is configured to accept an image; a preprocessing module, which is configured to preprocess the image, and the preprocessing content includes: increasing the number of channels of the image and standardizing the original low-level features of the image; a convolutional modulator with an attention mechanism, which is configured to perform multiple convolution operations on the preprocessed image, and use residual links to add the high-level features obtained after the last convolution operation to the original low-level features input to the convolution modulator during the first convolution to obtain a fused feature; a pooling layer, which is configured to perform a pooling operation on the fused features extracted by the convolutional modulator; and a fully connected layer, which is configured to output the pooled fused feature map to generate a segmentation result.

[0006] Secondly, a weakly supervised image segmentation method that takes into account global information is proposed, including: accepting an image; preprocessing the image, the preprocessing content includes: increasing the number of channels of the image and standardizing the original low-level features of the image; using a convolutional modulator with an attention mechanism to perform multiple convolution operations on the preprocessed image, and using residual links to add the high-level features obtained after the last convolution operation to the original low-level features input to the convolutional modulator during the first convolution to obtain fused features; using a pooling layer to pool the fused features extracted by the convolutional modulator; and using a fully connected layer to output the pooled fused feature map to generate a segmentation result.

[0007] In some examples, the convolutional modulator implements self-attention by performing Hadamard products on the features extracted by convolution with the original low-level features.

[0008] In some examples, the convolution modulator includes two branches, the first branch includes layer normalization and deformable convolution, and the second branch includes layer normalization, and the convolution modulator is configured as follows: the layer normalization of the first branch normalizes or standardizes the original low-level features, and the deformable convolution performs a deformable convolution operation on the normalized or standardized original low-level features; the layer normalization of the second branch normalizes or standardizes the original low-level features; the features extracted after the deformable convolution operation of the first branch are multiplied by the Hadamard product with the original low-level features after normalization or standardization of the second branch to obtain high-level features; finally, a layer normalization is used to normalize or standardize the high-level features.

[0009] In some examples, deformable convolution is configured to take into account global information of the image.

[0010] In a third aspect, a computer system is proposed, comprising: a processor; a memory comprising one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the weakly supervised image segmentation method that takes into account global information.

[0011] In a fourth aspect, a computer-readable storage medium is proposed for storing non-transitory computer-readable instructions, which, when executed by a computer, can implement the weakly supervised image segmentation method that takes into account global information. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 4 is an architecture diagram of a weakly supervised image segmentation model taking into account global information provided by one embodiment of the present invention.

[0013] Figure 2 This is a labeled sample of true color image data and remote sensing data provided by an embodiment of the present invention.

[0014] Figure 3 It will Figure 2 Convert the images and labels into the format required for deep learning.

[0015] Figure 4 This is a comparison of segmentation results provided by an embodiment of the present invention (the leftmost column is the true color image, the middle column is the segmentation result of the present invention, and the rightmost column is the segmentation result based on the residual network). DETAILED DESCRIPTION

[0016] The present invention improves CNN into a network that includes a self-attention mechanism (called: a weakly supervised image segmentation model that takes into account global information / a lightweight image segmentation model / model / network), combining the advantages of Transformer and CNN to improve image segmentation accuracy and efficiency. The specific convolutional modulator of the model of the present invention has a self-attention mechanism and a larger convolution kernel, which can well take into account the global characteristics of the image. In addition, compared with the transform model and the conventional ResNet, the model of the present invention is lightweight, runs faster than Transformer, is more robust than ResNet, and has high application value. The model of the present invention is applied to land use / cover mapping, which helps to explore the spatiotemporal variation characteristics of land features.

[0017] First combine Figure 1 This paper introduces a weakly supervised image segmentation model that takes into account global information. The model consists of an input layer, a preprocessing module, a convolutional modulator with an attention mechanism, a pooling layer, and a fully connected layer.

[0018] The input layer is configured to accept images.

[0019] The preprocessing module is configured to preprocess the image, including increasing the number of channels and normalizing the image's raw low-level features. Features include the image's physical contours, edges, color, texture, and other features, as well as abstract features related to categories. Increasing the number of channels means adding more features, and more features facilitate computer learning, thereby teaching the computer to recognize features. Increasing the number of channels is achieved by increasing the number of convolution kernels. For example, the number of channels can be increased to 64.

[0020] Standardize the features to reduce the impact of discrete features on the overall:

[0021]

[0022] in is the matrix of the image, is the mean of the minimum batch, is the variance of the minimum batch. However, forcing the data distribution in this way will cause some features to be unlearnable. Therefore, this paper introduces and , and let the model adaptively learn these two parameters and , to obtain the output of the BN layer.

[0023] As mentioned earlier, the features learned by CNNs are physical features such as outlines, edges, colors, and textures that are recognized in images, as well as abstract features related to categories. Normalization can unify the data of all features into a standard normal distribution, reducing the impact of features with large variance (discrete features) on the overall data distribution. This allows the model to focus more on common features, reducing the learning of discrete features and improving generalization.

[0024] The Convolutional Modulator (ConvMod) with attention mechanism is configured to perform convolution operations on the pre-processed image to extract high-level features. The Convolutional Modulator uses the unique Hadamard product to achieve self-attention. Figure 1The convolutional modulator contains two branches. The first branch contains layer normalization (LN) and deformable convolution (Dconv), and the second branch contains layer normalization (LN). The LN in the first branch normalizes or standardizes the original low-level features, and then Dconv performs a deformable convolution operation on the original low-level features after LN. The LN in the second branch normalizes or standardizes the original low-level features. The features extracted after the deformable convolution operation of the first branch are multiplied with the original low-level features after the LN operation of the second branch using the Hadamard product to obtain high-level features. Finally, another LN is used to normalize or standardize the high-level features. Note that the normalization in the convolutional modulator is different from the normalization of ordinary convolution. Experiments have found that using ordinary BN in the convolutional modulator will reduce accuracy.

[0025] The present invention directly uses k × k depth convolution to generate weights and reweighs the values ​​through Hadamard product:

[0026]

[0027] in, and is the weight matrix of two linear layers (two branches), represents the Hadamard product. represents a depthwise convolution with a kernel size of k×k. The convolutional modulation operation associates each spatial location (h, w) with all pixels within a square region centered at k×k, where h represents the image height and w represents the image width. Information exchange between channels is achieved through linear layers. The output at each spatial location is the weighted sum of all pixels within the square region.

[0028] The convolution modulator has a large convolution kernel, which can well take into account the global information of the image. For example, the size of Dconv is (11, 11). The convolution modulator performs multiple convolution operations on the preprocessed image to increase the amount of information and improve the accuracy of segmentation. In this embodiment, the preferred number is 2. The high-level features obtained after the last convolution operation are added to the corresponding low-level features input to the convolution modulator during the first convolution using residual links to obtain fused features. The following is the fusion formula:

[0029]

[0030] in is the output, Represents the features after convolution processing, Represents the original feature; As parameter, x passes through The output shape after processing is the same as that after convolution, and addition operations can be performed.

[0031] The pooling layer is configured to perform a pooling operation, such as global average pooling, on the fused features obtained by the convolutional modulator. Global average pooling is a pooling operation that aggregates information from the entire feature map. The global average pooling layer does not set a pooling window size; instead, it directly average pools the entire feature map. By averaging the feature values ​​of each channel, a pooled feature value corresponding to each channel is generated. The resulting pooled feature vector can be viewed as a global representation of the entire feature map.

[0032] The fully connected layer is configured to reshape the pooled fusion features and map the output to generate the segmentation result.

[0033] The following describes an embodiment of a weakly supervised image segmentation method that takes global information into account. This method is coupled with the aforementioned model, so only a brief description of the method is given below.

[0034] Step 1: Get the image.

[0035] Step 2: Preprocess the image, including increasing the number of image channels and standardizing the image features.

[0036] In step 3, the convolutional modulator with attention mechanism is used to perform multiple convolution operations on the preprocessed image, and the high-level features obtained after the last convolution operation are added to the low-level features input to the convolutional modulator during the first convolution using residual links to obtain fused features.

[0037] Step 5: Perform a pooling operation on the fused features obtained by the convolutional modulator, such as a global average pooling operation.

[0038] Step 6: Reshape the pooled fusion features and map the output to generate the segmentation result.

[0039] The present invention also provides an embodiment of a computer. The computer includes a processor and a memory. The memory is configured to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor is configured to execute the non-transitory computer-readable instructions. When executed by the processor, the non-transitory computer-readable instructions can perform one or more steps of the weakly supervised image segmentation method that considers global information described above. The memory and the processor can be interconnected via a bus system and / or other connection mechanisms.

[0040] For example, a processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other processing unit with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) can be an X86 or ARM architecture. The processor can be a general-purpose processor or a special-purpose processor, and can control other components in the computer to perform desired functions.

[0041] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory, a flash memory, and the like. One or more computer program modules may be stored on the computer-readable storage medium, and the processor may execute one or more computer program modules to implement various computer functions.

[0042] The present invention also provides an embodiment of a computer-readable storage medium, which is used to store non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, one or more steps in the above-mentioned weakly supervised image segmentation method taking into account global information can be implemented. That is, when the weakly supervised image segmentation method and model taking into account global information provided in the embodiment of the present application are implemented in the form of software and sold or used as an independent product, they can be stored in a computer-readable storage medium. For relevant instructions on the storage medium, please refer to the corresponding description of the memory in the computer above, which will not be repeated here.

[0043] Two types of data were selected, one of which is Figure 2 The RGB three-channel true color image shown in a has a size of 480×640; the other is Figure 2 The Landsat satellite image shown in c has 7 band information and a size of 375×301.

[0044] Using ENVI software Figure 2 The images shown in a and 2c are labeled, and the results are as follows Figure 2 b, 2d. Convert images and labels into the format required for deep learning, as shown in Figure 3 a, 3b, and then input them into the deep learning model for 10 rounds of model training and evaluation to verify the robustness of the model.

[0045] To evaluate the land cover classification results, the confusion matrix is ​​used to perform statistics on the predicted results and actual results of each classification, and the overall accuracy (OA) and Kappa coefficient are used as evaluation indicators. The calculation formulas for OA and Kappa coefficient are as follows:

[0046]

[0047] Карра

[0048] Where: is the number of samples classified correctly; N is the total number of samples; The sum of the number of correctly classified samples for each class divided by the total number of samples is the overall classification accuracy (OA); The Kappa coefficient is the sum of the true number of each category multiplied by the sum of the predicted number of that category, divided by the square of the total number of categories. The Kappa coefficient ranges from 0 to 1 and can be divided into five groups to represent different levels of consistency, as shown in Table 4.

[0049] Table 1 Kappa coefficient level classification

[0050]

[0051] The average Kappa coefficient of 10 experiments on true color images is 0.969, and OA is 0.975; under the remote sensing dataset, the Kappa coefficient is 92.46, and OA is 90.68; without increasing the network depth, the model and method of the present invention achieve good segmentation accuracy.

[0052] Figure 4 This is a comparison of segmentation results provided by an embodiment of the present invention (the leftmost column is the true color image, the middle column is the segmentation result of the present invention, and the rightmost column is the segmentation result based on the residual network).

[0053] The model of the present invention is compared with the residual network, and the results are referenced. Figure 4 . Figure 4 a. Figure 4 d is the original true color image, Figure 4 b. Figure 4 e is the model of the present invention respectively Figure 4 a. Figure 4 d is the result after segmentation, Figure 4 c. Figure 4 f is the use of ordinary network (residual network) to Figure 4 a. Figure 4 d is the result after segmentation. Figure 4 In a, 4b, and 4c, blue represents the background blue sky, red represents the tower, and green represents the trees. Figure 4In d, 4e, and 4f, green represents forest land, dark blue represents water bodies, light blue represents bare land, and red represents built-up land.

[0054] like Figure 4 In the segmentation result of the tower shown in c, ordinary convolution has some mis-segmentation, dividing part of the tower into blue sky and green plants. The convolution modulator based on residual link in the model of the present invention, due to its special self-attention mechanism and large convolution kernel, can well take into account global information and thus reduce the noise within large objects. The convolution modulator has a good ability to take into account global information, so it can better extract long-distance dependencies between objects. For example, it can extract dirt roads in mountainous areas when performing land use / cover mapping. Figure 4 As shown in the red rounded rectangle in e. Figure 4 e. In the yellow rectangle, the proposed model correctly segments the cultivated land and other information; while the common network shows a large number of water areas, which is incorrect. Figure 4 f. Continue to refer Figure 4 e. Figure 4 f, In the blue rectangle, the model of the present invention correctly segments the bare land, while the conventional network identifies it as built-up land. In addition, the segmentation network based on the residual convolutional modulator in the present invention has a smoother segmentation of the river.

[0055] The above demonstrates that the proposed weakly supervised image segmentation model and method, which considers global information, effectively improves image segmentation accuracy, particularly when performing land use / cover mapping, resulting in output that is more consistent with the actual surface. The proposed model is able to consider global information and extract long-range dependencies between objects, a feature that is particularly evident when segmenting large and slender objects. Furthermore, the model maintains good segmentation results even with small sample sizes.

[0056] Note that when using the model and method of the present invention for land use / cover mapping, a coordinate system needs to be added to the results to enable geographic analysis. The results show that the model can achieve good segmentation accuracy without increasing the network depth.

Claims

1. A weakly supervised image segmentation model that takes into account global information, characterized in that: include: An input layer, which is configured to accept an image; A preprocessing module is configured to preprocess the image, including increasing the number of channels of the image and normalizing the original low-level features of the image; A convolutional modulator with an attention mechanism is configured to perform multiple convolution operations on the preprocessed image, and use a residual link to add the high-level features obtained after the last convolution operation to the original low-level features input to the convolution modulator during the first convolution to obtain a fusion feature, wherein the convolutional modulator performs a Hadamard product on the features extracted by convolution and the original input features to achieve self-attention. The convolutional modulator includes two branches, the first branch includes layer normalization and deformable convolution, and the second branch includes layer normalization. The convolutional modulator is configured as follows: the layer normalization of the first branch normalizes or standardizes the original input features, and the deformable convolution performs a deformable convolution operation on the normalized or standardized original input features; the layer normalization of the second branch normalizes or standardizes the original input features; the features extracted after the deformable convolution operation of the first branch are multiplied by the Hadamard product with the normalized or standardized original input features of the second branch to obtain high-level features; finally, a layer normalization is used to normalize or standardize the high-level features; A pooling layer configured to perform a pooling operation on the fused features extracted by the convolutional modulator; The fully connected layer is configured to transform the size of the pooled fusion features and map the output to generate the segmentation result.

2. The weakly supervised image segmentation model taking into account global information according to claim 1, characterized in that The size of the deformable convolution is (11, 11), which can take into account the global information of the image.

3. A weakly supervised image segmentation method taking into account global information, characterized in that: include: Accept the image; Preprocess the image, including increasing the number of image channels and standardizing the original low-level features of the image; A convolution modulator with an attention mechanism is used to perform multiple convolution operations on the preprocessed image, and a residual link is used to add the high-level features obtained after the last convolution operation to the original low-level features input to the convolution modulator during the first convolution to obtain a fusion feature, wherein the convolution modulator performs a Hadamard product on the features extracted by convolution and the original input features to achieve self-attention. The convolution modulator includes two branches, the first branch includes layer normalization and deformable convolution, and the second branch includes layer normalization. The convolution modulator is configured as follows: the layer normalization of the first branch normalizes or standardizes the original input features, and the deformable convolution performs a deformable convolution operation on the normalized or standardized original input features; the layer normalization of the second branch normalizes or standardizes the original input features; the features extracted after the deformable convolution operation of the first branch are multiplied by the Hadamard product with the normalized or standardized original input features of the second branch to obtain high-level features; finally, a layer normalization is used to normalize or standardize the high-level features; Use the pooling layer to perform pooling operation on the fusion features extracted by the convolutional modulator; The fully connected layer is used to transform the size of the pooled fusion features and map the output to generate the segmentation result.

4. The weakly supervised image segmentation method according to claim 3, wherein: The size of the deformable convolution is (11, 11), which can take into account the global information of the image.

5. A computer system, characterized in that: include: processor; a memory comprising one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the weakly supervised image segmentation method taking into account global information as described in any one of claims 3-4.

6. A computer-readable storage medium for storing non-transitory computer-readable instructions, characterized in that: When the non-transitory computer-readable instructions are executed by a computer, the weakly supervised image segmentation method taking into account global information as described in any one of claims 3-4 can be implemented.

Images