Optical image segmentation method based on ResUnit + + model

Abstract

The invention discloses an optical image segmentation method based on a ResUnit + + model, and the method comprises the following steps: image preprocessing: marking a cloud layer label and a snow layer label on image data, and dividing the image data into a three-channel data set which is formed by stacking a background mask, a cloud layer mask and a snow layer mask; feature extraction: dividing the preprocessed data set into a training set and a test set, and inputting the training set into a ResUnit + + model for feature extraction so as to identify feature information corresponding to a background, a cloud layer or a snow layer in the image data; target segmentation and classification: taking the extracted feature information as an index of a final segmentation and classification task of a cloud layer and a snow layer, and outputting a segmented image of the image by a ResUnit + + model based on the extracted feature information so as to divide each pixel point in the image into a background, the cloud layer or the snow layer; in the feature fusion process, high-level semantic information can be better reserved, and meanwhile, a more accurate segmentation result is obtained in combination with low-level detail information.

Description

Technical Field

[0001] The present invention relates to the technical field of optical image recognition, and in particular to an optical image segmentation method based on a ResUnet++ model. Background Art

[0002] Accurate cloud and snow layer identification helps improve the accuracy of surface cover classification and supports applications such as weather forecasting, water resources management, and environmental monitoring. Traditional cloud and snow layer identification methods mainly rely on manual feature extraction and threshold segmentation, which have certain limitations, such as difficulty in feature extraction, poor robustness, and poor adaptability to complex scenes. In recent years, the rise of deep learning, especially convolutional neural networks (CNN), has provided new solutions for cloud and snow layer identification.

[0003] UNet and UNet++ are two commonly used convolutional neural network architectures, which are widely used in medical image segmentation and remote sensing image segmentation tasks. Through the encoder-decoder structure, they gradually extract high-level features of the image and aggregate contextual information to achieve accurate pixel-level segmentation.

[0004] The data labeling process using UNet and UNet++ architecture models requires a lot of manual intervention and is easily affected by subjective factors, resulting in inconsistent labeling, low quality of training data, and difficulty in learning accurate cloud and snow layer features. In addition, although UNet and UNet++ architecture models introduce skip connections and dense skip connections, their feature extraction and fusion capabilities are still insufficient when faced with very complex boundaries and details, and are prone to misclassification and omissions. Especially in the cloud and snow layer segmentation tasks, improper handling of complex boundaries and details will significantly reduce the accuracy of the model. Summary of the invention

[0005] The purpose of the present invention is to provide an optical image segmentation method based on the ResUnet++ model to solve the technical problem that in the prior art, in the segmentation task of cloud and snow layers, improper processing of complex boundaries and details will significantly reduce the accuracy of the model.

[0006] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0007] An optical image segmentation method based on the ResUnet++ model comprises the following steps:

[0008] Step 100, image preprocessing: marking cloud layer labels and snow layer labels on the image data, and dividing the image data into a three-channel data set stacked by background, cloud layer, and snow layer masks;

[0009] Step 200, feature extraction: the preprocessed data set is divided into a training set and a test set, and the training set is input into the ResUnet++ model for feature extraction to identify feature information corresponding to the background, cloud layer or snow layer in the image data;

[0010] Step 300, target segmentation and classification: The extracted feature information is used as an indicator for the final segmentation and classification task of the cloud layer and the snow layer. The ResUnet++ model outputs a segmented image of the image based on the extracted feature information so that each pixel in the image is divided into the background, cloud layer or snow layer.

[0011] As a preferred solution of the present invention, in step 100, the steps of dividing the image data into a three-channel data set stacked by background, cloud layer, and snow layer masks are as follows:

[0012] Step 101, loading image data: opening each input remote sensing image tif file, extracting data of different bands therefrom, wherein the extracted bands are red, green and blue bands, forming a basic RGB image;

[0013] Step 102, reading label data: loading label files of cloud layers and snow layers associated with each input image, wherein label data for indicating whether each pixel belongs to a cloud layer or a snow layer is set in the label file;

[0014] Step 103, stacking band data: combining the red, green and blue band data together to form three-channel image data;

[0015] Step 104, processing label data: reading the label files of cloud layer and snow layer into binary masks, where each pixel represents the background, or cloud layer and snow layer, respectively, to generate a multi-channel mask matrix;

[0016] Step 105, merging masks: stacking masks of different categories corresponding to the background, cloud layer and snow layer into a multi-channel matrix to meet the label requirements in the multi-class segmentation task.

[0017] As a preferred solution of the present invention, after the band data are stacked in step 103 to form three-channel image data, the image data is processed in the following manner:

[0018] Convert data format: convert the arrangement of image data from height-width-channel to channel-height-width;

[0019] Data type conversion: Convert red, green and blue band data into floating point format;

[0020] Normalization: Scale the pixel values ​​of image data from the original integer range to between 0 and 1.

[0021] As a preferred solution of the present invention, after the mask is merged in step 105, both the image data and the label data are converted into PyTorch tensors, and the image data and label data converted into PyTorch tensors are returned as a pair of input parameters and target parameters (x, y), so that the preprocessed background, cloud layer and snow layer are saved as a data set in the format of number (x, y).

[0022] As a preferred solution of the present invention, before step 200, image data and corresponding cloud layer labels and snow layer labels are obtained according to the index, and the image data and the corresponding cloud layer labels and snow layer labels are visualized to check the integrity of the image data and the corresponding cloud layer labels and snow layer labels before the feature extraction step.

[0023] As a preferred solution of the present invention, in step 200, the implementation steps of using the ResUnet++ model to extract features from the training set are specifically as follows:

[0024] Convolution operation: The ResUnet++ model processes the image data layer by layer through multiple convolutional layers to extract multi-scale spatial features, where the spatial features include the boundaries and texture information of clouds and snow layers;

[0025] Downsampling and upsampling: The ResUnet++ model gradually reduces the resolution of feature maps through downsampling layers during the convolution process, and restores the resolution through upsampling in the decoder stage.

[0026] As a preferred solution of the present invention, during the convolution operation, residual blocks are used to alleviate the gradient vanishing problem in the deep network, and all residual blocks are connected by jumps, so that the network can maintain the original input information when extracting high-order features;

[0027] The features of each channel are weighted and adjusted to enhance the ability to extract important features, so that the network can more accurately identify the features of different objects;

[0028] Using receptive fields of different scales for multi-scale feature extraction can capture richer contextual information.

[0029] As a preferred solution of the present invention, in step 300, the steps of dividing each pixel point in the image into background, cloud layer or snow layer are as follows:

[0030] The skip connection mechanism in the ResUnet++ model enables the high-resolution features of the shallow layer to be combined with the low-resolution features of the deep layer;

[0031] The ResUnet++ model outputs a multi-channel mask through the last convolutional layer, and normalizes the output multi-channel mask through an activation function to obtain the classification probability of each pixel of the image;

[0032] The cloud layer probability and snow layer probability in the multi-channel mask output by the ResUnet++ model are compared with the actual label data corresponding to the cloud layer and snow layer using the cross entropy loss function;

[0033] Adjust the weight of the ResUnet++ model according to the comparison result output by the cross entropy loss function, and repeat the above work until the comparison result output by the cross entropy loss function meets the expectation;

[0034] The adjusted ResUnet++ model outputs the classification results of background, cloud layer and snow layer.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The ResUnet++ model proposed in the present invention further optimizes feature fusion. During the feature fusion process, the ResUnet++ model can better retain high-level semantic information and combine low-level detail information to obtain more accurate segmentation results. Due to the introduction of residual blocks and improved feature fusion modules, the ResUnet++ model can perform more robustly when processing complex and noisy images, has stronger generalization ability, can improve training efficiency and model reasoning speed to a certain extent, and can usually provide faster training convergence speed and higher reasoning efficiency due to its effective network design and feature extraction capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0038] Figure 1 FIG. 4 is a flow chart of an optical image segmentation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, the present invention provides an optical image segmentation method based on the ResUnet++ model, comprising the following steps:

[0041] Step 100, image preprocessing: mark cloud layer labels and snow layer labels on the image data, and divide the image data into a three-channel data set stacked by background, cloud layer, and snow layer masks.

[0042] In step 100, the steps for dividing the image data into a three-channel data set stacked by background, cloud layer, and snow layer masks are as follows:

[0043] Step 101, loading image data: opening each input remote sensing image tif file, extracting data of different bands therefrom, wherein the extracted bands are red, green and blue bands, forming a basic RGB image.

[0044] Step 102, read label data: load label files of cloud layers and snow layers associated with each input image, wherein label data indicating whether each pixel belongs to a cloud layer or a snow layer is set in the label file.

[0045] Step 103, stacking band data: combining the red, green and blue band data together to form three-channel image data. If necessary, other bands, such as near-infrared bands, may be added later to expand the data dimension.

[0046] After the band data are stacked in step 103 to form three-channel image data, the image data is processed in the following manner:

[0047] Convert data format: Convert the arrangement of image data from height-width-channel to channel-height-width. This conversion is to adapt to the input format requirements of deep learning frameworks such as PyTorch.

[0048] Data Type Conversion: Convert the red, green, and blue band data to floating point format to maintain higher precision during subsequent processing.

[0049] Normalization: Scale the pixel values ​​of the image data from the original integer range to between 0 and 1. For example, convert the image data to a grayscale image and scale the pixel values ​​from the original integer range (such as 0 to 255) to between 0 and 1. The purpose of normalization is to accelerate the convergence process of the model and ensure that the numerical range of the input data is consistent.

[0050] Step 104, processing label data: reading the label files of the cloud layer and the snow layer into a binary mask, where each pixel represents the background, or the cloud layer and the snow layer, and generating a multi-channel mask matrix.

[0051] Step 105, merging masks: stacking masks of different categories corresponding to the background, cloud layer and snow layer into a multi-channel matrix to meet the label requirements in the multi-class segmentation task.

[0052] After merging the masks in step 105, both the image data and the label data are converted into PyTorch tensors to facilitate subsequent calculations in the ResUnet++ model, and the image data and label data converted into PyTorch tensors are returned as a pair of input parameters and target parameters (x, y), so that the preprocessed background, cloud layer and snow layer are saved as a data set in the format of (x, y).

[0053] Before step 200, the image data and the corresponding cloud layer labels and snow layer labels are obtained according to the index, and the image data and the corresponding cloud layer labels and snow layer labels are visualized to check the integrity of the image data and the corresponding cloud layer labels and snow layer labels before the feature extraction step, ensure that the distribution of the cloud and snow data input into the ResUnet++ model and the cloud and snow data corresponding to the original label data is consistent with expectations, and can intuitively discover possible errors or anomalies.

[0054] Step 200, feature extraction: the preprocessed data set is divided into a training set and a test set, and the training set is input into the ResUnet++ model for feature extraction to identify the feature information corresponding to the background, cloud layer or snow layer in the image data.

[0055] In step 200, the steps for implementing feature extraction from the training set using the ResUnet++ model are specifically as follows:

[0056] Convolution operation: The ResUnet++ model processes the image data layer by layer through multiple convolutional layers to extract multi-scale spatial features, where the spatial features include the boundaries and texture information of clouds and snow layers.

[0057] Downsampling and upsampling: The ResUnet++ model gradually reduces the resolution of the feature map through the downsampling layer during the convolution process, and restores the resolution through upsampling in the decoder stage, which can effectively preserve global and local features.

[0058] During the convolution operation, residual blocks are used to alleviate the gradient vanishing problem in deep networks. All residual blocks are connected through jumps, allowing the network to maintain the original input information while extracting high-order features.

[0059] The features of each channel are weighted and adjusted to enhance the ability to extract important features, so that the network can more accurately identify the features of different objects.

[0060] Using receptive fields of different scales for multi-scale feature extraction can capture richer contextual information.

[0061] Step 300, target segmentation and classification: The extracted feature information is used as an indicator for the final segmentation and classification task of the cloud layer and the snow layer. The ResUnet++ model outputs a segmented image of the image based on the extracted feature information so that each pixel in the image is divided into the background, cloud layer or snow layer.

[0062] In step 300, each pixel in the image is divided into background, cloud layer or snow layer by the following steps:

[0063] The skip connection mechanism in the ResUnet++ model enables the high-resolution features of the shallow layer to be combined with the low-resolution features of the deep layer;

[0064] The ResUnet++ model outputs a multi-channel mask through the last convolutional layer, and normalizes the output multi-channel mask through an activation function to obtain the classification probability of each pixel of the image;

[0065] The cloud layer probability and snow layer probability in the multi-channel mask output by the ResUnet++ model are compared with the actual label data corresponding to the cloud layer and snow layer using the cross entropy loss function;

[0066] Adjust the weight of the ResUnet++ model according to the comparison result output by the cross entropy loss function, and repeat the above work until the comparison result output by the cross entropy loss function meets the expectation;

[0067] The adjusted ResUnet++ model outputs the classification results of background, cloud layer and snow layer.

[0068] Among them, ResUNet++ is a deep learning network architecture for image segmentation tasks, which combines the advantages of residual network (ResNet) and U-Net, and introduces some improvements to enhance performance. The specific improvements are:

[0069] ResUNet++ uses residual blocks to enhance the training stability and effect of the network. By introducing the residual blocks of ResNet, ResUnet++ adds direct jump connections in each convolution module. These residual connections help alleviate the gradient vanishing problem, make the network easier to train, and improve the effect of gradient propagation. The residual blocks enable the features to retain more details and contextual information after passing through multiple convolutional layers, thereby improving the expressiveness and accuracy of the model.

[0070] Among them, the ResUNet++ model introduces dense connectivity based on the jump connection of U-Net, which improves the performance of the network through denser feature fusion; by introducing feature maps of different scales for fusion, it further improves the ability to capture multi-scale information.

[0071] That is, ResUnet++ further optimizes feature fusion, and by introducing DenseASPP and SE modules, multi-scale and global context information are better integrated. Such a design can more comprehensively capture the features of different levels of the image, thereby improving segmentation accuracy.

[0072] In the process of feature fusion, ResUnet++ can better retain high-level semantic information, while combining low-level detail information to obtain more accurate segmentation results. Due to the introduction of residual blocks and improved feature fusion modules, ResUnet++ can be more robust and have stronger generalization capabilities when processing complex and noisy images. Compared with UNet and UNet++, ResUnet++ provides higher accuracy and more stable training process, especially in tasks such as medical image segmentation.

[0073] Although ResUnet++ has a complex structure, the introduction of residual modules and optimized feature fusion strategies can improve the training efficiency and model reasoning speed to a certain extent. When processing large-scale data sets, ResUnet++ can usually provide faster training convergence speed and higher reasoning efficiency thanks to its effective network design and feature extraction capabilities.

[0074] When using the image data in the ResUnet++ model training set and the corresponding cloud layer labels and snow layer labels for image segmentation training, the callback function that saves the ResUnet++ model weights is used, and the learning rate scheduler is used to optimize the convergence path of the ResUnet++ model, and the Early Stopping mechanism is used to prevent the ResUnet++ model from overfitting.

[0075] The callback function is used to save the model weights when training the ResUnet++ model. According to the loss of the validation set, the model weights are dynamically saved during the training process to avoid poor performance of the model due to overfitting and other problems after training. It can also help save storage space.

[0076] Therefore, the ResUnet++ model proposed in this embodiment further optimizes feature fusion. During the feature fusion process, the ResUnet++ model can better retain high-level semantic information, and at the same time combine low-level detail information to obtain more accurate segmentation results. Due to the introduction of residual blocks and improved feature fusion modules, the ResUnet++ model can perform more robustly when processing complex and noisy images, has stronger generalization capabilities, and can improve the training efficiency and model reasoning speed to a certain extent. When processing large-scale data sets, thanks to its effective network design and feature extraction capabilities, it can usually provide faster training convergence speed and higher reasoning efficiency.

[0077] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. An optical image segmentation method based on the ResUnet++ model, characterized in that: The following steps are involved: Step 100, image preprocessing: marking cloud layer labels and snow layer labels on the image data, and dividing the image data into a three-channel data set stacked by background, cloud layer, and snow layer masks; Step 200, feature extraction: the preprocessed data set is divided into a training set and a test set, and the training set is input into the ResUnet++ model for feature extraction to identify feature information corresponding to the background, cloud layer or snow layer in the image data; Step 300, target segmentation and classification: The extracted feature information is used as an indicator for the final segmentation and classification task of the cloud layer and the snow layer. The ResUnet++ model outputs a segmented image of the image based on the extracted feature information so that each pixel in the image is divided into the background, cloud layer or snow layer.

2. The optical image segmentation method based on the ResUnet++ model according to claim 1, characterized in that: In step 100, the steps for dividing the image data into a three-channel data set stacked by background, cloud layer, and snow layer masks are as follows: Step 101, loading image data: opening each input remote sensing image tif file, extracting data of different bands therefrom, wherein the extracted bands are red, green and blue bands, forming a basic RGB image; Step 102, reading label data: loading label files of cloud layers and snow layers associated with each input image, wherein label data for indicating whether each pixel belongs to a cloud layer or a snow layer is set in the label file; Step 103, stacking band data: combining the red, green and blue band data together to form three-channel image data; Step 104, processing label data: reading the label files of cloud layer and snow layer into binary masks, where each pixel represents the background, or cloud layer and snow layer, respectively, to generate a multi-channel mask matrix; Step 105, merging masks: stacking masks of different categories corresponding to the background, cloud layer and snow layer into a multi-channel matrix to meet the label requirements in the multi-class segmentation task.

3. The optical image segmentation method based on the ResUnet++ model according to claim 2, characterized in that: After the band data are stacked in step 103 to form three-channel image data, the image data is processed in the following manner: Convert data format: convert the arrangement of image data from height-width-channel to channel-height-width; Data type conversion: Convert red, green and blue band data into floating point format; Normalization: Scale the pixel values ​​of image data from the original integer range to between 0 and 1.

4. The optical image segmentation method based on the ResUnet++ model according to claim 2, characterized in that: After merging the masks in step 105, both the image data and the label data are converted into PyTorch tensors, and the image data and label data converted into PyTorch tensors are returned as a pair of input parameters and target parameters (x, y), so that the preprocessed background, cloud layer and snow layer are saved as a data set in the format of (x, y).

5. The optical image segmentation method based on the ResUnet++ model according to claim 2, characterized in that: Prior to step 200, image data and corresponding cloud layer labels and snow layer labels are obtained according to the index, and the image data and the corresponding cloud layer labels and snow layer labels are visualized to check the integrity of the image data and the corresponding cloud layer labels and snow layer labels before the feature extraction step.

6. The optical image segmentation method based on the ResUnet++ model according to claim 2, characterized in that: In step 200, the steps for implementing feature extraction from the training set using the ResUnet++ model are specifically as follows: Convolution operation: The ResUnet++ model processes the image data layer by layer through multiple convolutional layers to extract multi-scale spatial features, where the spatial features include the boundaries and texture information of clouds and snow layers; Downsampling and upsampling: The ResUnet++ model gradually reduces the resolution of feature maps through downsampling layers during the convolution process, and restores the resolution through upsampling in the decoder stage.

7. The optical image segmentation method based on the ResUnet++ model according to claim 6, characterized in that: During the convolution operation, residual blocks are used to alleviate the gradient vanishing problem in deep networks. All residual blocks are connected through jumps, allowing the network to maintain the original input information while extracting high-order features. The features of each channel are weighted and adjusted to enhance the ability to extract important features, so that the network can more accurately identify the features of different objects; Using receptive fields of different scales for multi-scale feature extraction can capture richer contextual information.

8. The optical image segmentation method based on the ResUnet++ model according to claim 1, characterized in that: In step 300, each pixel in the image is divided into background, cloud layer or snow layer by the following steps: The skip connection mechanism in the ResUnet++ model enables the high-resolution features of the shallow layer to be combined with the low-resolution features of the deep layer; The ResUnet++ model outputs a multi-channel mask through the last convolutional layer, and normalizes the output multi-channel mask through an activation function to obtain the classification probability of each pixel of the image; The cloud layer probability and snow layer probability in the multi-channel mask output by the ResUnet++ model are compared with the actual label data corresponding to the cloud layer and snow layer using the cross entropy loss function; Adjust the weight of the ResUnet++ model according to the comparison result output by the cross entropy loss function, and repeat the above work until the comparison result output by the cross entropy loss function meets the expectation; The adjusted ResUnet++ model outputs the classification results of the background, cloud layer and snow layer.

Images