Optical image planting pattern identification method based on super-resolution reconstruction assistance

Abstract

The invention discloses an optical image planting pattern identification method based on super-resolution reconstruction assistance, and belongs to the technical field of agricultural monitoring. The problem that crop boundary recognition is difficult is solved. The method comprises the steps that a low-resolution remote sensing image and a high-resolution fusion image are collected and preprocessed, and an obtained data set is divided into a training set and a test set; constructing a feature extraction trunk, a multi-scale aggregation module and a feature enhancement module of the multi-task learning network; constructing a super-resolution reconstruction decoding branch of the multi-task learning network; constructing a land utilization classification decoding branch of the multi-task learning network; defining a loss function of the multi-task learning network; inputting the training set into the constructed multi-task learning network, constructing a joint loss function through a weighted summation mode, and carrying out joint training to obtain a trained multi-task learning network; and inputting a remote sensing image to be processed into the output trained multi-task learning network to output a high-resolution image and a land utilization classification result.

Description

Technical Field

[0001] This invention belongs to the field of agricultural monitoring technology, specifically relating to a method for identifying planting patterns in optical images based on super-resolution reconstruction assistance. Background Technology

[0002] With the rapid development of remote sensing technology, the Sentinel-2 satellite, due to its high temporal resolution, multispectral band configuration, and globally free and open data policy, has become an important data source for identifying agricultural planting patterns. However, its multispectral images have a spatial resolution of 10m or 20m, making it difficult to distinguish fragmented and complex intercropping planting patterns at the plot scale. This results in blurred crop boundaries, decreased accuracy in planting pattern classification, and a high proportion of mixed pixels. Under these conditions, the classification accuracy of traditional pixel-level semantic segmentation models is insufficient to meet the needs of refined agricultural monitoring.

[0003] To alleviate the limitations of low-resolution images in ground feature identification, super-resolution reconstruction technology has been gradually introduced into the field of remote sensing image processing. Existing research often employs single-image super-resolution networks to reconstruct Sentinel-2 images, and then inputs the reconstruction results into a semantic segmentation model. However, this two-stage sequential inference approach suffers from problems such as inconsistent objective functions, spectral information distortion, and high computational overhead. Furthermore, the reconstruction stage is prone to over-smoothing texture details, leading to the loss of crop boundary information, limited improvement in classification accuracy, and an inability to meet real-time processing requirements.

[0004] In recent years, multi-task learning frameworks have been widely used in computer vision, achieving joint optimization of multiple related tasks by sharing a core feature extraction mechanism. However, current multi-task methods are still in their early stages in remote sensing image processing, suffering from problems such as insufficient feature sharing, weak boundary perception capabilities, and high computational resource consumption, making them difficult to promote and apply in practical applications. Therefore, there is an urgent need for a method that can simultaneously achieve super-resolution reconstruction and planting pattern classification within a single model, improving crop boundary recognition accuracy, maintaining spectral consistency, and controlling model complexity through joint optimization strategies to meet the needs of large-scale, high-frequency, and refined crop planting pattern monitoring. Summary of the Invention

[0005] This invention aims to address the problems of insufficient spatial resolution of remote sensing images, difficulty in identifying crop boundaries, and limited classification accuracy. It proposes a planting pattern recognition method based on super-resolution reconstruction-assisted optical images.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A planting pattern recognition method based on super-resolution reconstruction-assisted optical image recognition includes the following steps:

[0008] S1. Collect low-resolution remote sensing images and high-resolution fused images of the same time period and the same scene area, preprocess them, manually classify and label the high-resolution fused images, and divide the resulting dataset into training set and test set.

[0009] S2. Construct a feature extraction backbone, a multi-scale aggregation module, and a feature enhancement module for a multi-task learning network. The feature extraction backbone, the multi-scale aggregation module, and the feature enhancement module are connected in sequence to extract deep and shallow features from the input low-resolution remote sensing image.

[0010] S3. Construct the super-resolution reconstruction decoding branch of the multi-task learning network, input the deep features and shallow features obtained in step S2 into the super-resolution reconstruction decoding branch, restore the image spatial resolution through upsampling operation, and output a high-resolution image;

[0011] S4. Construct a land use classification decoding branch of a multi-task learning network. Input the deep and shallow features obtained in step S2 into the land use classification decoding branch, perform pixel-by-pixel semantic category prediction through the decoder, and output the land use classification result.

[0012] S5. Define the super-resolution task loss function and the land use classification task loss function for multi-task learning networks;

[0013] S6. Input the training set obtained in step S1 into the multi-task learning network constructed in steps S2-S5, construct a joint loss function by weighted summation and perform joint training to obtain the trained multi-task learning network.

[0014] S7. Test the trained multi-task learning network obtained in step S6 using the test set, and then input the remote sensing image to be processed into the trained multi-task learning network to output high-resolution images and land use classification results.

[0015] Furthermore, the specific implementation method of step S1 includes the following steps:

[0016] S1.1. Select the study area, collect multispectral images from the European Space Agency's Sentinel-2B satellite of the same time period and the same scene area as low-resolution remote sensing images, and collect high-resolution fused images from China's Gaofen-2 GF-2 satellite as high-resolution images;

[0017] S1.2. Perform atmospheric correction, radiometric normalization and geometric registration on the low-resolution remote sensing image and high-resolution fused image obtained in step S1.1. Then, crop the preprocessed low-resolution remote sensing image into a 256×256 pixel low-resolution remote sensing image block and crop the high-resolution fused image into a 1024×1024 pixel high-resolution fused image block.

[0018] S1.3. Use ArcGIS Pro software to manually interpret the high-resolution fused image patches obtained in step S1.2 to generate land use classification label maps including rice, corn, soybeans and four other land features. Then, combine the low-resolution remote sensing image patches, the high-resolution fused image patches and the corresponding classification label maps to form training and test sets.

[0019] Furthermore, the specific implementation method of step S2 includes the following steps:

[0020] S2.1. Construct the feature extraction backbone of the multi-task learning network. Remove the downsampling operations in the fourth and fifth stages of the standard ResNet-101 model and use it as the feature extraction backbone. Input the low-resolution remote sensing image into the feature extraction backbone. The second stage directly outputs shallow features. Then, connect the multi-scale aggregation module after the last convolutional block of the ResNet-101 model to obtain the output features of the multi-scale aggregation module.

[0021] The multi-scale aggregation module includes four parallel branches: a 3×3 ordinary convolutional branch and three 3×3 dilated convolutional branches with dilation rates of 1, 2 and 3 respectively. The outputs of the four parallel branches are concatenated in the channel dimension, and then the number of channels is adjusted through a 1×1 convolutional layer to generate the output features of the multi-scale aggregation module.

[0022] S2.3. Construct a feature enhancement module (FMB). Apply a 1×1 convolution to the output features of the multi-scale aggregation module to expand the number of channels. Then, after channel segmentation, input the FMB into the Non-Local Information Modeling (EASA) branch and the Local Detail Estimation (LDE) branch for processing. After fusing the obtained non-local information features and local detail features through element-wise addition, input the FMB into a 1×1 convolutional layer to generate deep features.

[0023] Furthermore, the specific implementation method of step S2.3 includes the following steps:

[0024] S2.3.1. Construct the nonlocal information modeling branch EASA. First, perform 8x adaptive max pooling on the input features after channel segmentation. Then, use 3×3 depthwise convolution to extract the structural prior Xs at this low resolution. At the same time, calculate the spatial variance of the original image as a global statistical descriptor. Concatenate the spatial variance of the original image with Xs and then perform 1×1 convolution to generate the modulation image Xm. Finally, activate Xm with ReLU and restore it to the original resolution through upsampling. Multiply it element-wise with the input features so that each pixel obtains an adaptive weight from the global context, resulting in nonlocal information features.

[0025] S2.3.2. Construct the Local Detail Estimation (LDE) branch. First, perform a 3×3 depthwise convolution on the input features after channel segmentation. Then, increase the dimensionality through a 1×1 convolution and introduce ReLU nonlinear activation to enhance the response. Finally, pass a 1×1 convolution layer to obtain the local detail features.

[0026] Furthermore, the specific implementation method of step S3 includes the following steps:

[0027] S3.1. The transposed convolutional layer ConvSR is configured to have a main path and a residual path. The main path consists of a first 3×3 convolutional layer, a ReLU activation layer, and a second 3×3 convolutional layer connected in sequence. The residual path consists of a 1×1 convolutional layer. The outputs of the main path and the residual path are added element-wise to form the output of ConvSR. No batch normalization layer is set in ConvSR to avoid texture degradation.

[0028] S3.2. Setting up the super-resolution reconstruction decoding branch of the multi-task learning network: First, the deep features are upsampled by bilinear interpolation. The shallow features are adjusted by a 1×1 convolutional layer, a BN layer, and a ReLU layer. Then, the adjusted low-level features are concatenated with the upsampled deep features. The concatenated feature map is input into two decoding modules for decoding. After passing through four ConvSR modules and then through a 1×1 convolutional layer, a high-resolution map is output.

[0029] The decoding module includes a 3x3 convolutional layer, a BN layer, a ReLU layer, and a Dropout layer.

[0030] Furthermore, the specific implementation method of step S4 is as follows: first, bilinear interpolation upsampling is performed on the deep features, and the shallow features are adjusted through a 1×1 convolutional layer, a BN layer, and a ReLU layer. Then, the adjusted low-level features are concatenated with the upsampled deep features. The concatenated feature map is input into two decoding modules for decoding. Then, the fused feature map is finally upsampled by bilinear interpolation. After upsampling, it is passed through a 1×1 convolutional layer to generate the final semantic segmentation result.

[0031] The decoding module includes a 3x3 convolutional layer, a BN layer, a ReLU layer, and a Dropout layer.

[0032] Furthermore, the specific implementation method of step S5 includes the following steps:

[0033] S5.1. Define the super-resolution task loss function of a multi-task learning network as the pixel-level mean squared error. The expression is:

[0034]

[0035] in, Represents a true high-resolution image. This is the reconstructed image output by the network. H, W, and C represent the height, width, and number of channels of the image, respectively; i, j, and k represent the pixel indices in the height, width, and channel directions, respectively.

[0036] S5.2. Define the loss function for land use classification tasks as including cross-entropy loss and boundary enhancement loss. The expression for cross-entropy loss is:

[0037]

[0038] Where N is the total number of pixels, It is the number of land use categories, y n,c It is the true label of the nth pixel belonging to category c, and It is the probability value predicted by the model that the nth pixel belongs to class c;

[0039] Boundary Enhancement Loss The expression is:

[0040]

[0041] in, Indicates the boundary area. Indicates the predicted value. Indicates the one-hot label value. This represents the soft tag value.

[0042] S5.3. Overall Loss Function of Multi-Task Learning Networks for:

[0043]

[0044] in, , and These are the hyperparameters for pixel-level mean squared error, cross-entropy loss, and boundary enhancement loss, respectively.

[0045] Furthermore, during the training process in step S6, the Adam optimizer is used to perform end-to-end optimization of the network parameters. The initial learning rate is set to 0.0001 to 0.00001, the batch size is 8, and the number of iterations is 20. During the training process, data augmentation strategies such as random horizontal flipping, vertical flipping, and 90-degree rotation are introduced, and the ResNet pre-trained model is used as the initial weights.

[0046] Furthermore, in step S7, the remote sensing image to be processed is cut into non-overlapping segments of 256×256 pixels.

[0047] The beneficial effects of this invention are:

[0048] The present invention discloses an optical image planting pattern recognition method based on super-resolution reconstruction assistance. By constructing a multi-task learning framework, super-resolution reconstruction and land use classification tasks are jointly trained, so that the network can optimize the reconstruction of image details while also focusing on the accurate extraction of semantic information, thereby improving the generalization ability and practicality of the overall model.

[0049] The present invention provides a planting pattern recognition method for optical images based on super-resolution reconstruction, which improves the accuracy of boundary classification. By designing a boundary enhancement loss function that combines label smoothing and edge detection mechanisms, the overconfidence problem of the model in the boundary region of ground objects is effectively alleviated, and the classification accuracy and robustness at complex boundaries are significantly improved.

[0050] The present invention provides a planting pattern recognition method for optical images based on super-resolution reconstruction, which simplifies the processing flow. Compared with the traditional two-stage processing method, the present invention realizes end-to-end joint learning, simplifies the processing flow, avoids error accumulation in intermediate links, and improves processing efficiency.

[0051] The planting pattern recognition method based on super-resolution reconstruction-assisted optical image described in this invention has strong applicability. The method is based on publicly available Sentinel-2 and domestic Gaofen series satellite data, with a wide range of data sources. The method has strong universality and application promotion value, and is particularly suitable for practical application scenarios such as large-scale agricultural planting pattern monitoring. Attached Figure Description

[0052] Figure 1 This is a flowchart of a planting pattern recognition method for optical images based on super-resolution reconstruction assistance, as described in this invention.

[0053] Figure 2 This is a structural diagram of the multi-scale aggregation module described in this invention;

[0054] Figure 3 This is a structural diagram of the feature enhancement module described in this invention;

[0055] Figure 4 This is a structural diagram of the semantic segmentation decoder described in this invention;

[0056] Figure 5 This is a structural diagram of the super-resolution decoder described in this invention;

[0057] Figure 6 This is a diagram showing the planting pattern as described in this invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0059] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0060] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 - Appendix Figure 6 Detailed explanation is as follows:

[0061] Example 1:

[0062] A planting pattern recognition method based on super-resolution reconstruction-assisted optical image recognition includes the following steps:

[0063] S1. Collect low-resolution remote sensing images and high-resolution fused images of the same time period and the same scene area, preprocess them, manually classify and label the high-resolution fused images, and divide the resulting dataset into training set and test set.

[0064] Furthermore, the specific implementation method of step S1 includes the following steps:

[0065] S1.1. Select the study area, collect multispectral images from the European Space Agency's Sentinel-2B satellite of the same time period and the same scene area as low-resolution remote sensing images, and collect high-resolution fused images from China's Gaofen-2 GF-2 satellite as high-resolution images;

[0066] S1.2. Perform atmospheric correction, radiometric normalization and geometric registration on the low-resolution remote sensing image and high-resolution fused image obtained in step S1.1. Then, crop the preprocessed low-resolution remote sensing image into a 256×256 pixel low-resolution remote sensing image block and crop the high-resolution fused image into a 1024×1024 pixel high-resolution fused image block.

[0067] S1.3. Use ArcGIS Pro software to manually interpret the high-resolution fused image patches obtained in step S1.2 to generate land use classification label maps including rice, corn, soybeans and four other land features. Then, combine the low-resolution remote sensing image patches, the high-resolution fused image patches and the corresponding classification label maps to form training and test sets.

[0068] S2. Construct a feature extraction backbone, a multi-scale aggregation module, and a feature enhancement module for a multi-task learning network. The feature extraction backbone, the multi-scale aggregation module, and the feature enhancement module are connected in sequence to extract deep and shallow features from the input low-resolution remote sensing image.

[0069] Furthermore, the specific implementation method of step S2 includes the following steps:

[0070] S2.1. Construct the feature extraction backbone of the multi-task learning network. Remove the downsampling operations in the fourth and fifth stages of the standard ResNet-101 model and use it as the feature extraction backbone. Input the low-resolution remote sensing image into the feature extraction backbone. The second stage directly outputs shallow features. Then, connect the multi-scale aggregation module after the last convolutional block of the ResNet-101 model to obtain the output features of the multi-scale aggregation module.

[0071] Furthermore, the shared encoder uses ResNet-101 as its backbone to extract multi-level deep feature representations from the input Sentinel-2 low-resolution multispectral remote sensing images. ResNet-101 comprises 101 layers, and by introducing residual connection structures, it effectively alleviates the gradient degradation problem during training of deep networks, ensuring that the network can stably learn rich semantic information. To retain more spatial details while maintaining a large receptive field, the standard ResNet-101 structure was adaptively adjusted: the downsampling operations in the fourth and fifth stages were removed, resulting in the final output feature map of the encoder having a spatial resolution of 1 / 8 of the original input image. This high-resolution feature map is beneficial for the accurate perception of ground feature boundaries in subsequent tasks.

[0072] The multi-scale aggregation module comprises four parallel branches: one 3×3 ordinary convolutional branch and three 3×3 dilated convolutional branches with dilation rates of 1, 2, and 3, respectively. The outputs of these four parallel branches are concatenated along the channel dimension, and then the number of channels is adjusted using a 1×1 convolutional layer to generate the output features of the multi-scale aggregation module. The expression for the multi-scale aggregation layer is as follows:

[0073]

[0074] in, Indicates feature input, This represents the multi-scale features of the output. d represents the convolutional layer and d represents the expansion rate.

[0075] S2.3. Construct a feature enhancement module (FMB). Apply a 1×1 convolution to the output features of the multi-scale aggregation module to expand the number of channels. Then, after channel segmentation, input the FMB into the Non-Local Information Modeling (EASA) branch and the Local Detail Estimation (LDE) branch for processing. After fusing the obtained non-local information features and local detail features through element-wise addition, input the FMB into a 1×1 convolutional layer to generate deep features.

[0076] Furthermore, the specific implementation method of step S2.3 includes the following steps:

[0077] S2.3.1. Construct the nonlocal information modeling branch EASA. First, perform 8x adaptive max pooling on the input features after channel segmentation. Then, use 3×3 depthwise convolution to extract the structural prior Xs at this low resolution. At the same time, calculate the spatial variance of the original image as a global statistical descriptor. Concatenate the spatial variance of the original image with Xs and then perform 1×1 convolution to generate the modulation image Xm. Finally, activate Xm with ReLU and restore it to the original resolution through upsampling. Multiply it element-wise with the input features so that each pixel obtains an adaptive weight from the global context, resulting in nonlocal information features.

[0078] S2.3.2. Construct the Local Detail Estimation (LDE) branch. First, perform a 3×3 depthwise convolution on the input features after channel segmentation. Then, increase the dimensionality through a 1×1 convolution and introduce ReLU nonlinear activation to enhance the response. Finally, pass a 1×1 convolution layer to obtain the local detail features.

[0079] Furthermore, given input features ,in This represents the spatial size, and C is the number of channels. First, the normalized... A 1×1 convolution is applied to expand the number of channels, and then the channels are split into two parts through a channel splitting operation as inputs for the EASA and LDE branches:

[0080]

[0081] in, Represents L2 normalization, This represents a 1×1 convolutional layer. This indicates a channel splitting operation. and Features X and Y are processed in parallel using EASA and LDE branches to generate nonlocal features. and local features .Will and The elements are added together element-wise and then fed into a 1×1 convolutional layer to form the representative output of the feature enhancement module. This process can be formulated as follows:

[0082]

[0083] in, It is the output feature.

[0084] Low-frequency components are obtained through downsampling and then input into a 3×3 depthwise separable convolution to generate nonlocal structural information. :

[0085]

[0086] Where D represents adaptive max pooling with a scaling factor of 8. It is a 3×3 depthwise separable convolutional layer. This is to embed a global description to adjust the non-local representation. The variance of X is introduced as a statistical difference in spatial information, and it is combined with a nonlocal representation through a 1× convolution. merge:

[0087]

[0088]

[0089] in, X is the variance, and N is the total number of pixels. This represents the value of each pixel, where μ is the average of all pixel values. This represents the modulated features. This variance modulation mechanism helps to better explore nonlocal information.

[0090] The modulated features are used to aggregate the input features X in order to extract representative structural information. :

[0091]

[0092] in, This refers to the ReLU activation function. This indicates the nearest neighbor upsampling operation, and ⊙ represents the element-wise multiplication operation.

[0093] Local details are crucial for high-frequency reconstruction results. Given that EASA prioritizes the exploration of non-local structural information, a simple local detail estimation layer was developed to simultaneously capture local features. Specifically, extended depthwise separable convolutions with a 3×3 kernel size are used to encode local information from the input features Y. Enhanced local features are generated using two 1×1 convolutions with hidden ReLU activation functions. :

[0094]

[0095]

[0096] in, It is encoded local information.

[0097] S3. Construct the super-resolution reconstruction decoding branch of the multi-task learning network, input the deep features and shallow features obtained in step S2 into the super-resolution reconstruction decoding branch, restore the image spatial resolution through upsampling operation, and output a high-resolution image;

[0098] Furthermore, the specific implementation method of step S3 includes the following steps:

[0099] S3.1. The transposed convolutional layer ConvSR is configured to have a main path and a residual path. The main path consists of a first 3×3 convolutional layer, a ReLU activation layer, and a second 3×3 convolutional layer connected in sequence. The residual path consists of a 1×1 convolutional layer. The outputs of the main path and the residual path are added element-wise to form the output of ConvSR. No batch normalization layer is set in ConvSR to avoid texture degradation.

[0100] S3.2. Setting up the super-resolution reconstruction decoding branch of the multi-task learning network: First, the deep features are upsampled by bilinear interpolation. The shallow features are adjusted by a 1×1 convolutional layer, a BN layer, and a ReLU layer. Then, the adjusted low-level features are concatenated with the deep features upsampled by ConvSR. The concatenated feature map is input into two decoding modules for decoding. After passing through four ConvSR modules and then through a 1×1 convolutional layer, a high-resolution map is output.

[0101] The decoding module includes a 3x3 convolutional layer, a BN layer, a ReLU layer, and a Dropout layer.

[0102] S4. Construct a land use classification decoding branch of a multi-task learning network. Input the deep and shallow features obtained in step S2 into the land use classification decoding branch, perform pixel-by-pixel semantic category prediction through the decoder, and output the land use classification result.

[0103] Furthermore, the specific implementation method of step S4 is as follows: first, bilinear interpolation upsampling is performed on the deep features, and the shallow features are adjusted through a 1×1 convolutional layer, a BN layer, and a ReLU layer. Then, the adjusted low-level features are concatenated with the upsampled deep features. The concatenated feature map is input into two decoding modules for decoding. Then, the fused feature map is finally upsampled by bilinear interpolation. After upsampling, it is passed through a 1×1 convolutional layer to generate the final semantic segmentation result.

[0104] The decoding module includes a 3x3 convolutional layer, a BN layer, a ReLU layer, and a Dropout layer.

[0105] S5. Define the super-resolution task loss function and the land use classification task loss function for multi-task learning networks;

[0106] Furthermore, the specific implementation method of step S5 includes the following steps:

[0107] S5.1. Define the super-resolution task loss function of a multi-task learning network as the pixel-level mean squared error. The expression is:

[0108]

[0109] in, Represents a true high-resolution image. This is the reconstructed image output by the network. H, W, and C represent the height, width, and number of channels of the image, respectively; i, j, and k represent the pixel indices in the height, width, and channel directions, respectively.

[0110] S5.2. Define the loss function for land use classification tasks as including cross-entropy loss and boundary enhancement loss. The expression for cross-entropy loss is:

[0111]

[0112] Where N is the total number of pixels, It is the number of land use categories, y n,c It is the true label of the nth pixel belonging to category c, and It is the probability value predicted by the model that the nth pixel belongs to class c;

[0113] Boundary Enhancement Loss The expression is:

[0114]

[0115] in, Indicates the boundary area. Indicates the predicted value. Indicates the one-hot label value. This represents the soft tag value.

[0116] Furthermore, to address the issue of blurred or misclassified land cover boundaries in semantic segmentation of low-resolution remote sensing images, a boundary enhancement loss function combining label smoothing and image edge detection mechanisms is designed. The core idea of ​​this loss function is: in regular non-edge regions, one-hot encoded hard labels are used for training; in edge regions, soft labels are introduced for label smoothing to alleviate the model's overconfidence in difficult samples. The edge regions are determined by: performing edge detection on the real land use classification label map to identify the boundary pixels between different categories; in edge regions, smoothed soft labels are used to replace the original one-hot hard labels for calculating cross-entropy loss.

[0117] S5.3. Overall Loss Function of Multi-Task Learning Networks for:

[0118]

[0119] in, , and These are the hyperparameters for pixel-level mean squared error, cross-entropy loss, and boundary enhancement loss, respectively.

[0120] Furthermore, a joint loss function is constructed by weighted summation to jointly optimize the network parameters. The joint loss function enables the network to simultaneously focus on spatial detail reconstruction and semantic information extraction of the image during training, thereby improving the generalization ability and practicality of the overall model.

[0121] Furthermore, in semantic segmentation tasks of low-resolution remote sensing images, due to the lack of spatial details, models often struggle to accurately identify boundary regions between ground objects, leading to ambiguity or misclassification at class boundaries. To address this, this invention designs a novel loss function combining label smoothing and image edge detection mechanisms. This aims to employ differentiated labeling methods for different types of pixel regions, thereby improving the model's robustness and generalization ability in complex boundary regions. The core idea of ​​this loss function is to use one-hot labels for training in regular regions and introduce soft labels for label smoothing in edge regions, thus mitigating the model's overconfidence in difficult samples.

[0122] S6. Input the training set obtained in step S1 into the multi-task learning network constructed in steps S2-S5, construct a joint loss function by weighted summation and perform joint training to obtain the trained multi-task learning network.

[0123] Furthermore, during the training process in step S6, the Adam optimizer is used to perform end-to-end optimization of the network parameters. The initial learning rate is set to 0.0001 to 0.00001, the batch size is 8, and the number of iterations is 20. During the training process, data augmentation strategies such as random horizontal flipping, vertical flipping, and 90-degree rotation are introduced, and the ResNet pre-trained model is used as the initial weights.

[0124] S7. Test the trained multi-task learning network obtained in step S6 using the test set, and then input the remote sensing image to be processed into the trained multi-task learning network to output high-resolution images and land use classification results.

[0125] Furthermore, in step S7, the remote sensing image to be processed is cut into non-overlapping segments of 256×256 pixels.

[0126] Furthermore,

[0127] Given input features ,in This represents the spatial size, and C is the number of channels. First, the normalized... A 1×1 convolution is applied to expand the number of channels, and then the channels are split into two parts through a channel splitting operation as inputs for the EASA and LDE branches:

[0128]

[0129] here Represents L2 normalization, This represents a 1×1 convolutional layer. This indicates a channel splitting operation. and Features X and Y are processed in parallel using EASA and LDE branches to generate nonlocal features. and local features .Will and The elements are added together element-wise and then fed into a 1×1 convolutional layer to form the representative output of the feature enhancement module. This process can be formulated as follows:

[0130]

[0131] in It is the output feature.

[0132] Low-frequency components are obtained through downsampling and then input into a 3×3 depthwise separable convolution to generate nonlocal structural information. :

[0133]

[0134] Here, D represents adaptive max pooling with a scaling factor of 8. It is a 3×3 depthwise separable convolutional layer. This is to embed a global description to adjust the non-local representation. The variance of X is introduced as a statistical difference in spatial information, and it is combined with a nonlocal representation through a 1× convolution. merge:

[0135]

[0136]

[0137] here X is the variance, and N is the total number of pixels. This represents the value of each pixel, where μ is the average of all pixel values. This represents the modulated features. This variance modulation mechanism helps to better explore nonlocal information.

[0138] The modulated features are used to aggregate the input features X in order to extract representative structural information. :

[0139]

[0140] here This refers to the ReLU activation function. This indicates the nearest neighbor upsampling operation, and ⊙ represents the element-wise multiplication operation.

[0141] Local details are crucial for high-frequency reconstruction results. Given that EASA prioritizes the exploration of non-local structural information, a simple local detail estimation layer was developed to simultaneously capture local features. Specifically, extended depthwise separable convolutions with a 3×3 kernel size are used to encode local information from the input features Y. Enhanced local features are generated using two 1×1 convolutions with hidden ReLU activation functions. :

[0142]

[0143]

[0144] here It is encoded local information.

[0145] Semantic segmentation decoder

[0146] The semantic segmentation decoder is as follows Figure 4As shown, the feature map output by the feature enhancement module is first upsampled using bilinear interpolation to enlarge it to 1 / 4 the resolution of the input image. This initial upsampling step helps reduce the computational burden in subsequent processing. To further enhance spatial details, the decoder extracts low-level features from the intermediate layers of the backbone network. These features contain rich spatial details but less semantic information. The number of channels in the low-level features is adjusted using 1×1 convolutions. Subsequently, the adjusted low-level features are concatenated with the upsampled high-level features to fuse their advantages. This fusion strategy not only preserves the rich semantic information of the high-level features but also supplements the spatial details of the low-level features, significantly improving the ability to capture edges and fine-grained structures. The fused feature map is then decoded. The decoding module contains 3x3 convolutional layers, BN layers, ReLU layers, and Dropout layers.

[0147] Finally, the fused feature map is upsampled using bilinear interpolation to enlarge it to the original resolution of the input image. After upsampling, a 1×1 convolutional layer is used to adjust the number of channels in the feature map to match the number of target classes, generating the final semantic segmentation result. This 1×1 convolutional layer acts as a classification layer, mapping each pixel to its corresponding class label, resulting in the final semantic segmentation prediction map.

[0148] Super-resolution decoder

[0149] The super-resolution decoder is as follows Figure 5 As shown, the super-resolution decoder has a similar structure to the semantic segmentation decoder. The super-resolution decoder uses the ConvSR module for upsampling and adjusts the number of channels of the feature map to 4 through a regular convolutional layer in the final stage, and finally outputs a high-resolution image.

[0150] The ConvSR architecture is designed as follows: First, the size of the feature map is doubled using transposed convolution. The main path first convolves the input feature map with a 3×3 convolutional layer, then applies the ReLU activation function, followed by another 3×3 convolutional layer. The residual path performs channel matching on the input feature map using a 1×1 convolutional layer to ensure that the feature maps of the main and residual paths have the same number of channels. Feature fusion is achieved by element-wise addition of the outputs of the main and residual paths. To avoid interfering with the reconstructed texture, no batch normalization layer is introduced in the EDSR residual block. This super-resolution branch structure effectively alleviates the gradient vanishing problem in deep networks through cascaded transposed convolutions and the EDSR residual block design. Simultaneously, the EDSR residual block structure retains rich texture details and high-frequency information, contributing to the generation of high-quality super-resolution images.

[0151] The final output includes semantic segmentation results and super-resolution results. This multi-head output mechanism not only improves the flexibility of the inference stage, allowing the model to select the optimal output mode according to the actual task, but also provides diverse supervision signals for the design of the loss function, which helps to improve the overall learning effect and generalization ability of the model.

[0152] Multi-task loss function design

[0153] The multi-task learning framework proposed in this invention simultaneously optimizes two sub-tasks: image super-resolution and land use classification. To achieve an end-to-end learning process, corresponding loss functions are defined for each task, and a joint loss function is constructed through weighted summation to jointly optimize the network parameters.

[0154] For super-resolution tasks, pixel-level mean squared error (MSE) is used as the loss function. MSE measures the pixel difference between the high-resolution image generated by the model and the real high-resolution image, effectively guiding the network to recover image details. Its specific form is as follows:

[0155]

[0156] in Represents a true high-resolution image. This is the reconstructed image output by the network, where H, W, and C represent the image's height, width, and number of channels, respectively.

[0157] The goal of land use classification is to predict the semantic category of each pixel in the input image. Cross-entropy loss and boundary enhancement loss are used to measure the difference between the predicted result and the true label.

[0158]

[0159] in It is the total number of pixels. It refers to the number of land use categories. It is the true label of the nth pixel belonging to category c, and It is the probability value predicted by the model.

[0160] In semantic segmentation tasks of low-resolution remote sensing images, due to the lack of spatial details, models often struggle to accurately identify boundary regions between ground objects, leading to ambiguity or misclassification at class boundaries. To address this, this invention designs a novel loss function combining label smoothing and image edge detection mechanisms. This function aims to employ differentiated labeling methods for different types of pixel regions, thereby improving the model's robustness and generalization ability in complex boundary regions. The core idea of ​​this loss function is to use one-hot labels for training in regular regions and introduce soft labels for label smoothing in edge regions, thus mitigating the model's overconfidence in difficult samples.

[0161]

[0162] in Indicates the boundary area. Indicates the tag value. This represents the predicted value.

[0163] To simultaneously optimize both super-resolution and land use classification tasks, the above loss functions are linearly weighted and combined into a joint loss function:

[0164]

[0165] in , and These are hyperparameters that control the loss weights of the two tasks, used to balance the impact of different tasks on gradient updates. In this invention, these three parameters were determined through experimental verification to obtain optimal performance.

[0166] The joint loss function enables the network to simultaneously focus on spatial detail reconstruction and semantic information extraction of images during training, thereby improving the generalization ability and practicality of the overall model.

[0167] Furthermore, model training is performed, and the average loss curve on the training set, the overall accuracy (Acc) on the validation set, the average accuracy of each class (Acc_class), the average intersection-overall ratio (mIoU), and the frequency-weighted intersection-overall ratio (FWIoU) are used as criteria to evaluate the model's performance.

[0168] After training and comprehensive evaluation, the best-performing model is saved. The evaluation metrics and formulas are as follows:

[0169] 1. Overall accuracy

[0170] Overall accuracy refers to the proportion of all correctly classified pixels out of the total number of pixels, reflecting the overall accuracy of the model in classifying the entire image.

[0171] The formula is as follows:

[0172]

[0173] Wherein, TP represents a pixel correctly identified as a certain category; TN represents a pixel correctly identified as not belonging to that category; FP represents a pixel incorrectly identified as belonging to that category; and FN represents a pixel incorrectly not identified as belonging to that category.

[0174] 2. Category Average Accuracy

[0175] The class average accuracy is the result of averaging the accuracy calculated for each class separately. This method more fairly reflects the model's overall performance across all classes, and is especially suitable for situations with uneven sample distribution. The final calculation method is as follows:

[0176]

[0177] Where N represents the total number of categories.

[0178] 3. Average crossover ratio

[0179] mIoU is a widely used evaluation metric for semantic segmentation tasks. It measures the degree of overlap between the predicted region and the ground truth label region. mIoU is calculated based on the IoU for each class and averaged, effectively reflecting the model's accuracy in recognizing the boundaries of each class.

[0180] For the i-th category, its IoU is defined as:

[0181] ;

[0182] The average crossover-union ratio is:

[0183] ;

[0184] 4. Frequency-weighted intersection-union ratio

[0185] FWIoU is an improved form of mIoU, which assigns a weight to the IoU of each class, and the weight is determined by the frequency of that class in the dataset.

[0186] The calculation formula is as follows:

[0187]

[0188] in, This represents the total number of pixels in the i-th category across all images.

[0189] This specific implementation example selects Anhui Province, China as the study area and acquires Sentinel-2A / B satellite multispectral remote sensing images from April 2025. Four bands with a spatial resolution of 10 meters—blue, green, red, and near-infrared—are selected as network input data. Simultaneously, Gaofen-2 (GF-2) satellite fusion images from similar time periods (time interval ≤ 7 days) are acquired, and their corresponding panchromatic band images are used for multi-scale fusion processing to generate a high-resolution fusion image with a spatial resolution of 2.5 meters. This image serves as the supervision target image for the super-resolution reconstruction task. All images undergo the following preprocessing operations;

[0190] This invention conducts super-resolution reconstruction and land classification experiments based on deep learning methods. All model training and testing are completed in a unified hardware and software environment. The experimental platform configuration is as follows: Intel(R) Core(TM) i7-14700KF processor, 3400MHz, 32GB memory; NVIDIA RTX4070 GPU, 16GB video memory, supporting large-scale parallel computing tasks. The operating system is Windows 11, and the deep learning framework used is PyTorch 2.7.0, combined with CUDA 12.6 and cuDNN 8.9 for accelerated computation.

[0191] During model training, the optimizer employed the Adam algorithm, with an initial learning rate of 0.0001–0.00001 and a momentum parameter of 0.9–0.999. To prevent overfitting, data augmentation strategies were introduced during training, including random horizontal flipping, vertical flipping, and 90-degree rotation. The training batch size was set to 8, with 20 iterations, and model weights were saved every epoch. To accelerate convergence and improve generalization, a ResNet pre-trained model was used as the initial weights, and the learning rate was dynamically adjusted based on the actual training performance.

[0192] To facilitate understanding of the technical effects of this embodiment, the comparison between the method of this embodiment and the prior art is shown in Table 1.

[0193] Table 1

[0194]

[0195] As shown in Table 1, DeepLabSR in this embodiment performs excellently across all network performance metrics, demonstrating its powerful capabilities and high accuracy in handling semantic segmentation tasks in complex scenarios. These experimental results fully demonstrate the effectiveness and superiority of the DeepLabSR network design.

[0196] In addition, a qualitative analysis and comparative test was conducted using the optimal model from each algorithm to observe whether there were oversegmentation, undersegmentation, omissions, or missegmentation issues in the planting pattern extraction results. See the attached comparison chart. Figure 6 From the appendix Figure 6 It can be intuitively observed that the method of the present invention exhibits significantly superior performance. In terms of detail recognition, the classification is more refined, the edges are smoother and rounder, and the actual edges are closer to the actual edges, with fewer of the aforementioned segmentation defects. The accuracy and computational complexity evaluation of this embodiment are shown in Table 2.

[0197] Table 2

[0198]

[0199] As shown in Table 2, the results indicate that, compared to the sequential method of "super-resolution followed by segmentation", the present invention adopts multi-task end-to-end joint learning, which significantly reduces model complexity (40% reduction in parameters and 31% reduction in computation) while improving classification accuracy. It effectively avoids error accumulation and resource waste caused by step-by-step processing and achieves synergistic optimization of accuracy and efficiency.

[0200] The method described in this embodiment can be deployed on remote sensing image processing platforms or agricultural remote sensing monitoring systems, supporting automatic identification of crop planting patterns over large areas. This method is not limited to specific geographical regions or crop types, and can also be extended to other fine classification tasks based on medium- and low-resolution remote sensing data, such as urban green space identification, wetland monitoring, and land cover mapping, demonstrating good versatility and promotional value.

[0201] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0202] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for recognizing planting patterns in optical images based on super-resolution reconstruction, characterized in that, Includes the following steps: S1. Collect low-resolution remote sensing images and high-resolution fused images of the same time period and the same scene area, preprocess them, manually classify and label the high-resolution fused images, and divide the resulting dataset into training set and test set. S2. Construct a feature extraction backbone, a multi-scale aggregation module, and a feature enhancement module for a multi-task learning network. The feature extraction backbone, the multi-scale aggregation module, and the feature enhancement module are connected in sequence to extract deep and shallow features from the input low-resolution remote sensing image. S3. Construct the super-resolution reconstruction decoding branch of the multi-task learning network, input the deep features and shallow features obtained in step S2 into the super-resolution reconstruction decoding branch, restore the image spatial resolution through upsampling operation, and output a high-resolution image; S4. Construct a land use classification decoding branch of a multi-task learning network. Input the deep and shallow features obtained in step S2 into the land use classification decoding branch, perform pixel-by-pixel semantic category prediction through the decoder, and output the land use classification result. S5. Define the super-resolution task loss function and the land use classification task loss function for multi-task learning networks; S6. Input the training set obtained in step S1 into the multi-task learning network constructed in steps S2-S5, construct a joint loss function by weighted summation and perform joint training to obtain the trained multi-task learning network. S7. Test the trained multi-task learning network obtained in step S6 using the test set, and then input the remote sensing image to be processed into the trained multi-task learning network to output high-resolution images and land use classification results.

2. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 1, characterized in that, The specific implementation method of step S1 includes the following steps: S1.

1. Select the study area, collect multispectral images from the European Space Agency's Sentinel-2B satellite of the same time period and the same scene area as low-resolution remote sensing images, and collect high-resolution fused images from China's Gaofen-2 GF-2 satellite as high-resolution fused images; S1.

2. Perform atmospheric correction, radiometric normalization and geometric registration on the low-resolution remote sensing image and high-resolution fused image obtained in step S1.

1. Then, crop the preprocessed low-resolution remote sensing image into a 256×256 pixel low-resolution remote sensing image block and crop the high-resolution fused image into a 1024×1024 pixel high-resolution fused image block. S1.

3. Use ArcGIS Pro software to manually interpret the high-resolution fused image patches obtained in step S1.2 to generate land use classification label maps including rice, corn, soybeans and four other land features. Then, combine the low-resolution remote sensing image patches, the high-resolution fused image patches and the corresponding classification label maps to form training and test sets.

3. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 2, characterized in that, The specific implementation method of step S2 includes the following steps: S2.

1. Construct the feature extraction backbone of the multi-task learning network. Remove the downsampling operations in the fourth and fifth stages of the standard ResNet-101 model and use it as the feature extraction backbone. Input the low-resolution remote sensing image into the feature extraction backbone. The second stage directly outputs shallow features. Then, connect the multi-scale aggregation module after the last convolutional block of the ResNet-101 model to obtain the output features of the multi-scale aggregation module. The multi-scale aggregation module includes four parallel branches: a 3×3 ordinary convolutional branch and three 3×3 dilated convolutional branches with dilation rates of 1, 2 and 3 respectively. The outputs of the four parallel branches are concatenated in the channel dimension, and then the number of channels is adjusted through a 1×1 convolutional layer to generate the output features of the multi-scale aggregation module. S2.

3. Construct a feature enhancement module (FMB). Apply a 1×1 convolution to the output features of the multi-scale aggregation module to expand the number of channels. Then, after channel segmentation, input the FMB into the Non-Local Information Modeling (EASA) branch and the Local Detail Estimation (LDE) branch for processing. After fusing the obtained non-local information features and local detail features through element-wise addition, input the FMB into a 1×1 convolutional layer to generate deep features.

4. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 3, characterized in that, The specific implementation method of step S2.3 includes the following steps: S2.3.

1. Construct the nonlocal information modeling branch EASA. First, perform 8x adaptive max pooling on the input features after channel segmentation. Then, use 3×3 depthwise convolution to extract the structural prior Xs at this low resolution. At the same time, calculate the spatial variance of the original image as a global statistical descriptor. Concatenate the spatial variance of the original image with Xs and then perform 1×1 convolution to generate the modulation image Xm. Finally, activate Xm with ReLU and restore it to the original resolution through upsampling. Multiply it element-wise with the input features so that each pixel obtains an adaptive weight from the global context, resulting in nonlocal information features. S2.3.

2. Construct the Local Detail Estimation (LDE) branch. First, perform a 3×3 depthwise convolution on the input features after channel segmentation. Then, increase the dimensionality through a 1×1 convolution and introduce ReLU nonlinear activation to enhance the response. Finally, pass a 1×1 convolution layer to obtain the local detail features.

5. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 4, characterized in that, The specific implementation method of step S3 includes the following steps: S3.

1. The transposed convolutional layer ConvSR is configured to have a main path and a residual path. The main path consists of a first 3×3 convolutional layer, a ReLU activation layer, and a second 3×3 convolutional layer connected in sequence. The residual path consists of a 1×1 convolutional layer. The outputs of the main path and the residual path are added element-wise to form the output of ConvSR. No batch normalization layer is set in ConvSR to avoid texture degradation. S3.

2. Setting up the super-resolution reconstruction decoding branch of the multi-task learning network: First, the deep features are upsampled by bilinear interpolation. The shallow features are adjusted by a 1×1 convolutional layer, a BN layer, and a ReLU layer. Then, the adjusted low-level features are concatenated with the upsampled deep features. The concatenated feature map is input into two decoding modules for decoding. After passing through four ConvSR modules and then through a 1×1 convolutional layer, a high-resolution map is output. The decoding module includes a 3x3 convolutional layer, a BN layer, a ReLU layer, and a Dropout layer.

6. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 5, characterized in that, The specific implementation method of step S4 is as follows: First, the deep features are upsampled by bilinear interpolation. The shallow features are adjusted by a 1×1 convolutional layer, a BN layer, and a ReLU layer. Then, the adjusted low-level features are concatenated with the upsampled deep features. The concatenated feature map is input into two decoding modules for decoding. Then, the fused feature map is finally upsampled by bilinear interpolation. After upsampling, it is passed through a 1×1 convolutional layer to generate the final semantic segmentation result. The decoding module includes a 3x3 convolutional layer, a BN layer, a ReLU layer, and a Dropout layer.

7. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 6, characterized in that, The specific implementation method of step S5 includes the following steps: S5.

1. Define the super-resolution task loss function of a multi-task learning network as the pixel-level mean squared error. The expression is: in, Represents a true high-resolution image. This is the reconstructed image output by the network. H, W, and C represent the height, width, and number of channels of the image, respectively; i, j, and k represent the pixel indices in the height, width, and channel directions, respectively. S5.

2. Define the loss function for land use classification tasks as including cross-entropy loss and boundary enhancement loss. The expression for cross-entropy loss is: Where N is the total number of pixels, It is the number of land use categories, y n,c It is the true label of the nth pixel belonging to category c, and It is the probability value predicted by the model that the nth pixel belongs to class c; Boundary Enhancement Loss The expression is: in, Indicates the boundary area. Indicates the predicted value. Indicates the one-hot label value. This represents the soft tag value. S5.

3. Overall Loss Function of Multi-Task Learning Networks for: in, , and These are the hyperparameters for pixel-level mean squared error, cross-entropy loss, and boundary enhancement loss, respectively.

8. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 7, characterized in that, In step S6, the Adam optimizer is used to perform end-to-end optimization of the network parameters during training. The initial learning rate is set to 0.0001 to 0.00001, the batch size is 8, and the number of iterations is 20. During training, data augmentation strategies such as random horizontal flipping, vertical flipping, and 90-degree rotation are introduced, and the ResNet pre-trained model is used as the initial weights.

9. The method for identifying planting patterns in optical images based on super-resolution reconstruction assistance according to claim 8, characterized in that, In step S7, the remote sensing image to be processed is cut into non-overlapping segments of 256×256 pixels.

Images