A crop planting information extraction method based on a convolutional neural network
By integrating the attention mechanism and residual module, the URSE-Net model solves the problems of accuracy and efficiency in extracting crop planting information from high-resolution multispectral remote sensing images, and achieves high-precision extraction of crop planting information.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA AGRI UNIV
- Filing Date
- 2022-08-17
- Publication Date
- 2026-04-21
AI Technical Summary
Existing technologies suffer from low accuracy and low efficiency in extracting crop planting information from high-resolution multispectral remote sensing images. In particular, due to the special characteristics of satellite remote sensing images and the loss of image details during convolution and pooling processes in traditional CNN models, segmentation is inaccurate.
A method for extracting crop planting information based on convolutional neural networks is adopted. The URSE-Net model, which integrates attention mechanism and residual module, is combined with multispectral remote sensing image data. Through improved residual module, channel attention mechanism and batch normalization operation, the feature extraction capability and model generalization capability are improved.
It achieves high-precision extraction of crop planting information, improves the model's ability to extract deep features and resist overfitting, and enhances the accuracy and robustness of crop planting information extraction.
Smart Images
Figure CN115331104B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of crop planting information extraction from remote sensing images, specifically involving a method for extracting crop planting and arable land use information based on convolutional neural networks and using high-resolution remote sensing images as the data source. Background Technology
[0002] In recent years, with the rapid advancement of smart agriculture technologies, techniques based on high-resolution remote sensing imagery combined with deep learning have become increasingly important in yield prediction, crop information extraction, and planting information detection. Remote sensing imagery, especially multispectral remote sensing imagery, can better preserve the spectral characteristics of ground features. However, remote sensing imagery data is characterized by complex semantic features and numerous influencing factors. Therefore, efficiently segmenting crop planting information from high-resolution multispectral remote sensing imagery has become one of the most challenging research directions for crop information extraction.
[0003] The problem of crop pixel-level classification and the extraction of crop planting information are usually divided into traditional methods such as NDVI and WNDVI with manually set thresholds, and semantic segmentation methods based on computer deep learning technology.
[0004] Previous extraction methods often used mathematical models combined with remote sensing data to extract crop planting area information, but these methods suffered from low accuracy and efficiency. With the development of computer technology, the use of computer vision technology in related research has gained widespread acceptance. Image semantic segmentation technology is the foundation of computer vision technology. Currently, semantic segmentation technology is widely used in agricultural remote sensing, autonomous driving, robotics, and other fields. Convolutional Neural Network (CNN) technology is an important component of semantic segmentation technology. However, CNN technology suffers from inaccurate image segmentation due to the loss of image details during convolution and pooling. Jonathan et al. proposed the FCN model, which pioneered semantic-level image segmentation. Compared with traditional CNN methods, FCN can classify each pixel, but it suffers from the loss of details and smoothing issues during use. SegNet and U-Net models can overcome these problems better. The structures of SegNet and U-Net models are similar to those of FCN models, both exhibiting a clear "encode-decode" structure, and both achieving good performance on small datasets. Building upon this foundation, numerous novel image semantic segmentation models have been proposed, such as the Pyramid Scene Sparse Network Semantic Segmentation Model, the RefineNet model, and Google's DeepLab series. These models have demonstrated excellent performance on many large datasets. However, due to the unique characteristics of satellite remote sensing imagery, the aforementioned models typically do not perform ideally when applied to remote sensing images. Furthermore, since the training data for extracting farmland information generally only uses RGB three-band information, the richness of features it can preserve is limited.
[0005] For convolutional network models with traditional structures, such as U-Net, the degradation phenomenon becomes obvious as the model depth increases, the recognition error increases, and the accuracy decreases. In response to this phenomenon, He Kaiming et al. proposed the residual network model to solve the above problems. The residual structure achieves the goal of deepening the network while ensuring a suitable descent gradient by adding "spanning" identity mapping channels between non-adjacent feature maps. Summary of the Invention
[0006] To address various problems in existing crop planting information extraction methods, this invention provides a crop planting information extraction method based on convolutional neural networks. This method integrates a convolutional network model with attention mechanisms and residual modules, along with collaborative multispectral remote sensing image data, to extract crop planting information and improve the extraction efficiency.
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] A method for extracting crop planting information based on convolutional neural networks, wherein the method includes the following steps:
[0009] S1. Construct the processed raw multispectral satellite remote sensing image dataset:
[0010] S1.1. Raw remote sensing image data is collected by remote sensing satellites to form a raw multispectral satellite remote sensing image dataset;
[0011] The raw multispectral satellite remote sensing image dataset includes the following three types of multispectral remote sensing image data within farmland-covered areas:
[0012] 1. Multispectral remote sensing image data of farmland covered by crops;
[0013] 2. Multispectral remote sensing image data of farmland covered areas without crops; and
[0014] 3. Multispectral remote sensing image data of other coverage areas;
[0015] S1.2, Annotate the original remote sensing image data.
[0016] Farmland covered with crops is marked with a pixel value of 255, farmland covered without crops is marked with a pixel value of 200, and other types of coverage are marked with a pixel value of 0.
[0017] S1.3. Crop the original multispectral remote sensing image data in the original multispectral satellite remote sensing image dataset. The pixel size of the cropped image is 256×256.
[0018] S1.4 Divide the cropped original multispectral satellite remote sensing image dataset into training set, validation set and test set in a ratio of 8:1:1 to obtain the processed original multispectral satellite remote sensing image dataset.
[0019] S2, Data Augmentation:
[0020] The processed original multispectral remote sensing image dataset constructed in step S1 is enhanced to expand the number of images in the multispectral satellite remote sensing image dataset to 4 times the original number, resulting in a crop planting information dataset including farmland covered areas with crops, farmland covered areas without crops, and other types of covered areas.
[0021] S3. Model Building:
[0022] Based on the U-Net model structure, the URSE-Net convolutional neural network model is constructed by integrating the attention mechanism and residual module;
[0023] The URSE-Net convolutional neural network model consists of three parts: an encoder, intermediate layers, and a decoder. In the encoder part, an improved residual module is used to extract information from remote sensing image data. In the intermediate layers, two channel attention mechanisms, SE and ECA, are used. In the decoder part, each layer uses a "convolution-batch normalization-activation" operation once.
[0024] Step S3.1: Integrate the encoder with the fully pre-activated residual module.
[0025] The backbone network is built using a fully preactivated residual module that places batch normalization and activation operations before convolution operations. The structure of the fully preactivated residual module is: batch normalization layer, activation layer, and 3×3 convolutional layer.
[0026] During the fully pre-activated residual module processing, the number of channels in the feature map remains constant. As shown in Formula 1, after extraction by the backbone network, four feature maps are obtained, which are in the following order: encoder first feature map C1, encoder second feature map C2, encoder third feature map C3, and encoder fourth feature map C4. Their sizes are 256×256×64, 128×128×128, 64×64×256, and 32×32×512 pixels, respectively.
[0027]
[0028] In Formula 1: m is the size of the output feature map in pixels; w is the size of the input feature map in pixels; f is the filter size in pixels; and s is the stride in pixels.
[0029] The encoder is divided into five modules. Each module includes two residual module processing operations and one convolution operation. It also incorporates max pooling to compress the original image data size and increase the number of channels.
[0030] In the first module, the input image first undergoes a convolution operation to change the number of channels to 64, and then goes through two residual module operations to obtain a feature map with a pixel value of 256×256 and a number of channels of 64. This feature map is used as the first feature map C1 of the encoder and output to the intermediate layer.
[0031] The first feature map C1 of the encoder is halved in length and width by combining max pooling operation, and is input into the second module with a neuron dropout probability of 0.25. In the second module, the feature map undergoes two residual operations to obtain a feature map with a pixel value of 128×128 and a channel number of 128. This feature map is used as the second feature map C2 of the encoder and output to the intermediate layer.
[0032] The second feature map C2 of the encoder is halved in length and width by combining max pooling and is then input into the third module with a neuron dropout probability of 0.5. In the third module, the feature map undergoes two residual operations to obtain a feature map with 64×64 pixels and 256 channels. This feature map is then used as the third feature map C3 of the encoder and output to the intermediate layer.
[0033] The length and width of the encoder's third feature map C3 are halved by combining the max pooling operation, and then the feature map is input into the fourth module with a neuron dropout probability of 0.5. In the fourth module, the feature map undergoes two residual operations to obtain a feature map with a pixel size of 32×32 and a channel number of 512. This feature map is used as the fourth feature map C4 and output to the intermediate layer.
[0034] Combining max pooling, the width and height of the encoder's fourth feature map C4 are halved, and it is input into the fifth module with a neuron dropout probability of 0.5. In the fifth module, the feature map undergoes two residual operations to obtain the encoder's fifth feature map with a width and height pixel value of 16×16 and a channel number of 1024, which is then output to the decoder.
[0035] Step S3.2: Use the intermediate layer of the channel attention mechanism (SE module) and ECA module.
[0036] The encoder first feature map C1 obtained from the encoder first module and the encoder second feature map C2 obtained from the encoder second module are processed using the SE module; the encoder third feature map C3 obtained from the encoder third module and the encoder fourth feature map C4 obtained from the encoder fourth module are processed using the ECA module.
[0037] After the encoder's first feature map C1 and second feature map C2 are input into the SE module, they sequentially undergo global pooling, fully connected operation, activation operation, fully connected operation, and σ activation operation. During this process, the width × height × number of channels of their feature maps are as follows: 1. Same as the input; 2. Width and height both become 1, and the number of channels remains unchanged; 3. Width and height are 1, and the number of channels becomes one-sixteenth; 4. Width and height are 1, and the number of channels remains the same as the input. The output after the σ activation operation is multiplied by the input to obtain an output feature map with the same size and number of channels as the input feature map. This feature map is then output to the decoder section.
[0038] After the encoder's third feature map C3 and fourth feature map C4 are input, they first undergo a global average pooling operation, which sets their width and height to 1 while keeping the number of channels unchanged. Then, they undergo a one-dimensional convolution and σ activation function operation, keeping their width, height, and number of channels unchanged. The output of the output feature map is multiplied by the input feature map and then output to the decoder.
[0039] Step S3.3: In the decoder section, each small portion undergoes a "convolution-batch normalization-activation" operation followed by a random dropout operation.
[0040] The decoder structure also contains five modules, and each module contains only one "convolution-batch normalization-activation" operation. Combined with deconvolution, it realizes the restoration of image size and compression of the number of channels.
[0041] In the decoder section, the fifth feature map of the encoder is first deconvolutionally processed to obtain the fifth module feature map of the decoder with a length and width of 32×32 and a channel number of 512.
[0042] The feature map of the fifth module of the decoder is superimposed with the fourth feature map C4 of the encoder after being processed by the ECA module. After one "convolution-normalization-activation", the fourth feature map of the decoder is output as 32×32×512.
[0043] After deconvolution, the fourth feature map of the decoder becomes 64×64 in length and width and 256 in number of channels. It is then superimposed on the third feature map C3 of the encoder after being processed by the ECA module. After one "convolution-normalization-activation", the output is a 64×64×256 third feature map of the decoder.
[0044] After deconvolution, the decoder's third feature map becomes 128×128 in length and width, and the number of channels becomes 128. It is superimposed with the encoder's second feature map C2 after being processed by the SE module. After one "convolution-normalization-activation", the decoder's second feature map of 128×128×128 is output.
[0045] After deconvolution, the second feature map of the decoder becomes 256×256 in length and width, with 64 channels. It is superimposed with the first feature map C1 of the encoder after processing by the SE module. After one "convolution-normalization-activation", the first feature map of the decoder is output as 256×256×64. Finally, the prediction result is obtained after one convolution operation.
[0046] S4. Setting parameters:
[0047] Obtain training parameters from the model; using the training set data processed in step S2 as the data source, adjust the model's running parameters to achieve the best running performance and obtain the model training settings parameters.
[0048] S5. Obtain the model training results:
[0049] The weight parameters obtained after model training are saved, and the model built by S3 and the obtained training results are used to extract features from the dataset.
[0050] S6. Obtain the results of farmland planting information extraction:
[0051] By using a visualization method for model prediction results and employing pixel conversion technology to transform the model recognition results, a visualization of farmland planting information extraction is obtained and displayed.
[0052] In step S2, the data augmentation steps include:
[0053] S2.1 Rotation: Rotate the processed original remote sensing image by ±90 degrees;
[0054] S2.2 Flip: Flip the processed remote sensing image after the rotation in step S2.1 with the x-axis and y-axis as the center respectively.
[0055] In step S4, the parameters to be set include: optimizer, learning rate, batch size, and loss function.
[0056] In step S5, obtaining the model training results specifically includes: training the model with the training set, validating the training results with the validation set, and saving the set of results in which the model performs best on the test set. The criteria for determining the best performance are the loss value and the accuracy.
[0057] Specifically, in step S6, obtaining the farmland planting information extraction result includes: using the result model obtained in S5, extracting farmland planting information from the test set data, and using pixel conversion technology to visualize and evaluate the prediction result.
[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0059] In this application, a residual network module with pre-batch normalization and activation is used to replace the original structure in the encoder section to extract deep features of the network and avoid model degradation. In the intermediate processing section, two channel attention mechanisms are introduced to strengthen the channel weights of the feature map. In the decoder section, batch normalization is added to improve the generalization ability of the model while avoiding overfitting. The Dropout operation of randomly discarding neurons is introduced in the global model to effectively avoid overfitting. Attached Figure Description
[0060] Figure 1 Here is a diagram of the URSE-Net model structure;
[0061] Figure 2 This is a schematic diagram of the FPA residual module;
[0062] Figure 3 This is a schematic diagram of the attention mechanisms of SE and ECA. Detailed Implementation
[0063] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0064] A method for extracting crop planting information based on convolutional neural networks includes the following steps:
[0065] S1. Construct the processed raw multispectral satellite remote sensing image dataset:
[0066] S1.1. Raw remote sensing image data is collected by remote sensing satellites to form a raw multispectral satellite remote sensing image dataset.
[0067] The raw multispectral satellite remote sensing image dataset includes the following three types of multispectral remote sensing image data within farmland-covered areas:
[0068] 1. Multispectral remote sensing image data of farmland covered by crops;
[0069] 2. Multispectral remote sensing image data of farmland covered areas without crops; and
[0070] 3. Multispectral remote sensing image data of other coverage areas.
[0071] S1.2, Annotate the original remote sensing image data.
[0072] Farmland covered with crops is labeled with a pixel value of 255, farmland covered without crops is labeled with a pixel value of 200, and other types of coverage are labeled with a pixel value of 0.
[0073] S1.3. Crop the original multispectral remote sensing image data in the original multispectral satellite remote sensing image dataset. The pixel size of the cropped image is 256×256.
[0074] S1.4 Divide the cropped original multispectral satellite remote sensing image dataset into training set, validation set and test set in a ratio of 8:1:1 to obtain the processed original multispectral satellite remote sensing image dataset.
[0075] S2, Data Augmentation:
[0076] The processed original multispectral remote sensing image dataset constructed in step S1 is enhanced to expand the number of images in the multispectral satellite remote sensing image dataset to 4 times the original number, resulting in a crop planting information dataset including farmland covered areas with crops, farmland covered areas without crops, and other types of covered areas.
[0077] In step S2, the data augmentation steps include:
[0078] S2.1 Rotation: Rotate the processed original remote sensing image by ±90 degrees;
[0079] S2.2 Flip: Flip the processed remote sensing image after the rotation in step S2.1 with the x-axis and y-axis as the center respectively.
[0080] S3. Model Building:
[0081] Based on the U-Net model structure, the URSE-Net convolutional neural network model is constructed by integrating the attention mechanism and residual module.
[0082] In step S3, to address the issues of the traditional U-Net model's shallow network depth, weak deep feature extraction capability, and weak boundary information extraction capability during multi-class segmentation, an end-to-end semantic segmentation model URSE-Net is proposed. Combined with the improved residual module, the traditional U-Net model structure is optimized.
[0083] For convolutional network models like URSE-Net, overfitting becomes increasingly apparent as model depth increases. Traditionally, this is addressed through model fusion, but this increases training and testing time. To solve this problem, this invention incorporates a dropout operation before the model enters the next layer, temporarily and randomly discarding some neurons with a certain probability. Simultaneously, batch normalization is introduced in the decoder to prevent overfitting.
[0084] like Figure 1As shown, the URSE-Net convolutional neural network model consists of three parts: an encoder, intermediate layers, and a decoder. In the encoder, an improved residual module is used to extract information from the remote sensing image data; in the intermediate layers, two channel attention mechanisms, SE and ECA, are used; in the decoder, each layer uses a "convolution-batch normalization-activation" operation once.
[0085] Step S3.1: Integrate the encoder with the fully pre-activated residual module.
[0086] By comprehensively comparing the performance of different residual modules, this invention uses a fully pre-activated residual module (FPA) that places batch normalization and activation operations before convolution operations to build the backbone network. The structure of the FPA residual module is as follows: Figure 2 As shown.
[0087] like Figure 1 As shown, this invention uses fully pre-activated residual modules to replace the convolution operation in the traditional U-Net backbone network.
[0088] The basic structure of the fully pre-activated residual module is: batch normalization layer, activation layer, and 3×3 convolutional layer. The improved residual module has significantly improved feature extraction and generalization capabilities.
[0089] During the fully pre-activated residual module processing, the number of channels in the feature map remains constant. As shown in Formula 1, after extraction by the backbone network, four feature maps are obtained, which are in the following order: encoder first feature map C1, encoder second feature map C2, encoder third feature map C3, and encoder fourth feature map C4. Their dimensions are 256×256×64, 128×128×128, 64×64×256, and 32×32×512 pixels, respectively.
[0090]
[0091] In Formula 1: m is the size of the output feature map in pixels; w is the size of the input feature map in pixels; f is the filter size in pixels; and s is the stride in pixels.
[0092] The encoder consists of five modules, each containing two residual module processes and one convolution operation. It also incorporates max pooling to compress the original image data size and increase the number of channels. In the first module, the input image first undergoes a convolution operation to increase the number of channels to 64. Then, it passes through two residual module operations to obtain a feature map with a pixel value of 256×256 and 64 channels. This feature map is used as the encoder's first feature map C1 and output to the intermediate layer.
[0093] The first feature map C1 of the encoder is halved in length and width by combining max pooling operation, and is input into the second module with a neuron dropout probability of 0.25. In the second module, the feature map undergoes two residual operations to obtain a feature map with a pixel value of 128×128 and a channel number of 128. This feature map is used as the second feature map C2 of the encoder and output to the intermediate layer.
[0094] The second feature map C2 of the encoder is halved in length and width by combining max pooling and is then input into the third module with a neuron dropout probability of 0.5. In the third module, the feature map undergoes two residual operations to obtain a feature map with 64×64 pixels and 256 channels. This feature map is then used as the third feature map C3 of the encoder and output to the intermediate layer.
[0095] The length and width of the encoder's third feature map C3 are halved by combining the max pooling operation, and then the feature map is input into the fourth module with a neuron dropout probability of 0.5. In the fourth module, the feature map undergoes two residual operations to obtain a feature map with a pixel size of 32×32 and a channel number of 512. This feature map is used as the fourth feature map C4 and output to the intermediate layer.
[0096] Combining max pooling, the width and height of the encoder's fourth feature map C4 are halved, and it is input into the fifth module with a neuron dropout probability of 0.5. In the fifth module, the feature map undergoes two residual operations to obtain the encoder's fifth feature map with a width and height pixel value of 16×16 and a channel number of 1024, which is then output to the decoder.
[0097] Step S3.2: Use the intermediate layer of the channel attention mechanism (SE module) and ECA module.
[0098] The encoder first feature map C1 obtained from the encoder first module and the encoder second feature map C2 obtained from the encoder second module are processed using the SE module; the encoder third feature map C3 obtained from the encoder third module and the encoder fourth feature map C4 obtained from the encoder fourth module are processed using the ECA module.
[0099] Attention mechanisms are a way to achieve adaptive attention in networks. Common attention mechanisms include three types: channel attention, spatial attention, and a combination of both (channel-spatial attention). Representative implementations include the SE module, ECA module, and CBAM module.
[0100] Comparative experiments revealed that the following processing methods can significantly improve the model's information extraction capability: The SE module is used to process the encoder's first feature map C1 obtained from the first encoder module and the encoder's second feature map C2 obtained from the second encoder module; the ECA module is used to process the encoder's third feature map C3 obtained from the third encoder module and the encoder's fourth feature map C4 obtained from the fourth encoder module; and no attention processing is applied to the encoder's fifth feature map. The insertion positions of the SE module and the ECA module are as follows: Figure 3 As shown.
[0101] The SE module is a classic implementation of channel attention, mainly consisting of two parts: global average pooling (Squeeze) and channel weight generation (Extraction). For the input feature map, global average pooling (Squeeze) is first applied independently to each channel. Then, two fully-connected (FC) layers and a non-linear function (Sigmoid) are used to generate channel weights (Extraction). Finally, the weights are multiplied by the initial weights (Scale) to obtain a feature map with the same size as the input. When performing fully connected layers, the number of channels c is determined by Equation 2:
[0102]
[0103] In Formula 2: C is the number of channels in the input feature map, and r = 16.
[0104] In this invention, after the encoder's first feature map C1 and second feature map C2 are input to the SE module, they sequentially undergo global pooling, fully connected operation, activation operation, fully connected operation, and σ-activation operation. During this process, the width × height × number of channels of their feature maps are as follows: 1. Same as the input; 2. Width and height both become 1, and the number of channels remains unchanged; 3. Width and height are 1, and the number of channels becomes one-sixteenth; 4. Width and height are 1, and the number of channels remains the same as the input. The output after the σ-activation operation is multiplied by the input to obtain an output feature map with the same size and number of channels as the input feature map, and this feature map is output to the decoder section.
[0105] The ECA module is also one of the implementation methods for the channel attention mechanism. Compared to the SE module, the ECA module replaces the two fully connected layer calculations in the SE module with 1D convolutions and uses an adaptive convolution kernel to implement the channel attention mechanism. Compared to the SE module, the ECA module is more lightweight and requires fewer parameters. The size of its adaptive convolution kernel k is obtained from Equation 3:
[0106]
[0107] In Formula 3, k is the pixel size of the adaptive convolution kernel, C is the number of input channels, b is set to 1, γ is set to 2, and odd means that only odd numbers can be used.
[0108] In this invention, after the encoder's third feature map C3 and the encoder's fourth feature map C4 are input, they first undergo a global average pooling operation, which changes their width and height to 1 while keeping the number of channels unchanged. Then, they undergo a one-dimensional convolution and σ activation function operation, keeping their width, height, and number of channels unchanged. The output is multiplied by the input feature map to obtain the output feature map, which is then output to the decoder.
[0109] Step S3.3: In the decoder section, each small portion undergoes a "convolution-batch normalization-activation" operation followed by a random dropout operation.
[0110] The decoder section has a similar structure to the encoder section, also containing five modules. Each module contains only one "convolution-batch normalization-activation" operation, which, together with the deconvolution operation, restores the image size and compresses the number of channels.
[0111] In the decoder section, the fifth feature map of the encoder is first deconvolutionally processed to obtain the fifth module feature map of the decoder with a length and width of 32×32 and a channel number of 512.
[0112] The feature map of the fifth module of the decoder is superimposed with the fourth feature map C4 of the encoder after being processed by the ECA module. After one "convolution-normalization-activation", the fourth feature map of the decoder is output as 32×32×512.
[0113] After deconvolution, the fourth feature map of the decoder becomes 64×64 in length and width and 256 in number of channels. It is then superimposed on the third feature map C3 of the encoder after being processed by the ECA module. After one "convolution-normalization-activation", the output is a 64×64×256 third feature map of the decoder.
[0114] After deconvolution, the decoder's third feature map becomes 128×128 in length and width, and the number of channels becomes 128. It is superimposed with the encoder's second feature map C2 after being processed by the SE module. After one "convolution-normalization-activation", the decoder's second feature map of 128×128×128 is output.
[0115] After deconvolution, the second feature map of the decoder becomes 256×256 in length and width, with 64 channels. It is superimposed with the first feature map C1 of the encoder after processing by the SE module. After one "convolution-normalization-activation", the first feature map of the decoder is output as 256×256×64. Finally, the prediction result is obtained after one convolution operation.
[0116] S4. Setting parameters:
[0117] Obtain training parameters from the model; using the training set data processed in step S2 as the data source, adjust the model's running parameters to achieve the best running performance, and obtain the model training settings parameters.
[0118] In step S4, the parameters to be set include: optimizer, learning rate, batch size, and loss function.
[0119] S5. Obtain the model training results:
[0120] The weight parameters obtained after model training are saved, and the model built by S3 and the obtained training results are used to extract features from the dataset.
[0121] In step S5, obtaining the model training results specifically includes: training the model with the training set, validating the training results with the validation set, and saving the set of results in which the model performs best on the test set. The criteria for determining the best performance are the loss value and the accuracy.
[0122] S6. Obtain the results of farmland planting information extraction:
[0123] By using a visualization method for model prediction results and employing pixel conversion technology to transform the model recognition results, a visualization of farmland planting information extraction is obtained and displayed.
[0124] In step S6, obtaining the farmland planting information extraction results specifically includes: using the result model obtained in S5, extracting farmland planting information from the test set data, and using pixel conversion technology to visualize and evaluate the prediction results.
[0125] Example:
[0126] The crop planting information extraction method based on convolutional neural networks provided by the present invention includes the following steps:
[0127] S1. Construct the processed raw multispectral satellite remote sensing image dataset:
[0128] The original remote sensing image data was labeled, with farmland covered by crops labeled as pixel value 255, farmland covered by unplanted crops labeled as pixel value 200, and other types of coverage labeled as pixel value 0.
[0129] The satellite remote sensing data is cropped to a size of 256×256 pixels.
[0130] The cropped remote sensing images were divided into training, validation, and test sets in an 8:1:1 ratio, resulting in an original remote sensing image dataset containing 5300 training images, 530 validation images, and 530 test images.
[0131] S2, Data Augmentation:
[0132] S2.1 Rotation: Rotate the processed original remote sensing image by ±90 degrees;
[0133] S2.2 Flip: Flip the processed remote sensing image after the rotation in step S2.1 with the x-axis and y-axis as the center respectively.
[0134] S3. Model Construction: Based on the U-Net model structure, the URSE-Net convolutional neural network model is constructed by integrating the SE and ECA attention mechanism modules and the residual module.
[0135] The URSE-Net model consists of three parts: an encoder, an intermediate layer, and a decoder. (Reference) Figure 1 The encoder part consists of five modules, each containing two residual module processing operations and one convolution operation. It also incorporates max pooling to compress the original image data size and increase the number of channels. The intermediate layer uses the four feature maps C1, C2, C3, and C4 output from the encoder part and processes them through an attention mechanism. The decoder part has a similar structure to the encoder part, also containing five modules, and each module contains only one "convolution-batch normalization-activation" operation. Combined with deconvolution, it restores the image size and compresses the number of channels.
[0136] S4. Parameter Settings: The model optimizer uses the adaptive momentum stochastic optimization method (Adam), with a learning rate of e^(-π / 2). 4 The batch size was set to 8, and cross-entropy was used as the loss function to optimize the training of the model.
[0137] The maximum number of iterations is set to 100, and early stopping is set to stop training when the model stops training after 10 iterations if the validation set loss does not decrease, and saves the set of weights that performed best on the validation set.
[0138] S5. Obtain the model training results:
[0139] The parameters obtained after model training are saved to obtain a farmland planting information extraction model, and the model and training results are used to extract features from the dataset.
[0140] In step S5, obtaining the model training results specifically includes: training the model with the training set, validating the training results with the validation set, and saving the set of results in which the model performs best on the test set.
[0141] S6. Obtain the results of farmland planting information extraction:
[0142] By using a visualization method for model prediction results and employing pixel conversion technology to transform the model recognition results, a visualization of farmland planting information extraction is obtained and displayed.
[0143] In step S6, obtaining the farmland planting information extraction results specifically includes: using the result model obtained in S5, extracting farmland planting information from the test set data, and using pixel conversion technology to visualize and evaluate the prediction results.
[0144] Five crop planting information extraction methods—URSE-Net, DeepLabv3+, PSPNet, U-Net, and SegNet—were compared using a comparative validation method. Each method was compared using the same training, validation, and test sets. To better utilize model performance and considering hardware capabilities, batch sizes were set to 4, 8, 16, 32, and 64, and the best results from each method were selected for comparison.
[0145] Under the same conditions, the prediction results of U-Net, Segnet, PSPnet, and Deeplabv3+ were analyzed respectively.
[0146] Based on the characteristics of this study, namely achieving accurate classification at the pixel level, pixel accuracy (PA) and mean intersection over union (mIoU) are used as evaluation metrics when studying the overall accuracy.
[0147] Table 1 Evaluation Indicators for Each Model
[0148]
[0149] In summary, the URSE-Net model demonstrates higher pixel accuracy and more balanced precision when using multispectral remote sensing imagery for crop planting information extraction. The model also exhibits good generalization ability and robustness.
Claims
1. A method for crop planting information extraction based on a convolutional neural network, characterized in that: The method includes the following steps: S1. Construct the processed raw multispectral satellite remote sensing image dataset; S2, Data Augmentation: The processed original multispectral remote sensing image dataset constructed in step S1 is enhanced to expand the number of images in the multispectral satellite remote sensing image dataset to 4 times the original number, resulting in a crop planting information dataset including farmland covered areas with crops, farmland covered areas without crops, and other types of covered areas. S3. Model Building: Based on the U-Net model structure, the URSE-Net convolutional neural network model is constructed by integrating the attention mechanism and residual module; The URSE-Net convolutional neural network model consists of three parts: an encoder, intermediate layers, and a decoder. In the encoder part, an improved residual module is used to extract information from remote sensing image data. In the intermediate layers, two channel attention mechanisms, SE and ECA, are used. In the decoder part, each layer uses a "convolution-batch normalization-activation" operation once. Step S3.1: Integrate the encoder of the fully pre-activated residual module; The backbone network is built using a fully preactivated residual module that places batch normalization and activation operations before convolution operations. The structure of the fully preactivated residual module is: batch normalization layer, activation layer, and 3×3 convolutional layer. The encoder is divided into five modules. Each module includes two residual module processing operations and one convolution operation. It also incorporates max pooling to compress the original image data size and increase the number of channels. The modules are connected through max pooling and random drop operations; Step S3.2: Use the intermediate layer of the channel attention mechanism (SE module) and ECA module; The encoder first feature map C1 obtained from the encoder first module and the encoder second feature map C2 obtained from the encoder second module are processed using the SE module; the encoder third feature map C3 obtained from the encoder third module and the encoder fourth feature map C4 obtained from the encoder fourth module are processed using the ECA module. Step S3.3: In the decoder section, each small portion of the decoder section uses the "convolution-batch normalization-activation" operation and the random dropout operation once; The decoder structure also contains five modules, and each module contains only one "convolution-batch normalization-activation" operation and a random dropout operation. Combined with the deconvolution operation, it realizes the restoration of image size and the compression of the number of channels. S4. Setting parameters: Obtain training parameters from the model; using the training set data processed in step S2 as the data source, adjust the model's running parameters to achieve the best running performance and obtain the model training settings parameters. S5. Obtain the model training results: The weight parameters obtained after model training are saved, and the model built by S3 and the obtained training results are used to extract features from the dataset. S6. Obtain the results of farmland planting information extraction: By using a visualization method for model prediction results and employing pixel conversion technology to transform the model recognition results, a visualization of farmland planting information extraction is obtained and displayed.
2. The method for extracting crop planting information based on convolutional neural networks as described in claim 1, characterized in that: In step S2, the data augmentation steps include: S2.1 Rotation: Rotate the processed original remote sensing image by ±90 degrees; S2.2 Flip: Flip the processed remote sensing image after the rotation in step S2.1 with the x-axis and y-axis as the center respectively. 3.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: In step S4, the parameters to be set include: optimizer, learning rate, batch size, and loss function. 4.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: In step S5, obtaining the model training results specifically includes: training the model with the training set, validating the training results with the validation set, and saving the set of results in which the model performs best on the test set. The criteria for determining the best performance are the loss value and the accuracy. 5.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: In step S6, obtaining the farmland planting information extraction results specifically includes: using the result model obtained in S5, extracting farmland planting information from the test set data, and using pixel conversion technology to visualize and evaluate the prediction results. 6.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: Step S1 also includes the following steps: S1.
1. Raw remote sensing image data is collected by remote sensing satellites to form a raw multispectral satellite remote sensing image dataset; The raw multispectral satellite remote sensing image dataset includes the following three types of multispectral remote sensing image data within farmland-covered areas:
1. Multispectral remote sensing image data of farmland covered by crops; 2. Multispectral remote sensing image data of farmland covered areas without crops; and 3. Multispectral remote sensing image data of other coverage areas; S1.2, Annotate the original remote sensing image data; Farmland covered with crops is marked with a pixel value of 255, farmland covered without crops is marked with a pixel value of 200, and other types of coverage are marked with a pixel value of 0. S1.
3. Crop the original multispectral remote sensing image data in the original multispectral satellite remote sensing image dataset. The pixel size of the cropped image is 256×256. S1.4 Divide the cropped original multispectral satellite remote sensing image dataset into training set, validation set and test set in a ratio of 8:1:1 to obtain the processed original multispectral satellite remote sensing image dataset. 7.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: In step 3.1, during the fully pre-activated residual module processing, the number of channels of the feature map remains constant. As shown in Formula 1, after extraction by the backbone network, four feature maps are obtained, which are in the order of being obtained as encoder first feature map C1, encoder second feature map C2, encoder third feature map C3, and encoder fourth feature map C4, with pixel values of 256×256×64, 128×128×128, 64×64×256, and 32×32×512, respectively. Formula 1 In Formula 1: m is the size of the output feature map in pixels; w is the size of the input feature map in pixels; f is the filter size in pixels; s is the step size in pixels. 8.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: In step 3.1, in the first module, the input image first undergoes a convolution operation to change the number of channels to 64, and then undergoes two residual module operations to obtain a feature map with a pixel value of 256×256 and a number of channels of 64. This feature map is used as the first feature map C1 of the encoder and output to the intermediate layer. The first feature map C1 of the encoder is halved in length and width by combining max pooling operation, and is input into the second module with a neuron dropout probability of 0.
25. In the second module, the feature map undergoes two residual operations to obtain a feature map with a pixel value of 128×128 and a channel number of 128. This feature map is used as the second feature map C2 of the encoder and output to the intermediate layer. The second feature map C2 of the encoder is halved in length and width by combining max pooling and is then input into the third module with a neuron dropout probability of 0.
5. In the third module, the feature map undergoes two residual operations to obtain a feature map with 64×64 pixels and 256 channels. This feature map is then used as the third feature map C3 of the encoder and output to the intermediate layer. The length and width of the encoder's third feature map C3 are halved by combining the max pooling operation, and then the feature map is input into the fourth module with a neuron dropout probability of 0.
5. In the fourth module, the feature map undergoes two residual operations to obtain a feature map with a pixel size of 32×32 and a channel number of 512. This feature map is used as the fourth feature map C4 and output to the intermediate layer. Combined with max pooling, the width and height of the encoder's fourth feature map C4 are halved, and it is input into the fifth module with a neuron dropout probability of 0.
5. In the fifth module, the feature map undergoes two residual operations to obtain the encoder's fifth feature map, which has a length and width pixel value of 16×16 and a channel number of 1024, and is then output to the decoder section. 9.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: In step 3.2, after the encoder's first feature map C1 and second feature map C2 are input into the SE module, they sequentially undergo global pooling, fully connected operation, activation operation, fully connected operation, and σ activation operation. During this process, the width × height × number of channels of their feature maps are as follows:
1. Same as the input; 2. Width and height both become 1, and the number of channels remains unchanged; 3. Width and height are 1, and the number of channels becomes one-sixteenth; 4. Width and height are 1, and the number of channels remains consistent with the input. The output after the σ activation operation is multiplied by the input to obtain an output feature map with the same size and number of channels as the input feature map, and this feature map is output to the decoder section. After the encoder's third feature map C3 and fourth feature map C4 are input, they first undergo a global average pooling operation, which sets their width and height to 1 while keeping the number of channels unchanged. Then, they undergo a one-dimensional convolution and σ activation function operation, keeping their width, height, and number of channels unchanged. The output of these features map is multiplied by the input feature map to obtain the output feature map, which is then output to the decoder. 10.The crop planting information extraction method based on a convolutional neural network according to claim 1, wherein: In step 3.3, in the decoder part, the fifth feature map of the encoder is first deconvolutionally processed to obtain the fifth module feature map of the decoder with a length and width of 32×32 and a channel number of 512. The feature map of the fifth module of the decoder is superimposed with the fourth feature map C4 of the encoder after being processed by the ECA module. After one "convolution-batch normalization-activation" operation and a random dropout operation, a 32×32×512 fourth feature map of the decoder is output. After deconvolution, the fourth feature map of the decoder becomes 64×64 in length and width and 256 in number of channels. It is then superimposed on the third feature map C3 of the encoder after being processed by the ECA module. After one "convolution-batch normalization-activation" operation and random discarding operation, the third feature map of the decoder is output as 64×64×256. After deconvolution, the decoder's third feature map becomes 128×128 in length and width, and the number of channels becomes 128. It is superimposed with the encoder's second feature map C2 after processing by the SE module. After one "convolution-batch normalization-activation" operation and random dropout operation, the decoder's second feature map of 128×128×128 is output. After deconvolution, the second feature map of the decoder becomes 256×256 in length and width, with 64 channels. It is superimposed with the first feature map C1 of the encoder after processing by the SE module. After one "convolution-batch normalization-activation" operation and random dropout operation, the first feature map of the decoder is output as 256×256×64. Finally, the prediction result is obtained after one convolution operation.