Remote sensing image crop classification method considering global consistency and edge specificity
By adopting Fourier frequency domain convolutional neural network in remote sensing crop classification and using Fourier transform and adaptive filter, the problems of internal integrity and boundary accuracy in remote sensing crop classification are solved, and classification accuracy and model are improved Performance, supports intelligent agricultural production.
Patent Information
- Application Number
- CN202411056523.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-15
AI Technical Summary
The existing remote sensing crop classification methods are difficult to achieve internal complete and accurate high-precision recognition in complex remote sensing images, and have poor in-class consistency and small differences between classes, resulting in a decrease in classification errors and model generalization capabilities.
Using the Fourier frequency domain convolutional neural network (FSEU-Net) that takes into account global consistency and edge specificity, the AFSF module is built through fast Fourier transform and adaptive dynamic filters to improve the spectrum domain feature learning ability and enhance crops between classes and narrow intra-class differences.
It improves the accuracy and model performance of crop classification, enhances the ability to capture complex features, provides more accurate crop identification results, and supports intelligent agricultural production.
Smart Images

Figure CN118968171A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image processing, in particular to a remote sensing image crop classification method taking into account global consistency and edge specificity. Background Art
[0002] Accurate crop classification is crucial for agricultural applications. Crop identification and monitoring methods driven by remote sensing data have become the main way to systematically map crop distribution at local and global scales. High spatial resolution images provide new ideas for high-precision remote sensing crop classification. However, due to unfavorable factors such as the complexity of remote sensing imaging conditions and the differences in crop growth, the challenges of crop identification are increased, and a series of problems such as classification errors, boundary fragmentation, and poor model generalization ability often occur in the identification results.
[0003] Deep learning has achieved great success in the field of vision. It has been gradually applied to the field of remote sensing agriculture and has become one of the important methods for remote sensing crop classification. At present, deep learning methods for remote sensing crop classification can be mainly divided into methods based on pixel classification and methods based on semantic segmentation. However, these methods only learn information such as the shape, edge and texture of crops from the perspective of the spatial domain of remote sensing images to classify crops. Due to the differences in crop canopy level spectral reflectance of crops in different environments and management modes, the spectral characteristics and texture characteristics of different crops in complex remote sensing images are usually inconsistent, and these characteristics are often difficult to distinguish and describe in the spatial domain. Under the conditions of poor intra-class consistency in the same crop and small differences between different crop classes, it is difficult to achieve high-precision crop extraction with complete internal structure and accurate boundaries by deep learning crop extraction only from the perspective of the spatial domain of remote sensing images. Fourier transform can convert spatial description into frequency domain, frequency domain decomposition can capture feature information that is difficult to obtain directly in the image, and frequency domain filters can enhance or attenuate specific frequency domain components to expand the inter-class differences of different crops and reduce the intra-class differences of the same crops, thereby improving the accuracy of crop identification. Therefore, it is urgent and necessary to study the remote sensing crop classification technology of Fourier frequency domain convolutional neural network that takes into account global consistency and edge specificity.
[0004] Under current technical conditions, remote sensing crop classification technology based on deep learning faces the following challenges: 1) Due to the limitations of remote sensing image imaging conditions and crop growth environment, the spectral characteristics and texture characteristics of different crops vary greatly. The recognition results of traditional remote sensing crop classification methods have a series of problems such as classification errors, boundary fragmentation, and decreased model generalization ability; 2) Deep learning methods based on pixel or semantic segmentation are used for crop extraction. Under the conditions of poor intra-class consistency in the same crop and small differences between different crop classes, it is difficult to achieve high-precision crop extraction with complete internal structure and accurate boundaries; 3) There are few studies on using frequency domain learning to improve the internal integrity and edge accuracy of remote sensing deep learning models for crop classification, and there is a lack of available public crop image datasets. Therefore, how to improve model performance as much as possible under the premise of limited data is the primary problem to be solved in this field. Summary of the invention
[0005] In order to better utilize frequency domain learning to expand the inter-class differences of different crops and reduce the intra-class differences of the same crop, and improve the classification accuracy of crop development, the present invention provides a remote sensing image crop classification method that takes into account global consistency and edge specificity to solve the above problems.
[0006] To achieve the above object, the present invention adopts the following technical solutions, including:
[0007] The crop classification method of remote sensing images taking into account global consistency and edge specificity includes the following steps:
[0008] S1, acquisition and preprocessing of remote sensing images;
[0009] S2, using remote sensing images to create a crop sample set;
[0010] S3, using the crop sample set, trains a convolutional neural network model to perform semantic segmentation on crop plots in remote sensing images and output the types and distribution areas of crops;
[0011] S4, model testing, to achieve crop classification result output.
[0012] Preferably, the specific process of step S1 is as follows:
[0013] S11, collect remote sensing images for the research object, i.e., crops;
[0014] S11, preprocessing of remote sensing images, mainly including radiation calibration, atmospheric correction, geometric correction, image fusion and image mosaicking.
[0015] Preferably, the specific process of step S2 is as follows:.
[0016] S21, vector mapping and labeling of crops and background plots in the remote sensing image based on ground truth data and combined with manual visual interpretation to obtain a labeled image;
[0017] S22, cropping the remote sensing image and the corresponding annotated image respectively to obtain remote sensing image slices and corresponding annotated image slices; the remote sensing image slices and the corresponding annotated image slices constitute the crop samples;
[0018] S23, divide all sample data into training data set and test data set according to the proportion.
[0019] Preferably, in step S3, the convolutional neural network model adopts the FSEU-Net network.
[0020] Preferably, the structure of the FSEU-Net network is as follows:
[0021] The encoding part of the FSEU-Net network first performs feature coarse extraction and dimension upscaling through a depth-wise separable convolutional layer, and then sequentially connects four adaptive dynamic filter modules, namely AFSF modules, and designs a maximum pooling layer after each AFSF module for downsampling;
[0022] The decoding part of the FSEU-Net network also contains four AFSF modules connected in sequence, which use upsampling layers to gradually restore the original size of the input image;
[0023] Add skip connections between the corresponding AFSF modules in the encoding part and the decoding part to mix and connect low-level and high-level semantic information;
[0024] Finally, the result is output through a convolutional layer.
[0025] Preferably, the adaptive dynamic filtering module, ie, the AFSF module, includes a dynamic frequency domain filter that is a combination of low-pass filters and high-pass filters with different kernel sizes.
[0026] Preferably, the structure of the AFSF module is as follows:
[0027] First, a bottleneck layer is designed. Fast Fourier transform is used in the bottleneck layer to convert the spatial domain tensor into the spectral domain. The relatively smooth structural information inside the crop is converted into low-frequency signals, and the more detailed contour and texture information is converted into high-frequency signals.
[0028] Next, design the low-pass filter and high-pass filter in parallel:
[0029] The high-pass filter structure is as follows: the channels of the input signal are split into N groups, each channel group uses a grouped convolution layer with different convolution kernel sizes to simulate the cutoff frequencies in different high-frequency filters to filter low-frequency signals, and then a series layer is used to merge the channel results for high-frequency signal fusion;
[0030] The low-pass filter structure is as follows: multiple adaptive average pooling layers with different pooling window sizes are used as low-pass filters to filter high-frequency signals, and then a series layer is used to merge the channel results to perform low-frequency signal fusion;
[0031] Then, the channels are combined and fused across channels to reconstruct the frequency domain signal, with the combined signal as output;
[0032] Finally, a bottleneck layer is designed to transform back to the spatial domain through inverse Fourier transform.
[0033] Preferably, the specific process of step S4 is as follows:
[0034] S41, inputting the sample data to be tested into the model for semantic segmentation to obtain crop classification results;
[0035] S42, compare the crop classification results and the samples to be tested pixel by pixel, calculate the average intersection over union (mIoU), accuracy (OA), recall (Recall), precision (Precision), F1-score and Kappa coefficient of the classification results, and evaluate the accuracy of the model classification results. The specific calculation formula is as follows:
[0036]
[0037] Where TP represents the number of correctly predicted positive pixels, TN represents the number of correctly predicted negative pixels, FP represents the number of non-pixels incorrectly predicted as positive, FN represents the number of positive pixels incorrectly predicted as negative, and k represents the number of categories.
[0038] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the above-mentioned remote sensing image crop classification method taking into account global consistency and edge specificity.
[0039] An electronic device comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the remote sensing image crop classification method taking into account global consistency and edge specificity is implemented.
[0040] The advantages of the present invention are:
[0041] The present invention discloses a remote sensing crop classification method of a Fourier frequency domain convolutional neural network that takes into account global consistency and edge specificity. Aiming at the problems of high-precision crop identification such as internal fragmentation, boundary discontinuity, and category identification errors in the current high-resolution remote sensing crop classification results, the spectral domain feature learning is used to enhance the deep learning network's ability to capture complex crop features. The AFSF module is constructed using a Fourier transform strategy and an adaptive dynamic filter, and a convolutional neural network based on Fourier spectrum domain learning, FSEU-Net, is designed to enhance the differences between different crop classes and reduce the differences within the same crop class, enhance the separability between crops, improve the performance of the model, and provide an effective technical support for accurate statistics of intelligent agricultural production of global land area. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a flow chart of the remote sensing image crop classification method taking into account global consistency and edge specificity of the present invention.
[0043] Figure 2 Schematic diagram of the structure of the FSEU-Net network.
[0044] Figure 3 Schematic diagram of the structure of the adaptive dynamic filtering module (AFSF module). DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] Example 1
[0047] Depend on Figure 1 As shown in FIG. 1 , the crop classification method of remote sensing images taking into account global consistency and edge specificity includes the following steps:
[0048] S1, acquisition and preprocessing of remote sensing images;
[0049] S2, using remote sensing images to create a crop sample set;
[0050] S3, using the crop sample set, trains a convolutional neural network model to perform semantic segmentation on crop plots in remote sensing images to obtain the types and distribution areas of crops;
[0051] S4, model testing, to achieve crop classification result output.
[0052] The step S1 specifically includes the following contents:
[0053] S11, collect high temporal resolution remote sensing images for the research target crops;
[0054] S12, preprocessing of remote sensing images, mainly including radiation calibration, atmospheric correction, geometric correction, image fusion, and image mosaicking.
[0055] The step S2 specifically includes the following contents:
[0056] S21, vector mapping and labeling of soybean, corn and background plots in remote sensing images based on ground truth data and combined with manual visual interpretation;
[0057] S22, batch cropping the remote sensing images and the annotated images to obtain remote sensing image slices and corresponding annotated image slices, where the slice size is 256×256 pixels; the remote sensing image slices and the corresponding annotated image slices constitute the crop samples;
[0058] S23, divide all sample data into a training data set and a test data set in a ratio of 8:2 to produce a crop sample set.
[0059] In step S3, the convolutional neural network model uses the FSEU-Net network to perform semantic segmentation on soybean and corn planting plots in remote sensing images to obtain the types and distribution areas of crops. The FSEU-Net network structure includes a deep separable convolution layer, a dynamic frequency domain filter, a maximum pooling layer, and a skip connection.
[0060] Depend on Figure 2 As shown, specifically, the structure of the FSEU-Net network is as follows:
[0061] The encoding part of the FSEU-Net network first performs feature coarse extraction and dimension upscaling through a 1x1 depthwise separable convolutional layer to expand the feature tensor signal channel; then it sequentially connects four adaptive dynamic filter modules, namely AFSF modules, and designs a 2x2 maximum pooling layer with a step size of 2 after each AFSF module for downsampling;
[0062] The decoding part of the FSEU-Net network also contains four AFSF modules connected in sequence, which use upsampling layers to gradually restore the original size of the input image;
[0063] Add skip connections between the corresponding AFSF modules in the encoding part and the decoding part to mix and connect low-level and high-level semantic information, generate accurate masks, alleviate the gradient vanishing problem and capture effective features;
[0064] Finally, the result is output through a convolutional layer.
[0065] In the present invention, the model optimizer selects adam, the initial learning rate is 0.0001, the decay rate is 0.5, the batch size is 16, and the total number of training rounds is 60 rounds.
[0066] To enhance the semantic features, the adaptive dynamic filtering module (AFSF module) consists of low-pass filters and high-pass filters with different kernel sizes.
[0067] Depend on Figure 3 As shown, specifically, the structure of the AFSF module is as follows:
[0068] First, a bottleneck layer is designed. Fast Fourier transform is used in the bottleneck layer to convert the spatial domain tensor into the spectral domain. The relatively smooth structural information inside the crop is converted into low-frequency signals, and the more detailed contour and texture information is converted into high-frequency signals.
[0069] Next, design the low-pass filter and high-pass filter in parallel:
[0070] The high-pass filter structure is as follows: the channels of the input signal are split into N groups, and each channel group uses a grouped convolution layer with different convolution kernel sizes of 3x3, 5x5, and 7x7 to simulate the cutoff frequencies in different high-frequency filters to filter low-frequency signals; then a series layer is used to merge the channel results for high-frequency signal fusion;
[0071] The low-pass filter structure is as follows: multiple adaptive average pooling layers with different pooling window sizes are used as low-pass filters to filter high-frequency signals; then a series layer is used to merge channel results to perform low-frequency signal fusion;
[0072] Then, the channels are combined and fused across channels to reconstruct the frequency domain signal, with the combined signal as output;
[0073] Finally, a bottleneck layer is designed to transform back to the spatial domain through inverse Fourier transform, and its feature map size is the same as the input data.
[0074] In addition, the module is designed to fuse the input data as residual with the output data after inverse Fourier transformation to make up for the loss of local features caused by frequency domain feature extraction and realize the extraction of global and local features.
[0075] The step S4 specifically includes the following contents:
[0076] S41, inputting the sample data to be tested into the model for semantic segmentation to obtain crop classification results;
[0077] S42, use relevant software to compare the crop classification results and the samples to be tested pixel by pixel, calculate the mean intersection over union (mIoU), accuracy (OA), recall rate (Recall), precision rate (Precision), F1-score and Kappa coefficient of the classification results, and evaluate the accuracy of the model classification results. The specific calculation formula is as follows:
[0078]
[0079] Where TP represents the number of correctly predicted positive pixels, TN represents the number of correctly predicted negative pixels, FP represents the number of non-pixels incorrectly predicted as positive, FN represents the number of positive pixels incorrectly predicted as negative, and k represents the number of categories.
[0080] Example 2
[0081] The present invention provides a specific embodiment, and selects six farms, namely Ganhe, Guli, Yili, Najitun, Dahewan, and Dongfanghong, as the study area, and collects Planetscope data covering the study area in mid-August 2022. Taking the ground survey data in October 2022 as the true value, the remote sensing image is visually interpreted, and the cropped 1908 samples are used as the training data set, and the 477 sample data are used as the test data set. In addition, in order to compare the superiority of the model of the present invention compared with other commonly used models, Unet, Deeplab V3+, PSPNet, and MNFEnet are selected for comparative analysis, and the comparison results are shown in Table 1 below.
[0082] Table 1 Average accuracy evaluation results of different models
[0083]
[0084] Example 3
[0085] An electronic device comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the processor implements the remote sensing image crop classification method taking into account global consistency and edge specificity of the above-mentioned embodiment 1 when executing the computer program.
[0086] The electronic device of the embodiment of the present application can be the mobile device itself, or a stand-alone device independent of the mobile device, which can communicate with the mobile device to receive the collected input signals from them and send the selected target decision behavior to them.
[0087] The electronic device includes one or more processors and memory. The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may run the program instructions to implement the decision behavior decision method and / or other desired functions of the various embodiments of the present application described above.
[0088] Electronic devices may also include input devices and output devices.
[0089] Example 4
[0090] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the remote sensing image crop classification method that takes into account global consistency and edge specificity as described in the above-mentioned embodiment 1 of this specification.
[0091] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0092] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A remote sensing image crop classification method that takes into account global consistency and edge specificity. It is characterized in that The following steps are involved: S1, acquisition and preprocessing of remote sensing images; S2, using remote sensing images to create a crop sample set; S3, using the crop sample set, trains a convolutional neural network model to perform semantic segmentation on crop plots in remote sensing images and output the types and distribution areas of crops; S4, model testing, to achieve crop classification result output.
2. The remote sensing image crop classification method taking into account global consistency and edge specificity according to claim 1, It is characterized in that The specific process of step S1 is as follows: S11, collect remote sensing images for the research object, i.e., crops; S11, preprocessing of remote sensing images, mainly including radiation calibration, atmospheric correction, geometric correction, image fusion and image mosaicking.
3. The remote sensing image crop classification method taking into account global consistency and edge specificity according to claim 1, It is characterized in that The specific process of step S2 is as follows:. S21, vector mapping and labeling of crops and background plots in the remote sensing image based on ground truth data and combined with manual visual interpretation to obtain a labeled image; S22, cropping the remote sensing image and the corresponding annotated image respectively to obtain remote sensing image slices and corresponding annotated image slices; the remote sensing image slices and the corresponding annotated image slices constitute the crop samples; S23, divide all sample data into training data set and test data set according to the proportion.
4. The remote sensing image crop classification method taking into account global consistency and edge specificity according to claim 1, It is characterized in that In step S3, the convolutional neural network model adopts the FSEU-Net network.
5. The remote sensing image crop classification method taking into account global consistency and edge specificity according to claim 4, It is characterized in that The structure of the FSEU-Net network is as follows: The encoding part of the FSEU-Net network first performs feature coarse extraction and dimension upscaling through a depth-wise separable convolutional layer, and then sequentially connects four adaptive dynamic filter modules, namely AFSF modules, and designs a maximum pooling layer after each AFSF module for downsampling; The decoding part of the FSEU-Net network also contains four AFSF modules connected in sequence, which use upsampling layers to gradually restore the original size of the input image; Add skip connections between the corresponding AFSF modules in the encoding part and the decoding part to mix and connect low-level and high-level semantic information; Finally, the result is output through a convolutional layer.
6. The remote sensing image crop classification method taking into account global consistency and edge specificity according to claim 5, It is characterized in that The adaptive dynamic filtering module, AFSF module, includes a dynamic frequency domain filter composed of a combination of low-pass filters and high-pass filters with different kernel sizes.
7. The remote sensing image crop classification method taking into account global consistency and edge specificity according to claim 6, It is characterized in that The structure of the AFSF module is as follows: First, a bottleneck layer is designed. Fast Fourier transform is used in the bottleneck layer to convert the spatial domain tensor into the spectral domain. The relatively smooth structural information inside the crop is converted into low-frequency signals, and the more detailed contour and texture information is converted into high-frequency signals. Next, design the low-pass filter and high-pass filter in parallel: The high-pass filter structure is as follows: the channels of the input signal are split into N groups, each channel group uses a grouped convolution layer with different convolution kernel sizes to simulate the cutoff frequencies in different high-frequency filters to filter low-frequency signals, and then a series layer is used to merge the channel results for high-frequency signal fusion; The low-pass filter structure is as follows: multiple adaptive average pooling layers with different pooling window sizes are used as low-pass filters to filter high-frequency signals, and then a series layer is used to merge the channel results to perform low-frequency signal fusion; Then, the channels are combined and fused across channels to reconstruct the frequency domain signal, with the combined signal as output; Finally, a bottleneck layer is designed to transform back to the spatial domain through inverse Fourier transform.
8. The remote sensing image crop classification method taking into account global consistency and edge specificity according to claim 1, It is characterized in that The specific process of step S4 is as follows: S41, inputting the sample data to be tested into the model for semantic segmentation to obtain crop classification results; S42, compare the crop classification results and the samples to be tested pixel by pixel, calculate the average intersection over union (mIoU), accuracy (OA), recall (Recall), precision (Precision), F1-score and Kappa coefficient of the classification results, and evaluate the accuracy of the model classification results. The specific calculation formula is as follows: Where TP represents the number of correctly predicted positive pixels, TN represents the number of correctly predicted negative pixels, FP represents the number of non-pixels incorrectly predicted as positive, FN represents the number of positive pixels incorrectly predicted as negative, and k represents the number of categories.
9. A computer program product, It is characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, realizes the remote sensing image crop classification method taking into account global consistency and edge specificity as described in any one of claims 1 to 8.
10. An electronic device, It is characterized in that The invention comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the remote sensing image crop classification method taking into account global consistency and edge specificity as described in any one of claims 1 to 8 is implemented.
