Crop recognition method in foggy areas based on multi-source remote sensing data enhancement

Abstract

The present invention provides a method for identifying crops in foggy areas based on multi-source remote sensing data enhancement, and belongs to the technical field of image processing and recognition. In order to solve the problems in the existing crop remote sensing identification method that optical image data is difficult to obtain in foggy areas, and the recognition accuracy based only on radar images or single-period cloudless optical remote sensing images is low, the present invention introduces a deep learning algorithm, introduces a triple attention mechanism in a generative adversarial network, realizes data enhancement based on time series multi-source remote sensing data, utilizes the enhanced time series cloudless optical images, and constructs a deep spatiotemporal feature extraction network and a dual-branch deep neural network based on the differences in crop phenological characteristics, enhances the feature extraction and type discrimination capabilities of crop identification, and finally realizes accurate identification of crop types in foggy areas and efficient extraction of planting areas, which is of great significance to adhering to the principle of intensive resource utilization, scientifically and rationally utilizing cultivated land resources, and implementing the strictest cultivated land protection system.

Description

Technical Field

[0001] The invention relates to the technical field of image processing and recognition, and in particular to a method for identifying crops in foggy areas based on multi-source remote sensing data enhancement. Background Art

[0002] Crop variety identification is an important part of supporting the monitoring of cultivated land planting conditions. The identification results are an important basis for adhering to the principle of intensive resource utilization, scientifically and rationally utilizing cultivated land resources, and implementing the most stringent cultivated land protection system. Remote sensing technology has the characteristics of large coverage area and short acquisition cycle, and is widely used in crop identification and planting information extraction.

[0003] Existing crop remote sensing identification methods are usually based on multi-temporal and multi-spectral remote sensing images to extract spectral information, vegetation index and other features that are sensitive to crop growth conditions. However, optical remote sensing images are difficult to obtain in cloudy areas, and the crop identification accuracy based only on radar images or single-period cloud-free optical remote sensing images is low, resulting in low reliability and accuracy of crop identification in cloudy areas. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for crop identification in foggy areas based on multi-source remote sensing data enhancement, which is used to solve the problem of low reliability and accuracy of crop identification in foggy areas.

[0005] In order to achieve the above-mentioned object of the invention, in a first aspect, the present invention proposes a method for identifying crops in foggy areas based on multi-source remote sensing data enhancement, comprising:

[0006] Step S1, constructing and training a deep learning model based on cloud-free optical images and radar images;

[0007] Step S2: Generate cloud-free optical images based on the trained model and radar images of the cloud and fog area to achieve multi-source remote sensing data enhancement;

[0008] Step S3: constructing a deep spatiotemporal feature extraction network based on the generated time-series cloud-free optical images, and extracting crop identification features;

[0009] Step S4: construct a dual-branch deep neural network, input the crop identification features extracted in step S3, and carry out crop type identification and planting area extraction.

[0010] Furthermore, step S1 further comprises:

[0011] Step S11: acquiring radar images with similar time phases for the cloud-free optical images, and registering, resampling, and cropping the cloud-free optical images and radar images to a uniform resolution and size to obtain an image sample set;

[0012] Step S12: construct a deep learning model based on a generative adversarial network with an attention mechanism, and adopt a loss function that combines a structural similarity index, an L1 norm, and a triple attention loss;

[0013] Step S13: Based on the image sample set and deep learning model in steps S11-S12, training is performed to obtain the optimal model of Nash equilibrium.

[0014] Furthermore, step S12 further includes: constructing a generation network, adopting a U-net network structure that introduces a triple attention mechanism, which is composed of an encoder and a decoder, and the triple attention mechanism includes: a spatial attention module for capturing the spatial dependency of images, a channel attention module for learning the importance of data in different bands, and a temporal attention module for modeling the temporal association of multi-phase data; in each decoding, except for the first decoding, which is the input of the final feature map after the last encoding, the input of each other decoding is: the decoding feature map of the previous round, the encoding feature map of the corresponding size, and the feature map after adaptive weighting by the triple attention mechanism, wherein the adaptive The weighting process includes spatial dimension feature enhancement, channel dimension feature enhancement and temporal dimension feature enhancement; construct a discriminant network and use a Markov discriminator, that is, use the discriminant network to distinguish true from false for each image block of size N×N. If a real cloud-free optical image block is input, the output is 1, and if a generated cloud-free optical image block is input, the output is 0. The average of the true and false results of the full-frame image block is taken as the final discrimination result output of the image; construct a loss function and use a loss function that combines the structural similarity index, L1 norm and multiple attention loss. The multiple attention loss is used to optimize the feature learning of the three dimensions of space, channel and time, and the calculation formula is:

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] in, is the overall objective function, To generate the adversarial network objective function, is the loss function; and In the calculation formula, represents the cloud-free radar image in the image sample set, represents the cloud-free optical image in the image sample set, represents random noise, represents the generative network, represents the discriminant network, represents the minimum loss of the generated network G, Represents the maximum loss of the discriminant network; represents the natural logarithm, represents the expected value operation, is the distribution of real cloud-free image data, To generate the distribution of image data, It represents the expected calculation of x and y for the data sampled from the distribution of real cloud-free image data. Indicates the expected calculation of x and z based on the data sampled from the distribution of generated image data; It means that the discriminant network accepts input x and conditional information y, and outputs a probability value, which indicates the probability that the input data is the real data. It means that the generating network accepts input x and random noise z, and tries to generate data related to the conditional information y. Represents the evaluation of the discriminant network D on the output of the generative network G; In the calculation formula, is the L1 norm, is the structural similarity index, , , They are spatial attention loss, channel attention loss and temporal attention loss, which are used to optimize the feature learning of the corresponding dimensions. , , is the weight coefficient of each loss item; Represents the distribution of real cloud-free image data The data pairs (x, y) sampled from the image are obtained, and the distribution of the generated image data is obtained from Calculate the expected value of the variable z sampled from the In the calculation formula, represents the image block size, , express , The mean of , express , The variance of express , The covariance between , To avoid a constant with a denominator of zero; in the attention loss calculation formula, , , Represent the feature extraction functions of spatial dimension, channel dimension and temporal dimension respectively, , , They represent the attention graph generation functions of the corresponding dimensions respectively, Represents element-wise multiplication.

[0025] Furthermore, step S13 further includes: inputting the image sample set and random noise into the deep learning model constructed in S12, first fixing the discriminant network parameters, adjusting the generating network parameters, optimizing the generating network, then fixing the generating network parameters, adjusting the discriminant network parameters, optimizing the discriminant network, repeating this step until Nash equilibrium is reached to obtain the optimal model.

[0026] Furthermore, step S2 further comprises:

[0027] Step S21: for the original optical image disturbed by clouds and fog, a radar image with a phase smaller than a preset value is selected as an input source, and the deep learning optimal model obtained in S1 is input to generate a cloud-free optical image;

[0028] Step S22: Based on the data of the area not affected by cloud and fog in the original optical image and the corresponding data of the generated cloud-free optical image, a linear regression equation group is constructed to further enhance the generated cloud-free optical image. The calculation formula is:

[0029] ,

[0030] in, is the line number, is the column number, are channel categories, namely red band, green band, blue band, and near-infrared band. The original optical image Cells in the channel ( ) pixel value, for The initial enhancement factor of the channel (set to 1), To generate optical images Channel cell ( ), is a random error. After inputting all the data of the areas not affected by fog and cloud in the original image for regression analysis, the multi-source remote sensing data enhancement model for fog and cloud areas is obtained. The calculation formula is:

[0031]

[0032] in, It is the enhanced cloud-free optical image. Cells in the channel ( ), for The enhancement factor of the channel, Based on radar data , random noise , Deep Learning Optimal Model Generated cloud-free optical image Cells in the channel ( )’s pixel value.

[0033] Furthermore, step S3 further includes:

[0034] Step S31, constructing a three-dimensional convolutional neural network for extracting features from the enhanced time-series optical image;

[0035] Step S32: construct a spatiotemporal attention mechanism module, including a temporal attention submodule and a spatial attention submodule, for all channel features extracted in S31 Conduct dual attention enhancement in time and space;

[0036] Step S33: construct a feature aggregation module, and use a multi-scale feature pyramid structure to fuse spatiotemporal features at different levels.

[0037] Furthermore, step S31 further includes: inputting time-series optical image data of dimensions T×H×W×C, where T is the length of the time series, H is the image height, W is the image width, and C is the number of image channels, and outputting the feature map containing all channels. , the three-dimensional convolutional neural network includes three three-dimensional convolutional layers, and the calculation formula is:

[0038]

[0039] in, Indicates that the input is The first Layer feature map, Indicates The convolution kernel parameters of the layer, is the convolution kernel size, Represents input After the phase image, Layer cells ( ), is a bias term, and each convolutional layer is followed by a normalization layer and a ReLU activation function.

[0040] Furthermore, step S32 further includes: the temporal attention submodule uses a self-attention mechanism to enhance the relevance of features of different time steps, and the calculation formula is:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] in, , , are the three vectors in the self-attention mechanism, representing the query vector, key vector and numerical vector respectively. , , is a learnable linear transformation matrix, is the scaling factor, is the normalization function, Temporal attention weight matrix, It is the feature after temporal attention weighting, indicating the strength of association between different time steps;

[0047] The spatial attention submodule constructs a spatial attention map based on feature statistics, and the calculation formula is:

[0048]

[0049]

[0050]

[0051]

[0052] in, is the number of feature channels, and are the average and maximum feature maps of all channels respectively, is a 7×7 two-dimensional convolutional layer, is the sigmoid activation function, is the spatial attention map, represents broadcast multiplication, The features enhanced by temporal and spatial dual attention will be used as the input of the feature pyramid network.

[0053] Furthermore, step S33 further includes: Aggregate the time dimension and eliminate the time dimension by weighted average, where Layer characteristics The calculation formula is:

[0054]

[0055] in, is the length of the time series, is the time step The weight coefficient, the initial value can be 1 / , can be learned through back-propagation; Represents different levels of features;

[0056] Then, construct a feature pyramid, calculate layer by layer from top to bottom, and then fuse the features of feature pyramids at different levels:

[0057]

[0058]

[0059] in, Represents the feature pyramid Layer features, Indicates The learnable weight matrix of the layer, (·) represents bilinear interpolation upsampling, Indicates the fusion of the total Feature information at the layer scale, is a scalar weight coefficient, learned by back-propagation.

[0060] Furthermore, step S4 further comprises:

[0061] Step S41, constructing a dual-branch deep neural network, including a recognition branch and a segmentation branch;

[0062] Step S42: Use a multi-task joint training strategy to optimize network parameters, and then perform connected domain analysis and optimization processing on the network output results to obtain the final crop recognition and area extraction results.

[0063] Furthermore, step S41 further includes: constructing a recognition branch network of a full convolutional network structure, inputting the recognition branch network obtained in step S33 , output class probability map , calculate the category probability of each pixel position, where the loss function is:

[0064]

[0065] in, , are the feature map space sizes, is the normalization coefficient, is the number of channels, For the cell ( ) The label value of the channel value after one-hot encoding is 0 or 1. The class probability predicted by the network is the cell ( ) Middle The category prediction probability value of the channel, is a logarithmic operation;

[0066] The encoder-decoder structure is used to construct the segmentation branch network. The number of layers of the encoding block and the decoding block is the same. Each encoding block contains two convolutional layers and one maximum pooling layer. Each decoding block contains a transposed convolutional layer and two convolutional layers. Finally, a single-channel segmentation mask is output. The loss function is:

[0067]

[0068] in, , are the feature map space sizes, is the segmentation probability predicted by the network, is the true segmentation label.

[0069] Furthermore, step S42 further includes: optimizing network parameters by adopting a multi-task joint training strategy, and training using an Adam optimizer, wherein the loss function is a weighted combination of the two-branch deep neural network loss function in step S41:

[0070]

[0071] in, , is the weight coefficient, which is used to balance the contribution of the recognition task and the segmentation task;

[0072] Then, the connectivity analysis is performed on the network output results to optimize the segmentation results, and finally the crop type recognition and planting area extraction are realized.

[0073] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of any method described in the first aspect.

[0074] In order to solve the problems in the existing crop remote sensing identification methods, such as the difficulty in obtaining optical image data in foggy areas, and the low recognition accuracy based only on radar images or single-period cloud-free optical remote sensing images, the present invention introduces a deep learning algorithm, introduces an attention mechanism in a generative adversarial network, and realizes data enhancement based on time-series multi-source remote sensing data. By utilizing the enhanced time-series cloud-free optical images, a deep spatiotemporal feature extraction network and a dual-branch deep neural network are constructed based on the differences in crop phenological characteristics, and the feature extraction and type discrimination capabilities for crop identification are enhanced, ultimately achieving accurate identification of crop types in foggy areas and efficient extraction of planting areas, which has important scientific and practical guiding significance for adhering to the principle of intensive resource utilization, scientifically and rationally utilizing cultivated land resources, and implementing the strictest cultivated land protection system.

[0075] The beneficial effect of the above-mentioned further scheme is: when identifying the crop types in foggy areas, in order to make full use of the characteristic information that is sensitive to the phenological differences of crops, the present invention generates a time-series cloud-free optical image through steps S1-S2, realizes multi-source remote sensing data enhancement in foggy areas, and solves the difficulty in obtaining optical image data, and then further extracts the spatiotemporal characteristics coupled with the growth cycle of crops through steps S3-S4, thereby realizing accurate identification of crop types and efficient extraction of areas in foggy areas based on multi-source remote sensing data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0077] Figure 1 The present invention is a flowchart of a method for identifying crops in foggy areas based on multi-source remote sensing data enhancement.

[0078] Figure 2 It is a schematic diagram of the data enhancement process based on the deep learning model in the present invention. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the solutions in the embodiments of the present invention are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0080] It should be noted that the embodiments of the present invention and the features therein may be combined with each other as long as there is no conflict between the combinations.

[0081] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0082] like Figure 1 As shown in the first embodiment of the present invention, the present invention discloses a method for identifying crops in foggy areas based on multi-source remote sensing data enhancement, comprising the following steps:

[0083] Step S1, constructing a multi-source remote sensing data enhanced deep learning model based on cloud-free optical images and radar images and training the model;

[0084] Step S2: inputting the radar image of the cloud area into the trained deep learning model to generate a cloud-free optical image;

[0085] Step S3: constructing a deep spatiotemporal feature extraction network based on the generated time-series cloud-free optical images, and extracting crop identification features;

[0086] Step S4: construct a dual-branch deep neural network, input the crop identification features extracted in S3, and carry out crop type identification and planting area extraction.

[0087] like Figure 2 As shown, further, step S1 further includes:

[0088] Step S11, for the cloud-free optical image, obtain a radar image with a similar time phase, and align the cloud-free optical image and the radar image, resample them to a uniform resolution, and crop them to a uniform size to obtain an image sample set. In this embodiment, the spatial resolution of the image sample set is 10 meters, the size is 256×256, and the time difference between the optical image and the radar image is less than a preset value, which may be 10 days. Those skilled in the art should know that the smaller the preset time of the time difference, the smaller the difference in ground information obtained by the optical image and the radar image of the same area. In another embodiment of the present application, the preset value may also be set to 7 days or 5 days.

[0089] Step S12: construct a deep learning model based on a generative adversarial network with an attention mechanism, and adopt a loss function combining a structural similarity index, an L1 norm, and multiple attention losses;

[0090] Step S13: Based on the image sample set and deep learning model in steps S11-S12, training is performed to obtain the optimal model of Nash equilibrium.

[0091] Further, in step S12, a generation network is constructed, and a U-net network structure with a triple attention mechanism is adopted. In this embodiment, the total number of layers of the generation network is 16, wherein the encoder is composed of 8 convolutional layers, and the decoder is composed of 8 deconvolutional layers. The convolution kernel size of the encoder convolutional layer is 4×4, the stride is 2, all convolutional layers adopt the reflect filling mode, and use the batch norm layer. The activation function uses the Leaky-ReLU function. In the triple attention mechanism, the spatial attention module extracts spatial features through two convolutional layers with a convolution kernel size of 7×7, and uses the sigmoid function to generate a spatial attention map. The channel attention module adopts global average pooling and maximum pooling. Through the MLP layer, the number of channels is reduced by 16 to learn the relationship between channels. The temporal attention module uses a three-dimensional convolutional layer with a convolution kernel size of 3×3× 3. Processing time series information. The kernel size of the deconvolution layer of the decoder is 4×4, and the stride is 2. In order to prevent overfitting, the Dropout operation is used. Except for the first decoding, the final feature map after 8 encodings is input. For each decoding, the decoding feature map of the previous round and the encoding feature map of the corresponding size, as well as the feature map adaptively weighted by the triple attention mechanism are input; a discriminant network is constructed, and a Markov discriminator is used. The convolution kernel size is 4×4. The discriminant network is used to distinguish the true and false of each image block of size 30×30, and the average of the true and false results of the full-frame image block is taken as the final discrimination result output of the image; a loss function is constructed, and a loss function combining the structural similarity index, the L1 norm and the triple attention loss is used. The triple attention loss is used to optimize the feature learning of the three dimensions of space, channel and time series. The calculation formula is:

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] in, is the overall objective function, To generate the adversarial network objective function, is the loss function; and In the calculation formula, represents the cloud-free radar image in the image sample set, represents the cloud-free optical image in the image sample set, represents random noise, represents the generative network, represents the discriminant network, represents the minimum loss of the generated network G, Represents the maximum loss of the discriminant network; represents the natural logarithm, represents the expected value operation, is the distribution of real cloud-free image data, To generate the distribution of image data, It represents the expected calculation of x and y for the data sampled from the distribution of real cloud-free image data. Indicates the expected calculation of x and z based on the data sampled from the distribution of generated image data; It means that the discriminant network accepts input x and conditional information y, and outputs a probability value, which indicates the probability that the input data is the real data. It means that the generating network accepts input x and random noise z, and tries to generate data related to the conditional information y. Represents the evaluation of the discriminant network D on the output of the generative network G; In the calculation formula, is the L1 norm, is the structural similarity index, , , They are spatial attention loss, channel attention loss and temporal attention loss, which are used to optimize the feature learning of the corresponding dimensions. , , is the weight coefficient of each loss item; Represents the distribution of real cloud-free image data The data pairs (x, y) sampled from the image are obtained, and the distribution of the generated image data is obtained from Calculate the expected value of the variable z sampled from .

[0102] In this embodiment, , , They are 100, -100, and 10 respectively; In the calculation formula, represents the image block size. In this embodiment, N is 30. , express , The mean of , express , The variance of express , The covariance between , To avoid a constant with a denominator of zero, in this embodiment, the calculation formula is:

[0103]

[0104]

[0105] is 0.01, is 0.03, Indicates the pixel value range of the image, which is 0~255 after uniform stretching. is 255;

[0106] In the calculation formula of attention loss, , , Represent the feature extraction functions of spatial dimension, channel dimension and temporal dimension respectively, , , They represent the attention graph generation functions of the corresponding dimensions respectively, Represents element-wise multiplication.

[0107] Furthermore, step S13 further includes: inputting the image sample set and random noise into the deep learning model constructed in S12, first fixing the discriminant network parameters, adjusting the generating network parameters, optimizing the generating network, then fixing the generating network parameters, adjusting the discriminant network parameters, optimizing the discriminant network, repeating this step until Nash equilibrium is reached to obtain the optimal model.

[0108] In this embodiment, a total of 13,250 pairs of radar images and optical images are trained, and the number of training rounds is 35, which can achieve Nash equilibrium. The structural similarity between the generated cloud-free optical image and the original cloud-free image exceeds 0.8, and the peak signal-to-noise ratio exceeds 30db.

[0109] When constructing a deep learning model, this embodiment adds the structural similarity index and triple attention loss to the loss function to improve the texture effect and detail quality of the generated image, taking into account that relying solely on the L1 norm may lead to blurred texture of the generated image, low spatial dependency and modeling correlation, and other problems.

[0110] Furthermore, step S2 further comprises:

[0111] Step S21: For the original optical image disturbed by clouds and fog, in this embodiment, a radar image that is no more than 5 days different from the original image is selected as the input source, and the deep learning optimal model obtained in S1 is input to generate a cloud-free optical image;

[0112] Step S22: Based on the data of the area not affected by cloud and fog in the original optical image and the corresponding data of the generated cloud-free optical image, a linear regression equation group is constructed to further enhance the generated cloud-free optical image. The calculation formula is:

[0113] ,

[0114] in, is the line number, is the column number, are channel categories, namely red band, green band, blue band, and near-infrared band. The original optical image Cells in the channel ( ) pixel value, for The initial enhancement factor of the channel (set to 1), To generate optical images Channel cell ( ), is a random error. After inputting all the data of the areas not affected by fog and cloud in the original image for regression analysis, the multi-source remote sensing data enhancement model for fog and cloud areas is obtained. The calculation formula is:

[0115]

[0116] in, It is the enhanced cloud-free optical image. Cells in the channel ( ), for The enhancement factor of the channel, Based on radar data , random noise , Deep Learning Optimal Model Generated cloud-free optical image Cells in the channel ( )’s pixel value.

[0117] When enhancing multi-source remote sensing data, this embodiment takes into account the numerical and distribution differences between the generated optical image and the original optical image at the pixel level, introduces the original cloud-free area as a pseudo-invariant area, constructs a multi-source remote sensing data enhancement model, further enhances the data information of the generated optical image, and solves the problem of obtaining optical image data in foggy areas.

[0118] Furthermore, step S3 further includes:

[0119] Step S31, constructing a three-dimensional convolutional neural network for extracting features from the enhanced time-series optical image;

[0120] Step S32: construct a spatiotemporal attention mechanism module, including a temporal attention submodule and a spatial attention submodule, for all channel features extracted in S31 Conduct dual attention enhancement in time and space;

[0121] Step S33: construct a feature aggregation module, and use a multi-scale feature pyramid structure to fuse spatiotemporal features at different levels.

[0122] Furthermore, step S31 further includes: inputting time-series optical image data of dimensions T×H×W×C, wherein T is the length of the time series, in this embodiment, T is 16, indicating that 16 consecutive periods of image data are used, H is the image height, W is the image width, in this embodiment, the image size is 256×256 pixels, C is the number of image channels, in this embodiment, including four band channels of red, green, blue and near infrared, and the output is a feature map containing all channels The three-dimensional convolutional neural network contains three three-dimensional convolutional layers, the convolution kernel size is 3×3×3, the step size is 1, the padding is 1, and the number of channels C is expanded from the initial 4 channels to 64, 128, and 256 channels through convolution operations. The calculation formula is:

[0123]

[0124] in, Indicates that the input is The first Layer feature map, Indicates The convolution kernel parameters of the layer, is the convolution kernel size, Represents input After the phase image, Layer cells ( ), is a bias term, and each convolutional layer is followed by a normalization layer and a ReLU activation function.

[0125] Furthermore, step S32 further includes: the temporal attention submodule uses a self-attention mechanism to enhance the relevance of features of different time steps, and the calculation formula is:

[0126]

[0127]

[0128]

[0129]

[0130]

[0131] in, , , are the three vectors in the self-attention mechanism, representing the query vector, key vector and numerical vector respectively. , , is a learnable linear transformation matrix. In this embodiment, the dimension is 256×256. is the scaling factor, which is 256 in this embodiment. is the normalization function, Temporal attention weight matrix, Represents the strength of association between different time steps, which is a feature weighted by temporal attention;

[0132] The spatial attention submodule constructs a spatial attention map based on feature statistics, and the calculation formula is:

[0133]

[0134]

[0135]

[0136]

[0137] in, is the number of characteristic channels, which is 256 in this embodiment, and are the average and maximum feature maps of all channels respectively, is a 7×7 two-dimensional convolutional layer, is the sigmoid activation function, is the spatial attention map, represents broadcast multiplication, The features enhanced by temporal and spatial dual attention will be used as the input of the feature pyramid network.

[0138] Furthermore, step S33 further includes: processing the features output from step S32 Aggregate the time dimension and eliminate the time dimension through weighted average. The dimensions are C× ×H×W, in this embodiment, C, are 256 and 16 respectively, H and W are the feature map space sizes, and Layer characteristics The calculation formula is:

[0139]

[0140] in, is the length of the time series, is the time step The weight coefficient, the initial value can be 1 / , can be learned through back-propagation; Indicates different feature levels, with values ​​of 1, 2, 3, The dimensions are 256×64×64, The dimensions are 256×32×32, The dimensions are 256×16×16.

[0141] Then, construct a feature pyramid, calculate layer by layer from top to bottom, and then fuse the features of feature pyramids at different levels:

[0142]

[0143]

[0144] in, Represents the feature pyramid Layer features, Indicates The learnable weight matrix of the layer, () indicates bilinear interpolation upsampling, Indicates the fusion of the total Feature information at the layer scale, is a scalar weight coefficient, learned by back-propagation.

[0145] In this embodiment, the top layer It can be expressed as: ,in, is a 256×256 learnable weight matrix used to adjust the importance between channels. The dimension is 256×16×16, capturing large-scale semantic information; the middle layer It can be expressed as: ,in, is a 256×256 learnable weight matrix; () indicates bilinear interpolation upsampling, which expands 16×16 to 32×32. The final dimension is 256×32×32, which contains medium-scale structural information; the bottom layer It can be expressed as: ,in, is a 256×256 learnable weight matrix; () Expand 32×32 to 64×64; The final dimension is 256×64×64, which retains rich spatial detail information; The initial value is 1 / 3, The feature information of the above three scales is integrated, and the final dimension is 256×64×64, which serves as the shared feature input for subsequent crop recognition and segmentation tasks.

[0146] Furthermore, step S4 further comprises:

[0147] Step S41, constructing a dual-branch deep neural network, including a recognition branch and a segmentation branch;

[0148] Step S42: Use a multi-task joint training strategy to optimize network parameters, and then perform connected domain analysis and optimization processing on the network output results to obtain the final crop recognition and area extraction results.

[0149] Furthermore, step S41 further includes: constructing a recognition branch network of a full convolutional network structure. In this embodiment, four convolutional layers are used, and the convolution kernel size is 3×3. The input obtained in S33 The number of channels is The 256 will be 128, 64, 32, N, where N is the number of crop categories, and the output category probability map , with a dimension of N×64×64, the category probability of each pixel position is calculated, where the loss function is:

[0150]

[0151] in, , are the feature map space sizes, both of which are 64 in this embodiment. is the spatial position index. In this embodiment, the value range is 1 to 64. is the normalization coefficient, is the number of channels, is the crop category index, For the cell ( ) The label value of the channel value after one-hot encoding is 0 or 1. The class probability predicted by the network is the cell ( ) Middle The category prediction probability value of the channel, To find the logarithm operation;

[0152] The encoder-decoder structure is used to construct the segmentation branch network. The number of layers of the encoding block is consistent with that of the decoding block. In this embodiment, each encoding block contains two 3×3 convolutional layers and one maximum pooling layer, and each decoding block contains one transposed convolutional layer and two 3×3 convolutional layers. The number of channels is doubled in the encoding process (from 64 to 128, 256, and 512), and is halved in the decoding process to finally output a single-channel segmentation mask. The loss function is:

[0153]

[0154] in, , are the feature map space sizes, both of which are 64 in this embodiment. is the segmentation probability predicted by the network, is the true segmentation label.

[0155] Furthermore, step S42 further includes: optimizing network parameters using a multi-task joint training strategy, setting the batch size to 16, using the Adam optimizer for training, with an initial learning rate of 0.001, a learning rate decay of half every 30 epochs, and a total number of training rounds of 100, wherein the loss function is a weighted combination of the two-branch deep neural network loss function in S41:

[0156]

[0157] in, , is the weight coefficient, which is 0.7 and 0.3 in this embodiment, and is used to balance the contribution of the recognition task and the segmentation task;

[0158] Then, the results after the network output are subjected to connectivity analysis to optimize the segmentation results. In this embodiment, the segmentation mask is binarized with a threshold of 0.5, and the 8-neighborhood connectivity analysis algorithm is used to extract the complete crop area, mark different connected areas, calculate the number of pixels in each connected area, and convert the actual area in combination with the spatial resolution. At the same time, the predicted category with the largest proportion in each connected area is counted as the crop type of the area. The minimum area threshold of the connected domain analysis is set to 100 pixels, and areas smaller than this threshold will be filtered out to reduce the impact of noise, and finally realize crop type recognition and planting area extraction.

[0159] When identifying crop types, this embodiment takes into account the efficiency of identification and the accuracy of the results. By constructing a deep spatiotemporal feature extraction network and a dual-branch deep neural network, the feature extraction capability for crop identification is enhanced, and the extraction accuracy and efficiency of crop types and planting areas are improved.

[0160] In order to solve the problems in existing crop remote sensing identification methods, such as the difficulty in obtaining optical image data in foggy areas and the low recognition accuracy based only on radar images or single-period cloud-free optical remote sensing images, the present invention introduces a deep learning algorithm, considers the detail texture completeness of the generated image and the numerical and distribution characteristics of the pixel level, and realizes data enhancement based on time-series multi-source remote sensing data; considers the changes in the phenological characteristics of crops, constructs a deep spatiotemporal feature extraction network and a dual-branch deep neural network, extracts crop spatiotemporal features based on the enhanced time-series cloud-free optical images, and finally realizes accurate identification of crop types in foggy areas and efficient extraction of planting areas.

[0161] The main features and advantages of the present invention are shown and described above. It is obvious to those skilled in the art that the present invention is limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0162] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in the embodiments may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for crop identification in foggy areas based on multi-source remote sensing data enhancement, characterized in that: include: Step S1, constructing a multi-source remote sensing data enhanced deep learning model based on cloud-free optical images and radar images and training the model; The step S1 further comprises: Step S11: acquiring radar images with similar time phases for the cloud-free optical images, and registering, resampling, and cropping the cloud-free optical images and radar images to a uniform resolution and size to obtain an image sample set; Step S12: construct a deep learning model based on a generative adversarial network with an attention mechanism, and the loss function of the model adopts a loss function combining a structural similarity index, an L1 norm, and a triple attention loss; Step S13: Based on the image sample set and the deep learning model in steps S11-S12, training is performed to obtain an optimal model of Nash equilibrium; Step S2: inputting the radar image of the cloud area into the trained deep learning model to generate a cloud-free optical image; the step S2 further includes: Step S21, for the original optical image disturbed by clouds and fog, select a radar image whose phase is less than a preset value as an input source, input the multi-source remote sensing data enhancement model trained in step S1, and generate a cloud-free optical image; Step S22: Based on the data of the area not affected by cloud and fog in the original optical image and the generated cloud-free optical image data, a linear regression equation group is constructed to further enhance the generated cloud-free optical image. The calculation formula is: in, is the line number, is the column number, are channel categories, namely red band, green band, blue band, and near-infrared band. The original optical image Cells in the channel ( ) pixel value, for The initial enhancement factor of the channel (set to 1), To generate optical images Channel cell ( ), is a random error. After inputting all the data of the areas not affected by fog and cloud in the original image for regression analysis, the multi-source remote sensing data enhancement model for fog and cloud areas is obtained. The calculation formula is: in, It is the enhanced cloud-free optical image. Cells in the channel ( ), for The enhancement factor of the channel, Based on radar data , random noise , Deep Learning Optimal Model Generated cloud-free optical image Cells in the channel ( )’s pixel value; Step S3: constructing a deep spatiotemporal feature extraction network based on the generated time-series cloud-free optical images, and extracting crop identification features; Step S4: construct a dual-branch deep neural network, input the crop identification features extracted in step S3, and carry out crop type identification and planting area extraction.

2. The method for crop identification in foggy areas based on multi-source remote sensing data enhancement according to claim 1 is characterized in that: The step S12 further comprises: The generative network is constructed using a U-net network structure that introduces a triple attention mechanism, which consists of an encoder and a decoder. The triple attention mechanism includes: a spatial attention module for capturing the spatial dependency of images, a channel attention module for learning the importance of data in different bands, and a temporal attention module for modeling the temporal association of multi-phase data; in each decoding, except for the first decoding, which is the input of the final feature map after the last encoding, the input of each other decoding is: the decoding feature map of the previous round, the encoding feature map of the corresponding size, and the feature map after adaptive weighting by the triple attention mechanism, where the adaptive weighting process includes spatial Dimension feature enhancement, channel dimension feature enhancement and temporal dimension feature enhancement; construct a discriminant network and use a Markov discriminator, that is, use the discriminant network to make a true or false judgment on each image block of size N×N. If a real cloud-free optical image block is input, the output is 1, and if a generated cloud-free optical image block is input, the output is 0. The average value of the true and false results of the full-frame image block is taken as the final discrimination result output of the image; construct a loss function and use a loss function that combines the structural similarity index, L1 norm and triple attention loss. The triple attention loss is used to optimize the feature learning of the three dimensions of space, channel and time series. The calculation formula is: in, is the overall objective function, To generate the adversarial network objective function, is the loss function; and In the calculation formula, represents the cloud-free radar image in the image sample set, represents the cloud-free optical image in the image sample set, represents random noise, represents the generative network, represents the discriminant network, represents the minimum loss of the generated network G, Represents the maximum loss of the discriminant network; represents the natural logarithm, represents the expected value operation, is the distribution of real cloud-free image data, To generate the distribution of image data, It represents the expected calculation of x and y for the data sampled from the distribution of real cloud-free image data. Indicates the expected calculation of x and z based on the data sampled from the distribution of generated image data; It means that the discriminant network accepts input x and conditional information y, and outputs a probability value, which indicates the probability that the input data is real data. It means that the generating network accepts input x and random noise z, and tries to generate data related to the conditional information y. Represents the evaluation of the discriminant network D on the output of the generative network G; In the calculation formula, is the L1 norm, is the structural similarity index, , , They are spatial attention loss, channel attention loss and temporal attention loss, which are used to optimize the feature learning of the corresponding dimensions. , , is the weight coefficient of each loss item; Represents the distribution of real cloud-free image data The data pairs (x, y) sampled from the image are obtained, and the distribution of the generated image data is obtained from Calculate the expected value of the variable z sampled from the In the calculation formula, represents the image block size, , express , The mean of , express , The variance of express , The covariance between , To avoid a constant with a denominator of zero; in the attention loss calculation formula, , , Represent the feature extraction functions of spatial dimension, channel dimension and temporal dimension respectively, , , They represent the attention graph generation functions of the corresponding dimensions respectively, Represents element-wise multiplication.

3. The method for crop identification in foggy areas based on multi-source remote sensing data enhancement according to claim 1 is characterized in that: The step S13 further comprises: The image sample set and random noise are input into the model constructed in step S12. The discriminant network parameters are first fixed, the generating network parameters are adjusted, and the generating network is optimized. Then, the generating network parameters are fixed, the discriminant network parameters are adjusted, and the discriminant network is optimized. This step is repeated until Nash equilibrium is reached to obtain the optimal model.

4. The method for crop identification in foggy areas based on multi-source remote sensing data enhancement according to claim 1 is characterized in that: The step S3 further comprises: Step S31, construct a three-dimensional convolutional neural network for feature extraction of the enhanced time-series optical image, the input is the time-series optical image data of T×H×W×C dimensions, where T is the time series length, H is the image height, W is the image width, C is the number of image channels, and the output is the feature extraction of all channels. , the three-dimensional convolutional neural network includes three three-dimensional convolutional layers, and the calculation formula is: in, Indicates that the input is The first Layer feature map, Indicates The convolution kernel parameters of the layer, is the convolution kernel size, Represents input After the phase image, Layer cells ( ), is a bias term, and each convolutional layer is followed by a normalization layer and a ReLU activation function; Step S32: construct a spatiotemporal attention mechanism module, including a temporal attention submodule and a spatial attention submodule, for all channel features extracted in S31 Conduct dual attention enhancement in time and space; Step S33: construct a feature aggregation module, and use a multi-scale feature pyramid structure to fuse spatiotemporal features at different levels.

5. The method for identifying crops in foggy areas based on multi-source remote sensing data enhancement according to claim 4 is characterized in that: The step S32 further includes: The temporal attention submodule adopts the self-attention mechanism to enhance the correlation of features at different time steps. The calculation formula is: in, , , are the three vectors in the self-attention mechanism, representing the query vector, key vector and numerical vector respectively. , , is a learnable linear transformation matrix, is the scaling factor, is the normalization function, Temporal attention weight matrix, It is the feature after temporal attention weighting, indicating the strength of association between different time steps; The spatial attention submodule constructs a spatial attention map based on feature statistics, and the calculation formula is: in, is the number of feature channels, and are the average and maximum feature maps of all channels respectively, is a 7×7 two-dimensional convolutional layer, is the sigmoid activation function, is the spatial attention map, represents broadcast multiplication, The features enhanced by temporal and spatial dual attention will be used as the input of the feature pyramid network.

6. The method for identifying crops in foggy areas based on multi-source remote sensing data enhancement according to claim 4 is characterized in that: The step S33 further comprises: The output of step S32 Aggregate the time dimension and eliminate the time dimension by weighted average, where Layer characteristics The calculation formula is: in, is the length of the time series, is the time step The weight coefficient, the initial value can be 1 / , can be learned through back-propagation; Represents different levels of features; Then, construct a feature pyramid, calculate layer by layer from top to bottom, and then fuse the features of feature pyramids at different levels: in, Represents the feature pyramid Layer features, Indicates The learnable weight matrix of the layer, (·) represents bilinear interpolation upsampling, Indicates the fusion of the total Feature information at the layer scale, is a scalar weight coefficient, learned by back-propagation.

7. The method for identifying crops in foggy areas based on multi-source remote sensing data enhancement according to claim 6 is characterized in that: The step S4 further comprises: Step S41, constructing a dual-branch deep neural network, including a recognition branch and a segmentation branch; Step S42: Use a multi-task joint training strategy to optimize network parameters, and then perform connected domain analysis and optimization processing on the network output results to obtain the final crop recognition and area extraction results.

8. The method for identifying crops in foggy areas based on multi-source remote sensing data enhancement according to claim 7 is characterized in that: The step S41 further comprises: Construct the recognition branch network of the full convolutional network structure and input the one obtained in step S33 , output class probability map , calculate the category probability of each pixel position, where the loss function is: in, , are the feature map space sizes, is the normalization coefficient, is the number of channels, For the cell ( ) The label value of the channel value after one-hot encoding is 0 or 1. The class probability predicted by the network is the cell ( ) Middle The category prediction probability value of the channel, is a logarithmic operation; The encoder-decoder structure is used to construct the segmentation branch network. The number of layers of the encoding block and the decoding block is the same. Each encoding block contains two convolutional layers and one maximum pooling layer. Each decoding block contains a transposed convolutional layer and two convolutional layers. Finally, a single-channel segmentation mask is output. The loss function is: in, , are the feature map space sizes, is the segmentation probability predicted by the network, is the true segmentation label.

9. The method for identifying crops in foggy areas based on multi-source remote sensing data enhancement according to claim 8, characterized in that: The step S42 further includes: The multi-task joint training strategy is adopted to optimize the network parameters, and the Adam optimizer is used for training, where the loss function is a weighted combination of the two-branch deep neural network loss function in step S41: in, , is the weight coefficient, which is used to balance the contribution of the recognition task and the segmentation task; Connectivity analysis is performed on the network output results to optimize the segmentation results, ultimately achieving crop type recognition and planting area extraction.

10. A computer device, characterized in that: The method comprises a memory, a processor and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps in the method according to any one of claims 1 to 9.

Images