Sparse sample crop drawing method based on weak supervision semantic segmentation

Abstract

The invention discloses a sparse sample crop drawing method based on weak supervision semantic segmentation. The method comprises the following steps: acquiring monthly synthesis time sequence reflectivity data in a research area; obtaining sparse dotted crop labels in the research area, and processing monthly synthesis time sequence reflectivity data to obtain image edge labels; constructing a time sequence semantic segmentation model; constructing a multi-task weakly supervised learning network, constructing a total loss function, and training the time sequence semantic segmentation model and the multi-task weakly supervised learning network; and obtaining monthly synthesis time sequence reflectivity data, image edge labels and sparse point crop labels of a to-be-predicted region, and obtaining final crop categories of all sample points of the to-be-predicted region by using a time sequence semantic segmentation model and a multi-task weak supervised learning network. Compared with a traditional method, the method has the advantages that the performance is better in different crop category classification, and the F1-score of each crop is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural remote sensing ground object information extraction, and in particular to a sparse sample crop mapping method based on weakly supervised semantic segmentation, which is used for sparse small sample crop mapping. Background Art

[0002] The growing population has greatly increased the demand for food. However, blind expansion of cultivated land and increased farming intensity may have adverse effects on the ecological environment. Therefore, it is crucial to effectively balance food production and ecological environmental protection to achieve sustainable agricultural development. Timely grasping of crop spatial distribution information is the key to achieving sustainable agriculture for assessing food production, formulating agricultural production policies, and optimizing agricultural resource allocation. Traditionally, crop spatial distribution information is mainly obtained through field surveys by census personnel. However, the high cost of capital and manpower investment has led to a long update cycle and poor effectiveness of census data, which is difficult to meet actual needs.

[0003] The rapid development of remote sensing satellite technology has provided an effective means for large-scale automated and efficient identification of crops. Existing remote sensing mapping studies mostly use threshold methods or machine learning to mine the unique phenological characteristics of crop growth stages to distinguish different crop categories. Among them, the threshold method is based on phenological prior knowledge and uses single or multi-period remote sensing image features to construct classification decision rules and thresholds for crop identification. However, the threshold method is often limited to the spectral characteristics of a single period and is difficult to effectively deal with the phenological shift problem caused by differences in climate, topography, soil conditions and agricultural practices. In contrast, machine learning methods, such as decision trees, random forests and support vector machines, can automatically establish mathematical relationships between remote sensing features and target crop categories, improving the robustness of the model to deal with phenological migration problems. However, traditional machine learning algorithms usually treat multi-temporal information as multiple independent data, ignoring their temporal dependencies, and are difficult to deal with the "same spectrum, different objects" problem. In recent years, deep learning methods, such as recurrent neural networks (RNNs), convolutional neural networks (CNNs) and self-attention networks, have been widely used in various crop mapping applications with their powerful high-dimensional feature mining capabilities. However, deep learning models are highly dependent on a large number of high-quality dense surface labels. Existing field sample collection strategies are limited by terrain and sampling costs, and can only obtain sparse point labels, which is difficult to meet the training requirements of deep learning models, resulting in reduced crop recognition accuracy. Therefore, how to design a suitable weakly supervised learning framework to efficiently utilize limited sparse weak samples is the key to achieving efficient automatic crop recognition. Summary of the invention

[0004] The purpose of the present invention is to address the above-mentioned problems existing in the prior art and to provide a sparse sample crop mapping method based on weakly supervised semantic segmentation, so as to solve the problems that it is difficult to efficiently obtain sufficient and densely distributed high-quality samples in large-area crop remote sensing mapping, and the actual sample distribution is sparse in space and insufficient in number, resulting in low crop mapping accuracy.

[0005] In order to achieve the above object, the following is the technical solution of the present invention:

[0006] A sparse sample crop mapping method based on weakly supervised semantic segmentation includes the following steps:

[0007] Step S1: Obtain monthly synthetic time series reflectivity data in the study area;

[0008] Step S2: Obtain sparse point crop labels in the study area and process the monthly synthetic time series reflectivity data to obtain image edge labels;

[0009] Step S3: construct a time series semantic segmentation model, which generates spatiotemporal spectral features based on monthly synthetic time series reflectivity data;

[0010] Step S4: Construct a multi-task weakly supervised learning network, which includes a Segmentation task head module, an Edge task head module, and an Expansion task head module.

[0011] In the Segmentation task head module: based on sparse point crop labels and spatiotemporal spectral features, the predicted probability values ​​of different crop categories at all sampling points in the study area are generated;

[0012] In the Edge task head module: based on the image edge labels and spatiotemporal spectral features, the predicted edge strength of all sampling points in the study area is generated;

[0013] In the Expansion task head module: based on the image edge labels and the dense pseudo labels constructed by the predicted probability values ​​in the Segmentation task head module whose predicted probability values ​​are greater than the set adjustable threshold, the final predicted probability values ​​of different crop categories for all sample points in the study area are generated;

[0014] Step S5: construct a total loss function to train the temporal semantic segmentation model and the multi-task weakly supervised learning network;

[0015] Step S6: Obtain monthly synthetic time series reflectance data, image edge labels, and sparse point crop labels of the area to be predicted, and use the time series semantic segmentation model and multi-task weakly supervised learning network to obtain the final crop category of all sample points in the area to be predicted.

[0016] The monthly synthetic time series reflectivity data in step S1 as described above is obtained based on the following steps:

[0017] Sentinel-2 remote sensing images of the study area were collected, and the Sentinel-2 cloud probability products provided by the GEE platform were used to remove pixels with corresponding cloud probabilities higher than 65% to obtain the de-clouded Sentinel-2 reflectivity data. The de-clouded Sentinel-2 reflectivity data were median-synthesized to obtain monthly synthetic time-series reflectivity data.

[0018] As described above, the image edge label in step S2 is obtained based on the following steps:

[0019] The monthly synthetic time series reflectivity data is processed based on Sobel filtering to generate the edge gradient map of the corresponding month. The edge gradient map is normalized to generate the edge intensity map of each month. The edge intensity map is mean-synthesized to generate the image edge label.

[0020] As described above, the temporal semantic segmentation model in step S3 includes a multi-scale spatial feature encoder, a temporal feature encoder, and a multi-stage feature decoder.

[0021] In the multi-scale spatial feature encoder: DownConv modules with different numbers of series are used to extract spatial feature maps of different scales corresponding to the synthetic reflectance data of each month, and the spatial feature maps of the same scale and obtained in different months are connected along the time dimension to obtain the temporal spatial feature map of the scale. The temporal spatial feature maps of each scale are generated as multi-scale temporal spatial feature maps;

[0022] In the temporal feature encoder: the temporal spatial feature map is converted into a three-dimensional vector of dimension (HW)×T×C, where T is the number of months of the temporal spatial feature, C is the number of feature channels, H and W are the height and width of the temporal spatial feature map, and the three-dimensional vector is passed through two layers of bidirectional BiLSTM modules to generate a spatiotemporal feature sequence corresponding to the scale. The spatiotemporal feature sequence corresponding to the scale is input into a 1×1 convolutional layer with a softmax activation function to obtain the attention weights of different months. The features of different months in the spatiotemporal feature sequence corresponding to the scale are multiplied by the corresponding attention weights and then weighted summed to generate the spatiotemporal feature map corresponding to the scale.

[0023] The multi-stage feature decoder includes a dilated spatial pyramid pooling module and multiple joint attention mechanism modules. n is the serial number of the scale, and the value range of n is 1 to N. N is the total number of scales. Scales 1 to N are successively deeper. The spatiotemporal feature map corresponding to scale N is input into the dilated spatial pyramid pooling module. The feature map output by the dilated spatial pyramid pooling module passes through the upsampling module corresponding to scale N to obtain the decoding feature map corresponding to scale N.

[0024] For scale 1-scale (N-1): the spatiotemporal feature map corresponding to scale n is concatenated with the decoding feature map corresponding to scale n+1, and then the feature fusion is performed through the corresponding joint attention mechanism module, and then the decoding feature map corresponding to scale n is obtained through the upsampling module corresponding to scale n;

[0025] The decoded feature map corresponding to scale 1 is used as the spatiotemporal spectral feature output by the temporal semantic segmentation model.

[0026] As mentioned above, the atrous spatial pyramid pooling module includes three atrous convolution modules and one pooling module, and the joint attention mechanism module includes a channel attention module and a spatial attention module.

[0027] As mentioned above, the total loss function L is based on the following formula:

[0028] L=L seg +L exp +λ e L edge +λ c L con

[0029] Among them, L seg is the loss function of the Segmentation task head module, L edge is the edge loss function of the Edge task head module, L exp is the loss function of the Expansion task head module, L con is the consistency loss function between the predicted probability values ​​of different crop categories by the Expansion task head module and the Segmentation task head module, λ e and λ c are the weight coefficients of edge detection loss and consistency loss respectively.

[0030] As mentioned above, the loss function L of the Segmentation task head module seg Based on the following formula:

[0031]

[0032] Among them, n g and k represent the total number of sample points and the number of crop categories of sparse point crop labels, respectively. is the predicted probability value of the cth crop category of the ith sample point of the sparse point label by the Segmentation task head module, y i,c is the actual probability value of the cth crop category of the ith sample point of the sparse point crop label, and γ is an adjustable coefficient with a value range greater than zero.

[0033] As mentioned above, the edge loss function L of the Edge task head module edge Based on the following formula:

[0034]

[0035] Among them, j represents the sample point number in the study area, is the predicted edge strength of the jth sample point by the Edge task head module, b j is the actual image edge strength of the jth sample point of the image edge label, is the first type marginal loss function, It is the second type of marginal loss function.

[0036] As mentioned above, the loss function L of the expansion task head module exp and consistency loss L con Based on the following formula:

[0037]

[0038] Among them, |||| 2 Indicates square value calculation, n e represents the number of non-zero sample points of dense pseudo-labels, m is the sequence number of non-zero sample points of dense pseudo-labels, is the predicted probability value of the cth crop category of the mth sample point with dense pseudo labels that is non-zero, E m,c is the probability value of the cth crop category of the mth sample point with dense pseudo labels, The Segmentation task head module calculates and outputs the predicted probability value of the cth crop category at the jth sampling point in the study area. is the predicted probability value of the cth crop category at the jth sample point in the study area by the Expansion task head module, and H and W are the height and width of the input monthly synthetic time series reflectivity data.

[0039] As mentioned above, dense pseudo labels are obtained based on the following formula:

[0040]

[0041] E j,c =0else

[0042] Among them, τ a and τ b are two adjustable thresholds ranging from 0 to 1. The Segmentation task head module calculates and outputs the predicted probability value of the cth crop category at the jth sampling point in the study area. The Segmentation task head module calculates and outputs the maximum predicted probability value of various crop categories at the j-th sampling point in the study area. The Segmentation task head module calculates and outputs the crop category corresponding to the maximum predicted probability value of various crop categories at the jth sampling point in the study area, b j is the actual image edge strength of the jth sample point of the image edge label; E j,c is the dense pseudo label of the cth crop category at the jth sampling point in the study area.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. We built a new temporal semantic segmentation model U-TempoNet that combines convolutional neural networks and long short-term memory networks. This model can deeply explore the complex spatial and temporal characteristics of crops in temporal remote sensing data, thereby achieving more efficient and accurate crop category recognition.

[0045] 2. The edge detection task is introduced to obtain the edge strength of the image, which enhances the perception of the spatial structure information of crops, enabling it to more accurately restore the boundaries of crop fields, thereby improving the accuracy of the segmentation results.

[0046] 3. By introducing the pseudo-label supervision task, dense pseudo-labels are constructed based on the image edge labels and the predicted probability values ​​in the Segmentation task head module that are greater than the set adjustable threshold, and supervised training is performed. This improves the richness of the model's training samples, further deepens the understanding and distinction of crop categories, and enhances its generalization ability in practical applications.

[0047] Through the integration of the above innovative technologies, the present invention effectively improves the accuracy of crop category recognition based on sparse point crop labels and provides more reliable technical support for large-scale remote sensing agricultural monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the present invention;

[0049] Figure 2 is the distribution map of the study area of ​​the present invention; wherein, (A) is the distribution map of study area A; (B) is the distribution map of study area B; (C) is the distribution map of study area C;

[0050] Figure 3 This is a structural diagram of the temporal semantic segmentation model of the present invention;

[0051] Figure 4 This is a multi-task weakly supervised learning framework diagram of the present invention;

[0052] Figure 5 The crop classification accuracy evaluation results of the present invention are shown in Figure 1. (a) represents the crop classification accuracy evaluation results of the present invention in the Jianghan Plain test area; (b) represents the crop classification accuracy evaluation results of the present invention in the Songnen Plain test area; (c) represents the crop classification accuracy evaluation results of the present invention in the Sanjiang Plain test area. DETAILED DESCRIPTION

[0053] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0054] like Figure 1 As shown, a sparse sample crop mapping method based on weakly supervised semantic segmentation includes the following steps:

[0055] Step S1: Based on the GEE (Google Earth Engine) platform, obtain the Sentinel-2 remote sensing image in the study area and perform cloud removal to obtain monthly synthetic time series reflectance data:

[0056] Step S1.1: Based on the GEE platform, Sentinel-2 remote sensing images of study areas A, B and C were collected (the distribution of study areas and field sampling points is shown in Figure 1 ). The collection time range is from January 1, 2021 to December 31, 2021. In order to capture the complex spatiotemporal characteristics of areas with strong heterogeneity, this paper selects four spectral bands (i.e., blue light, green light, red light, and near-infrared bands) of Sentinel-2 remote sensing images with a spatial resolution of 10 meters as research data;

[0057] Step S1.2: For the collected Sentinel-2 remote sensing images, use the Sentinel-2 cloud probability product provided by the GEE platform to remove pixels with corresponding cloud probabilities higher than 65% to obtain the cloud-free Sentinel-2 reflectivity data. The monthly median synthesis method is used to perform median synthesis on the cloud-free Sentinel-2 reflectivity data for each month from January 1 to December 31, 2021 to obtain the monthly synthetic time series reflectivity data.

[0058] Step S2: Obtain sparse point crop labels in the study area (which can be obtained through field sampling by ground personnel or known sparse point crop labels). Process the monthly synthetic time series reflectivity data based on the Sobel operator to obtain image edge labels:

[0059] Step S2.1: Ground personnel went to the study area for field sampling in late April and late August 2021, respectively. They used handheld GPS instruments to record the geographic coordinates of the crop categories in the study area and obtain sparse point crop labels of the pixels corresponding to the geographic coordinates;

[0060] Step S2.2: Process the monthly synthetic time series reflectivity data obtained in step S1 based on Sobel filtering, and generate edge gradient maps of the corresponding months one by one. Then, these edge gradient maps are normalized to the range of 0 to 1 by 2% linear stretching to generate edge intensity maps for each month. Finally, these edge intensity maps are mean synthesized to generate high-quality image edge labels. By aggregating the mean of multi-temporal information, the impact of clouds and rain is effectively reduced, thereby ensuring the high quality of image edge labels.

[0061] Step S3: The present invention couples a convolutional neural network (CNN) and a long short-term memory network (LSTM) to construct a temporal semantic segmentation model (U-TempoNet) specifically for monthly synthetic temporal reflectivity data. The overall structure of the model is as follows: Figure 3 As shown in Figure 1, it includes a multi-scale spatial feature encoder, a temporal feature encoder, and a multi-stage feature decoder. This model can not only capture the spatial features in remote sensing data, but also fully explore the temporal variation features therein, thereby obtaining robust and representative "space-time spectrum" features:

[0062] S3.1 Multi-scale spatial feature encoder: For the monthly synthetic time series reflectivity data obtained in step S1, DownConv modules with different numbers of cascades are used to extract spatial feature maps of different scales corresponding to the synthetic reflectivity data of each month (such as Figure 3). The DownConv module includes three Conv2DN layers. The first Conv2DN layer uses a convolution operation with a stride of 2 to reduce the resolution of the input feature map to 1 / 2 of the original. The feature map output by the first Conv2DN layer is input to the second Conv2DN layer, and the output feature map of the second Conv2DN layer is input to the third Conv2DN layer. The output feature map of the third Conv2DN layer and the output feature map of the second Conv2DN layer are added together as the output spatial feature map of the DownConv module. Through this additive jump connection, the DownConv module can extract deeper features while avoiding the gradient disappearance caused by the model being too deep. The spatial feature map of the corresponding scale generated by each DownConv module is halved in height and width compared to the previous scale, while the number of channels is doubled. Use different numbers of serially connected DownConv modules to process the monthly synthetic time series reflectivity data of each month respectively, and obtain spatial feature maps of different scales for each month. Connect the spatial feature maps of the same scale but obtained in different months along the time dimension to obtain the time series spatial feature map of that scale. The time series spatial feature maps of each scale are generated as multi-scale time series spatial feature maps. The dimension of the time series spatial feature map is T×C×H×W, where T is the number of months, C is the number of spatial feature map channels, and H and W are the height and width of the time series spatial feature map.

[0063] S3.2 Temporal feature encoder: For the temporal spatial feature maps of each scale obtained in step S3.1, the AtBiLSTM module is used to model the temporal dependencies between feature maps of different months. AtBiLSTM wraps two layers of bidirectional LSTM (BiLSTM) modules and a 1×1 convolutional layer with softmax activation function. Given a temporal spatial feature map of dimension T×C×H×W, where T is the number of months, C is the number of channels of the spatial feature map, and H and W are the height and width of the temporal spatial feature map. First, the temporal spatial feature map is converted into a three-dimensional vector of dimension (HW)×T×C, which can be regarded as converting the H×W temporal spatial features in the temporal spatial feature map into a one-dimensional vector, with each temporal spatial feature having T months and C feature channels. Subsequently, a two-layer bidirectional BiLSTM module is used to explore the temporal dependencies in these temporal spatial features, and the three-dimensional vector passes through the two-layer bidirectional BiLSTM module to generate a temporal and spatial feature sequence corresponding to the scale. The spatiotemporal feature sequence corresponding to the scale is input into a 1×1 convolutional layer with a softmax activation function to learn the attention weights of different months. The probability value output by the softmax activation function is used as the attention weight of each month. Finally, the features of different months in the spatiotemporal feature sequence corresponding to the scale are multiplied by the corresponding attention weights and then weighted summed to generate the spatiotemporal feature map corresponding to the scale. The spatiotemporal feature maps of each scale generate the final multi-scale spatiotemporal feature map.

[0064] S3.3 Multi-stage feature decoder: For the spatiotemporal feature maps of various scales obtained in step S3.2, since spatiotemporal feature maps of different scales can capture different aspects of the target crop. Although the spatiotemporal feature maps at deeper levels have lower resolutions, they have richer contextual semantic information. On the other hand, the spatiotemporal feature maps at lower levels contain more spatial details, which facilitates more refined analysis. In order to effectively fuse information with different resolutions and semantic richness, a multi-stage feature decoder is used to deeply fuse the spatiotemporal feature maps of different scales, thereby obtaining robust and representative spatiotemporal spectral features.

[0065] The multi-stage feature decoder contains an atrous spatial pyramid pooling (ASPP) module and multiple joint attention mechanism modules (JA). The atrous spatial pyramid pooling module includes three atrous convolution modules and one pooling module. The dilation rates of the three atrous convolution layers are set to 1, 2, and 4 respectively to expand the receptive field and capture more contextual information. The joint attention mechanism module includes a channel attention module and a spatial attention module. The channel attention module and the spatial attention module process the input features of the joint attention mechanism module respectively. The output features of the channel attention module are multiplied with the input features of the joint attention mechanism module as the feature channel weight vector to obtain the first intermediate feature. The output features of the spatial attention module are multiplied with the input features of the joint attention mechanism module as the feature space weight vector to obtain the second intermediate feature. The first intermediate feature and the second intermediate feature are multiplied as the output features of the joint attention mechanism module. Channel attention readjusts the contributions of features of different channels by assigning attention weights to them, while spatial attention highlights salient areas by assigning attention weights to different positions.

[0066] n is the serial number of the scale, and the value range of n is 1 to N. N is the total number of scales. Scales 1 to N are successively deeper. The spatiotemporal feature map corresponding to scale N is input into the dilated spatial pyramid pooling module. The feature map output by the dilated spatial pyramid pooling module passes through the upsampling module corresponding to scale N to obtain the decoding feature map corresponding to scale N.

[0067] For scale 1 to scale (N-1): the spatiotemporal feature map corresponding to scale n is concatenated with the decoding feature map corresponding to scale n+1, and then the feature fusion is performed through the corresponding joint attention mechanism module, and then the decoding feature map corresponding to scale n is obtained through the upsampling module corresponding to scale n;

[0068] Finally, the decoded feature map corresponding to scale 1 is used as the spatiotemporal spectral feature output by the temporal semantic segmentation model.

[0069] Step S4: In order to improve the performance of the temporal semantic segmentation model constructed in step S3 under sparse sample conditions, the present invention designs a multi-task weakly supervised learning network based on multi-task collaborative training ( Figure 4 ). The multi-task weakly supervised learning network includes a Segmentation task head module, an Edge task head module, and an Expansion task head module. Each task head module includes two 3×3 convolutional layers.

[0070] Among them, the Segmentation task head module performs the semantic segmentation task, uses the sparse point crop labels obtained in step S2.1 and the robust and representative spatiotemporal spectral features obtained in step S3.3 for training, identifies different crop categories, and generates predicted probability values ​​of different crop categories for all sampling points (pixel points) in the study area; the Edge task head module uses the image edge labels obtained in step S2.2 as labels, and is also trained in combination with the spatiotemporal spectral features obtained in step S3.3 to perform image edge detection tasks, assist the temporal semantic segmentation model in learning the spatial structure information of crops, and obtain the predicted edge strength of all sampling points in the study area; the Expansion task head module uses the image edge labels obtained in step S2.2 and the predicted probability values ​​with higher predicted probability values ​​(greater than the set adjustable threshold) in the Segmentation task head module to construct dense pseudo-labels, combines spatiotemporal spectral features and dense pseudo-labels for supervised training, and outputs the predicted probability values ​​of different crop categories for all sample points in the study area, thereby improving the crop recognition accuracy of the model under sparse and few sample conditions.

[0071] S4.1: For the Segmentation task head module. The present invention uses Focal loss to calculate the loss function L of the Segmentation task head module seg (Formula 1), reducing the impact of class imbalance. In formula (1), y and p s They represent the sparse point crop labels obtained in step S2.1 and the predicted probability values ​​of the Segmentation module respectively. g and k represent the total number of sample points and the number of crop categories of sparse point crop labels, respectively. i and c represent the sample point number and the crop category number of sparse point crop labels, respectively.

[0072]

[0073] Among them, p s i,c is the predicted probability value of the cth crop category of the ith sample point of the sparse point label obtained in step S2.1 by the Segmentation task head module, y i,c is the actual probability value of the cth crop category of the i-th sample point of the sparse point crop label obtained in step S2.1, and γ is an adjustable coefficient in the focal loss with a value range greater than zero. The Segmentation task head module calculates and outputs the predicted probability values ​​of all sampling points in the study area. The sample points of the sparse point label are only some of the sampling points in the study area. The sample points of the sparse point label are numbered i, and the sample points in the study area are numbered j.

[0074] S4.2: For the Edge task head module, the edge loss function L is constructed based on Tanimoto loss (Formula 2) edge (Formula 4). Where b and They represent the image edge label obtained in step S2.2 and the predicted edge strength of the Edge task head module. t Represents the loss function of the Edge task head module (Tanimoto loss function), which can effectively deal with the problem of unbalanced number of edge pixels and non-edge pixels in edge detection tasks.

[0075]

[0076] Among them, j represents the serial number of the sample point in the study area, is the predicted edge strength of the jth sample point by the Edge task head module, b j is the actual image edge strength of the jth sample point of the image edge label obtained in step S2.2. The sampling points of the image edge label are all the sampling points in the study area. is the first type marginal loss function, It is the second type of marginal loss function.

[0077] S4.3: For the Expansion task head module, the results with higher predicted probability values ​​in the Segmentation task head module are used as the label amplification pool. On this basis, the sparse point crop labels are amplified, and the generated dense pseudo labels are used for training the Expansion task head module. In order to improve the amplification accuracy of dense pseudo labels and remove the influence of mixed pixels at the boundary of the image, the image edge labels obtained in S2.2 are introduced to constrain the amplification range of generating dense pseudo labels ( Figure 4 The generation process of dense pseudo labels E∈[0,k] (where 1 to k represent different categories of crops and 0 represents unlabeled pixels) constrained by image edge labels is shown in formula (5).

[0078]

[0079] Among them, τ a and τ b There are two adjustable thresholds ranging from 0 to 1, which are used to constrain the range of dense pseudo-label expansion. Their values ​​are set to 0.99 and 0.15 respectively. The Expansion task head module also constructs the loss function L based on Focal loss. exp (Formula 6). The Segmentation task head module calculates and outputs the predicted probability value of the cth crop category at the jth sampling point in the study area. The Segmentation task head module calculates and outputs the maximum predicted probability value of various crop categories at the j-th sampling point in the study area. The Segmentation task head module calculates and outputs the crop category corresponding to the maximum predicted probability value of various crop categories at the jth sampling point in the study area, b j is the actual image edge strength of the jth sample point of the image edge label obtained in step S2.2; E j,c is the dense pseudo label of the cth crop category at the jth sampling point in the study area.

[0080] In order to enforce consistency constraints between the Segmentation task head module and the Expansion task head module, the present invention uses the dropout method to perturb the "space-time spectrum" feature extracted in step S3.3 before inputting it into the Expansion task head module. After the perturbation, the "space-time spectrum" feature is input into the Expansion task head module for training. The consistency loss function L between the predicted probability values ​​of different crop categories by the Expansion task head module and the Segmentation task head module is con It can be expressed by formula (7).

[0081]

[0082] In formula (6), |||| 2 Indicates the calculation of the square value, n e represents the number of non-zero sample points in the dense pseudo-label, m is the sequence number of the non-zero sample points in the dense pseudo-label, is the predicted probability value of the cth crop category of the mth sample point in the dense pseudo-label by the Expansion task head module, E m,c is the probability value of the cth crop category for the mth sample point in the dense pseudo-label, The Segmentation task head module calculates and outputs the predicted probability value of the cth crop category at the jth sampling point in the study area. is the predicted probability value of the cth crop category at the jth sample point in the study area by the Expansion task head module. In formula (7), H and W are the height and width of the input monthly synthetic time series reflectance data. The final total loss function L is shown in formula (8), λ e and λ care the weight coefficients of edge detection loss and consistency loss, respectively. By coordinating multiple groups of tasks for joint training, the temporal semantic segmentation model can effectively utilize the spatial structure information of crops and the rich contextual semantic information of dense pseudo-labels, making up for the lack of sparse sample data, thereby improving the accuracy of crop mapping.

[0083] L=L seg +L exp +λ e L edge +λ c L con (8)

[0084] Step S5: Under the condition of minimizing the total loss function, supervised training is performed on the temporal semantic segmentation model of step S3 and the multi-task weakly supervised learning network of step S4.

[0085] Step S6, obtain the monthly synthetic time series reflectivity data and image edge labels of the area to be predicted, obtain the sparse point crop labels in the study area, use the time series semantic segmentation model and the multi-task weakly supervised learning network for prediction, obtain the final prediction probability values ​​of different crop categories of all sample points in the area to be predicted, and then process them through the argmax function to obtain the final crop category of all sample points in the area to be predicted.

[0086] The final crop classification of all sample points in the predicted area is compared with the crop recognition results of two traditional machine learning models (Random Forest RF and Support Vector Machine SVM) and three deep learning models (DCM, Transformer, and Patch-based CNN). The results are evaluated using indicators such as F1-score, User's Accuracy, and Producer's Accuracy. Figure 5 Compared with these methods, the method of the present invention has higher accuracy and better recognition effect.

[0087] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A sparse sample crop mapping method based on weakly supervised semantic segmentation, characterized in that: The following steps are involved: Step S1: Obtain monthly synthetic time series reflectivity data in the study area; Step S2: Obtain sparse point crop labels in the study area and process the monthly synthetic time series reflectivity data to obtain image edge labels; Step S3: construct a time series semantic segmentation model, which generates spatiotemporal spectral features based on monthly synthetic time series reflectivity data; Step S4: Construct a multi-task weakly supervised learning network, which includes a Segmentation task head module, an Edge task head module, and an Expansion task head module. In the Segmentation task head module: based on sparse point crop labels and spatiotemporal spectral features, the predicted probability values ​​of different crop categories at all sampling points in the study area are generated; In the Edge task head module: based on the image edge labels and spatiotemporal spectral features, the predicted edge strength of all sampling points in the study area is generated; In the Expansion task head module: based on the image edge labels and the dense pseudo labels constructed by the predicted probability values ​​in the Segmentation task head module whose predicted probability values ​​are greater than the set adjustable threshold, the final predicted probability values ​​of different crop categories for all sample points in the study area are generated; Step S5: construct a total loss function to train the temporal semantic segmentation model and the multi-task weakly supervised learning network; Step S6: Obtain monthly synthetic time series reflectance data, image edge labels, and sparse point crop labels of the area to be predicted, and use the time series semantic segmentation model and multi-task weakly supervised learning network to obtain the final crop category of all sample points in the area to be predicted.

2. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 1, characterized in that: The monthly synthetic time series reflectivity data in step S1 is obtained based on the following steps: Sentinel-2 remote sensing images of the study area were collected, and the Sentinel-2 cloud probability products provided by the GEE platform were used to remove pixels with corresponding cloud probabilities higher than 65% to obtain the de-clouded Sentinel-2 reflectivity data. The de-clouded Sentinel-2 reflectivity data were median-synthesized to obtain monthly synthetic time-series reflectivity data.

3. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 1, characterized in that: The image edge label in step S2 is obtained based on the following steps: The monthly synthetic time series reflectivity data is processed based on Sobel filtering to generate the edge gradient map of the corresponding month. The edge gradient map is normalized to generate the edge intensity map of each month. The edge intensity map is mean-synthesized to generate the image edge label.

4. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 1, characterized in that: The temporal semantic segmentation model in step S3 includes a multi-scale spatial feature encoder, a temporal feature encoder, and a multi-stage feature decoder. In the multi-scale spatial feature encoder: DownConv modules with different numbers of series connections are used to extract spatial feature maps of different scales corresponding to the synthetic reflectance data of each month. The spatial feature maps of the same scale and obtained in different months are connected along the time dimension to obtain the temporal spatial feature map of the scale. The temporal spatial feature maps of each scale are generated as multi-scale temporal spatial feature maps. In the temporal feature encoder: the temporal spatial feature map is converted into a three-dimensional vector of dimension (HW)×T×C, where T is the number of months of the temporal spatial feature, C is the number of feature channels, H and W are the height and width of the temporal spatial feature map, and the three-dimensional vector is passed through two layers of bidirectional BiLSTM modules to generate a spatiotemporal feature sequence corresponding to the scale. The spatiotemporal feature sequence corresponding to the scale is input into a 1×1 convolutional layer with a softmax activation function to obtain the attention weights of different months. The features of different months in the spatiotemporal feature sequence corresponding to the scale are multiplied by the corresponding attention weights and then weighted summed to generate the spatiotemporal feature map corresponding to the scale. The multi-stage feature decoder includes a dilated spatial pyramid pooling module and multiple joint attention mechanism modules. n is the serial number of the scale, and the value range of n is 1 to N. N is the total number of scales. Scales 1 to N are successively deeper. The spatiotemporal feature map corresponding to scale N is input into the dilated spatial pyramid pooling module. The feature map output by the dilated spatial pyramid pooling module passes through the upsampling module corresponding to scale N to obtain the decoding feature map corresponding to scale N. For scale 1-scale (N-1): the spatiotemporal feature map corresponding to scale n is concatenated with the decoding feature map corresponding to scale n+1, and then the feature fusion is performed through the corresponding joint attention mechanism module, and then the decoding feature map corresponding to scale n is obtained through the upsampling module corresponding to scale n; The decoded feature map corresponding to scale 1 is used as the spatiotemporal spectral feature output by the temporal semantic segmentation model.

5. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 4, characterized in that: The atrous spatial pyramid pooling module includes three atrous convolution modules and one pooling module, and the joint attention mechanism module includes a channel attention module and a spatial attention module.

6. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 1, characterized in that: The total loss function L is based on the following formula: L=L seg +L exp +λ e L edge +λ c L con Among them, L seg is the loss function of the Segmentation task head module, L edge is the edge loss function of the Edge task head module, L exp is the loss function of the Expansion task head module, L con is the consistency loss function between the predicted probability values ​​of different crop categories by the Expansion task head module and the Segmentation task head module, λ e and λ c are the weight coefficients of edge detection loss and consistency loss respectively.

7. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 6, characterized in that: The loss function L of the Segmentation task head module seg Based on the following formula: Among them, n g and k represent the total number of sample points and the number of crop categories of sparse point crop labels, respectively. (is the predicted probability value of the cth crop category of the ith sample point of the sparse point label by the Segmentation task head module, y i,c is the actual probability value of the cth crop category of the ith sample point of the sparse point crop label, and γ is an adjustable coefficient with a value range greater than zero.

8. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 6, characterized in that: The edge loss function L of the Edge task head module is edge Based on the following formula: Among them, j represents the sample point number in the study area, is the predicted edge strength of the jth sample point by the Edge task head module, b j is the actual image edge strength of the jth sample point of the image edge label, is the first type marginal loss function, It is the second type of marginal loss function.

9. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 6, characterized in that: The loss function L of the expansion task head module exp and consistency loss L con Based on the following formula: Among them, |||| 2 Indicates square value calculation, n e represents the number of non-zero sample points of dense pseudo-labels, m is the sequence number of non-zero sample points of dense pseudo-labels, is the predicted probability value of the cth crop category of the mth sample point with dense pseudo labels that is non-zero, E m,c is the probability value of the cth crop category of the mth sample point with dense pseudo labels, The Segmentation task head module calculates and outputs the predicted probability value of the cth crop category at the jth sampling point in the study area. is the predicted probability value of the cth crop category at the jth sample point in the study area by the Expansion task head module, and H and W are the height and width of the input monthly synthetic time series reflectivity data.

10. The sparse sample crop mapping method based on weakly supervised semantic segmentation according to claim 1, characterized in that: The dense pseudo-labels are obtained based on the following formula: AND j,c =0else Among them, τ a and τ b are two adjustable thresholds ranging from 0 to 1. The Segmentation task head module calculates and outputs the predicted probability value of the cth crop category at the jth sampling point in the study area. The Segmentation task head module calculates and outputs the maximum predicted probability value of various crop categories at the j-th sampling point in the study area. The Segmentation task head module calculates and outputs the crop category corresponding to the maximum predicted probability value of various crop categories at the jth sampling point in the study area, b j is the actual image edge strength of the jth sample point of the image edge label; E j,c is the dense pseudo label of the cth crop category at the jth sampling point in the study area.

Images