Crop classification method, device and equipment based on vegetation index and remote sensing image

Abstract

The present invention provides a crop classification method, apparatus, and device based on vegetation index and remote sensing imagery, relating to the technical field of crop classification. The method comprises: obtaining time-series remote sensing image data of a study area, and determining target remote sensing image data and target vegetation index data corresponding to multiple specified time periods based on the remote sensing image time-series data; determining crop classification results based on the target remote sensing image data and target vegetation index data using a pre-trained global crop recognition model; wherein pseudo-training labels are determined based on first prediction results output by a local crop recognition model for the training vegetation index data. The present invention can fully utilize vegetation indices and remote sensing images to improve the generalization capability and accuracy of the global crop recognition model. Embodiments of the present invention have the characteristics of a wide inference range, fast inference speed, high model accuracy, and strong generalization capability.

Description

Technical Field

[0001] The present invention relates to the technical field of crop classification, and in particular to a crop classification method, device and equipment based on vegetation index and remote sensing imagery. Background Art

[0002] The crop identification method based on remote sensing images has the characteristics of wide identification range, real-time, high efficiency and low cost. It has been widely used in grain census and disaster statistics, crop asset assessment and other businesses, and can play an important role in food security, emergency management and rural digital inclusive finance.

[0003] Currently, crop identification methods based on remote sensing images include: traditional machine learning methods based on remote sensing image vegetation indices, including support vector machines and random forests; and deep learning methods based on remote sensing image time series data, including TempConv and ConvLSTM. Traditional machine learning methods based on remote sensing image vegetation indices are object-centric learning methods. Due to the small number of model parameters used, different models need to be constructed for different terrains and different phenological zones (township / county level). This leads to poor generalization and high training and inference costs for large-scale crop identification. Semantic segmentation methods based on remote sensing image time series data are pixel-centric and utilize recurrent neural networks or channel attention mechanisms for crop representation learning. However, due to the simple network structure and the inability of recurrent neural networks to run in parallel, training is slow and requires a large amount of video memory. Furthermore, they do not make good use of the vegetation indices in traditional machine learning methods. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a crop classification method, device and equipment based on vegetation index and remote sensing imagery, which can make full use of vegetation index and remote sensing imagery to improve the generalization ability and accuracy of the global crop recognition model. The embodiments of the present invention have the characteristics of wide inference range, fast inference speed, high model accuracy and strong generalization ability.

[0005] In a first aspect, an embodiment of the present invention provides a crop classification method based on vegetation index and remote sensing imagery, comprising:

[0006] Acquire remote sensing image time series data of the study area, and determine target remote sensing image data and target vegetation index data corresponding to multiple specified periods based on the remote sensing image time series data;

[0007] Determining the crop classification result corresponding to the study area based on the target remote sensing image data and the target vegetation index data corresponding to each of the specified periods using a pre-trained global crop recognition model;

[0008] Among them, the training data set used to train the global crop recognition model includes training remote sensing image data, training vegetation index data and pseudo training labels, and the pseudo training labels are determined based on the first prediction result output by the local crop recognition model for the training vegetation index data.

[0009] In one embodiment, the step of training the global crop recognition model includes:

[0010] Dividing the study area into a plurality of sub-areas, and determining training remote sensing image data, training vegetation index data, and initial training labels corresponding to the sub-areas based on the remote sensing image time series data or historical remote sensing image time series data corresponding to the sub-areas;

[0011] Training a local crop recognition model corresponding to the sub-region using the training vegetation index data corresponding to the sub-region and the initial training label, so as to output a first prediction result for the training vegetation index data corresponding to the sub-region using the trained local crop recognition model;

[0012] Determine a pseudo training label corresponding to the sub-region based on the first prediction result;

[0013] The global crop recognition model is trained using the training remote sensing image data, the training vegetation index data, and the pseudo training labels corresponding to each of the sub-regions until a preset condition is met.

[0014] In one embodiment, determining the pseudo training label corresponding to the sub-region based on the first prediction result includes:

[0015] Based on a preset transition zone width, determining image edges and non-image edges from the training remote sensing image data corresponding to the sub-region;

[0016] The ratio between the number of pixels contained in the edge of the image and the number of pixels contained in the training remote sensing image data is used as a smoothness index;

[0017] Using the first prediction result as a hard label, and performing edge label smoothing processing on the hard label at the edge of the image using the smoothing index to obtain a soft label at the edge of the image;

[0018] Based on the hard labels at the non-image edges and the soft labels at the image edges, pseudo training labels corresponding to the sub-regions are constructed.

[0019] In one embodiment, the step of performing edge label smoothing on the hard labels at the edges of the image using the smoothing index includes:

[0020] The hard labels at the edge of the image are smoothed according to the following formula:

[0021]

[0022] Among them, K represents crop classification, α represents smoothing index, and y k represents a hard label, which includes the probability that the sample belongs to class k, y k ∈{0,1}, represents a soft label, which includes the smoothed probability that the sample belongs to class k.

[0023] In one embodiment, the step of training the global crop recognition model using the training remote sensing image data corresponding to each of the sub-regions, the training vegetation index data, and the pseudo training labels includes:

[0024] Performing dimensional unification processing on the training remote sensing image data and the training vegetation index data corresponding to each of the sub-regions;

[0025] Determining a second prediction result based on the training remote sensing image data and the training vegetation index data after dimensional uniformity processing using a global crop recognition model;

[0026] Model parameters of the global crop recognition model are adjusted based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges.

[0027] In one embodiment, the step of adjusting the model parameters of the global crop recognition model based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges further includes:

[0028] For each sample in the study area, if the sample is located at the edge of the image, determining a cross entropy loss value corresponding to the sample based on the soft label and the second prediction result; if the sample is not located at the edge of the image, determining a cross entropy loss value corresponding to the sample based on the hard label and the second prediction result;

[0029] The mean of the cross entropy loss values ​​corresponding to all the samples is determined as the target cross entropy loss value;

[0030] Determining a DiceLoss loss value based on the second prediction result, the hard label at the non-image edge, and the soft label at the image edge;

[0031] A total loss value is obtained by weighted summing the target cross entropy loss value and the DiceLoss loss value, and the model parameters of the global crop recognition model are adjusted using the total loss value.

[0032] In one embodiment, the method further comprises:

[0033] determining an evaluation index corresponding to each crop in the study area based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges;

[0034] If the evaluation index corresponding to the crop is lower than the preset threshold, the number of the training remote sensing image data and the training vegetation index data corresponding to the crop category in the training data set is adjusted, and the local crop recognition model and the global crop recognition model are continued to be trained until the evaluation index corresponding to each crop is higher than the preset threshold.

[0035] In a second aspect, an embodiment of the present invention further provides a crop classification device based on vegetation index and remote sensing imagery, comprising:

[0036] A data acquisition module is used to acquire remote sensing image time series data of a study area, and determine target remote sensing image data and target vegetation index data corresponding to a plurality of specified periods based on the remote sensing image time series data;

[0037] A crop recognition module is configured to determine a crop classification result corresponding to the study area based on the target remote sensing image data and the target vegetation index data corresponding to each of the specified periods using a pre-trained global crop recognition model;

[0038] Among them, the training data set used to train the global crop recognition model includes training remote sensing image data, training vegetation index data and pseudo training labels, and the pseudo training labels are determined based on the first prediction result output by the local crop recognition model for the training vegetation index data.

[0039] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0041] The crop classification method, apparatus, and device based on vegetation index and remote sensing imagery provided by the embodiments of the present invention first acquires time-series remote sensing imagery data for a study area. Then, based on the time-series remote sensing imagery data, target remote sensing imagery data and target vegetation index data corresponding to multiple specified time periods are determined. Finally, a pre-trained global crop recognition model is used to determine the crop classification result corresponding to the study area based on the target remote sensing imagery data and target vegetation index data corresponding to each specified time period. The training dataset used to train the global crop recognition model includes training remote sensing imagery data, training vegetation index data, and pseudo-training labels. The pseudo-training labels are determined based on the first prediction results output by a local crop recognition model for the training vegetation index data. This method, which combines remote sensing imagery and vegetation index as input, can address the problem of insufficient training labels and fully utilize the texture features of remote sensing imagery and the semantic features of vegetation index, enabling faster convergence and higher accuracy of the global crop recognition model. Furthermore, this method utilizes limited, noisy annotated data in the development process. The pseudo-training labels are generated locally using the first prediction results of the local crop recognition model, reducing annotation costs and shortening the development cycle. Therefore, the embodiments of the present invention can make full use of vegetation indices and remote sensing images to improve the generalization ability and accuracy of the global crop recognition model, and have the characteristics of wide reasoning range, fast reasoning speed, high model accuracy, and strong generalization ability.

[0042] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of a process flow of a crop classification method based on vegetation index and remote sensing imagery provided by an embodiment of the present invention;

[0046] Figure 2A schematic diagram of time series data of remote sensing images of a block in Hebei provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of data of an area to be inferred provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a remote sensing image of a certain area and its corresponding prediction results provided by an embodiment of the present invention;

[0049] Figure 5 A schematic flow chart of another crop classification method based on vegetation index and remote sensing imagery provided by an embodiment of the present invention;

[0050] Figure 6 A schematic structural diagram of a crop classification device based on vegetation index and remote sensing imagery provided by an embodiment of the present invention;

[0051] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] Currently, traditional machine learning methods based on remote sensing image vegetation indices suffer from poor generalization and high training and inference costs for large-scale crop identification. Semantic segmentation methods based on remote sensing image time series data also suffer from slow training and large video memory requirements, while also failing to fully utilize the vegetation indices used in traditional machine learning methods. Therefore, the present invention provides a crop classification method, apparatus, and device based on vegetation indices and remote sensing images. These methods fully utilize vegetation indices and remote sensing images to improve the generalization and accuracy of global crop identification models. These embodiments have the advantages of a wide inference range, fast inference speed, high model accuracy, and strong generalization.

[0054] To facilitate understanding of this embodiment, a crop classification method based on vegetation index and remote sensing image disclosed in an embodiment of the present invention is first described in detail. Figure 1 The flowchart of a crop classification method based on vegetation index and remote sensing image is shown. The method mainly includes the following steps S102 to S104:

[0055] Step S102: acquiring remote sensing image time series data of the study area, and determining target remote sensing image data and target vegetation index data corresponding to a plurality of specified periods based on the remote sensing image time series data.

[0056] Among them, remote sensing image time series data is remote sensing images of multiple periods. The specified period can be the key phenological period of the studied crops. The target remote sensing image data can include remote sensing images within the key phenological period of the studied crops. The target vegetation index data is NDVI data calculated based on the target remote sensing image data.

[0057] In one embodiment, during the data preparation stage, it is necessary to download and select Sentinel 2 tiles corresponding to the study area as remote sensing image time series data, and then screen the remote sensing image time series data according to the key phenological period of the study crop and the tile effective value ratio. At the same time, the remote sensing image time series data is aligned with the coordinates and resolution to obtain the processed remote sensing images within the key phenological period (which can be remote sensing images in raster form), and then select a period of processed remote sensing images from each month as the target remote sensing image data corresponding to each month, and calculate the target vegetation index data based on the numerical value of the specified band in the target remote sensing image data.

[0058] Step S104 , using the pre-trained global crop recognition model, the crop classification result corresponding to the study area is determined based on the target remote sensing image data and target vegetation index data corresponding to each designated period.

[0059] Among them, the training data set used to train the global crop recognition model includes training remote sensing image data, training vegetation index data and pseudo training labels. The pseudo training labels are determined based on the first prediction results output by the local crop recognition model for the training vegetation index data. The pseudo training labels include soft labels at the image edges and hard labels at the non-image edges. The hard labels are also the first prediction results, and the soft labels are the results of edge label smoothing processing on the first prediction results (hard labels).

[0060] In one embodiment, during the model training phase, the study area can be divided into multiple sub-areas, each corresponding to a local crop recognition model. Each local crop recognition model is trained using training vegetation index data and initial training labels. The trained local crop recognition model then outputs a first prediction result for the training vegetation index data and generates pseudo-training labels. Furthermore, the training remote sensing image data, training vegetation index data, and pseudo-training labels are used to train a global crop recognition model for the entire study area. Once the model training phase is complete, the trained global crop recognition model can be used to output crop classification results based on the target remote sensing image data and target vegetation index data.

[0061] The crop classification method based on vegetation index and remote sensing imagery provided by the embodiment of the present invention combines remote sensing imagery and vegetation index as input, can compensate for the problem of insufficient training labels, and fully utilizes the texture features of remote sensing imagery and the semantic features of vegetation index, which can make the global crop recognition model converge faster and more accurately. In addition, the development process uses limited, noisy annotated data, and within a local range, the first prediction results of the local crop recognition model are used to produce pseudo-training labels, which can reduce labeling costs and shorten the development cycle. Therefore, the embodiment of the present invention can fully utilize vegetation index and remote sensing imagery to improve the generalization ability and accuracy of the global crop recognition model, and has the characteristics of a wide inference range, fast inference speed, high model accuracy, and strong generalization ability.

[0062] For ease of understanding, an embodiment of the present invention provides a specific implementation of a crop classification method based on vegetation index and remote sensing images.

[0063] Regarding the aforementioned step S102, the embodiment of the present invention provides an implementation method for obtaining remote sensing image time series data of a study area and determining target remote sensing image data and target vegetation index data corresponding to multiple specified periods based on the remote sensing image time series data. Specifically:

[0064] In one example, the embodiment of the present invention mainly uses the 10m resolution Sentinel-2 remote sensing image time series data, taking a block in Hebei as an example, see Figure 2 The diagram shows a time series data of remote sensing images of a block in Hebei Province. Figure 2 (a) in the figure shows all the time series data of all Sentinel 2 tiles after alignment. Figure 2 (b) in the figure is the result after time series data screening, interpolation, and extrapolation.

[0065] First, during the data preparation stage, it is necessary to download and select the Sentinel-2 tiles corresponding to the study area.

[0066] Then, the time series data is filtered based on the key phenological periods of the crops under study and the effective value ratio of the tiles. In actual applications, due to the relationship between satellite scanning strips, some tiles only have partial effective values. Time series data with effective value ratios higher than the preset ratio can be retained, and time series data with effective value ratios lower than the preset ratio can be eliminated.

[0067] At the same time, the coordinates and resolutions of the time series data are aligned to obtain processed remote sensing images of key phenological periods, such as processed remote sensing images from April to October.

[0068] Finally, for each month, a batch of processed remote sensing image data is selected as the target remote sensing image data from the multiple processed remote sensing images corresponding to that month. In addition, if there is no valid data for some data months, the target remote sensing image data for that month can be obtained through interpolation or linear extrapolation.

[0069] On this basis, the corresponding target vegetation index data can be calculated based on the acquired target remote sensing image data.

[0070] Before executing the aforementioned step S104, the global crop recognition model needs to be trained in advance. The embodiment of the present invention provides an implementation method for training the global crop recognition model, see the following steps 1 to 4:

[0071] Step 1: Divide the study area into multiple sub-areas, and determine the training remote sensing image data, training vegetation index data and initial training labels corresponding to the sub-areas based on the remote sensing image time series data or historical remote sensing image time series data corresponding to the sub-areas.

[0072] In practical applications, the training remote sensing image data, training vegetation index data, and initial training labels required for training can be determined from the above-mentioned remote sensing image time series data, or from historical remote sensing image time series data from previous years.

[0073] The embodiment of the present invention takes remote sensing image time series data as an example and provides an implementation method for determining the training remote sensing image data, training vegetation index data, and initial training labels corresponding to a sub-region. Specifically:

[0074] The supervised method requires a portion of labeled data. The main method is to make 512x512 slices in the area to be inferred (i.e., sub-area), randomly select 20% of the sliced ​​remote sensing image time series data as training remote sensing image data, and then do a small amount of object annotation for some crops in the training remote sensing image data. During the annotation process, it is necessary to cover all the crop categories to be studied in the training remote sensing image data, such as Figure 3 As shown in the labeling object (row 3-2), if there is a historical crop label, it can be used as a labeling reference (row 3-1) to obtain the required initial training label. Figure 3 This is a data diagram of the area to be inferred. Rows 1-2 are time series data, row 3-1 are reference labels, row 3-2 are object annotations, and row 3-3 are prediction results.

[0075] In one example, the vegetation index NDVI can be calculated from Sentinel-2 time-series remote sensing image data, and the vegetation index NDVI is the training vegetation index data.

[0076] Step 2: Using the training vegetation index data and initial training labels corresponding to the sub-region, train the local crop recognition model corresponding to the sub-region, so as to output a first prediction result for the training vegetation index data corresponding to the sub-region using the trained local crop recognition model.

[0077] In one embodiment, the purpose of step 2 is to perform crop recognition training based on a machine learning model of vegetation index in a local area. Specifically, the traditional machine learning crop recognition method used in this step uses the vegetation index NDVI (that is, the above-mentioned training vegetation index data) calculated from the Sentinel-2 time-series remote sensing image data as input. Regional division is performed based on cities, counties and districts, and each sub-region obtained by the division corresponds to a local crop recognition model (such as a machine learning model such as random forest, XGBoost, etc.). For each sub-region, the training vegetation index data corresponding to the sub-region is input into the local crop recognition model corresponding to the sub-region, so that the local crop recognition model outputs a corresponding prediction result, and a loss value is determined based on the prediction result and the initial training label corresponding to the sub-region to train the local crop recognition model corresponding to the sub-region.

[0078] Optionally, the training steps of the various local crop recognition models may be performed in parallel.

[0079] Furthermore, after the local crop recognition model corresponding to each sub-region is trained, the aforementioned selected 512x512 verification slice can be inferred by region using the local crop recognition model to obtain a first prediction result.

[0080] Step 3: Determine the pseudo training labels corresponding to the sub-regions based on the first prediction results. This step is the training data preparation stage of the image segmentation deep learning model. First, the selected 512x512 validation slice is used and the machine learning model in the third step is used to infer the region and the first prediction results (such as Figure 3 Line 3-3) performs edge label smoothing, which serves as pseudo training labels. Edge label smoothing empirically divides the hard and easy sample regions and applies a soft labeling approach to the hard samples to provide a certain degree of stability. Hard samples can be understood as samples at the edge of the image.

[0081] For specific implementation, please refer to steps 3.1 to 3.4 below:

[0082] Step 3.1: Based on the preset transition zone width, determine the image edge and non-image edge from the training remote sensing image data corresponding to the sub-region.

[0083] In one example, in an image segmentation task, the classification result of each pixel depends largely on the surrounding pixels. The difficulty of predicting different pixels in the image is different. It has been verified that the prediction confidence of the image at the boundary and classification edge area is lower and more prone to errors. A transition zone with a width of w is set to divide the area between the image boundary and the boundary of different crop categories, which is defined as the image edge. The image edge U e The pixels inside the transition band are considered as difficult samples; the area outside the transition band is also the non-image edge, and the pixels inside the non-image edge are considered as easy samples.

[0084] In step 3.2, the ratio between the number of pixels contained in the edge of the image and the number of pixels contained in the training remote sensing image data is used as the smoothness index, that is, the smoothness index: the number of pixels contained in the edge of the image / the total number of pixels contained in the image (equivalent to the area of ​​the edge of the image / the area of ​​the image).

[0085] In one example, the degree of smoothing of the edge label smoothing process depends on the ratio α of edge region pixels in the label to the total input pixels, and the ratio α is also the smoothing index.

[0086] In step 3.3, the first prediction result is used as a hard label, and the hard label at the edge of the image is smoothed using a smoothing index to obtain a soft label at the edge of the image.

[0087] In one example, the embodiment of the present invention does not process hard labels located at non-image edges. However, hard labels located at image edges need to be smoothed to form "soft labels." Specifically, edge label smoothing can be performed on hard labels located at image edges according to the following formula:

[0088]

[0089] Among them, K represents crop classification, α represents smoothing index, and y k Represents a hard label, which includes the probability that the sample belongs to class k, y k ∈{0,1}, Represents a soft label, which includes the smoothed probability that the sample belongs to class k.

[0090] In step 3.4, based on the hard labels at the non-image edges and the soft labels at the image edges, the pseudo training labels corresponding to the sub-regions are constructed.

[0091] The embodiment of the present invention can effectively improve the accuracy of the global crop recognition model by performing edge label smoothing processing on hard labels at the edges of the image.

[0092] In step 4, the global crop recognition model is trained using the training remote sensing image data, training vegetation index data, and pseudo-training labels corresponding to each sub-region until the preset conditions are met. This step is the model training phase of the image segmentation deep learning model. Supervised training is performed using the noisy labeled data obtained in step 3. Training remote sensing image data (pseudo-color data, mainly including near-infrared band 8, red band 4, green band 3, and blue band 2) from April to October and training vegetation index data are used as input. The addition of vegetation index introduces expert knowledge to a certain extent, which accelerates model convergence and enriches semantic information, improving the mean precision (mF1) by 3%.

[0093] For specific implementation, please refer to steps 4.1 to 4.3 below:

[0094] Step 4.1: Perform dimensional unification processing on the training remote sensing image data and training vegetation index data corresponding to each sub-region.

[0095] In one example, it should be pointed out that the input data here is a 4x7x512x512 matrix, and each channel represents the band, phenology, data slice length, and width respectively. The dimensions of the vegetation index NDVI and the original remote sensing image data are inconsistent. The value range of NDVI is [-1,1], but the value range of the original remote sensing image data DN is [0,19k], so different normalization schemes are adopted for the two data. Specifically, input the sentinel B2348 (band 2,3,4,8) data, first calculate NDVI = (DN8-N4+ps) / (DN8+N4+ps), where eps is a constant. In the embodiment of the present invention, eps is a very small positive number to prevent the denominator from being 0 abnormally, and then use the minimum value -1 and the maximum value 1 for truncation; secondly, use the reflectivity corresponding to the values ​​of bands 3, 4, and 8 (B348) instead of the DN value, that is, R i =N i / 10000,i=3,4,8where DN i Indicates the value of band i; finally, NDVI and R i ,i=3,4,8 are merged as the input of the global crop recognition model. Then the channel and the phenological channel are merged to obtain a 28x512x512 input, which contains phenological information and band information.

[0096] In step 4.2, a second prediction result is determined by using the global crop recognition model based on the training remote sensing image data and the training vegetation index data after dimensional uniformity processing.

[0097] The global crop recognition model framework uses UPerNet and ConvNeXt-base as its backbone. ConvNeXt, a fully convolutional feature extractor designed to benchmark the Swin-Transformer, offers the efficiency advantages of convolution while matching the accuracy of the Transformer. In fact, using an A40 graphics card, ConvNeXt can achieve an inference speed of 32.6 km² / s on Sentinel 10m resolution data, demonstrating exceptional efficiency. The separable convolution design employed in ConvNeXt, deepwise convolution, excels at selecting pixel features (physical space), while pointwise convolution excels at extracting channel information (temporal phenology).

[0098] The training remote sensing image data and the training vegetation index data obtained in step 4.1 are input into the above-mentioned global crop recognition model to obtain the second prediction result output by the global crop recognition model.

[0099] In step 4.3, the model parameters of the global crop recognition model are adjusted based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges. For details, see (1) to (4) below:

[0100] (1) For each sample in the study area, if the sample is located at the edge of the image, the cross entropy loss value corresponding to the sample is determined based on the soft label and the second prediction result; if the sample is not located at the edge of the image, the cross entropy loss value corresponding to the sample is determined based on the hard label and the second prediction result. Specifically, the target cross entropy loss value H(y,p) can be determined according to the following formula:

[0101]

[0102]

[0103] Among them, δ e Indicates whether the sample belongs to the edge area U e ,y k , denote the hard labels of easy samples and the soft labels of difficult samples, respectively, p k is the second prediction result.

[0104] (2) The mean of the cross entropy loss values ​​corresponding to all samples is determined as the target cross entropy loss value.

[0105] (3) Based on the second prediction result, the hard labels at the non-image edge, and the soft labels at the image edge, a DiceLoss loss value is determined. For specific implementation, reference may be made to the prior art explanation of the DiceLoss loss value, which will not be further described in the present embodiment.

[0106] (4) The target cross entropy loss value and the DiceLoss loss value are weightedly summed to obtain the total loss value, and the total loss value is used to adjust the model parameters of the global crop recognition model.

[0107] In practical applications, crop growth samples are subject to severe variations in "same crop, different spectra" and "different crop, same spectra" due to the influence of geographical and climatic conditions. OHEMPixelSampler's online mining method for difficult samples is used to avoid overfitting. A weighted sum of CrossEntropyLoss (CE) and DiceLoss loss functions is used as the loss function. This dual loss function stabilizes training, preserving both category competitiveness and block information to a greater extent. Label smoothing is used for difficult samples at the edge, making them equivalent to Soft-CrossEntropyLoss (SCE) and DiceLoss. Using soft-loss can mitigate overfitting for incorrect labels.

[0108] In one example, the above steps 1 to 4 can be repeatedly performed to iteratively train the local crop recognition model and the global crop recognition model. During the iterative training process, some crops with poor performance can also be additionally labeled, including: (a) determining the evaluation index corresponding to each crop in the study area based on the second prediction result, the hard label at the non-image edge, and the soft label at the image edge; (b) if the evaluation index corresponding to a certain crop is lower than a preset threshold, adjusting the number of training remote sensing image data and training vegetation index data corresponding to the crop category in the training data set, and continuing to train the local crop recognition model and the global crop recognition model until the evaluation index corresponding to each crop is higher than the preset threshold.

[0109] For example, the accuracy of the second prediction result can be calculated. If the accuracy is lower than a preset threshold, it is determined that additional annotation is needed for the crop category, and the amount of training remote sensing image data and training vegetation index data corresponding to the samples in the training dataset is adjusted. The local crop recognition model and the global crop recognition model are iteratively trained using the adjusted training remote sensing image data and training vegetation index data according to steps 1 to 4 above until the accuracy of each crop exceeds the preset threshold. Iterative training can then be stopped. Alternatively, iterative training can be stopped until the accuracy cannot be improved. This step determines whether additional samples are needed and the type of additional samples based on the accuracy performance of each crop, thereby improving the overall accuracy of the model, preventing model overfitting, and improving the overall recognition accuracy of all crops.

[0110] After training the global crop recognition model using steps 1 to 4 above, the target remote sensing image data and target vegetation index data can be input into the global crop recognition model to infer the crop types contained in the study area through the global crop recognition model, and complete the model prediction and crop type vectorization in the study area. The effect is as follows: Figure 4 The figure shows a schematic diagram of a remote sensing image of a certain area and its corresponding prediction results.

[0111] In summary, the embodiment of the present invention is a crop recognition method based on remote sensing image time series data and vegetation index. It combines remote sensing image time series data and vegetation index as input, adopts the UPerNet encoding and decoding framework, uses ConvNeXt, one of the most advanced convolutional neural networks currently, to extract features, and the decoder obtains the probability map to obtain the crop category. This method has the advantages of a wide inference range, fast inference speed, high model accuracy, and strong generalization ability. In addition, in the development process, limited, noisy labeled data is used. Within a local range, the prediction results of the traditional machine learning method based on the vegetation index are used, and the edge label smoothing method is used to produce pseudo labels. This can greatly speed up the labeling of the full amount of element training samples required for semantic segmentation and shorten the development cycle.

[0112] Furthermore, the embodiment of the present invention also provides an application example of a crop classification method based on vegetation index and remote sensing imagery, see Figure 5 The flowchart of another crop classification method based on vegetation index and remote sensing imagery is shown in FIG, including:

[0113] (1) Model development phase: Acquisition of remote sensing time series data; partial crop labeling; regional division; training of regional models (i.e., local crop identification models) based on vegetation indices calculated from remote sensing time series data; regional model inference; pseudo-labeling; training of segmentation models (i.e., global crop identification models) based on vegetation indices calculated from remote sensing time series data; segmentation model evaluation; and re-labeling of crops with poor evaluation results.

[0114] (2) Model inference stage: obtaining remote sensing time series data; inputting remote sensing time series data and vegetation index calculated based on remote sensing time series data into the segmentation model; crop identification.

[0115] On this basis, the embodiment of the present invention provides experimental results. Table 1 records the detailed experiments and results. The experimental labels are the results of machine learning model inference. The verification accuracy at the sampling points is approximately 78.72%, which is approximately 21.28% noise. ELS represents the edge label smoothing loss, and SCE represents the smoothed label loss, both of which represent "soft labels." It can be seen that the "soft label" SCE+Dice has a 1.08-point improvement over the "hard label" CE+Dice, and the edge smoothing ELS+Dice has a 1.1-point improvement, which fully demonstrates the effectiveness of edge label smoothing in noisy label training. It also proves the effectiveness of the dual-label loss function; the dual-label loss (ELS vs. ELS+Dice) has a 0.35-point improvement, demonstrating the effectiveness of the dual loss function; NDVI replacing Sentinel B2 data has a 3-point improvement, proving the effectiveness of the crop index method.

[0116] Table 1

[0117]

[0118] The second row is the Baseline, the numbers in brackets in column F1 indicate the growth relative to the Baseline, and the bold numbers indicate the optimal results.

[0119] Compared with the prior art, the embodiments of the present invention have at least one of the following advantages:

[0120] 1. Using remote sensing images and vegetation indices as input can make up for the problem of insufficient labeled data for training samples. By fully utilizing the texture features of image plots and the semantic features of vegetation indices, the model converges faster and achieves higher accuracy.

[0121] 2. Using machine learning methods to create pseudo labels as a pre-step can reduce labeling costs. At the same time, edge label smoothing can avoid edge error noise interference. At the same time, it can be iterated to dynamically supplement a small amount of labels to improve model accuracy.

[0122] 3. The embodiment of the present invention applies the advanced ConvNeXt-based image segmentation solution to the crop identification based on phenological time series, fully explores the spatiotemporal feature extraction capability of deep separable convolution, and completes the full-factor classification of crops. ConvNeXt is a fully convolutional feature extractor designed with deep separable convolution, with smaller parameters and operations, fast inference speed, and can be used for inference over a large range.

[0123] Specifically, (1) compared with the related patent "County-level crop yield estimation method based on CNN-LSTM", the difference between the embodiment of the present invention and the patent is that the patent not only uses remote sensing data, but also uses other environmental data that affect crops. On the other hand, it uses the histogram statistics method to extract more features of county crops and converts them into tensors as inputs of the deep network CNN-LSTM. The input is more complex, and the deep learning model CNN-LSTM is also different from this solution; (2) compared with the related patent "Dryland crop identification method based on remote sensing time series data", the difference between the embodiment of the present invention and the patent is that the patent only uses traditional vegetation index, combines the vegetation index characteristics of dryland crops, and constructs a vegetation index that is completely based on the environment and the object to be studied. It requires a lot of expert knowledge and needs to adjust the threshold according to the region. The scalability is relatively poor, and the accuracy is not very high compared with the deep learning method; (3) compared with the related patent "Remote sensing image classification method and device", the embodiment of the present invention and the patent are different. The difference between the two patents is that the essence of the patent is a multi-model fusion solution, which selects targeted machine learning models for different research areas, and relies on relatively fine-grained (district, county) regional division to divide the ground objects into pure areas, mixed areas, and non-target areas. It also requires some threshold debugging based on the region. It is not directly data-driven and is not suitable for large-scale promotion and application. (4) Compared with the related patent "Classification method and device based on hyperspectral remote sensing image", the difference between the embodiment of the present invention and the patent is that the input of the patent is hyperspectral data, and it needs to use the spectral characteristics of some crops. It is not as accurate as the vegetation index based on phenological data, and it is not easy to label crop information under hyperspectral conditions, so it is not suitable for industrial applications. (5) Compared with the related patent "Farmland and mechanized farming road boundary line extraction method and system", the difference between the embodiment of the present invention and the patent is mainly that although the patent also uses ConvNeXt, it mainly performs road interpretation, and its input is still visible light data, and no phenological information is used.

[0124] Based on the above embodiments, the present invention provides a crop classification device based on vegetation index and remote sensing image. Figure 6 The structure diagram of a crop classification device based on vegetation index and remote sensing image is shown in FIG. The device mainly includes the following parts:

[0125] The data acquisition module 602 is used to acquire remote sensing image time series data of the study area, and determine target remote sensing image data and target vegetation index data corresponding to multiple specified periods based on the remote sensing image time series data;

[0126] The crop identification module 604 is configured to determine the crop classification result corresponding to the study area based on the target remote sensing image data and target vegetation index data corresponding to each specified period using a pre-trained global crop identification model;

[0127] Among them, the training data set used to train the global crop recognition model includes training remote sensing image data, training vegetation index data and pseudo training labels. The pseudo training labels are determined based on the first prediction result output by the local crop recognition model for the training vegetation index data.

[0128] The crop classification method based on vegetation index and remote sensing imagery provided by the embodiment of the present invention combines remote sensing imagery and vegetation index as input, can compensate for the problem of insufficient training labels, and fully utilizes the texture features of remote sensing imagery and the semantic features of vegetation index, which can make the global crop recognition model converge faster and more accurately. In addition, the development process uses limited, noisy annotated data, and within a local range, the first prediction results of the local crop recognition model are used to produce pseudo-training labels, which can reduce labeling costs and shorten the development cycle. Therefore, the embodiment of the present invention can fully utilize vegetation index and remote sensing imagery to improve the generalization ability and accuracy of the global crop recognition model, and has the characteristics of a wide inference range, fast inference speed, high model accuracy, and strong generalization ability.

[0129] In one embodiment, a training module is further included for:

[0130] The study area is divided into multiple sub-areas, and based on the remote sensing image time series data or historical remote sensing image time series data corresponding to the sub-areas, the training remote sensing image data, training vegetation index data and initial training labels corresponding to the sub-areas are determined;

[0131] Using the training vegetation index data and the initial training labels corresponding to the sub-region, training a local crop recognition model corresponding to the sub-region, so as to output a first prediction result for the training vegetation index data corresponding to the sub-region using the trained local crop recognition model;

[0132] Determine a pseudo training label corresponding to the sub-region based on the first prediction result;

[0133] The global crop recognition model is trained using the training remote sensing image data, training vegetation index data and pseudo training labels corresponding to each sub-region until the preset conditions are met.

[0134] In one embodiment, the training module is further configured to:

[0135] Based on a preset transition zone width, determining image edges and non-image edges from the training remote sensing image data corresponding to the sub-region;

[0136] The ratio between the number of pixels contained in the edge of the image and the number of pixels contained in the training remote sensing image data is used as a smoothness index;

[0137] The first prediction result is used as a hard label, and the hard label at the edge of the image is smoothed using a smoothing index to obtain a soft label at the edge of the image;

[0138] Based on the hard labels at non-image edges and the soft labels at image edges, pseudo training labels corresponding to the sub-regions are constructed.

[0139] In one embodiment, the training module is further configured to:

[0140] The hard labels at the edge of the image are smoothed according to the following formula:

[0141]

[0142] Among them, K represents crop classification, α represents smoothing index, and y k Represents a hard label, which includes the probability that the sample belongs to class k, y k ∈{0,1}, Represents a soft label, which includes the smoothed probability that the sample belongs to class k.

[0143] In one embodiment, the training module is further configured to:

[0144] Perform dimensional unification processing on the training remote sensing image data and training vegetation index data corresponding to each sub-region;

[0145] Determine a second prediction result through a global crop recognition model based on the dimensionally unified training remote sensing image data and the training vegetation index data;

[0146] Model parameters of the global crop recognition model are adjusted based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges.

[0147] In one embodiment, the training module is further configured to:

[0148] For each sample in the study area, if the sample is located at the edge of the image, the cross entropy loss value corresponding to the sample is determined based on the soft label and the second prediction result; if the sample is not located at the edge of the image, the cross entropy loss value corresponding to the sample is determined based on the hard label and the second prediction result;

[0149] The mean of the cross entropy loss values ​​corresponding to all samples is determined as the target cross entropy loss value;

[0150] Determine a DiceLoss loss value based on the second prediction result, the hard label at the non-image edge, and the soft label at the image edge;

[0151] The target cross entropy loss value and the DiceLoss loss value are weightedly summed to obtain the total loss value, and the total loss value is used to adjust the model parameters of the global crop recognition model.

[0152] In one embodiment, the system further includes a sample adjustment module for:

[0153] determining an evaluation index corresponding to each crop in the study area based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges;

[0154] If the evaluation index corresponding to the crop is lower than the preset threshold, the number of the training remote sensing image data and the training vegetation index data corresponding to the crop category in the training data set is adjusted, and the local crop recognition model and the global crop recognition model are continued to be trained until the evaluation index corresponding to each crop is higher than the preset threshold.

[0155] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0156] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0157] Figure 7 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 70, a memory 71, a bus 72 and a communication interface 73, wherein the processor 70, the communication interface 73 and the memory 71 are connected via the bus 72; the processor 70 is used to execute an executable module stored in the memory 71, such as a computer program.

[0158] The memory 71 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 73 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0159] The bus 72 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0160] Among them, the memory 71 is used to store programs, and the processor 70 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 70 or implemented by the processor 70.

[0161] The processor 70 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 70. The processor 70 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 71 , and the processor 70 reads the information in the memory 71 and completes the steps of the above method in combination with its hardware.

[0162] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the previous method embodiment and will not be repeated here.

[0163] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0164] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A crop classification method based on vegetation index and remote sensing imagery, characterized in that: include: Acquire remote sensing image time series data of the study area, and determine target remote sensing image data and target vegetation index data corresponding to multiple specified periods based on the remote sensing image time series data; Determining the crop classification result corresponding to the study area based on the target remote sensing image data and the target vegetation index data corresponding to each of the specified periods using a pre-trained global crop recognition model; The training data set used to train the global crop recognition model includes training remote sensing image data, training vegetation index data, and pseudo training labels, wherein the pseudo training labels are determined based on a first prediction result output by the local crop recognition model for the training vegetation index data; The training steps of the global crop recognition model include: Dividing the study area into a plurality of sub-areas, and determining training remote sensing image data, training vegetation index data, and initial training labels corresponding to the sub-areas based on the remote sensing image time series data or historical remote sensing image time series data corresponding to the sub-areas; Training a local crop recognition model corresponding to the sub-region using the training vegetation index data corresponding to the sub-region and the initial training label, so as to output a first prediction result for the training vegetation index data corresponding to the sub-region using the trained local crop recognition model; Determine a pseudo training label corresponding to the sub-region based on the first prediction result; The global crop recognition model is trained using the training remote sensing image data, the training vegetation index data, and the pseudo training labels corresponding to each of the sub-regions until a preset condition is met.

2. The crop classification method based on vegetation index and remote sensing imagery according to claim 1, characterized in that: Determining a pseudo training label corresponding to the sub-region based on the first prediction result includes: Based on a preset transition zone width, determining image edges and non-image edges from the training remote sensing image data corresponding to the sub-region; The ratio between the number of pixels contained in the edge of the image and the number of pixels contained in the training remote sensing image data is used as a smoothness index; Using the first prediction result as a hard label, and performing edge label smoothing processing on the hard label at the edge of the image using the smoothing index to obtain a soft label at the edge of the image; Based on the hard labels at the non-image edges and the soft labels at the image edges, pseudo training labels corresponding to the sub-regions are constructed.

3. The crop classification method based on vegetation index and remote sensing imagery according to claim 2, characterized in that: The step of performing edge label smoothing processing on the hard labels at the edges of the image using the smoothing index comprises: The hard labels at the edge of the image are smoothed using the following formula: ； in, Indicates crop classification, represents the smoothing index, Represents a hard label, the hard label includes the sample belongs to The probability of the class, , Represents a soft label, the soft label includes the sample belongs to The smoothed probability of the class.

4. The crop classification method based on vegetation index and remote sensing imagery according to claim 2, characterized in that: The step of training the global crop recognition model using the training remote sensing image data corresponding to each of the sub-regions, the training vegetation index data, and the pseudo training labels comprises: Performing dimensional unification processing on the training remote sensing image data and the training vegetation index data corresponding to each of the sub-regions; Determining a second prediction result based on the training remote sensing image data and the training vegetation index data after dimensional uniformity processing using a global crop recognition model; Model parameters of the global crop recognition model are adjusted based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges.

5. The crop classification method based on vegetation index and remote sensing imagery according to claim 4, characterized in that: The step of adjusting the model parameters of the global crop recognition model based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges further includes: For each sample in the study area, if the sample is located at the edge of the image, determining a cross entropy loss value corresponding to the sample based on the soft label and the second prediction result; if the sample is not located at the edge of the image, determining a cross entropy loss value corresponding to the sample based on the hard label and the second prediction result; The mean of the cross entropy loss values ​​corresponding to all the samples is determined as the target cross entropy loss value; Determining a DiceLoss loss value based on the second prediction result, the hard label at the non-image edge, and the soft label at the image edge; A total loss value is obtained by weighted summing the target cross entropy loss value and the DiceLoss loss value, and the model parameters of the global crop recognition model are adjusted using the total loss value.

6. The crop classification method based on vegetation index and remote sensing imagery according to claim 4, characterized in that: The method further comprises: determining an evaluation index corresponding to each crop in the study area based on the second prediction result, the hard labels at the non-image edges, and the soft labels at the image edges; If the evaluation index corresponding to the crop is lower than the preset threshold, the number of the training remote sensing image data and the training vegetation index data corresponding to the crop category in the training data set is adjusted, and the local crop recognition model and the global crop recognition model are continued to be trained until the evaluation index corresponding to each crop is higher than the preset threshold.

7. A crop classification device based on vegetation index and remote sensing image, characterized in that: include: A data acquisition module is used to acquire remote sensing image time series data of a study area, and determine target remote sensing image data and target vegetation index data corresponding to a plurality of specified periods based on the remote sensing image time series data; A crop recognition module is configured to determine a crop classification result corresponding to the study area based on the target remote sensing image data and the target vegetation index data corresponding to each of the specified periods using a pre-trained global crop recognition model; The training data set used to train the global crop recognition model includes training remote sensing image data, training vegetation index data, and pseudo training labels, wherein the pseudo training labels are determined based on a first prediction result output by the local crop recognition model for the training vegetation index data; Also includes training modules for: Dividing the study area into a plurality of sub-areas, and determining training remote sensing image data, training vegetation index data, and initial training labels corresponding to the sub-areas based on the remote sensing image time series data or historical remote sensing image time series data corresponding to the sub-areas; Training a local crop recognition model corresponding to the sub-region using the training vegetation index data corresponding to the sub-region and the initial training label, so as to output a first prediction result for the training vegetation index data corresponding to the sub-region using the trained local crop recognition model; Determine a pseudo training label corresponding to the sub-region based on the first prediction result; The global crop recognition model is trained using the training remote sensing image data, the training vegetation index data, and the pseudo training labels corresponding to each of the sub-regions until a preset condition is met.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 6.

Images