Crop identification and yield prediction method based on self-attention remote sensing feature extraction and reinforcement learning

Abstract

The invention provides a crop identification and yield prediction method based on self-attention remote sensing feature extraction and reinforcement learning. The crop identification and yield prediction method provided by the invention comprises the following steps: preprocessing remote sensing data of a target area to generate time sequence data; the feature extraction network processes the time sequence data by using a self-attention mechanism to generate high-dimensional time sequence remote sensing features, the high-dimensional time sequence remote sensing features are subjected to dimension increasing to obtain feature data features, and the feature data features are provided for a prediction network and a confidence network; the feature extraction network also generates a prediction remote sensing image according to the high-dimensional time sequence remote sensing features; and the prediction network generates predicted types and yields of the crops according to the characteristics of the characteristic data.

Description

Technical Field

[0001] The present application relates to remote sensing technology, and in particular, to a crop identification and yield prediction method based on self-attention extraction of remote sensing features and reinforcement learning, and an information processing device thereof. Background Art

[0002] Remote sensing data is crucial for crop monitoring, climate change analysis, and environmental assessments. Remote sensing data encompasses a variety of spectral information, including visible light, infrared, and radar bands, reflecting the physical and chemical properties of land features. By analyzing remote sensing data, scientists can effectively predict key information such as land cover and crop yields. However, due to the complex characteristics of remote sensing data and the strong correlations between multiple bands, existing remote sensing feature modeling methods have numerous limitations.

[0003] Existing technologies primarily use traditional time series models, such as long short-term memory (LSTM) and gated recurrent units (GRU). These models primarily rely on time series information and ignore high-order correlations between multiple bands. For example, the paper "Time Series Remote Sensing Crop Classification Based on the Bi-LSTM Model" (Huang Chong and Hou Xiangjun, Chinese Journal of Agricultural Sciences) uses Sentinel-2 satellite imagery available throughout the year as a data source to construct an annual time series NDVI dataset. Using a recurrent neural network architecture, they build a bidirectional long short-term memory (Bi-LSTM) model for structured time series data. The paper explores the potential of deep learning models combined with remote sensing time series for refined crop classification and early growing season mapping.

[0004] Another approach is to extract spatial features from remote sensing data using convolutional neural networks (CNNs). For example, patent ZL201811571333.4 describes a GL-CNN remote sensing image scene classification method based on frequency band feature fusion. This method extracts high-frequency and low-frequency subbands from the original image through a non-subsampled wavelet transform. The high-frequency subbands of the original image are then fused with the frequency band features of the original image to obtain a fused high-frequency subband. Spectral analysis is then used to analyze the high- and low-frequency components of the sample image, consisting of the low-frequency subbands of the original image and the fused high-frequency subbands. The sample fusion of the fused high-frequency and low-frequency subbands is then achieved through the stationary interval analysis of the spectral angular energy distribution curve. Finally, the convolutional neural network is guided to automatically extract high-level features from the high- and low-frequency subbands of the image to achieve scene classification.

[0005] In recent years, reinforcement learning (RL) methods have seen increasing application in various forecasting tasks, particularly in decision-making and forecasting in complex dynamic environments. The actor-critic architecture, a common approach in reinforcement learning, divides the model into two relatively independent sub-models: a policy network (actor) that outputs the optimal action strategy based on the current state, and a value network (critic) that evaluates various strategies and assigns values. Both networks are optimized independently. This separation of policy and value training strategy adapts to complex and diverse environments, enabling the model to perform better in these environments. The AC (actor-critic) architecture typically uses TD-error (temporal difference error) as a learning objective, allowing for rapid, small-step adjustments to the strategy during the learning process for more efficient model optimization. Typical applications include robotic control and recommender systems, but its application in agricultural yield forecasting remains relatively limited.

[0006] Remote sensing data has complex characteristics, and there are many difficulties in using remote sensing data to monitor crop growth, specifically:

[0007] Remote sensing images are composed of the reflection intensity of surface objects in various bands, with a pixel resolution between 0.1 and 10 meters, which basically loses the detailed information of surface objects.

[0008] For crops, their production cycle and growth are related to the crop type, region, altitude, precipitation, soil conditions, etc. The growth characteristics of crops coexist with both regularity in large areas and local randomness.

[0009] Data loss is inevitable and unpredictable due to satellite cycle and cloud cover.

[0010] Crops and other vegetation have similarities in all bands and need to be distinguished based on temporal characteristics by taking into account the differences in their growth cycles from other vegetation.

[0011] While existing time series modeling methods, such as LSTM and GRU, can extract features from time series data to a certain extent, their ability to model global information is limited. These methods rely on the step-by-step transmission of sequences, leading to long-range dependencies and failing to simultaneously consider correlations between distant bands. Furthermore, they have limited ability to handle missing data, typically requiring complete time series input or complex interpolation algorithms to fill in missing data.

[0012] While models like CNNs can extract local features, they are unable to process the temporal nature of remote sensing data and are insufficient in mining deep correlations between bands. Furthermore, existing technologies struggle to effectively handle the dynamic changes and multidimensional characteristics of remote sensing data, significantly impacting prediction accuracy when data is missing. Summary of the Invention

[0013] The present invention aims to overcome the defects in the existing technology and proposes a crop species identification and yield prediction method based on self-attention extraction of remote sensing features and reinforcement learning. High-dimensional features are extracted from the time series of remote sensing features through a feature extraction network with a self-attention mechanism, and the crop species identification and yield prediction model is decomposed into a prediction network and a confidence network, which are optimized separately to achieve crop species identification and yield prediction.

[0014] A feature extraction network with a decoder-only architecture is used to model the self-attention of time-series remote sensing data, simultaneously capturing deep connections between different time points, bands, and indices. The DOY (Day of Year) feature is used as a temporal position code to provide location information for each time point in the time-series remote sensing data, enabling the feature extraction network to process the data and empowering the model with time-series modeling capabilities. The decoder-only architecture eliminates long-range dependencies and does not require the completeness of the time series, enabling the model to achieve predictive capabilities early in crop growth, with prediction accuracy gradually improving over time.

[0015] A prediction network and a confidence network are designed. The prediction network is used to identify crop species and predict yields, while the confidence network is evaluated based on the confidence of the current prediction. The two networks share the high-dimensional remote sensing features extracted from time-series remote sensing data by the feature extraction network, and each is optimized with an independent objective function.

[0016] The feature extraction network also outputs predictions for future remote sensing images, enabling the model to simultaneously predict yields and provide visual forecasts of remote sensing images as a basis for inference for crop species identification and yield prediction. Using a deep learning model and a well-designed dropout mechanism ensures that the model maintains a certain level of predictive power even in situations with missing data.

[0017] Through this design of a feature extraction network based on the self-attention mechanism and a combination of a prediction network and a confidence network, it is possible to interpret and predict the yield of a variety of crops under various complex geographical and climatic conditions.

[0018] According to a first aspect of the present application, a method for crop identification and yield prediction is provided, comprising:

[0019] Preprocessing remote sensing data of the target area to generate time series data;

[0020] The feature extraction network processes the time series data using a self-attention mechanism to generate high-dimensional time series remote sensing features, increases the dimension of the high-dimensional time series remote sensing features to obtain data features, and provides the data features to the prediction network;

[0021] The feature extraction network also generates a predicted remote sensing image based on the high-dimensional time series remote sensing features;

[0022] The prediction network generates a prediction of the type and yield of the crop according to the data features;

[0023] The process of training the feature extraction network and the prediction network includes:

[0024] At each time step, the time series data SS of the training sample t and SS t+1 Provided to the feature extraction network, the feature extraction network outputs data features f t and f t+1 , and predict remote sensing images PS; the prediction network outputs a prediction result; based on the prediction result output by the prediction network, a reward r is generated t ; The confidence network is based on the reward r t , data features f t+1 With f t Calculate the TD-error, expressed as δ; where t represents the current time step;

[0025] In each training round, the following steps S1 to S4 are also performed:

[0026] S1. Using the δ of multiple time steps of the training round, calculate the objective function of the belief network JV(w) = ∑δ 2 , where w represents the parameters of the belief network;

[0027] S2. Calculate the objective function J(θ)=∑(log(prob)δ) of the prediction network, where prob represents the probability distribution corresponding to the prediction result output by the prediction network, and θ is a parameter of the prediction network;

[0028] S3. Calculate the objective function J(k)=∑(Y-PS) of the feature extraction network 2 , where Y represents the real remote sensing image, PS represents the predicted remote sensing image generated by the feature extraction network, and k is the parameter of the feature extraction network;

[0029] S4. Update the feature extraction network, update the prediction network, and update the confidence network.

[0030] According to the second aspect of the present application, an information processing device according to the second aspect of the present application is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the method according to the first aspect of the present application when executing the program. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present application, together with the preferred mode of use and further objects and advantages thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, in which:

[0032] Figure 1 The remote sensing data structure of an embodiment of the present application is shown.

[0033] Figure 2 The feature extraction network of the embodiment of the present application is shown.

[0034] Figure 3 Showing the decoder layers in the feature extraction network.

[0035] Figure 4 Showing the masked multi-head self-attention layer in the decoder layer.

[0036] Figure 5 The overall structure of the crop recognition and yield prediction model based on self-attention extraction of remote sensing features and reinforcement learning is demonstrated.

[0037] Figure 6 The overall structure of the crop identification and yield prediction model of another embodiment of the present application is shown.

[0038] Figure 7 A flowchart showing how to use the trained ensemble model to predict crop types and yields.

[0039] Figure 8 This is the interpretation result of remote sensing data on the distribution of winter wheat planting areas in a certain area of ​​Henan in 2023.

[0040] Figure 9 This is an image prediction obtained based on remote sensing images in the embodiment of the present application.

[0041] Figure 10 This is the result of crop type prediction in the embodiment of the present application.

[0042] Figure 11 This is the result of crop yield prediction in the embodiment of the present application. DETAILED DESCRIPTION

[0043] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0044] Remote sensing data is obtained from satellites, drones, and other means. Remote sensing data has multiple dimensions, such as data obtained in different bands (visible light, infrared, microwave, etc.) and from different sources (satellites, drones, balloons, etc.).

[0045] Figure 1 The remote sensing data structure of an embodiment of the present application is shown.

[0046] Figure 1 In the remote sensing data, for example, 30 fields are included, and each field is distinguished by a "number". The fields numbered 1-13 are remote sensing data of different bands. Remote sensing data can be in a variety of forms, such as data series or data sets, and the remote sensing data of each band can also be presented in the form of remote sensing images. The fields numbered 14-20 are different indexes extracted from the remote sensing data. By analyzing and processing remote sensing data, a variety of vegetation indices (such as normalized difference preparation index, green normalized difference preparation index, wide range dynamic vegetation index, enhanced vegetation index, etc.) are also obtained. These vegetation indices can also be used as one of the dimensions of remote sensing data.

[0047] Field 21 of the remote sensing data is DOY (Day of Year), which represents the time when the remote sensing data is collected. It is also used to identify the position of the remote sensing data in the time series remote sensing data (also called time series data), and also reflects the relative order relationship between multiple remote sensing data at different times. Since the seasonality of the crop growth cycle is very obvious and there are slight differences in different regions, this application uses the day of year (DOY) as the time position encoding (Positional Encoding) corresponding to the remote sensing data when generating time series data. The time series data includes remote sensing data at multiple time points, and the remote sensing data at each time point is also attached with the day of year (DOY) information. Therefore, the time series data can be provided to the feature extraction network of the self-attention mechanism that is originally unable to process time series data, so as to enhance the feature extraction ability of the feature extraction network for time series data. The time position encoding can retain the time sequence information, ensure that the feature extraction network can perceive the order of different time points when processing time series data, and enhance the feature extraction network's understanding of different seasons and annual cycles. It is particularly suitable for processing remote sensing data of the crop growth cycle.

[0048] The remote sensing data may also include one or more dimensions representing crop species. The crop species may be identified from the remote sensing data or obtained based on external information. Figure 1 In the example, field 22 of the remote sensing data represents the crop category. Field 23 of the input data represents the yield per unit area, which is the actual crop yield of the plot corresponding to the input data obtained based on historical data.

[0049] Optionally, the remote sensing data further includes dimensions representing parameters such as latitude and longitude, altitude, average annual rainfall and / or soil (fields 24-30). The remote sensing data may further include dimensions representing the area of ​​a plot of land.

[0050] Remote sensing data has a time attribute. For example, for a certain plot of land, a remote sensing satellite flies over the plot of land every 5 days, so that a copy of remote sensing data of the plot of land can be obtained every 5 days. Taking into account weather factors (cloud cover, etc.), a copy of high-quality remote sensing data of the plot of land can be obtained every 10-20 days. The day of year (DOY) of the remote sensing data represents the time when the remote sensing data is generated, which serves as the time attribute of the remote sensing data. Thus, the remote sensing data can form time series data with several days as intervals. In this application, time series data is, for example, multiple copies of data at different time points. Figure 1 The length of the time series data is denoted as N, where N is a positive integer. Typically, N is the number of remote sensing data within a specified time period (such as 1 year). For example, a piece of remote sensing data of a certain plot of land can be obtained every 10 days, and there are 36 pieces of remote sensing data in 1 year, and N=36. The time period usually represents a growth cycle of a crop. In this application, the time series data formed by multiple remote sensing data within a time period is used as the input of the feature extraction network, so that the obtained time series data carries the key information that distinguishes this crop from that crop, thereby improving the accuracy of crop type identification and yield prediction.

[0051] To better uncover the underlying patterns in remote sensing data, the feature extraction network of this invention processes remote sensing data to generate high-dimensional time-series remote sensing features. By expanding band data to a higher dimension, it can capture higher-level correlations between bands in remote sensing data, further enhancing the model's representational capabilities.

[0052] Optionally, preprocess the remote sensing data, including:

[0053] Normalize each band of remote sensing data to between 0 and 1;

[0054] Use remote sensing data to generate multiple remote sensing indices as features to expand the dimension of remote sensing data;

[0055] Embed the day of year (DOY) as both a feature (a dimension of remote sensing data) and a temporal position in the feature extraction network.

[0056] Optionally, meteorological data can also be added to the remote sensing data as a dimension of the remote sensing data.

[0057] Figure 2 The feature extraction network of the embodiment of the present application is shown.

[0058] The feature extraction network processes time series data generated by remote sensing data through a self-attention mechanism, leveraging global information to model the complex relationships between time series data. The self-attention mechanism can model global dependencies for each time point and band in the input time series data, regardless of the distance between points in the sequence, thus overcoming the long-range dependency issues inherent in traditional time series models.

[0059] The feature extraction network uses a neural network model with a decoder-only architecture, such as the decoder in the Transformer model. The time series data at time t formed by remote sensing data (denoted as S1, S2, ... S i …S t ) as the input of the feature extraction network, where S i Represents the token provided to the decoder or called extended remote sensing data (to distinguish it from the remote sensing data of each band), which has the following characteristics: Figure 1 The remote sensing data structure shown. i The corresponding day-of-year (DOY) data is attached to the remote sensing data through location embedding and input into the feature extraction network. Unlike the traditional encoder-decoder structure, the decoder-only architecture directly models and predicts historical data through a self-attention mechanism.

[0060] The feature extraction network outputs the predicted time series data SS at time t+1 t+1 '=(S2', S3', ... S t+1 '). Each data in the output time series data has the same Figure 1 The same structure is used for remote sensing data. In other words, each data in the output time series includes remote sensing band data and, optionally, a prediction of vegetation index.

[0061] In one embodiment, the remote sensing data of different bands is remote sensing imagery, and the data output by the feature extraction network also includes predicted remote sensing images of each band. By predicting remote sensing images, users can estimate crop yields based on their own experience and obtain the reasoning process or prediction basis for the crop yield predicted by the model of this application. The model is designed to output the predicted remote sensing images as intermediate results, thereby achieving interpretability or comprehensibility of the prediction model in a visual manner, enabling the model to output the reasoning process and improving user acceptance of the model.

[0062] The feature extraction network extracts the time series data SS from the input t =(S1, S2, ... S i …S t)Generate high-dimensional time series remote sensing features. The output layer of the feature extraction network ( Figure 2 "Linear Transformation 2") generates predicted time series data SS based on high-dimensional time series remote sensing features t+1 '=(S2', S3', ... S t+1 The data features obtained from the dimensionality increase of high-dimensional time series remote sensing features are also fed into the prediction network and the confidence network to identify the crop species represented by the remote sensing data and predict crop yields. The prediction network and confidence network will be discussed in detail later.

[0063] Transformer is the mainstream model framework in the field of natural language processing. Large language models based on Transformer usually adopt an encoder-decoder architecture. The Decoder-Only architecture is a simplified Transformer model architecture. Unlike the traditional encoder-decoder architecture, it only contains the decoder part and omits the encoder (Encoder). When processing natural language, the input of the Transformer is, for example, a sentence including multiple words. Obviously, time series data is different from sentences. However, in this application, by adding the day of year (DOY) as location information to the time series data of remote sensing data, the time series data has a structure similar to that of a sentence in form, so that it can be received by the decoder of the Transformer architecture.

[0064] In this application, the feature extraction network includes multiple (e.g., 16) decoder layers of a decoder-only architecture stacked together. These decoder layers process the input time series data and generate high-dimensional time series remote sensing features. Figure 3 This figure shows the decoder layers in the feature extraction network. Each decoder layer mainly includes a masked multi-head self-attention layer and a fully connected layer. Figure 4 Showing the masked multi-head self-attention layer in the decoder layer. Figure 4 In the example, Q, K, and V represent query (Q), key (K), and value (V), respectively. Q = X·W Q , K=X·W K , V=X·W V , where X represents the input vector (usually the embedding vector plus the positional encoding), W Q 、W K and W V is a learnable weight matrix.

[0065] The input of the feature extraction network includes time series data (S1, S2, ... S i ).

[0066] Masked Multi-Head Self-Attention Layer: This layer uses a self-attention mechanism to process global dependencies in time series data, allowing interactions between different time points (day of year, DOY) and different wavelengths. The core of self-attention is to correlate features across all input time points through a weighted mechanism, thereby uncovering deep connections in the data.

[0067] The output of the feature extraction network includes remote sensing data or index prediction results at future time points.

[0068] For example, the input time series data includes A time points, and the output time series data also includes A time points, where A represents the number of time points in all or part of the crop's growth cycle. Assuming the crop growth cycle runs from October of the current year to June of the following year, the input data may include data for January 1st, February 1st, and March 1st, and the output data may be forecasts for February 1st, March 1st, and April 1st, respectively. Alternatively, the input data may include data for January 1st, February 1st, March 1st, April 1st, and May 1st, and the output data may be forecasts for February 1st, March 1st, April 1st, May 1st, and June 1st, respectively. Crops mature on June 1st. In an application, the forecast data for June 1st can be used alone to identify crop species and predict crop yield, or the forecast data for April 1st, May 1st, and June 1st can be used together to identify crop species and predict yield. By selecting different parts of the high-dimensional time series remote sensing features output by the feature extraction network corresponding to different time points, the data features provided to the prediction network / confidence network are generated to change the prediction data used. Data features representing fewer time points help reduce the complexity and training cost of the prediction network / confidence network, while data features representing more time points help the prediction network / confidence network explore the associations between different time points of the time series remote sensing features and improve prediction results.

[0069] As another example, the input data includes data for November 1st, December 1st, January 1st of the following year, February 1st of the following year, and March 1st of the following year. The output is a prediction for December 1st, January 1st of the following year, February 1st of the following year, March 1st of the following year, and April 1st of the following year, respectively. Selecting a time range during the growing cycle when the remote sensing characteristics of crops change most strongly is beneficial for crop identification. For example, for wheat identification, selecting remote sensing data from October to December yields good results.

[0070] Train a feature extraction network to optimize its accuracy in predicting time series data.

[0071] Figure 5 The overall structure of the crop recognition and yield prediction model based on self-attention extraction of remote sensing features and reinforcement learning is demonstrated.

[0072] The overall model of this application includes a feature extraction network, a confidence network and a prediction network.

[0073] The input provided to the overall model is the time series data SS obtained from remote sensing data according to the previous embodiment. t =(S1, S2, ... S i …S t ), specifically, the input data is provided to the feature extraction network, which generates high-dimensional time series remote sensing features. The feature extraction network also uses high-dimensional time series remote sensing features to generate two outputs, one of which is the output time series data SS t+1 '=(S2', S3', ... S t+1 '), and the second is the data features after dimensionality increase (denoted as f). The output time series data represents the remote sensing data, index and other optional data at each future time point predicted by the feature extraction network. The output time series data is obtained by linear transformation 2 (see also Figure 2 ) is obtained by processing high-dimensional time series remote sensing features. By increasing the dimension, it also makes the input time series data SS t =(S1, S2, ... S i …S t ) includes no matter how many copies of data are included, the data features f after dimensionality increase have the same structure to facilitate use.

[0074] The data features f after dimensionality increase are provided to the confidence network and the prediction network as input data for both networks. This allows the confidence network and the prediction network to share the model structure of the feature extraction network, saving 25% of computing resources and reducing model training and inference time by 40%, making it suitable for large-scale remote sensing data interpretation tasks. The prediction network is responsible for identifying crop types and predicting yields based on the input data features f. The confidence network is based on the prediction errors of crop types and yields and the data features of two adjacent time points (denoted as f). t With f t+1 ) The expectation of future error is used as the confidence function to evaluate the prediction error. Through the time difference (TD-error) mechanism, the prediction network and the confidence network are optimized respectively. The data features of the two adjacent time points are recorded as f t With f t+1 , where f t The input time series data SS at time t t =(S1, S2, ... S i …S t ) is generated by the feature extraction network, and f t+1 The input time series data SS at time t+1 t =(S1, S2, ... Si …S t 、S t+1 ) is generated by the feature extraction network. For two adjacent time points, the input time series data at the later time point is compared with the input time series data at the previous time point, and an additional data S t+1 The model of this application learns to make predictions based on a small amount of early remote sensing data (fewer time points in the time series data) and improves prediction accuracy over time by using more remote sensing data (more time points in the time series data).

[0075] Denote the processing of the feature extraction network as f t =Trans(SS t ), where SS t represents the time series data input at time t, f t Represents the data characteristics at time t.

[0076] Denote the processing of the prediction network as a t ,p←Pred(f t ), where Pred() represents the prediction network for the input data feature f t Process and get result a t ,p, where a t represents the predicted crop yield, and p represents the identified crop type.

[0077] The process of the confidence network is expressed as Eval(f t ), the confidence network calculates the TD error (TD-error)δ=r based on the error t +λ*Eval(f t+1 )-Eval(f t ), provides the prediction network as the confidence (δ), where r t Represents the reward at time t, and Eval() represents the belief network for the data feature f at time t t The value evaluation of is defined as the expected cumulative reward generated by the current strategy when inputting data feature f. λ represents the confidence level in the predicted reward, ranging from 0 to 1. A λ value of 1 indicates absolute confidence in the predicted reward, while a λ value of 0 indicates that only the current reward is considered, regardless of future rewards. For example, λ = 1 can be used. The reward is derived by combining the error in crop type identification and the error in yield prediction at time t. The prediction network uses the received confidence level (δ) to update its parameters.

[0078] In this application, the fully connected layer of the prediction network is fitted with Pred(), and the fully connected layer of the confidence network is fitted with Eval().

[0079] According to an embodiment of the present application, the confidence network and the prediction network are implemented as a reinforcement learning model of the Actor-Critic architecture.

[0080] The reinforcement learning model of the Actor-Critic architecture consists of an Actor and a Critic. The Actor observes the environmental features as a state (data features f t ), and predict the types and future yields of crops, and use the prediction results as actions (action=(a t ,p)) is input into the environment. The error between the actual crop type / yield and the predicted value is obtained from the environment as a reward (reward, r). The Critic is responsible for evaluating the actions (predicted results) generated by the Actor and outputting a value function (Eval()) that represents the expected value of future accumulated rewards.

[0081] Since the concept of "environment" includes the natural environment in which crops grow, and also includes the unit in the reinforcement learning model that generates environmental states and calculates rewards, in order to distinguish them, the former is called the "real environment" and the latter is called the "environmental agent". The environmental agent can be implemented by, for example, a computer program that generates environmental states (data features f) from remote sensing data of the crop growth environment. t ) and calculates the reward (r) and provides it to the Actor and Critic. In this application, the prediction network acts as the Actor and the confidence network acts as the Critic.

[0082] The prediction network (Actor) is based on the current state (f t ), generating the relationship between crop type (p) and yield (a t ), which acts as an action generated by the prediction network and acts on the environment. In the present invention, the action includes the prediction of the future yield of crops, and its effect on the real environment does not directly affect the yield of crops. However, the yield predictions generated at different stages of the crop production cycle will have an indirect impact on the real environment. For example, a pessimistic prediction of yield may prompt people to increase investment or impose more manual intervention on the crop growth process, thereby changing the final actual yield of crops. The action generated by the prediction network is provided to the environmental agent, which uses the received action to calculate the reward and calculates the state (f at the next time point) at the next time point based on the remote sensing data at the next time point. t+1 ).

[0083] The task of the Critic network is to make a decision based on the current state (f t) and the reward r given by the environment agent, and calculates Eval() to evaluate the contribution of the prediction network's action-generating strategy to future rewards. The confidence network is optimized using TD-error (temporal difference error), making its estimate of value more accurate.

[0084] State: In the present invention, remote sensing data from multiple time points (S1, S2, ... S t )Generate time series data (denoted as SS t ), and then generate high-dimensional time series remote sensing features through feature extraction network, and then generate data features (f t ) as the state at time point t provided to the confidence network and prediction network (denoted as f t ). Status(f t ) has a time attribute. For example, for a certain plot of land, a piece of remote sensing data of the plot can be obtained every 5 days. Taking weather factors into account, a piece of high-quality remote sensing data of the plot can be obtained every 10-20 days. Thus, the remote sensing data forms a time series data with several days as the interval (S1, S2, ... S t ), where the first remote sensing data S1 comes from the beginning of the crop growth cycle, and the remote sensing data S t From the current time point t. Similarly, time series data (SS t+1 ) from remote sensing data (S1, S2, ... S t ,S t+1 ) is obtained from the time series data (SS t+1 )Generate data features (f t+1 ) represents the state at time point t+1 provided to the confidence network and prediction network. To facilitate processing by the feature extraction network, each time series data set can include the same number of time points, and remote sensing data that has not yet been obtained at the current time (t) can be occupied by a specified value. Thus, as time passes, data features (f) representing the state are generated from time series data with updated time points and carry more information reflecting the crop type and yield. By training the overall model of the present invention using data features obtained from time series data at different time points in the crop growth cycle, the overall model can make predictions based on early data features and make more accurate predictions based on later data features.

[0085] Optionally, the output of the prediction network is provided to both the environmental agent of the present invention and the real environment, for example, by publishing the crop yields predicted by the prediction network to agricultural practitioners in the real environment to influence the real environment.

[0086] Crop types and yields are used to calculate rewards.

[0087] Crop types are coded to facilitate reward calculation, ensuring that the calculated reward is of the same or similar order of magnitude as the reward derived from the predicted yield. Table 1 shows examples of crop type coding. In Table 1, each predicted crop type is assigned a different code value, such as an integer between 0 and 7.

[0088] Crop yields are converted to "relative yields" and then encoded. This encoding is used to calculate rewards and also eliminates the impact of larger actual yield values ​​on reward calculations. Relative yields are generated by preprocessing the actual crop yields. Relative yield is the ratio of the yield (predicted or actual) of a certain crop in a certain location in a certain year (or a certain crop growing season) to the average yield of that location over the years (or multiple crop growing seasons). Using relative yields also allows the model to adapt to yield predictions for different types of crops and different regions. Table 2 provides an example of encoding relative yields. Through crop yield encoding, the relative yield representing the ratio is converted into a code value. The code value of the crop yield code is, for example, an integer between 0 and 6. The code value of the crop yield code has a similar value range to the code value of the crop type code, making it easier to combine the two prediction settlements to calculate rewards.

[0089] Table 1

[0090]

[0091] Table 2

[0092]

[0093] Reward is given based on the difference between the predicted output and the actual output. The specific formula is:

[0094] r=r x +r y

[0095] Among them, r x is the reward for crop type discrimination, r y is the output forecast reward, and the calculation formula is

[0096] r x = {Total number of identified crop types (correct), 0 (incorrect)}

[0097] r y = Total yield class - ABS (forecast crop yield code - actual crop yield code)

[0098] This design allows two rewards r x and r yThe output levels are roughly on the same order of magnitude to avoid bias during training. It also rewards close output levels, enhancing the robustness of the model.

[0099] In calculating the crop type discrimination reward r x When the prediction network predicts the correct crop types, the number of correctly predicted crop types is taken as r x For example, if there is only one type of crop in the data sample and the prediction network correctly predicts its type, then r x = 1. If there are three kinds of crops in the data sample, and the prediction network correctly predicts the types of two of them, but incorrectly predicts the type of the other, then r x =2.

[0100] In calculating the output prediction reward r y When , "Total Yield Grades" is the total number of possible values ​​for the crop yield code. In the example in Table 2, "Total Yield Grades" is 7. ABS() represents a function that calculates absolute values. The predicted crop yield code comes from the prediction network's output, while the actual crop yield code comes from preprocessed data samples. This is calculated by mapping the actual yield of a particular crop in a particular location in a particular year (or crop growing season) to the average yield of that location over the past years (or multiple crop growing seasons) to the crop yield code.

[0101] Continue to read Figure 5 In this application, at time t, the time series data SS formed by multiple remote sensing data from the beginning of the current crop growth cycle to the current is obtained from the environmental agent. t =(S1,S2,…S t ), feature extraction network processing time series data SS t , generate high-dimensional time series remote sensing features, and obtain data features f after dimensionality increase t As the state of the reinforcement learning model at time t, it is provided to the prediction network and the confidence network. The prediction network (Actor) is based on the received data features f t Output predicted crop type p and yield a t As the action at time t, it is provided to the environment agent. The environment agent generates a reward r at time t based on the received action t And provide the remote sensing data of the next moment (t+1) to form the time series data SS t+1 =(S1,S2,…S t+1 ).

[0102] The environmental agent obtains the remote sensing data of the next moment from the data sample based on time t. The current moment t and the next moment t+1 are adjacent. For example, if the data sample is generated from the remote sensing data obtained from the satellite every 10 days, then the moment t+1 is 10 days later than the moment t. Therefore, the environmental agent of this application obtains the remote sensing data S from the data sample. t The data sample of the next moment represented by the current moment t is used to generate remote sensing data S t+1 This is different from the reinforcement learning method of the prior art. In the reinforcement learning model of the prior art, the environment generates the state of the next moment based on the current state and the impact of the action at the current moment on the environment. For example, in a chess game, the action is a move, and the new situation of the chess game after the move forms the state of the next moment. The environmental agent of the present invention does not need to simulate the state change of the environment, but uses the data samples generated from the remote sensing data to generate the remote sensing data S at time t+1. t+1 , and then generate the time data series SS at time t+1 t+1 .

[0103] In the present invention, the environmental agent calculates the predicted crop types and yields based on the received actions and the actual crop types and yields according to the aforementioned formula r=r x +r y Calculate the reward r at time t. In addition to using the predicted value output by the prediction network, the calculation of reward r requires the actual crop type and actual yield at time t+1. In agricultural production, except for the time when crops mature and are harvested, there are no corresponding crop types and actual yields for most of these times. In this application, the final yield of the crop in the data sample for that year (or season) is used as the crop type and actual yield corresponding to each data sample in the data sample time series of that year (or season), and is used to calculate the reward r.

[0104] Optionally, some of the predictions output by the prediction network include multiple crops, each with its own predicted yield and actual yield. y When the yield of each crop is calculated, r y , and use its statistical values ​​(sum, average, weighted average) to calculate the reward r.

[0105] The environment agent outputs the reward r corresponding to the output of the prediction network at time t t Provided to the confidence network. The confidence network is based on the data feature f at time t t Output value evaluation Eval(f t ). Eval() represents the value evaluation function of the confidence network. The confidence network also evaluates the value of the state of the two adjacent moments Eval(f t+1 ) and Eval(f t) and the reward r at time t t Calculate TD-error, expressed as δ = r t +λ*Eval(f t+1 )-Eval(f t ). The TD-error calculated at each moment is used to update the parameters of the prediction network and the confidence network in subsequent training.

[0106] During training, the feature extraction network outputs the predicted time data sequence from the high-dimensional time series features at time t. Next, the reinforcement learning model advances to the next moment. Using the next moment data feature f t+1 Provided to the prediction network and confidence network, and repeat the above process, the feature extraction network outputs the data feature f t+2 , the prediction network outputs the prediction result (crop type and yield) at the next moment, and the reward r at the next moment t+1 , and the confidence network based on the data features f t+2 and data features f t+1 Value evaluation Eval(f t+2 ) and Eval(f t+1 ) calculates the TD-error at the next moment. This process is called a time step.

[0107] The training process of a reinforcement learning model consists of multiple epochs, each of which includes multiple time steps. The number of time steps is, for example, the number of data samples in a time series obtained from remote sensing data over a crop growing season. After one epoch of learning, for example, the feature extraction network, prediction network, and confidence network are updated. Further epochs of learning are also performed.

[0108] During training, the feature extraction network is trained by comparing real remote sensing data at each time point in the data sample with the predictions of the feature extraction network. It is important to understand that while time series data from the complete crop growth cycle can be used during the training phase, during the prediction phase, since predictions are often made before the harvest season and remote sensing data from the complete growth cycle is not yet available, time series data from incomplete growth cycles can be used to identify crop species and predict yields. Optionally, time series data from incomplete crop growth cycles can also be used during the training process.

[0109] Model training

[0110] Table 3 shows the training process of the overall model according to the present invention.

[0111] One epoch of training consists of multiple time steps, where t represents the time step. At each time step, the environment agent obtains remote sensing data from the data sample and generates input time series data SS t .

[0112] Temporal difference (TD-error) is used to optimize the prediction network and confidence network. TD-error is calculated using rewards. The learning objectives of the prediction network are to maximize policy value, the confidence network is to minimize prediction error, and the feature extraction network is to minimize prediction error for remote sensing data. Network parameters are updated based on TD-error and the backpropagation algorithm.

[0113] See Table 3, a t and p represent the prediction results output by the prediction network at time t, including the predicted crop type p and yield a t SS t Represents the time series data input at time t, which comes from remote sensing data. t Represents the reward at time t, which is obtained by the difference between the prediction result of the prediction network and the real historical data. t Represents data features and is the output of the feature extraction network. prob represents the prediction result a output by the prediction network t The probability distribution corresponding to p is provided by the prediction network. Y represents the actual remote sensing data, and PS represents the remote sensing data predicted by the feature extraction network. Optionally, the feature extraction network predicts and outputs a time series that includes multiple sets of predicted remote sensing data at different time points, while the actual remote sensing data also includes multiple sets of corresponding data.

[0114] At each time step in the training process, the feature extraction network is trained based on the input time series data SS t Output data features f t (Step 3 of Table 3), data feature f t Provided to the prediction network, the prediction network outputs the prediction result a t and p (step 4 of Table 3). In Table 3, Pred() represents the processing performed by the prediction network. The prediction result a output by the prediction network t and p are provided to the environment agent, which then uses the prediction result a t and p generate the time series data S at time t+1 t+1 and the reward r at time t t (Step 5 of Table 3). As an example, the environment agent obtains the time series data S at time t+1 based on the time series of remote sensing data in the training data. t+1 , and calculate the reward r according to the reward calculation method provided by the present invention t In Table 3, ENV() represents the processing performed by the environment agent. The feature extraction network is again based on the input time series data SS t+1 Output data features f t+1 (Step 6 of Table 3). The confidence network is based on the data feature f t+1 With ft Calculate the TD-error, denoted as δ (step 7 in Table 3). Eval() represents the processing performed by the belief network. Generate a state transition tuple (f t ,(a t ,p),r t ,f t+1 ), which is used to subsequently update the parameters of the prediction network and the confidence network.

[0115] A training epoch includes multiple time steps. The number of time steps (denoted as N) is, for example, the number of remote sensing data in a remote sensing data time series within a crop growing season. Alternatively, the number of time steps in a training epoch can take other values.

[0116] For each time step of training, repeat steps 2 to 8 of Table 3. If the current time step is t, it is used to generate data features f t The time series data is SS t , then the next time step is t+1, which is used to generate the data feature f for the next time step t+1 The time series sequence is SS t+1 Therefore, in the later time steps, the time series data used include more remote sensing data (closer to the end of the crop growth cycle). In each growing season of crops, there is only one set of real crop species and real yield data as a data sample for training the reinforcement learning model. When calculating the reward at each time step of a training round (step 5 of Table 3), regardless of whether the time series sequence of the current time step is SS t Regardless of the number of remote sensing data points, rewards and errors are calculated using the same crop species and yield data from the same growing season. Temporal difference (TD-error) is used to optimize the prediction network and confidence network, enabling the reinforcement learning model to learn to achieve more accurate identification and prediction results based on more remote sensing data from the same growing season.

[0117] Using the N time steps of a crop growing season, calculate the objective function of the confidence network JV(w) = ∑δ 2 , where w represents the confidence network parameter, and the squares of the N δ values ​​obtained over N time steps are summed (step 9 of Table 3). Calculate the objective function of the prediction network as J(θ) = ∑(log(prob)δ) (step 10 of Table 3), with N summation terms. Calculate the objective function of the feature extraction network as J(k) = ∑(Y-PS) 2(Step 11 of Table 3), where the sum is the number of remote sensing data points output by the feature extraction network in N time steps. The feature extraction network is updated accordingly (Step 12 of Table 3), the confidence network is updated (Step 13 of Table 3), and the feature extraction network is updated (Step 14 of Table 3). The goal of learning the prediction network is to maximize the confidence level, the goal of learning the confidence network is to minimize the error in future error estimates, and the goal of training the feature extraction network is to minimize the mean squared error of future remote sensing image predictions.

[0118] Optionally, the calculation and update of the objective function of the feature extraction network in steps 11 and 12 may occur after steps 3 and / or 6, thereby updating the feature extraction network multiple times in one training round.

[0119] In steps 9 and 10 of Table 3, and the corresponding steps 13 and 14, the state transition tuples of a data batch are sampled to update the parameters of the prediction network and the confidence network. A data batch includes, for example, 256 data items. A data batch can include other amounts of data. The data in a data batch can include data from different plots, different crop growing seasons, and different types of crops, making the trained model more versatile, eliminating the need to train a dedicated model for each plot or each crop.

[0120] The present invention has no mandatory requirements for the number of training epochs, batch size, or time step. These are adjusted based on the natural spacing of the data and the speed of model convergence. For example, since Sentinel satellites pass over a fixed ground point once every five days, one time step is used, with every five days of satellite remote sensing data as one training batch. The number of training epochs is 256 samples, the number of training epochs is 5000, and the learning rate is set to 1e-5, with the learning rate reduced to 10% every 2000 steps.

[0121] Alternatively, the number of cycles from step 2 to step 7 in Table 3 represents a complete growth cycle of a crop. For example, the growth cycle of wheat is about 240 days, and a remote sensing data set is obtained every 5 days. The complete time series data consists of 48 remote sensing data sets. 48 time steps are used to input the time series data SS with lengths from 1 to 48. t For example, if we take the natural year as the growth cycle of crops and obtain a remote sensing data set every 5 days, the complete time series data has 74 remote sensing data sets. Accordingly, one round of training has 74 time steps. In some cases, the remote sensing data of some time points are missing (for example, blocked by clouds), and the input time series data SS t This may reflect missing data.

[0122] Table 3

[0123]

[0124] The present invention allows for data loss through a model pruning mechanism. Due to cloud cover and other objective factors, data from various remote sensing bands cannot always be obtained periodically. The pruning code design allows for data loss in certain bands or time points.

[0125] Figure 6 The overall structure of the crop identification and yield prediction model of another embodiment of the present application is shown.

[0126] exist Figure 6 In the embodiment, the feature extraction network is a decoder in the Transformer model of the Decoder Only architecture, and the time series data SS provided to the feature extraction network is t =(S1, S2, S2, ... S t ) includes multiple remote sensing images (denoted as image 1, image 2, ... image t). These remote sensing images are a sequence of remote sensing images of a certain area at different time points in the crop growth cycle, sorted in time. For example, image 1 is from March 1, image 2 is from March 15, image 3 is from March 30, and so on. The time series data output by the feature extraction network includes multiple predicted remote sensing images (denoted as image prediction 2, image prediction 3, ... image prediction t+1). By outputting predicted remote sensing images, users can identify the reasoning process or basis for the overall model of this application to make predictions on crop types and yields. For example, the image predictions in the later period in the time series data relatively intuitively reflect the growth of crops, so that users can infer the prediction basis of the prediction model based on the growth of crops they see.

[0127] Figure 6 In , the input time series data provided to the feature prediction network includes t remote sensing images, and the output time series data also includes t image predictions.

[0128] In the traditional actor-critic architecture, the action A generated by an actor acts on the environment, and the rewards corresponding to different actions can vary significantly. In the crop yield prediction scenario of the present invention, the actor's prediction of crop yield and variety is output to the environment as action A. This does not generally affect the actual yield, thus lacking a closed-loop "action-environment-reward" mechanism. Therefore, the traditional understanding is that the actor-critic architecture cannot be applied to the application scenario of crop yield prediction.

[0129] However, in the inventor's opinion, the prediction of future crop yields will have an impact on the future to a certain extent. For example, a pessimistic prediction of yields may prompt people to increase investment, so that the actual yield will be higher than the predicted yield. From this perspective, these implicit influencing factors can also be attributed to the environment in the prediction task, so that the behavior of predicting crop yields has the effect of acting on the environment to a certain extent. The crop yield prediction model of the present invention is not mathematically equivalent to the traditional method. The design method of optimizing the prediction network and the confidence network separately is essentially a framework that separates prediction and evaluation, so that the value function can measure the uncertainty of the current state, which is lacking in traditional regression methods, thereby adapting to more complex scenarios without considering implicit variables (meteorology, soil, altitude, time, etc.).

[0130] Moreover, in traditional reinforcement learning models, each state processed has the same structure (e.g., different chess situations). However, in the present invention, at each time step in training, the data features f provided to the prediction network and the confidence network are t Corresponding to time data series SS of different lengths t , as time step t advances, longer time data series SS t Not only corresponding to different data features f t , and the data feature f t The amount of information carried in increases, which helps to obtain more accurate prediction results as time t progresses. When applying the trained prediction network, the data features f t Can also be generated from time data series SS of different lengths t , so that the present invention can be used to identify crop species and predict yields regardless of when in the crop growth cycle.

[0131] Figure 7 A flowchart showing how to use the trained ensemble model to predict crop types and yields.

[0132] First, a target area is selected and remote sensing data of the target area is acquired. The acquired remote sensing data may be multiple copies, each representing remote sensing data at multiple consecutive time points. The multiple copies of remote sensing data are, for example, all available remote sensing data of the target area from the beginning of the current crop growth cycle to the present. When the crop growth cycle is based on a natural year, the multiple copies of remote sensing data are all available remote sensing data from the beginning of the year or after sowing to the present. Optionally, the remote sensing data also includes day-of-year (DOY) information to indicate the temporal position of the generation time of the remote sensing data in the crop growth cycle.

[0133] Preprocess the remote sensing data to generate time series data for input to the feature extraction network. This preprocessing includes, for example, normalizing the data across bands, generating one or more remote sensing indices to increase the dimensionality of the input data, and embedding year-day information. Optionally, preprocessing also addresses missing data and supplements it with weather and geolocation data.

[0134] The feature extraction network generates high-dimensional time series remote sensing features based on the input time series data. The feature extraction network generates predicted remote sensing images from the high-dimensional time series remote sensing features, and also increases the dimension of the high-dimensional time series remote sensing features to obtain data features (f).

[0135] The data features (f) are provided to the prediction network, which generates predicted crop types and yields based on the data features.

[0136] In the early stages of the crop growth cycle, the input time series data includes relatively early and smaller amounts of remote sensing data, and the predictions made by the feature extraction network and the prediction network may not be accurate. As time goes by, more remote sensing data of the target area is available. Using more remote sensing data to generate time series data and provide it to the feature network, the feature extraction network and the prediction network can make more accurate predictions accordingly. Figure 7 The process shown can be executed multiple times, for example, monthly. Figure 7 The processing flow is used to obtain the predicted value of the target area. As more remote sensing data of the current growth cycle are collected, the prediction results will become more accurate.

[0137] When predicting crop types and yields, there is no need to use a confidence network. Optionally, when the types and actual yields of crops currently planted in the target area are obtained, the training process shown in Table 3 can also be used to optimize the overall network provided by the present invention. For example, see Figure 7 By comparing the predicted crop types and yields with the actual values ​​to generate rewards, the rewards can be further used to calculate the prediction error (TD-Error, denoted as δ), so that the confidence network's prediction of confidence can be corrected. Different uses of δ and confidence can be used to update the prediction network and the confidence network.

[0138] As time goes by, more real remote sensing data is obtained, which can also be used to optimize the feature extraction network.

[0139] The present invention includes the following key points.

[0140] Self-attention mechanism extracts time series remote sensing features: This mechanism extracts bands, remote sensing indices, geographic information, and temporal features from time series remote sensing data, enabling global information modeling. This overcomes the limitations of traditional time series models and improves feature extraction capabilities for complex time series data. This objectively reduces the need for production data samples, enabling small sample size training.

[0141] The feature extraction network is combined with the reinforcement learning model to predict crop yields using time series data, taking into account both high-dimensional feature extraction capabilities and the ability to train for complex scenarios.

[0142] Training an independent remote sensing prediction module makes the reasoning process visible: The model is designed to output the predicted remote sensing image as an intermediate result, thereby achieving the interpretability of the prediction model in a visual way, enabling the model to have the output capability of the reasoning process and improving user acceptance of the model.

[0143] Reuse feature extraction network design: The confidence network and prediction network share the model structure of the feature extraction network, saving 25% of computing resources and reducing model training and inference time by 40%, making it suitable for large-scale remote sensing interpretation tasks.

[0144] Model pruning enhances robustness: The Dropout layer, designed during training, imparts robustness to the model, allowing for missing data in certain bands or time points without affecting the model's overall predictive power, thus enhancing its practicality. During training, the Dropout layer randomly discards input elements to simulate missing data, enabling the model to learn to predict even with some data missing.

[0145] Dimensionality increase processing: By performing dimensionality increase processing on remote sensing band data, deeper correlations in the data can be mined, thereby improving the accuracy of modeling and predictive capabilities.

[0146] Time and position encoding: Time and position encoding of day-of-year (DOY) features can more accurately capture the cyclical growth patterns of crops.

[0147] The value function design based on TD-error achieves rapid iteration and improved accuracy. It not only enables the model to start predictions in the early stages of crop planting, but also optimizes the value function with time as the independent variable so that the value function can adapt to different prediction scenarios and time, thereby improving the model's performance in long-term predictions, thereby achieving "less accurate early predictions" and "more accurate late predictions" in one model.

[0148] In addition, relative yield is designed in the present invention. The average yield of cultivated land of this grade in the region is used as a benchmark to predict relative yield, thereby realizing yield prediction for different regions and soil characteristics. Future yield is used as a long-term delayed incentive, so that the model can make reasonable predictions in long-term tasks.

[0149] Compared with existing crop yield prediction methods, the present invention also has the following significant advantages:

[0150] Global information modeling: The self-attention mechanism enables the model to simultaneously capture the global dependencies between different bands and different time points, greatly improving the modeling ability of complex time series data.

[0151] Robustness to missing data: Through pruning, this method tolerates partial data loss in remote sensing data, enhancing the model's practicality. The feature extraction network can be trained independently, making the remote sensing feature extraction portion independent of yield samples, thus supporting training feasibility with smaller yield samples.

[0152] Capturing periodicity and temporal features: Through temporal position encoding and DOY features, the model performs better when processing seasonal and periodic data, and is particularly suitable for crop growth prediction in agricultural remote sensing scenarios.

[0153] High-dimensional data modeling: Dimensionality increase enables the model to better explore the deep correlations between bands, improving the modeling accuracy of remote sensing data.

[0154] Enhanced interpretability: The model design allows the predicted remote sensing image to be output as an intermediate result, thereby achieving the interpretability of the model in a visual way, enabling the model to have the output capability of the reasoning process, improving user acceptance of the model, and promoting its widespread application in actual production.

[0155] Resource saving: The shared Transformer model structure design saves computing resources by 25% and reduces model training and inference time by 40%, making it suitable for large-scale remote sensing interpretation tasks.

[0156] The application prospects of this invention are broad. The remote sensing time series features extracted by this model can be further used to identify crop types, crop growth monitoring and yield prediction, environmental monitoring, climate change prediction and analysis, etc., providing efficient and accurate prediction models in the fields of agriculture, meteorology, etc., significantly improving the utilization efficiency of remote sensing data.

[0157] Figure 8 This is the interpretation of remote sensing data showing the distribution of winter wheat planting areas in a certain region of Henan Province in 2023. Green represents winter wheat planting areas, red represents other areas, larger red areas represent towns, smaller areas represent villages, and lines represent rivers.

[0158] Figure 9 This is an image prediction obtained based on remote sensing images in the embodiment of the present application.

[0159] Figure 9 The three columns represent three sequential time points, separated by approximately two weeks. In each column, the first row of images is the original RGB remote sensing image, the second row is the actual NDVI (vegetation cover index) calculated based on the remote sensing bands, the third row is the predicted NDVI output by the feature extraction network, which can be used as a visualization tool to check the final results, and the fourth row is the predicted crop planting distribution output by the feature extraction network.

[0160] Figure 10 This is the result of crop type prediction in the embodiment of the present application.

[0161] The classification results of crop types are statistically analyzed, such as Figure 10 The horizontal axis represents time, using six temporally consecutive remote sensing images. The percentage line indicates the proportion of valid data for the pixel position to be tested in each remote sensing image. That is, 100% of the pixel positions have valid data in at least one remote sensing image, and only 10% of the pixel positions have valid data in all six remote sensing images. Figure 10 The upper left graph shows the accuracy metric, the upper right graph shows the precision, the lower left graph shows the recall rate, and the lower right graph shows the F1 harmonic mean of the accuracy and precision. As can be seen, most of these metrics improve over time, demonstrating that the present invention can identify crops early in their growth and improve both accuracy and recall over time.

[0162] Figure 11 This is the result of crop yield prediction in the embodiment of the present application.

[0163] The prediction results of crop yield are as follows: Figure 11 As shown, the horizontal axis is time, and 6 temporally continuous remote sensing images are used. Figure 11 The left figure is the mean square error (MSE) of the relative yield, the middle figure is the mean absolute error (MAE) of the relative yield, and the right figure is the MSE, MAE and R2 indicators of the last frame prediction of all pixels. It can be seen that the model of the present invention gradually becomes more accurate in predicting yield over time.

[0164] The innovations of the present invention include:

[0165] 1. Use machine learning and deep learning methods to extract the deep features of time-series remote sensing images, replacing the traditional exponential formula method, so that the model does not rely on empirical exponential formulas, allowing the model to discover patterns independently. The upper limit of the model's capabilities depends on the richness of the data.

[0166] Second, use the self-attention mechanism to extract time series remote sensing features, replacing LSTM, allowing for partial data missing and allowing early prediction.

[0167] Third, the prediction model is divided into an independent prediction network and a confidence network, replacing a single prediction model, enhancing the model's ability to understand complex situations, and realizing a universal prediction model for multiple crops in different regions. The reliability of the prediction can be judged by the confidence level.

[0168] Fourth, an independent third network (feature extraction network) is used to output predictions of future remote sensing images, changing the previous black box prediction paradigm, thereby achieving output of the reasoning process and making the predictions explainable.

[0169] Table 4 Comparison of the present invention with other methods

[0170]

[0171] Although the present application has been described with reference to examples, this is for illustrative purposes only and is not intended to limit the present application, and changes, additions and / or deletions to the embodiments may be made without departing from the scope of the present application.

[0172] Those skilled in the art to which these embodiments relate and who benefit from the teachings presented in the above description and the associated drawings will recognize many modifications and other embodiments of the present application described herein. Therefore, it should be understood that this application is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. A method for crop identification and yield prediction, comprising: Preprocessing remote sensing data of the target area to generate time series data; The feature extraction network processes the time series data using a self-attention mechanism to generate high-dimensional time series remote sensing features, increases the dimension of the high-dimensional time series remote sensing features to obtain data features, and provides the data features to the prediction network; The feature extraction network also generates a predicted remote sensing image based on the high-dimensional time series remote sensing features; The prediction network generates a prediction of the type and yield of the crop according to the data features; The process of training the feature extraction network and the prediction network includes: At each time step, the time series data SS of the training sample t and SS t+1 Provided to the feature extraction network, the feature extraction network outputs data features f t and f t+1 , and predict remote sensing images PS; the prediction network outputs a prediction result; based on the prediction result output by the prediction network, a reward r is generated t ; The confidence network is based on the reward r t , data features f t+1 With f t Calculate the TD-error, expressed as δ; where t represents the current time step; In each training round, the following steps S1 to S4 are also performed: S1. Using the δ of multiple time steps of the training round, calculate the objective function of the belief network JV(w) = ∑δ 2 , where w represents the parameters of the belief network; S2. Calculate the objective function J(θ)=∑(log(prob)δ) of the prediction network, where prob represents the probability distribution corresponding to the prediction result output by the prediction network, and θ is a parameter of the prediction network; S3. Calculate the objective function J(k)=∑(Y-PS) of the feature extraction network 2 , where Y represents the real remote sensing image, PS represents the predicted remote sensing image generated by the feature extraction network, and k is the parameter of the feature extraction network; S4. Update the feature extraction network, update the prediction network, and update the confidence network.

2. The method according to claim 1, wherein The process of training the feature extraction network, the prediction network, and the confidence network includes a plurality of training rounds, each training round including a plurality of time steps; At each time step, perform the following steps A1 to A5: A1. The time series data SS of the training sample t Provided to the feature extraction network, the feature extraction network outputs data features f t , and predict remote sensing image PS; A2. Data feature f t Provided to the prediction network, the prediction network outputs a prediction result; A3. Generate time series data SS for the next time step t+1 , and generate a reward r based on the prediction result output by the prediction network t ; Among them, r t Represents the reward at time step t, which is obtained by the difference between the prediction result output by the prediction network and the actual historical data; A4. Convert the time series data SS t+1 Provided to the feature extraction network, the feature extraction network outputs data features f t+1 ; A5. According to the reward r t , data features f t+1 With f t Calculate TD-error, expressed as δ, δ = r t +λ*Eval(f t+1 )-Eval(f t ), where Eval() represents the processing performed by the confidence network, and λ represents a hyperparameter; wherein the prediction results include the predicted crop type p and yield a t .

3. The method according to claim 2, wherein: in, The feature extraction network is a neural network with a decoder-only architecture, and the prediction network and the confidence network are each a multi-layer neural network; The training samples include time series data generated based on historical real remote sensing data and corresponding crop type and yield data.

4. The method according to claim 3, wherein: The feature extraction network is a decoder in the Transformer model, including multiple stacked decoder layers. It processes the global dependencies of time series data through a multi-head self-attention mechanism, allowing remote sensing data at multiple time points and multiple bands to interact with each other. The feature extraction network generates predicted time series data based on the high-dimensional time series remote sensing features, wherein the predicted time series data includes predicted remote sensing images representing multiple future time points and multiple bands within the growth cycle of the crop; The structure of the data features output by the feature extraction network does not change with the structure of the time series data; During the training of the feature extraction network, part of the input time series data is randomly discarded through the Dropout layer to simulate data missing.

5. The method according to claim 4, wherein The preprocessing of the remote sensing data of the target area to generate time series data includes: Acquiring remote sensing data at multiple time points within the growth cycle of the crop, wherein the remote sensing data at each time point includes year and day attributes; Normalize the data of each band of remote sensing data at each time point to between 0 and 1; Using remote sensing data at each time point to generate a variety of remote sensing indices as features, expanding the dimension of remote sensing data; Generate extended remote sensing data at each time point, the extended remote sensing data including normalized band data of remote sensing data, one or more remote sensing indices, and annual and daily attributes; Combine the extended remote sensing data at each time point to obtain time series data; When the feature extraction network processes the time series data to generate high-dimensional time series remote sensing features, the year and day attributes of each extended remote sensing data of the time series data are also used as position codes for input of the feature extraction network.

6. The method according to claim 5, wherein: The preprocessing of the remote sensing data of the target area to generate time series data further includes: The time series data is formed by sorting the time represented by the year and day attributes of each extended remote sensing data of the time series data; Mark missing one or more time points in time series data.

7. The method according to claim 6, wherein: In step S1, the objective function of the confidence network JV(w)=∑δ 2 The number of summations in is the number of time steps in the current training epoch; In step S2, the number of summation terms of the objective function J(θ)=∑(log(prob)δ) of the prediction network is the number of time steps in the current training round; In step S3, the objective function of the feature extraction network is J(k)=∑(Y-PS) 2 The number of summation items is the number of predicted remote sensing images output by the feature extraction network at each time step in the current training round.

8. The method according to claim 7, wherein: The number of time steps in each training round is the number of time points N of remote sensing data within the growing season of the crop; In each training round, it also includes a state transition tuple (f t ,(a t ,p),r t ,f t+1 ); The remote sensing data includes remote sensing image data of multiple bands at multiple time points within the crop growth cycle.

9. The method according to claim 8, wherein The time series data used in each time step has a different end time point; The time series data SS for the next time step is generated t+1 , including obtaining the time series data SS at time step t t The extended remote sensing data of the next time point at the end time point is appended to the time series data SS of time step t t The time series data SS obtained later t+1 .

10. An information processing device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images