Crop planting optimization decision method based on digital twinning and reinforcement learning

Abstract

The present application relates to the field of crop planting optimization decision, and particularly relates to a crop planting optimization decision method based on digital twinning and reinforcement learning. The technical scheme comprises: collecting greenhouse multi-modal data, including greenhouse environment data, crop data, soil data and management data; cleaning, aligning and normalizing the collected data to form a structured multi-modal time series data set; identifying the growth cycle of crops based on the collected data; constructing a crop-environment digital twinning model that is real-time mapped with the physical greenhouse environment; constructing a digital twinning simulation space; reinforcing learning in the digital twinning simulation space and optimizing the strategy to dynamically and adaptively generate a globally optimal control strategy. The present application is suitable for crop planting optimization decision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of crop planting optimization decision, and in particular to a crop planting optimization decision method based on digital twinning and reinforcement learning. BACKGROUND

[0002] With the development of Internet of Things, big data and artificial intelligence technology, precision agriculture has become an important direction of modern agriculture. Traditional crop planting management mainly relies on the experience of growers, or passive and extensive control based on simple sensor thresholds (such as turning on the fan when the temperature is higher than 30℃, and starting irrigation when the soil humidity is lower than 50%). These methods have the following problems: 1) The control strategy is static and isolated, and the coupling effect between multiple environmental factors (temperature, light, humidity, CO2 concentration, soil nutrients, etc.) is not considered; 2) The control method is "reactive" rather than "predictive", and often intervenes after the crop appears stress, which cannot achieve optimal growth; 3) It is difficult to find a globally optimal control strategy, and it is difficult to achieve the best balance between water, fertilizer, electricity and crop yield, quality.

[0003] To solve the above problems, some researches begin to apply crop growth models for growth simulation, or use machine learning for pest and disease identification and growth prediction. Although the automatic system improves the decision-making ability, there are still the following problems.

[0004] Most automatic systems use control logic based on fixed thresholds (for example, turn on the ventilation when the temperature is higher than 30℃). This way is reactive and cannot adapt to the dynamic needs of crops in different growth stages (such as seedling stage, flowering stage, fruiting stage), and cannot handle the coupling effect between multiple variables, such as the synergistic promotion of high light and high carbon dioxide concentration on photosynthesis.

[0005] Complex long-period management (such as water and fertilizer integration scheme during the whole growth period) highly depends on the experience of planting experts. This experience is difficult to quantify, replicate and optimize, and when facing extreme weather or new varieties, the trial and error cost is high and the risk is great.

[0006] Simple crop growth mechanism models have fixed parameters and cannot accurately reflect the real growth conditions of specific varieties in specific greenhouses; while simple data-driven models (such as machine learning prediction) lack biological mechanism constraints, and have poor generalization ability and interpretability.

[0007] Existing technical solutions are mostly open-loop prediction or single-step control, and lack a global optimal control strategy generation method that can maximize the final yield or benefit during the entire growth period. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of the prior art and provide a crop planting optimization decision-making method based on digital twinning and reinforcement learning, which realizes dynamic self-adaptive generation of a globally optimal control strategy.

[0009] The present application achieves the above-mentioned purpose by adopting the following technical solutions, and provides a crop planting optimization decision-making method based on digital twinning and reinforcement learning, comprising:

[0010] S1, collecting greenhouse multi-modal data, including greenhouse environment data, crop data, soil data and management data;

[0011] S2, cleaning, aligning and normalizing the collected data to form a structured multi-modal time series data set;

[0012] S3, identifying the growth cycle of the crop based on the collected data;

[0013] S4, constructing a crop-environment digital twinning model that is real-time mapped with the physical greenhouse environment;

[0014] S5, constructing a digital twinning simulation space;

[0015] The digital twinning is formalized as a Markov decision process, defined as a tuple , wherein, represents the state space, represents the action space, represents the state transition probability function, represents the reward function, represents the discount factor, and the Markov decision process is the training environment of the reinforcement learning agent, and the core is the state transition function , which describes the probability of the system transitioning to the next state after performing action in the current state ;

[0016] S6, reinforcement learning strategy optimization;

[0017] A deep reinforcement learning agent interacts with the digital twinning for interactive learning, and the definition of the state space S is:

[0018] The state at time is defined as:

[0019] ;

[0020] , wherein, represents the current state, represents the current time growth stage, current time accumulated dry matter, current time leaf area index, current time temperature, current time humidity, current time light intensity, current time carbon dioxide concentration, current time soil nitrogen content, temperature prediction value for the next 24 hours, light prediction value for the next 24 hours;

[0021] Definition of action space A:

[0022] ;

[0023] wherein, action performed in the current state, temperature set value, humidity set value, light compensation duration, irrigation amount, fertilizer concentration;

[0024] Design of reward function R:

[0025] ;

[0026] wherein, reward, single-step dry matter increment simulated by digital twin, which is a direct proxy indicator of yield, electric energy cost consumed by performing action , water and fertilizer cost consumed, penalty term generated when environmental indicators in the next state exceed the suitable range for crops, respectively represent the adjustable weight coefficients of the corresponding terms;

[0027] Policy learning:

[0028] The goal of the agent is to learn a policy to maximize the expected cumulative reward , offset for a time step, represents the expected cumulative reward;

[0029] The strategy network is optimized by the PPO algorithm, and the core is the iterative update of the value function, which follows the Bellman optimal equation:

[0030] ;

[0031] Through simulation training in the digital twin simulation space for multiple growth cycles, the agent finally learns the optimal action-value function and the optimal strategy , E represents the expectation of the total return that can be obtained in the future under the condition of taking action in the current state .

[0032] Further, the optimization decision method further comprises:

[0033] S7, closed-loop control and feedback optimization;

[0034] Deploy and run the trained optimal strategy ;

[0035] Forward control:

[0036] Every other decision cycle, collect the real state , input into the policy network , get the optimal action , parse into specific instructions to drive the physical actuator;

[0037] Feedback optimization:

[0038] At the same time, the collected real state is used as a new training sample to update or fine-tune the data-driven correction sub-model online.

[0039] Further, in step S1, collecting greenhouse environment data, crop data, soil data and management data specifically includes:

[0040] Greenhouse environment data collection:

[0041] Through temperature and humidity sensors, light intensity sensors and carbon dioxide concentration sensors, real-time collection of greenhouse air temperature, humidity, light and carbon dioxide concentration data;

[0042] Crop data collection:

[0043] Through RGB camera, 3D camera, hyperspectral camera and leaf temperature sensor, the plant height, leaf area index, canopy structure, leaf color, fruit number and size and leaf temperature data of crops are collected;

[0044] Soil data collection:

[0045] Through soil sensors, the pH value, conductivity, temperature and humidity and nutrient content data of the soil are collected;

[0046] Management data collection:

[0047] Through touch screen terminal or mobile terminal App, the planting time, irrigation amount, fertilizer concentration and pesticide spraying data are manually recorded.

[0048] Further, step S3 specifically comprises:

[0049] The collected crop data and the planting start date of the crop are input into the pre-trained convolutional neural network, the phenological characteristics of the crop are identified through the pre-trained convolutional neural network, and the current growth stage of the crop is comprehensively judged based on the phenological model of effective accumulated temperature.

[0050] Further, step S4 specifically comprises:

[0051] The crop-environment digital twin model comprises the following coupled submodels:

[0052] Crop growth mechanism submodel:

[0053] The core of the crop growth mechanism submodel is dry matter accumulation, and the daily dry matter increment is calculated by photosynthesis efficiency:

[0054] ;

[0055] In the formula, represents the daily dry matter increment, represents the light energy utilization efficiency, (PAR) represents the photosynthetically active radiation, represents the extinction coefficient, which describes the degree of canopy shading, (LAI) represents the leaf area index, and are temperature, carbon dioxide concentration and nitrogen stress correction functions of photosynthesis, respectively;

[0056] Data-driven correction submodel:

[0057] Using real-time collected data, the set parameters of the crop growth mechanism submodel are corrected online through Kalman filtering or long short-term memory network.

[0058] The beneficial effects of the present application are:

[0059] The present application constructs a high-fidelity crop-environment dynamic digital twin that combines crop growth mechanisms and real-time multi-modal data, and creatively uses it as a virtual training environment for reinforcement learning agents. In this digital twin, the agent learns a strategy that aims to maximize the cumulative reward (comprehensive yield and cost) throughout the growing season, autonomously explores and masters a set of long-period, multi-variable, dynamic coordination global optimal control strategies. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of a crop planting optimization decision method based on digital twinning and reinforcement learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0061] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0062] The present application provides a crop planting optimization decision method based on digital twinning and reinforcement learning, as shown in Figure 1 specifically includes:

[0063] S1, collect multi-modal data of the greenhouse, including greenhouse environment data, crop data, soil data and management data;

[0064] Greenhouse environment data collection:

[0065] Through the temperature and humidity sensor, the light intensity sensor and the carbon dioxide concentration sensor, the air temperature, humidity, light and carbon dioxide concentration data in the greenhouse are collected in real time;

[0066] Crop data collection:

[0067] Through the RGB camera, 3D camera, hyperspectral camera and leaf temperature sensor, non-destructive data such as plant height, leaf area index, canopy structure, leaf color, fruit number and size, and leaf temperature of the crop are collected;

[0068] Soil data collection:

[0069] Through the soil sensor, the pH value, conductivity, temperature and humidity, and nutrient content (nitrogen, phosphorus, potassium) data of the soil are collected;

[0070] Management data collection:

[0071] The planting time (year-month-day), irrigation amount (L), fertilizer concentration (g / L) and pesticide spraying times are recorded manually through a touch screen terminal or a mobile terminal App.

[0072] The plant morphology data can also be acquired by using other sensors such as a laser radar.

[0073] S2, the collected data is cleaned (abnormal values are removed), aligned (time stamps are unified) and normalized to form a structured multi-modal time series data set, which provides a data basis for subsequent construction and synchronization of a digital twin model.

[0074] S3, the growth cycle of the crop is identified based on the collected data;

[0075] The crop images are collected by the RGB camera deployed in the greenhouse at regular time intervals, combined with the planting start date, and input into a pre-trained convolutional neural network to identify the key phenological characteristics (such as the number of flower buds and the number of fruits) of the crop, and the growth stage of the current crop is comprehensively judged by combining the effective accumulated temperature-based phenological model (such as seedling, vegetative growth period, flowering period, fruit setting period and maturity period).

[0076] S4, a crop-environment digital twin model that is real-time mapped with the physical greenhouse environment is constructed;

[0077] The crop-environment digital twin model includes the following coupled sub-models;

[0078] Crop growth mechanism sub-model:

[0079] The core of the crop growth mechanism sub-model is dry matter accumulation, and the daily dry matter increment is calculated by photosynthetic efficiency:

[0080] ;

[0081] In the formula, represents the daily dry matter increment, represents the light energy utilization efficiency, represents the photosynthetically active radiation, represents the extinction coefficient, which describes the degree of light shielding by the canopy, represents the leaf area index, and are stress correction functions (value range 0-1) of temperature, carbon dioxide concentration and nitrogen on photosynthesis, respectively;

[0082] The crop growth mechanism sub-model can be replaced by other mature models, such as WOFOST (World Food Studies, mechanism type crop growth model), DSSAT (Decision Support System for Agrotechnology Transfer, comprehensive type agricultural technology decision support system) or TOMGRO (Tomato Growth Model, greenhouse tomato growth special model) and the like.

[0083] Data-driven correction sub-model:

[0084] Using real-time collected data, the set parameters of the crop growth mechanism sub-model are corrected online through Kalman filtering or long short-term memory network, such as light energy utilization efficiency and extinction coefficient , to ensure that the model output is synchronized with the real world. The long short-term memory network is selected because it has the ability to capture long-term temporal dependencies, and can learn the dynamic rules of parameter changes from historical environmental and growth data sequences, which is crucial for simulating the crop growth process with long-term memory effect.

[0085] For example, the long short-term memory network receives historical environmental data sequences and crop growth data sequences, predicts the next time model parameter correction amount, and dynamically adjusts the mechanism model to better fit the growth curve under specific varieties and environments.

[0086] In addition to selecting the long short-term memory network, other time series prediction models such as GRU (Gated Recurrent Unit, gated recurrent unit), Transformer, etc. can also be selected.

[0087] S5, constructing a digital twin simulation space;

[0088] The digital twin is formalized as a Markov decision process, defined as a tuple , wherein represents a state space, represents an action space, represents a state transition probability function, represents a reward function, represents a discount factor, the Markov decision process is the training environment of the reinforcement learning agent, and the core is the state transition function , which describes the probability of the system moving to the next state after performing action in state ; in the present application, the function is determined and calculated by the crop-environment digital twin model.

[0089] S6, reinforcement learning policy optimization;

[0090] An intelligent agent of deep reinforcement learning interacts with a digital twin to learn, in which the agent represents the subject of performing actions, interacting with the environment and learning, and in the present application, the agent represents a deep neural network model.

[0091] Definition of state space S:

[0092] State At time is defined as:

[0093] ;

[0094] In the formula, represents the current state, represents the current time growth stage, represents the current time accumulation of dry matter, represents the current time leaf area index, represents the current time temperature, represents the current time humidity, represents the current time light intensity, represents the current time carbon dioxide concentration, represents the current time soil nitrogen content, represents the temperature prediction value of the next 24 hours, represents the light prediction value of the next 24 hours;

[0095] Definition of action space A:

[0096] ;

[0097] In the formula, represents the action performed by the current state, represents the temperature setting value, represents the humidity setting value, represents the light supplement duration, represents the irrigation amount, represents the fertilization concentration;

[0098] Design of reward function R:

[0099] ;

[0100] wherein, represents the reward, represents the single-step dry matter increment simulated by the digital twin, which is a direct proxy of the yield, represents the action consumed electrical energy cost, represents the water and fertilizer cost consumed, represents the next state penalty term generated when the environmental indicator exceeds the suitable range of the crop, respectively represent the adjustable weight coefficients of the corresponding terms.

[0101] The design of the reward function of the present application can be adjusted. In addition to maximizing the yield, it can also be set to maximize the quality (such as the sugar-acid ratio), maximize the profit (considering the selling price and the cost comprehensively), or minimize the carbon emissions, etc. other optimization goals.

[0102] Policy learning:

[0103] The goal of the agent is to learn a policy to maximize the expected cumulative reward , represents the offset of a time step, i.e. represents the reward of the kth step, represents the expected cumulative reward.

[0104] It is realized by optimizing the policy network through the PPO algorithm, and the core is the iterative update of the value function, which follows the Bellman optimality equation:

[0105] ;

[0106] wherein, represents the optimal value of taking action in state , represents the optimal action selected in the next state , and E represents the expected total return that can be obtained in the future under the condition of taking action in the current state .

[0107] In the framework of reinforcement learning, the response of the environment (i.e. state transition and reward) is not necessarily completely determined, but often random and uncertain. Even in the same state , taking the same action , the environment can transfer to multiple different next states , and each transfer occurs with different probabilities.

[0108] Therefore, instead of considering only one single future outcome, we must consider all possible future outcomes and take a weighted average of them according to their probabilities. This is the mathematical expectation The work done.

[0109] In this invention, this uncertainty is reflected in the following aspects:

[0110] Biological randomness: The crop itself is a living being, and its response to environmental regulation has inherent biological randomness. For example, even if the same light and water and fertilizer are given (action ), the dry matter increment of a single crop (affecting the next state ) may fluctuate within a small range.

[0111] Uncertainty of the environment model: Although the digital twin model is high-fidelity, it is still an approximation of the real world. The model itself may contain stochastic processes (e.g., stochastic occurrence of simulated diseases), or the weather forecast itself is probabilistic. For example, performing the action of "ventilation cooling", the temperature that can ultimately be reached (next state) will be affected by uncontrollable random factors such as outdoor wind speed.

[0112] Sensor noise: The sensor readings that measure the next state may also have random noise.

[0113] Uncertainty of rewards: The immediate reward itself may also be uncertain. For example, the cost of electricity for performing the action of supplemental lighting may be affected by time-of-use electricity prices, and the price at a certain time in the future may be a random variable.

[0114] Expanding the expectation : Suppose that after performing action from state , it can transition to different next states , each with a probability of , and the corresponding reward obtained is .

[0115] Then the Bellman optimality equation can be written more specifically as:

[0116] ;

[0117] The above expansion clearly shows that:

[0118] The value of is the weighted average of the total return of all possible futures, and the weight is the probability of each future (i.e., transition to state The introduction of E is expected, and the technical scheme of the present application considers the inherent randomness and uncertainty in the agricultural production process. The goal of the agent is not to obtain the highest return in a lucky case, but to learn a strategy that performs optimally in the long term, on average. The value calculated by the agent represents the most robust and reliable long-term value expectation of performing the action in the current state, and the control strategy finally generated has stronger robustness and practical applicability. The value calculated by the agent represents the most robust and reliable long-term value expectation of performing the action in the current state, and the control strategy finally generated has stronger robustness and practical applicability.

[0119] The PPO (Proximal Policy Optimization) algorithm is preferred in the present application, because it limits the difference between the new and old strategies during policy updating by introducing a clipping objective function, effectively avoiding the training collapse problem caused by too large policy updating step, thereby ensuring the stability and convergence efficiency of reinforcement learning in a complex agricultural environment model.

[0120] By simulating multiple growth cycles in the digital twin simulation space, the agent finally learns the optimal action-value function and the optimal strategy .

[0121] S7, closed-loop control and feedback optimization;

[0122] Deploy and run the trained optimal strategy .

[0123] Forward control:

[0124] Every other decision cycle, collect the real state , input the policy network , get the optimal action , and parse it into specific instructions to drive the physical actuator . Feedback optimization:

[0125] At the same time, the collected real state

[0126] is used as a new training sample to update or fine-tune the data-driven correction sub-model online.

[0127] The present application realizes the transition from "passive reaction" to "active optimization": unlike the fixed threshold control of the prior art, the present application performs global optimization in the digital twin through reinforcement learning, and can find a dynamic and coordinated multivariate optimal control strategy that is difficult for human experts to formulate, thereby maximizing crop yield and resource utilization efficiency.

[0128] ​​​The application realizes high-precision and forward-looking prediction control: the digital twin combines mechanism and data, and can accurately predict the long-term impact of control actions. Reinforcement learning agents can use these predictions to make forward-looking decisions and create a stable and optimal environment for crop growth.

[0129] The application provides a safe and low-cost strategy verification platform: the digital twin provides a virtual test field for complex control strategies. Researchers and growers can safely, quickly and cost-effectively conduct massive experiments and optimization without affecting physical crops, greatly reducing decision-making risks and research and development costs.

[0130] The application realizes highly adaptive and personalized planting: the framework of the application is universal. By replacing or adjusting the crop model and environmental model parameters in the digital twin and retraining, it can quickly adapt to different crop varieties, different greenhouse structures and different climate conditions, realizing personalized and precise planting of 'one crop, one strategy' and 'one greenhouse, one strategy'.

[0131] The above is only the preferred embodiment of the application, and it should be understood that the application is not limited to the form disclosed herein, and should not be considered as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the application shall be within the protection scope of the claims attached to the application.

Claims

1. A crop planting optimization decision method based on digital twinning and reinforcement learning, characterized in that, Comprise: S1, collect greenhouse multi-modal data, including greenhouse environment data, crop data, soil data and management data; S2, clean, align and normalize the collected data to form a structured multi-modal time series dataset; S3, identify the growth cycle of the crop based on the collected data; S4, construct a crop-environment digital twin model that maps in real time with the physical greenhouse environment; S5, construct a digital twin simulation space; The digital twin is formalized as a Markov decision process defined as a tuple , where denotes the state space, denotes the action space, denotes the state transition probability function, denotes the reward function, denotes the discount factor, the Markov decision process is the training environment for the reinforcement learning agent, the core is the state transition function , which describes the probability of transitioning to the next state after performing an action in the current state ; S6, optimize reinforcement learning strategy in the digital twin simulation space; Through a deep reinforcement learning agent and a digital twin, the state space S is defined: State At time is defined as: ； wherein, represents the current state, represents the current time a growth stage, represents the current time a dry matter accumulation, represents the current time a leaf area index, represents the current time a temperature, represents the current time a humidity, represents the current time an amount of light, represents the current time a carbon dioxide concentration, represents the current time a soil nitrogen content, represents a temperature prediction value for the next 24 hours, represents a light prediction value for the next 24 hours; The definition of action space A is: ； In the formula, represents an action performed under the current state, represents a temperature setting value, represents a humidity setting value, represents a light compensation duration, represents an irrigation amount, represents a fertilization concentration; The design of reward function R is: ； wherein, represents a reward, represents a single-step dry matter increment simulated by the digital twin, which is a direct proxy of yield, represents performing an action cost of consumed electricity, represents cost of consumed water and fertilizer, represents a next state penalty term generated when the environmental indicator exceeds the suitable range for the crop, respectively represent adjustable weight coefficients for the corresponding terms. Strategy learning: The goal of the agent is to learn a policy that maximizes the expected cumulative reward , denotes a shift in time steps, denotes the expected cumulative reward; Optimize the strategy network through PPO algorithm, the core is the iterative update of value function, follow the Bellman optimal equation: ； In the formula, Indicates the state Take action below The optimal value Indicates the next state Select the optimal action E represents the current state. Take action Under the given conditions, the expected total return that may be obtained in the future; Through simulation training of multiple growth cycles in the digital twin simulation space, the intelligent agent finally learns the optimal action-value function and the optimal strategy .

2. The crop planting optimization decision method based on digital twin and reinforcement learning according to claim 1, characterized in that, The optimization decision method also includes: S7, closed-loop control and feedback optimization; deploying the trained optimal policy run a deployment; Forward control: Every other decision cycle, collect real state , input policy network, get optimal action , input policy network, get optimal action , input policy network, get optimal action , input policy network, get optimal action Feedback optimization: At the same time, the real state As new training samples, for online updating or fine-tuning of the data-driven correction sub-model.

3. The crop planting optimization decision method based on digital twin and reinforcement learning according to claim 1, characterized in that, In step S1, the collection of greenhouse environment data, crop data, soil data and management data specifically includes: Greenhouse environment data collection: Through temperature and humidity sensors, light intensity sensors and carbon dioxide concentration sensors, real-time collection of greenhouse air temperature, humidity, light and carbon dioxide concentration data; Crop data collection: Through RGB cameras, 3D cameras, hyperspectral cameras and leaf temperature sensors, collect crop height, leaf area index, canopy structure, leaf color, fruit number and size, and leaf temperature data; Soil data collection: Through soil sensors to collect soil pH, conductivity, temperature and humidity, and nutrient content data; Management data collection: Through touch screen terminal or mobile App manual record planting time, irrigation amount, fertilizer concentration and pesticide data.

4. The crop planting optimization decision method based on digital twin and reinforcement learning according to claim 1, characterized in that, Step S3 specifically includes: Input the collected crop data and the planting start date of the crop into the pre-trained convolutional neural network, identify the phenological characteristics of the crop through the pre-trained convolutional neural network, and then judge the growth stage of the current crop based on the phenological model of effective accumulated temperature.

5. The crop planting optimization decision method based on digital twin and reinforcement learning according to claim 1, characterized in that, Step S4 specifically includes: The crop-environment digital twin model includes the following coupled submodels; Crop growth mechanism submodel: The core of the crop growth mechanism submodel is dry matter accumulation, and the daily dry matter increment is calculated by photosynthetic efficiency: ； wherein represents the daily increment of dry matter, represents the light use efficiency, represents the photosynthetically active radiation, represents the extinction coefficient, which describes the degree of shading of the canopy by the light, represents the leaf area index, are the stress correction functions for temperature, carbon dioxide concentration and nitrogen on photosynthesis, respectively; Data-driven correction submodel: Use real-time collected data to correct the set parameters of the crop growth mechanism submodel online through Kalman filtering or long short-term memory network.

Images