Dynamic control method for tea drying equipment based on deep reinforcement learning

Abstract

The invention discloses a tea drying equipment dynamic control method based on deep reinforcement learning, comprising the following steps: S1, collecting parameters of each control unit of tea drying equipment, and generating a state observation data set; s2, constructing an equipment graph structure, and outputting equipment graph structure information; s3, constructing an action value calculation network containing a dried graph neural network structure based on an improved MADDPG algorithm, and generating a global joint feature embedding set; s4, constructing a strategy network set, and generating a control action set; s5, inputting the global joint feature embedded set and the control action set into an action value prediction module, and outputting an action value prediction result; s6, the improved MADDPG algorithm is used for executing action value loss function calculation and parameter reverse updating; and S7, deploying the trained strategy network set to a tea drying equipment control system. According to the method, the MADDPG and drying graph neural network is improved, and dynamic closed-loop regulation and control of the tea leaf drying equipment are realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing and process control technology, and in particular to a dynamic control method for tea drying equipment based on deep reinforcement learning. Background Art

[0002] Currently, tea drying equipment, as a key piece of equipment in the tea processing process, plays a decisive role in drying quality, energy efficiency, and process consistency. Traditional tea drying equipment primarily relies on fixed parameter settings and empirical control methods, adjusting heating power, air supply speed, and dehumidification opening manually or through simple temperature and humidity sensor feedback to achieve the best tea drying results. This type of control method relies heavily on operator experience. When the process environment fluctuates significantly or the equipment structure is complex, it is often difficult to achieve precise control of drying quality. It is also difficult to balance energy consumption optimization with process stability, which can easily lead to uneven drying, large moisture content deviations, and low energy efficiency.

[0003] With the development of intelligent manufacturing and artificial intelligence technology, some research and applications have introduced tea drying control schemes based on traditional deep reinforcement learning algorithms. For example, existing technologies use deterministic policy gradient or deep deterministic policy gradient methods to achieve dynamic control of tea drying equipment, attempting to use neural networks to learn the mapping relationship between working condition changes and actions, and assist in the adaptive optimization of adjustment parameters. However, most of these methods only perform policy optimization within a single control unit or a local process parameter range, and lack the ability to fully model and characterize the physical coupling relationship between multiple control units of tea drying equipment, resulting in limited effectiveness in the process of multi-unit collaborative control. In addition, when dealing with complex physical field interactions, this type of reinforcement learning method often finds it difficult to effectively capture the coupling dynamics between multiple physical fields such as heat flow, humidity, and airflow. The policy generalization ability and stability are difficult to meet the high requirements of the tea drying process for drying quality consistency and energy efficiency.

[0004] In recent years, graph neural networks have been applied to certain extent in the fields of multi-agent collaborative control and physical field modeling, and are used to enhance the model's ability to learn the interactive relationships of multi-node systems. However, in the existing tea drying process control schemes, there is no public technology that combines graph neural networks with deep reinforcement learning methods to dynamically control the physical coupling relationship between heat and moisture in multiple control units of tea drying equipment. Existing multi-agent control schemes based on graph neural networks are mostly used in the fields of drone formations and robot clusters, and the relevant technologies are difficult to directly apply to the actual needs of the tea drying process. A complete solution has not yet been formed for the graph neural network and reinforcement learning fusion control method specially designed for the specific physical field characteristics, working condition change characteristics and control objectives of the tea drying process.

[0005] Therefore, how to provide a dynamic control method for tea drying equipment based on deep reinforcement learning is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] One purpose of the present invention is to propose a dynamic control method for tea drying equipment based on deep reinforcement learning. The present invention makes full use of the improved MADDPG algorithm and the specially designed drying graph neural network structure, and describes in detail the process of constructing the physical coupling relationship of multiple control units of tea drying equipment, dynamically generating physical action edge weights, aggregating working condition characteristics and outputting multi-unit collaborative control strategies. It has the advantages of high drying quality consistency, high energy efficiency utilization and high process control accuracy.

[0007] A dynamic control method for tea drying equipment based on deep reinforcement learning according to an embodiment of the present invention includes the following steps:

[0008] S1. Collect parameters of each control unit of the tea drying device to generate a state observation data set;

[0009] S2. Build a device graph structure based on the state observation data set and the device physical structure information, and output the device graph structure information;

[0010] S3. Based on the improved MADDPG algorithm, an action value calculation network including a drying graph neural network structure is constructed, which takes the state observation data set and the device graph structure information as input. The drying graph neural network structure includes a dynamic edge weight generation module, a physical field perception aggregation operator, and an edge weight dynamic update mechanism to generate a global joint feature embedding set;

[0011] S4. Constructing a set of policy networks based on the improved MADDPG algorithm, wherein each policy network in the set of policy networks takes the local observation data of the corresponding control unit as input and generates a set of control actions;

[0012] S5. Input the global joint feature embedding set and the control action set into the action value prediction module of the improved MADDPG algorithm, and output the action value prediction result;

[0013] S6. Based on the action value prediction results and the global reward function, the improved MADDPG algorithm is used to perform action value loss function calculation and parameter reverse update, and the parameters of the action value calculation network and the policy network set are updated;

[0014] S7. Deploy the trained policy network set to the tea drying equipment control system, receive the state observation data set in real time, and output the control action set.

[0015] Optionally, the state observation data set includes the state observation vector of each control unit at each sampling moment and the corresponding time label and unit number identifier; the state observation vector includes temperature observation values, humidity observation values, wind speed observation values, moisture content observation values ​​and energy consumption parameters; the time label identifies the sampling moment of the state observation vector, and the unit number identifier identifies the control unit number corresponding to the state observation vector.

[0016] Optionally, the S2 specifically includes:

[0017] S21, each control unit is used as a node of the device graph structure model, and the node number is consistent with the unit number identification in the state observation data set;

[0018] S22. Generate a physical action edge set based on the state observation vector and physical action relationship corresponding to each node, where each edge in the physical action edge set represents a physical coupling relationship between nodes based on heat flow transfer, humidity diffusion, or air flow;

[0019] S23. Generate a device graph structure adjacency matrix A. The device graph structure adjacency matrix is ​​an N×N matrix, where N is the total number of control units and each matrix element a ij Indicates whether there is a physical interaction edge between the i-th control unit and the j-th control unit. When there is a physical interaction edge between the i-th control unit and the j-th control unit, a ij The value is 1. When there is no physical interaction edge between the i-th control unit and the j-th control unit, a ij The value of is 0;

[0020] S24. The device graph structure adjacency matrix, the node set, and the physical action edge set are integrated into the device graph structure information.

[0021] Optionally, the S3 specifically includes:

[0022] S31. Construct an action value calculation network including a drying graph neural network structure based on an improved MADDPG algorithm, taking the state observation data set and the device graph structure information as input. The drying graph neural network structure includes a dynamic edge weight generation module, a physical field perception aggregation operator, and an edge weight dynamic update mechanism.

[0023] S32. In the drying graph neural network structure, a dynamic edge weight generation module is used to calculate physical action edge weights based on state observation vectors in the state observation data set to obtain a physical action edge weight set.

[0024] S33. Perform weighted aggregation on the neighbor control unit feature vectors according to the physical action edge weights using a physical field perception aggregation operator to generate an updated control unit feature vector, thereby forming a multi-unit collaborative control feature embedding for the tea drying equipment.

[0025] S34. Recalculate the physical action edge weight after each layer of aggregation through the edge weight dynamic update mechanism to reflect the real-time changing physical action state of the tea drying condition and dynamically adjust the information transmission weight

[0026] S35. Perform graph-level aggregation operations on the control unit feature vectors after aggregation of each layer to generate a global joint feature embedding set.

[0027] Optionally, the S32 specifically includes:

[0028] S321, extract the state observation vector o of each control unit in the state observation data set. i =[T i ,H i ,V i ,M i ,E i ], where T i is the temperature observation value, H i is the humidity observation value, V i is the wind speed observation value, M i is the observed value of moisture content, E i is the energy consumption parameter;

[0029] S322, according to the physical action edge set in the device graph structure information, determine the node pair set with physical action edges, the node pair is (i, j), and extract the state observation vector o of the corresponding node pair i and o j ;

[0030] S323. For each node pair (i, j) with a physical interaction edge, calculate the physical interaction edge weight e based on the state observation vector of the node pair. ij , the physical action edge weight e ij Calculate and generate based on the physical parameter difference or weighted combination of the temperature observation value, humidity observation value, wind speed observation value, moisture content observation value and energy consumption parameter in the state observation vector of the i-th control unit and the j-th control unit;

[0031] S324. Integrate the physical action edge weight results of all node pairs into a physical action edge weight set.

[0032] Optionally, the S33 specifically includes:

[0033] S331. Determine a neighbor node set N(i) of each node based on the physical interaction edge weight set, where the neighbor node set N(i) includes all node numbers that have physical interaction edges with the i-th control unit;

[0034] S332, for each node's l-th layer feature vector And the neighbor node feature vectors, perform weighted aggregation based on the corresponding physical action edge weights, and generate the feature vector of the i-th control unit at the l+1 layer:

[0035] S333, the weighted aggregation operation is as follows: the eigenvector of the node is obtained by linearly transforming its own eigenvector through the parameter matrix, adding the eigenvectors of neighboring nodes through the parameter matrix and weighted summing them according to the corresponding physical action edge weights, and then processing them through the activation function to obtain the updated eigenvector;

[0036] S334. Perform weighted aggregation operations on all nodes to generate updated control unit feature vectors, form multi-unit collaborative control feature embedding of tea drying equipment, and retain the aggregated node feature vectors.

[0037] Optionally, the S4 specifically includes:

[0038] S41. Constructing a set of policy networks based on the improved MADDPG algorithm, where each policy network in the set of policy networks corresponds to a control unit of the tea drying equipment;

[0039] S42, using the local observation data of the corresponding control unit in the state observation data set as the input of each strategy network;

[0040] S43, each strategy network is based on the corresponding local state observation vector o i Generate control action a of the i-th control unit i ;

[0041] S44, the control action set is represented by A={a1, a2,…, a i ,…,a N}, where A represents the control action set, a i Represents the control action of the i-th control unit, N is the total number of control units, and outputs the control action set.

[0042] Optionally, the value prediction module includes a global feature embedding input layer, an action fusion layer and a value output layer; the global feature embedding input layer receives a global joint feature embedding set; the action fusion layer calculates the fused feature representation of the global joint feature embedding set and the control action set through a fusion function; the value output layer receives the fused feature representation, performs forward propagation operation on the fused feature through a value calculation function, and outputs the action value prediction result.

[0043] Optionally, the S6 specifically includes:

[0044] S61. Generate a global reward function r based on the tea drying quality index, the equipment energy consumption index, and the temperature and humidity deviation index, wherein the global reward function includes a weighted combination of a tea drying quality reward item, an equipment energy consumption penalty item, and a temperature and humidity deviation penalty item;

[0045] S62. Combine the global reward function r with the action value prediction result to construct an action value loss function:

[0046]

[0047] Among them, L(φ) is the action value loss function, φ represents the parameters of the action value calculation network, Q(H G ,A) is the action value prediction result in the current global state, H G represents the global joint feature embedding set, A represents the control action set, r is the global reward function, γ is the discount factor, Q(H ′ G ,A ′ ) is the action value prediction result in the next state, H ′ G represents the global joint feature embedding set generated based on the next state, A ′ represents the set of control actions in the next state, is the expectation operator;

[0048] S63, performing back propagation calculation based on the action value loss function L(φ), and updating the parameter φ of the action value calculation network;

[0049] S64. Synchronously executing parameter updates of the policy network set based on the improved MADDPG algorithm, wherein each policy network in the policy network set completes parameter optimization by maximizing the expected action value prediction result.

[0050] Optionally, the S7 specifically includes:

[0051] S71. Deploy the trained policy network set to the tea drying equipment control system. Each policy network corresponds to a control unit of the equipment and establishes a communication connection with the control interface of the heating unit, air supply unit, and dehumidification unit of the equipment.

[0052] S72, the strategy network set receives in real time the state observation vector of each control unit in the state observation data set;

[0053] S73. Each strategy network calculates a control action for the corresponding control unit based on the state observation vector to generate a control action set, wherein the control action set includes a heating power control action for driving each heating unit of the tea drying equipment, an air supply speed control action for each air supply unit, and a dehumidification opening control action for each dehumidification unit;

[0054] S74: Output the control action set to drive the actuators of the heating unit, air supply unit, and dehumidification unit.

[0055] The beneficial effects of the present invention are:

[0056] The present invention proposes a dynamic control method for tea drying equipment based on deep reinforcement learning. By constructing an action value calculation network containing a drying graph neural network structure, it can effectively model the physical coupling relationship between the control units in the tea drying equipment based on heat flow transfer, humidity diffusion and airflow, and form a multi-unit collaborative control feature representation oriented to process physical characteristics. By utilizing the dynamic edge weight generation module, the physical field perception aggregation operator and the edge weight dynamic update mechanism, the dynamic aggregation of the working condition characteristics of different control unit states is realized, and the adaptability of the action value calculation network to the changes in complex working conditions of multiple physical fields is effectively improved. The policy network set constructed based on the improved MADDPG algorithm can generate a control action set based on the local observation data of each control unit, and input it into the action value prediction module together with the global joint feature embedding set to complete the accurate prediction of the action value of the control action set under the current working conditions.

[0057] The present invention realizes a comprehensive evaluation of the control effect of the tea drying process by designing a global reward function that combines tea drying quality indicators, equipment energy consumption indicators and temperature and humidity deviation indicators, which helps to improve the consistency of drying quality and reduce energy consumption levels. In the dynamic control stage, the present invention deploys a set of trained policy networks to realize dynamic closed-loop control of the heating unit, air supply unit and dehumidification unit of the tea drying equipment, and can adaptively generate heating power control actions, air supply speed control actions and dehumidification opening control actions under different working conditions, thereby improving energy utilization efficiency while ensuring the quality of tea drying. The technical solution of the present invention can realize the coordinated optimization of multiple control units in the tea drying process, improve the accuracy and stability of dynamic control, and meet the comprehensive requirements of the tea drying process for quality consistency, energy efficiency and adaptability to working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0059] Figure 1 This is a flow chart of a dynamic control method for tea drying equipment based on deep reinforcement learning proposed by the present invention;

[0060] Figure 2 This is a schematic diagram of the drying graph neural network structure in the dynamic control method of tea drying equipment based on deep reinforcement learning proposed in the present invention;

[0061] Figure 3 This is a structural block diagram of the improved MADDPG action value calculation network and strategy network set in the dynamic control method of tea drying equipment based on deep reinforcement learning proposed in this invention. DETAILED DESCRIPTION

[0062] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0063] refer to Figure 1-3 , a dynamic control method for tea drying equipment based on deep reinforcement learning, comprising the following steps:

[0064] S1. Collect parameters of each control unit of the tea drying device to generate a state observation data set;

[0065] S2. Build a device graph structure based on the state observation data set and the device physical structure information, and output the device graph structure information;

[0066] S3. Based on the improved MADDPG algorithm, an action value calculation network including a drying graph neural network structure is constructed, which takes the state observation data set and the device graph structure information as input. The drying graph neural network structure includes a dynamic edge weight generation module, a physical field perception aggregation operator, and an edge weight dynamic update mechanism to generate a global joint feature embedding set;

[0067] S4. Constructing a set of policy networks based on the improved MADDPG algorithm, wherein each policy network in the set of policy networks takes the local observation data of the corresponding control unit as input and generates a set of control actions;

[0068] S5. Input the global joint feature embedding set and the control action set into the action value prediction module of the improved MADDPG algorithm, and output the action value prediction result;

[0069] S6. Based on the action value prediction results and the global reward function, the improved MADDPG algorithm is used to perform action value loss function calculation and parameter reverse update, and the parameters of the action value calculation network and the policy network set are updated;

[0070] S7. Deploy the trained policy network set to the tea drying equipment control system, receive the state observation data set in real time, and output the control action set to achieve dynamic closed-loop control of the tea drying process.

[0071] The present invention proposes a dynamic control method for tea drying equipment based on deep reinforcement learning. By combining the improved MADDPG algorithm with the drying graph neural network structure, dynamic modeling of the physical coupling relationship between heat flow transfer, humidity diffusion and airflow action of multiple control units of tea drying equipment is achieved. Based on the state observation data set and equipment graph structure information, this method generates physical action edge weights and aggregates working condition characteristics to form a global joint feature embedding set, which is combined with the strategy network set to output a control action set for driving heating power, air supply speed and dehumidification opening to achieve dynamic closed-loop control of the tea drying process. This method has obvious advantages in improving the consistency of drying quality, reducing energy consumption and improving process control accuracy, and can meet the comprehensive requirements of the tea drying process for high efficiency, stability and quality control.

[0072] In this embodiment, it is characterized in that the state observation data set includes the state observation vector of each control unit at each sampling moment and the corresponding time label and unit number identifier; the state observation vector includes temperature observation value, humidity observation value, wind speed observation value, moisture content observation value and energy consumption parameter; the time label is the sampling moment of the identification state observation vector, and the unit number identifier is the control unit number corresponding to the identification state observation vector.

[0073] This method accurately characterizes the operating conditions of multiple units in a tea drying machine by collecting the state observation vectors of each control unit at each sampling moment, along with the corresponding time tags and unit number identifiers. The state observation vectors cover temperature, humidity, wind speed, moisture content, and energy consumption parameters, helping to dynamically capture changes in process conditions. This method improves the accuracy of operating condition perception, supports real-time and targeted control action generation, and helps optimize drying quality and energy efficiency.

[0074] In this embodiment, S2 specifically includes:

[0075] S21, each control unit is used as a node of the device graph structure model, and the node number is consistent with the unit number identification in the state observation data set;

[0076] S22. Generate a physical action edge set based on the state observation vector and physical action relationship corresponding to each node, where each edge in the physical action edge set represents a physical coupling relationship between nodes based on heat flow transfer, humidity diffusion, or air flow;

[0077] S23. Generate a device graph structure adjacency matrix A. The device graph structure adjacency matrix is ​​an N×N matrix, where N is the total number of control units and each matrix element a ij Indicates whether there is a physical interaction edge between the i-th control unit and the j-th control unit. When there is a physical interaction edge between the i-th control unit and the j-th control unit, a ijThe value is 1. When there is no physical interaction edge between the i-th control unit and the j-th control unit, a ij The value of is 0:

[0078]

[0079] Among them, a ij Indicates the existence of the physical interaction edge between node i and node j, where i and j are node numbers;

[0080] S24. Integrate the device graph structure adjacency matrix, the node set, and the physical action edge set into device graph structure information, where each node in the node set corresponds to a control unit, the physical action edge set represents the physical relationship between the nodes, and the device graph structure adjacency matrix is ​​used to identify the existence status of the physical action edges between the nodes.

[0081] The present invention ensures the consistency of the model and data by treating the control units as nodes of the equipment graph structure model, and the node numbers are consistent with the unit numbers in the state observation data set. On this basis, a set of physical action edges is generated based on the state observation vectors and physical action relationships corresponding to each node, where each edge represents a physical coupling relationship between nodes based on heat flow transfer, humidity diffusion or air flow. A device graph structure adjacency matrix is ​​further constructed. The adjacency matrix is ​​a square matrix determined by the total number of control units and is used to identify whether there is a physical action edge between any two nodes. If a physical action edge exists between two nodes in the adjacency matrix, the value of the corresponding matrix element is one, which is used to indicate that there is a physical coupling relationship between the two nodes; if there is no physical action edge between two nodes, the value of the corresponding matrix element is zero, which is used to indicate that there is no physical coupling relationship between the two nodes. This method can accurately express the physical action state between control units, improve the accuracy of working condition modeling and the adaptability of dynamic control, and help to achieve efficient modeling of equipment physical action relationships and fine optimization of control strategies.

[0082] In this embodiment, S3 specifically includes:

[0083] S31. Construct an action value calculation network based on an improved MADDPG algorithm, including a drying graph neural network structure, taking the state observation data set and the equipment graph structure information as input. The drying graph neural network structure includes a dynamic edge weight generation module, a physical field-aware aggregation operator, and an edge weight dynamic update mechanism. The drying graph neural network structure is used to model the physical coupling relationship between the control units of the tea drying equipment under the effects of heat flow transfer, humidity diffusion, and airflow;

[0084] S32. In the drying graph neural network structure, a dynamic edge weight generation module calculates physical action edge weights based on state observation vectors in the state observation data set to obtain a physical action edge weight set, wherein the physical action edge weights represent the intensity of heat and moisture coupling between control units;

[0085] S33. Perform weighted aggregation on the neighbor control unit feature vectors according to the physical action edge weights using a physical field perception aggregation operator to generate an updated control unit feature vector, thereby forming a multi-unit collaborative control feature embedding for the tea drying equipment.

[0086] S34. Recalculate the physical action edge weight after each layer of aggregation through the edge weight dynamic update mechanism to reflect the real-time changing physical action state of the tea drying condition and dynamically adjust the information transmission weight

[0087] S35. Perform a graph-level aggregation operation on the control unit feature vectors after aggregation of each layer to generate a global joint feature embedding set. The global joint feature embedding set is used for the subsequent action value prediction module to output the action value prediction result to characterize the global value of the control action set under the current tea drying condition.

[0088] The present invention realizes dynamic modeling of the physical coupling relationship between the control units of tea drying equipment under the action of heat flow transfer, humidity diffusion and airflow by constructing an action value calculation network including a drying graph neural network structure. The drying graph neural network structure includes a dynamic edge weight generation module, a physical field perception aggregation operator and an edge weight dynamic update mechanism, which are used to calculate the physical action edge weights, weighted aggregation control unit feature vectors and dynamically update the edge weights to reflect changes in working conditions. This method generates a global joint feature embedding set for action value prediction, which helps to improve the accuracy of drying condition modeling, enhance the adaptability and accuracy of control strategies, and support the realization of drying quality consistency and energy efficiency optimization goals.

[0089] In this embodiment, the S32 specifically includes:

[0090] S321, extract the state observation vector o of each control unit in the state observation data set. i =[T i ,H i ,V i ,M i ,E i ], where T i is the temperature observation value, H i is the humidity observation value, V i is the wind speed observation value, M i is the observed value of moisture content, E i is the energy consumption parameter;

[0091] S322, according to the physical action edge set in the device graph structure information, determine the node pair set with physical action edges, the node pair is (i, j), and extract the state observation vector o of the corresponding node pair i and o j ;

[0092] S323. For each node pair (i, j) with a physical interaction edge, calculate the physical interaction edge weight e based on the state observation vector of the node pair. ij , the physical action edge weight e ij Calculate and generate based on the physical parameter difference or weighted combination of the temperature observation value, humidity observation value, wind speed observation value, moisture content observation value and energy consumption parameter in the state observation vector of the i-th control unit and the j-th control unit;

[0093] S324. Integrate the physical action edge weight results of all node pairs into a physical action edge weight set, which is used for subsequent physical field perception aggregation operators to perform weighted aggregation operations.

[0094] The present invention extracts the state observation vector of each control unit in the state observation data set, and combines the temperature observation value, humidity observation value, wind speed observation value, moisture content observation value and energy consumption parameters to accurately characterize the working condition of the control unit. The physical action edge set is determined in the equipment graph structure information, and the node pair state observation vector is extracted to calculate the physical action edge weight. The physical action edge weight is calculated based on the weighted combination of the physical parameter differences in the state observation vectors between the control units, and is used to dynamically describe the strength of the physical coupling relationship between the control units. The physical action edge weight set is used as the physical action edge weight set for the weighted aggregation operation of the subsequent physical field perception aggregation operator. This method helps to improve the accuracy of working condition modeling and the pertinence of the control strategy.

[0095] In this embodiment, the S33 specifically includes:

[0096] S331. Determine a neighbor node set N(i) of each node based on the physical interaction edge weight set, where the neighbor node set N(i) includes all node numbers that have physical interaction edges with the i-th control unit;

[0097] S332, for each node's l-th layer feature vector And the neighbor node feature vectors, perform weighted aggregation based on the corresponding physical action edge weights, and generate the feature vector of the i-th control unit at the l+1 layer:

[0098] S333, the weighted aggregation operation is: the eigenvector of a node is the result of linear transformation of its own eigenvector by the parameter matrix, and the result of linear transformation of the eigenvector of its neighboring nodes by the parameter matrix and weighted summation according to the corresponding physical action edge weight, and then the updated eigenvector is obtained after activation function processing:

[0099]

[0100] in, represents the updated feature vector of the i-th control unit in the physical field perception aggregation operator of the l+1 layer, represents the eigenvector of the i-th control unit in the physical field perception aggregation operator of the l-th layer, W (l) represents the learnable parameter matrix for the eigenvector of the ith control unit itself in the lth layer aggregation operation, It represents the feature vector of the neighbor control unit j in the lth layer that has a physical interaction edge with the i-th control unit, represents the physical action edge weight of the lth layer, M (l) represents the learnable parameter matrix for the feature vector of the neighbor control unit in the l-th layer aggregation operation, N(i) represents the set of neighbor control units of the i-th control unit, σ represents the activation function, i represents the node number corresponding to a control unit in the device graph structure information, j represents the node number corresponding to the neighbor control unit with which the i-th control unit has a physical interaction edge in the device graph structure information, and l represents the layer number of the physical field-aware aggregation operator in the drying graph neural network structure;

[0101] S334. Perform weighted aggregation operations on all nodes to generate updated control unit feature vectors, form multi-unit collaborative control feature embedding of tea drying equipment, and retain the aggregated node feature vectors.

[0102] The present invention determines the set of neighbor nodes based on the set of physical action edge weights, and performs weighted aggregation operations on each node according to the neighbor node feature vectors to generate an updated feature vector. In the aggregation calculation, the updated feature vector of each node is obtained by adding the result of the linear transformation of its own feature vector through the parameter matrix and the weighted summation of the feature vectors of the neighbor nodes after the linear transformation of the parameter matrix according to the physical action edge weights, and then processing it through the activation function. The process is expressed in mathematical expressions as the updated feature vector is equal to the sum of the result of the linear transformation of its own features plus the weighted summation of the features of the neighbor nodes, and then passing through the activation function. This method improves the accuracy of the dynamic update of node features and the working condition perception ability, and helps to enhance the adaptability and accuracy of the control strategy of tea drying equipment.

[0103] In this embodiment, the S4 specifically includes:

[0104] S41. Constructing a set of policy networks based on the improved MADDPG algorithm, where each policy network in the set of policy networks corresponds to a control unit of the tea drying equipment;

[0105] S42, using the local observation data of the corresponding control unit in the state observation data set as the input of each strategy network;

[0106] S43, each strategy network is based on the corresponding local state observation vector o i Generate control action a of the i-th control unit i ;

[0107] S44, the control action set is represented by A={a1, a2,…, a i ,…,a N}, where A represents the control action set, a i Represents the control action of the i-th control unit, N is the total number of control units, and outputs the control action set.

[0108] The present invention constructs a set of policy networks based on the improved MADDPG algorithm, so that each control unit corresponds to an independent policy network for generating control actions. Each policy network takes the local state observation data of the corresponding control unit as input and generates control actions based on the state observation vector. The control action set is represented in the form of a set, which is composed of the control actions of each control unit and is used to drive the heating, air supply and dehumidification functions of the tea drying equipment. This method improves the pertinence and real-time nature of the control actions, helps to achieve dynamic closed-loop control and refined process management of the tea drying process, and enhances the equipment's adaptability to changes in complex working conditions.

[0109] In this embodiment, it is characterized in that the value prediction module includes a global feature embedding input layer, an action fusion layer and a value output layer; the global feature embedding input layer receives a global joint feature embedding set; the action fusion layer calculates the fusion feature representation of the global joint feature embedding set and the control action set through a fusion function; the value output layer receives the fusion feature representation, performs forward propagation operation on the fusion feature through a value calculation function, and outputs the action value prediction result.

[0110] This invention uses a value prediction module to evaluate the value of a set of control actions under current operating conditions. The module comprises a global feature embedding input layer, an action fusion layer, and a value output layer. The global feature embedding input layer receives the global joint feature embedding set. The action fusion layer calculates the fused feature representation of the control action using a fusion function. The value output layer performs forward propagation on the fused features and outputs the action value prediction result. This method improves the accuracy of action value calculation and helps optimize dynamic control strategies.

[0111] In this embodiment, S6 specifically includes:

[0112] S61. Generate a global reward function r based on the tea drying quality index, the equipment energy consumption index, and the temperature and humidity deviation index. The global reward function includes a weighted combination of a tea drying quality reward item, an equipment energy consumption penalty item, and a temperature and humidity deviation penalty item, and is used to measure the comprehensive control effect of the current control action set under the tea drying working condition.

[0113] S62. Combine the global reward function r with the action value prediction result to construct an action value loss function:

[0114]

[0115] Among them, L(φ) is the action value loss function, φ represents the parameters of the action value calculation network, Q(H G ,A) is the action value prediction result in the current global state, H G represents the global joint feature embedding set, A represents the control action set, r is the global reward function, γ is the discount factor, Q(H ′ G ,A ′ ) is the action value prediction result in the next state, H ′ G represents the global joint feature embedding set generated based on the next state, A ′ represents the set of control actions in the next state, is the expectation operator;

[0116] S63, performing back propagation calculation based on the action value loss function L(φ), and updating the parameter φ of the action value calculation network;

[0117] S64. Synchronously executing parameter updates of the policy network set based on the improved MADDPG algorithm, wherein each policy network in the policy network set completes parameter optimization by maximizing the expected action value prediction result.

[0118] The present invention achieves a comprehensive evaluation of the control effect by generating a global reward function based on tea drying quality indicators, equipment energy consumption indicators and temperature and humidity deviation indicators. The global reward function is formed by a weighted combination of various indicators and is used to balance the comprehensive control results of the current control action set under tea drying conditions. The action value loss function combines the action value prediction result with the global reward function to measure the error between the prediction and the expectation. The loss function is expressed as the square expectation of the difference between the action value prediction result and the action value prediction result after the global reward function plus the upper and lower state discounts. This method improves the value calculation accuracy and provides a basis for the optimization of dynamic control strategies.

[0119] In this embodiment, the S7 specifically includes:

[0120] S71. Deploy the trained policy network set to the tea drying equipment control system. Each policy network corresponds to a control unit of the equipment and establishes a communication connection with the control interface of the heating unit, air supply unit, and dehumidification unit of the equipment.

[0121] S72, the strategy network set receives in real time the state observation vector of each control unit in the state observation data set;

[0122] S73. Each strategy network calculates a control action for the corresponding control unit based on the state observation vector to generate a control action set, wherein the control action set includes a heating power control action for driving each heating unit of the tea drying equipment, an air supply speed control action for each air supply unit, and a dehumidification opening control action for each dehumidification unit;

[0123] S74. Output the control action set to drive the actuators of each heating unit, air supply unit and dehumidification unit to achieve dynamic closed-loop control of the tea drying process.

[0124] The present invention implements real-time dynamic control of each control unit by deploying a trained strategy network set to the tea drying equipment control system. The strategy network set receives the state observation vector of each control unit and calculates the corresponding control action based on the observation data. The control action set includes heating power control action, air supply speed control action and dehumidification opening control action, which are used to drive the actuators of the heating unit, air supply unit and dehumidification unit. This method supports dynamic closed-loop control of tea drying equipment under complex working conditions, improves the control response speed and control accuracy, and helps to achieve the comprehensive goals of drying quality consistency and energy consumption optimization.

[0125] Example 1:

[0126] In order to verify the feasibility of the present invention in practice, the present invention is applied to the tea drying production line of a tea production enterprise. The production line is equipped with a plurality of independently controlled heating units, air supply units and dehumidification units, which are used to complete the drying and processing tasks of large quantities of tea. In this scenario, the type of tea is a certain type of green tea widely grown locally, with an initial moisture content of about 70% and a target moisture content of less than 5%. Under such working conditions, traditional tea drying equipment mainly relies on the operator to set the parameters of each control unit based on experience. However, due to the complex working conditions of the equipment, the uneven stacking state of the tea leaves, and the changes in the ambient temperature and humidity, there are often problems such as uneven drying of some tea leaves, large fluctuations in the moisture content distribution, and high energy consumption. This is particularly evident in large batches and continuous production.

[0127] In this application scenario, the dynamic control method of tea drying equipment based on deep reinforcement learning proposed in the present invention is adopted. First, sensors are placed on each control unit of the equipment to collect temperature, humidity, wind speed, moisture content and energy consumption data, and a state observation data set is constructed in real time. Based on the state observation data set and the physical structure information of the equipment, an equipment graph structure is constructed, and the dynamic generation of physical action edge weights and the aggregation of working condition characteristics are realized through the drying graph neural network structure. During the production process, the policy network set receives the input of the state observation data set in real time, and dynamically outputs the heating power, air supply speed and dehumidification opening control actions. The system runs continuously for 12 batches under actual production conditions, with a drying capacity of 500 kilograms per batch, all using the same raw materials and target process indicators. During the drying process, the system generates a global reward function in real time based on indicators such as drying quality, energy consumption and temperature and humidity deviation. The action value calculation network and policy network parameters are continuously optimized and updated through the improved MADDPG algorithm to form an adaptive closed-loop control capability.

[0128] In the experimental data, the solution of the present invention was compared with the traditional manual experience control solution. The results showed that the solution of the present invention showed obvious advantages in terms of drying quality consistency, moisture content fluctuation range, energy consumption and production rhythm. In terms of drying quality consistency, the standard deviation of moisture content using the solution of the present invention was controlled within 0.8%, while the standard deviation of moisture content of the traditional solution was about 2.5%. In terms of energy consumption, the average energy consumption per batch of the solution of the present invention was about 310 kWh, while the average energy consumption per batch of the traditional solution was about 400 kWh. In terms of production rhythm, the time required for the solution of the present invention to complete a single batch of drying was about 130 minutes, while the average time for the traditional solution was 160 minutes. In terms of temperature and humidity deviation, the solution of the present invention made the absolute value of the deviation between the internal temperature of the equipment and the target temperature less than 1.2 degrees Celsius on average, and the absolute value of the deviation between the humidity and the target humidity less than 2.0% on average. The data table summarizes the comparison of key performance indicators under the above production conditions, verifying that the present invention has the beneficial effects of improving drying quality consistency, reducing energy consumption and improving production efficiency under actual working conditions.

[0129] Table 1 Comparison of key indicators of tea drying process

[0130]

[0131] As can be seen from the above table, the present invention has significant improvements in multiple key performance indicators of the tea drying process compared with the traditional manual experience control method. First, in terms of consistency of drying quality, the standard deviation of the moisture content of each batch of finished tea leaves in the actual production process of 12 batches of the present invention is controlled within 0.8%, while the standard deviation of the moisture content of the traditional solution under the same working conditions is 2.5% on average, and the fluctuation range is significantly larger. This shows that the dynamic control method based on deep reinforcement learning proposed in the present invention can more accurately control the heating power, air supply speed and dehumidification opening, making the tea drying quality more uniform and consistent, which helps to improve the market competitiveness of finished tea.

[0132] Secondly, in terms of energy consumption indicators, the average energy consumption of the solution of the present invention during each batch of drying is about 310 kWh, which is about 22.5% lower than the average energy consumption of 400 kWh of the traditional solution. This is due to the synergy between the improved MADDPG algorithm and the drying graph neural network structure in the present invention, which can perceive the changes in working conditions in real time and dynamically optimize the action output of each control unit to achieve efficient energy utilization. Further comparing the production efficiency, the average time for the solution of the present invention to complete a single batch of drying is 130 minutes, which is about 18.8% shorter than the 160 minutes of the traditional solution. While ensuring the drying quality, it significantly improves the production rhythm and meets the needs of tea processing companies for efficient production.

[0133] In addition, in terms of process deviation control, the absolute value of the deviation between the internal temperature of the equipment and the target temperature in the solution of the present invention is only 1.2 degrees Celsius on average, and the absolute value of the deviation between the humidity and the target humidity is 2.0% on average, which are significantly better than the traditional solution levels of 3.5 degrees Celsius and 5.0%, respectively. This shows that the present invention can achieve high-precision control of process parameters in a complex and changeable production environment, further ensuring the stability of the drying process and the consistency of the quality of the finished product. Comprehensive data show that the method proposed in the present invention has good industrial application value in terms of drying quality, energy efficiency and production efficiency, and provides a feasible solution for the intelligent upgrade of tea drying equipment.

[0134] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A dynamic control method for tea drying equipment based on deep reinforcement learning, characterized in that: The steps include: S1. Collect parameters of each control unit of the tea drying device to generate a state observation data set; S2. Build a device graph structure based on the state observation data set and the device physical structure information, and output the device graph structure information; S3. Based on the improved MADDPG algorithm, an action value calculation network including a drying graph neural network structure is constructed, which takes the state observation data set and the device graph structure information as input. The drying graph neural network structure includes a dynamic edge weight generation module, a physical field perception aggregation operator, and an edge weight dynamic update mechanism to generate a global joint feature embedding set; S4. Constructing a set of policy networks based on the improved MADDPG algorithm, wherein each policy network in the set of policy networks takes the local observation data of the corresponding control unit as input and generates a set of control actions; S5. Input the global joint feature embedding set and the control action set into the action value prediction module of the improved MADDPG algorithm, and output the action value prediction result; S6. Based on the action value prediction results and the global reward function, the improved MADDPG algorithm is used to perform action value loss function calculation and parameter reverse update, and the parameters of the action value calculation network and the policy network set are updated; S7. Deploy the trained policy network set to the tea drying equipment control system, receive the state observation data set in real time, and output the control action set.

2. A dynamic control method for tea drying equipment based on deep reinforcement learning according to claim 1, characterized in that: The state observation data set includes the state observation vector of each control unit at each sampling moment and the corresponding time tag and unit number identifier; the state observation vector includes temperature observation values, humidity observation values, wind speed observation values, moisture content observation values ​​and energy consumption parameters; the time tag is the sampling moment of the state observation vector, and the unit number identifier is the control unit number corresponding to the state observation vector.

3. The dynamic control method of tea drying equipment based on deep reinforcement learning according to claim 1 is characterized in that: The S2 specifically includes: S21, each control unit is used as a node of the device graph structure model, and the node number is consistent with the unit number identification in the state observation data set; S22. Generate a physical action edge set based on the state observation vector and physical action relationship corresponding to each node, where each edge in the physical action edge set represents a physical coupling relationship between nodes based on heat flow transfer, humidity diffusion, or air flow; S23. Generate a device graph structure adjacency matrix A. The device graph structure adjacency matrix is ​​an N×N matrix, where N is the total number of control units and each matrix element a ij Indicates whether there is a physical interaction edge between the i-th control unit and the j-th control unit. When there is a physical interaction edge between the i-th control unit and the j-th control unit, a ij The value is 1. When there is no physical interaction edge between the i-th control unit and the j-th control unit, a ij The value of is 0; S24. The device graph structure adjacency matrix, the node set, and the physical action edge set are integrated into the device graph structure information.

4. The dynamic control method of tea drying equipment based on deep reinforcement learning according to claim 1 is characterized in that: The S3 specifically includes: S31. Construct an action value calculation network including a drying graph neural network structure based on an improved MADDPG algorithm, taking the state observation data set and the device graph structure information as input. The drying graph neural network structure includes a dynamic edge weight generation module, a physical field perception aggregation operator, and an edge weight dynamic update mechanism. S32. In the drying graph neural network structure, a dynamic edge weight generation module is used to calculate physical action edge weights based on state observation vectors in the state observation data set to obtain a physical action edge weight set. S33. Perform weighted aggregation on the neighbor control unit feature vectors according to the physical action edge weights using a physical field perception aggregation operator to generate an updated control unit feature vector, thereby forming a multi-unit collaborative control feature embedding for the tea drying equipment. S34. Recalculate the physical action edge weight after each layer of aggregation through the edge weight dynamic update mechanism to reflect the real-time changing physical action state of the tea drying condition and dynamically adjust the information transmission weight S35. Perform graph-level aggregation operations on the control unit feature vectors after aggregation of each layer to generate a global joint feature embedding set.

5. A dynamic control method for tea drying equipment based on deep reinforcement learning according to claim 4, characterized in that: The S32 specifically includes: S321, extract the state observation vector o of each control unit in the state observation data set. i =[T i ,H i ,V i ,M i ,E i ], where T i is the temperature observation value, H i is the humidity observation value, V i is the wind speed observation value, M i is the observed value of moisture content, E i is the energy consumption parameter; S322, according to the physical action edge set in the device graph structure information, determine the node pair set with physical action edges, the node pair is (i, j), and extract the state observation vector o of the corresponding node pair i and o j ; S323. For each node pair (i, j) with a physical interaction edge, calculate the physical interaction edge weight e based on the state observation vector of the node pair. ij , the physical action edge weight e ij Calculate and generate based on the physical parameter difference or weighted combination of the temperature observation value, humidity observation value, wind speed observation value, moisture content observation value and energy consumption parameter in the state observation vector of the i-th control unit and the j-th control unit; S324. Integrate the physical action edge weight results of all node pairs into a physical action edge weight set.

6. A tea drying equipment dynamic control method based on deep reinforcement learning according to claim 4, characterized in that: The S33 specifically includes: S331. Determine a neighbor node set N(i) of each node based on the physical interaction edge weight set, where the neighbor node set N(i) includes all node numbers that have physical interaction edges with the i-th control unit; S332, for each node's l-th layer feature vector And the neighbor node feature vectors, perform weighted aggregation based on the corresponding physical action edge weights, and generate the feature vector of the i-th control unit at the l+1 layer: S333, the weighted aggregation operation is as follows: the eigenvector of the node is obtained by linearly transforming its own eigenvector through the parameter matrix, adding the eigenvectors of neighboring nodes through the parameter matrix and weighted summing them according to the corresponding physical action edge weights, and then processing them through the activation function to obtain the updated eigenvector; S334. Perform weighted aggregation operations on all nodes to generate updated control unit feature vectors, form multi-unit collaborative control feature embedding of tea drying equipment, and retain the aggregated node feature vectors.

7. The dynamic control method of tea drying equipment based on deep reinforcement learning according to claim 1 is characterized in that: The S4 specifically includes: S41. Constructing a set of policy networks based on the improved MADDPG algorithm, where each policy network in the set of policy networks corresponds to a control unit of the tea drying equipment; S42, using the local observation data of the corresponding control unit in the state observation data set as the input of each strategy network; S43, each strategy network is based on the corresponding local state observation vector o i Generate control action a of the i-th control unit i ; S44, the control action set is represented by A={a1, a2,…, a i ,…,a N }, where A represents the control action set, a i Represents the control action of the i-th control unit, N is the total number of control units, and outputs the control action set.

8. The method for dynamic control of tea drying equipment based on deep reinforcement learning according to claim 1, characterized in that: The value prediction module includes a global feature embedding input layer, an action fusion layer and a value output layer; the global feature embedding input layer receives a global joint feature embedding set; the action fusion layer calculates the fused feature representation of the global joint feature embedding set and the control action set through a fusion function; the value output layer receives the fused feature representation, performs forward propagation operation on the fused feature through a value calculation function, and outputs the action value prediction result.

9. The dynamic control method of tea drying equipment based on deep reinforcement learning according to claim 1, characterized in that: The S6 specifically includes: S61. Generate a global reward function r based on the tea drying quality index, the equipment energy consumption index, and the temperature and humidity deviation index, wherein the global reward function includes a weighted combination of a tea drying quality reward item, an equipment energy consumption penalty item, and a temperature and humidity deviation penalty item; S62. Combine the global reward function r with the action value prediction result to construct an action value loss function: Among them, L(φ) is the action value loss function, φ represents the parameters of the action value calculation network, Q(H G ,A) is the action value prediction result in the current global state, H G represents the global joint feature embedding set, A represents the control action set, r is the global reward function, γ is the discount factor, Q(H ′ G ,A ′ ) is the action value prediction result in the next state, H ′ G represents the global joint feature embedding set generated based on the next state, A ′ represents the set of control actions in the next state, is the expectation operator; S63, performing back propagation calculation based on the action value loss function L(φ), and updating the parameter φ of the action value calculation network; S64. Synchronously executing parameter updates of the policy network set based on the improved MADDPG algorithm, wherein each policy network in the policy network set completes parameter optimization by maximizing the expected action value prediction result.

10. The dynamic control method of tea drying equipment based on deep reinforcement learning according to claim 1, characterized in that: The S7 specifically includes: S71. Deploy the trained policy network set to the tea drying equipment control system. Each policy network corresponds to a control unit of the equipment and establishes a communication connection with the control interface of the heating unit, air supply unit, and dehumidification unit of the equipment. S72, the strategy network set receives in real time the state observation vector of each control unit in the state observation data set; S73. Each strategy network calculates a control action for the corresponding control unit based on the state observation vector to generate a control action set, wherein the control action set includes a heating power control action for driving each heating unit of the tea drying equipment, an air supply speed control action for each air supply unit, and a dehumidification opening control action for each dehumidification unit; S74: Output the control action set to drive the actuators of the heating unit, air supply unit, and dehumidification unit.

Images