A method and system for temperature control in a feed pelleting production process

Abstract

The present application belongs to the technical field of intelligent control of industrial processes, and particularly relates to a temperature control method and system in a feed granulation production process. The method comprises: obtaining state data, calculating an instant reward value based on a preset production target; a strategy network outputs an action and superimposes confidence-driven adaptive exploration noise to generate an execution action, and a state transition tuple is stored in an experience replay pool; a composite priority weighted with an absolute time sequence difference error and a state sensitivity is used for sampling; a target Q value update network is constructed using a smaller value of a double target Q value network, and a strategy network is updated with a delay, and a motion gradient regularization term positively correlated with the state sensitivity is introduced in a loss function; and finally a deterministic action control execution mechanism is output. The present application effectively overcomes the problems of slow convergence of traditional exploration, insufficient learning of key states and control instruction jitter, and guarantees production stability and product quality consistency.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for industrial processes. More specifically, this invention relates to a temperature control method and system for the feed pelleting production process. Background Technology

[0002] In the continuous processes of raw material conveying, crushing, mixing, pelleting, and cooling in feed production lines, the control of pelleting process parameters directly affects product uniformity and quality. The various units within the system exhibit dynamic coupling and nonlinear characteristics. Traditional mechanistic modeling control struggles to accurately reflect transient behavior and cannot adapt to real-time disturbances in raw materials and operating conditions. Therefore, intelligent control methods with autonomous learning and adaptive capabilities have become an urgent industry need.

[0003] Deep reinforcement learning, through continuous interaction with the environment and trial-and-error learning, can autonomously seek optimization without relying on precise mathematical models, making it well-suited for industrial continuous process control. Existing technical solutions for applying this method to feed production line control typically follow a general framework: an agent composed of a policy network and a value function network, resampling historical data using an experience replay buffer, guiding policy updates with temporal difference errors, and exploring policies through random perturbations in the action space.

[0004] However, when faced with the high-dimensional continuous state, strong time-varying response, and safety boundary constraints of feed production lines, such general solutions have the following shortcomings: First, the exploration mechanism uses indiscriminate random perturbation, which cannot adaptively adjust the exploration intensity according to the agent's understanding of the current state region, resulting in a large number of invalid trials and errors in the early stages of learning, and a slow algorithm convergence speed. Second, experience replay only evaluates the sample value based on temporal difference error, without distinguishing the response sensitivity differences in different regions of the state space, resulting in insufficient learning near key process points that affect product uniformity, such as conditioning temperature and steam addition amount, making it difficult to accurately grasp the operational boundaries. Third, policy updates lack direct constraints on the smoothness of the policy function. When the gradient of the value function is large, the control commands output by the policy network are prone to drastic fluctuations in adjacent time steps. This control jitter is applied to actuators such as pellet mill drives and regulating valves, which will aggravate equipment wear and affect production safety.

[0005] In summary, existing deep reinforcement learning-based feed production line control methods still have room for improvement in terms of learning efficiency, sufficiency of learning in critical state areas, and stability of control command output. Summary of the Invention

[0006] To address the technical problems of existing deep reinforcement learning-based feed production line control methods, which suffer from slow learning convergence, insufficient learning of key process boundaries, and severe control command jitter due to the use of indiscriminate random exploration mechanisms, failure to distinguish state sensitivity differences during experience playback, and lack of policy output smoothness constraints, this invention provides the following technical solution: In a first aspect, the present invention provides a temperature control method for a feed pelleting production process, comprising: acquiring process state data of the feed pelleting production process, the process state data including at least parameters reflecting the temperature of the pellet mill and control parameters affecting the temperature regulation of the pellet mill; calculating an immediate reward value based on the state data and a preset production target; generating an execution action by outputting an action using a policy network and superimposing exploration noise; storing a state transition tuple containing the state data, the execution action, the reward, the next state, and a task end flag into an experience replay pool; and, based on a composite priority index weighted by absolute temporal difference error and state sensitivity value, calculating an immediate reward value from an experience replay pool. State transition tuples are sampled in the playback pool, wherein the state sensitivity value is obtained by calculating the gradient norm of the first Q-value function with respect to the state data; the network is updated using the state transition tuples: the next action is calculated based on the target policy network and the next state, the smaller value of the outputs of the two target Q-value networks is taken to construct the target Q-value, and then the two Q-value networks are updated; the policy network is updated with a delay by optimizing a loss function that includes a Q-value term and a weighted action gradient regularization term, wherein the weight of the regularization term is positively correlated with the state sensitivity value; the deterministic action output by the policy network in the current state is sent as a control command to the actuator.

[0007] This invention achieves closed-loop intelligent control of feed pelleting temperature based on deep reinforcement learning. It collects parameters reflecting pellet mill temperature and control parameters affecting temperature regulation, using immediate reward values ​​to guide the policy network to autonomously learn the optimal control logic. An adaptive exploration-based noise-balanced trial-and-error approach is employed, combining absolute temporal difference error and state sensitivity as composite priority indicators to select high-value training samples. A dual Q-value network and delayed update mechanism stabilize the training process. Furthermore, in regions where Q-values ​​are sensitive to state changes, action gradient regularization constraints with dynamically adjusted weights based on state sensitivity suppress drastic fluctuations in control commands. Finally, precise control commands for raw material flow and steam addition are output, effectively improving learning efficiency and control stability, ensuring product quality and stable production operation.

[0008] Preferably, the step of generating an execution action by using the policy network to output an action and superimposing exploration noise includes: using the policy network to output a deterministic action for the current state; calculating the exponential moving average of the absolute values ​​of the historical time-series difference errors of the two Q-value networks; adding a preset minimum positive number to each of the two exponential moving averages, taking the reciprocal to calculate the confidence level, and selecting the smallest one as the final confidence level; adding 1 to the final confidence level, taking the reciprocal to obtain a noise scaling factor, and sampling a noise from a standard normal distribution; multiplying the sampled noise by the noise scaling factor and superimposing it onto the deterministic action to generate an execution action and executing it.

[0009] By utilizing the exponentially moving average of the historical time-series difference errors of two Q-value networks, the confidence level of the current policy is calculated by taking the reciprocal and the smaller value, and the scaling of action noise is dynamically adjusted accordingly. When the error is large, the low confidence corresponds to enhanced noise perturbation, prompting the agent to expand its exploration range to discover better policies; as the error decreases and the confidence increases, the noise amplitude automatically decreases, making the action output more deterministic. This adaptive exploration mechanism effectively coordinates the balance between trial and error and utilization, improves the effectiveness of training samples, accelerates policy network convergence, and enhances the stability of control action output.

[0010] Preferably, the sampling of state transition tuples from the experience replay pool based on the composite priority index weighted by absolute temporal difference error and state sensitivity value includes: for each state transition tuple in the experience replay pool, calculating the absolute temporal difference error; calculating the gradient of the first Q-value function with respect to the state vector and obtaining the L2 norm as the state sensitivity value; multiplying the absolute temporal difference error and the state sensitivity value by preset first and second weighting coefficients respectively and then summing them to obtain the composite priority index; calculating the probability of each tuple being sampled based on the composite priority index; and extracting a batch of tuples from the experience replay pool without repetition for network updates based on the probability.

[0011] By combining the absolute temporal difference error with the Q-value's sensitivity gradient to state changes into a composite priority, the network prioritizes both samples with large prediction biases and key regions that strongly respond to state perturbations during empirical replay. This guides the network to learn high-value and high-sensitivity transition tuples first. This sampling strategy effectively improves the utilization efficiency of training data, accelerates the accurate fitting of the strategy to important state spaces, and enhances the output stability of control commands under critical operating conditions.

[0012] Preferably, the step of calculating the next action based on the target policy network and the next state, constructing the target Q-value by taking the smaller value of the outputs of the two target Q-value networks, and then updating the two Q-value networks includes: calculating the next action through the target policy network based on the next state in the sampled tuple; adding a pruned noise to the next action to achieve target policy smoothing; calculating the Q-value of the next state using the two target Q-value networks; selecting the smaller of the two target Q-values ​​and calculating the target Q-value together with the immediate reward value and the discount factor; and simultaneously updating the parameters of the two Q-value networks using the mean squared error loss function and the gradient descent method.

[0013] By utilizing a target policy network to calculate the expected action based on the next state and adding pruned noise to smooth the Q-value estimation, the smaller value from the output of a dual-target Q-value network is then used to construct the target Q-value, effectively suppressing the overestimation bias of a single network. Simultaneous updates to the two Q-value networks through mean squared error loss make the value assessment more accurate and reliable. This mechanism improves training stability during reinforcement learning, reduces policy fluctuations caused by estimation bias, and thus enhances the accuracy and safety of the final control commands.

[0014] Preferably, the delayed update policy network is implemented by optimizing a loss function that includes a Q-value term and a weighted action gradient regularization term, including: obtaining the state from the sampled tuples and calculating the corresponding action using the current policy network; calculating the gradient of the first Q-value function with respect to the state and obtaining the L2 norm as the state sensitivity value; calculating the weight coefficient of the regularization term based on the state sensitivity value and a preset reference gradient value; calculating the square of the gradient norm of the policy network output action with respect to the state as the regularization term; constructing a loss function that includes a Q-value term and a weighted regularization term, and updating the policy network parameters using gradient descent.

[0015] Preferably, the process status data includes raw material flow rate, crusher current, granulator temperature and steam valve opening; the four sets of data collected are used to form a four-dimensional state vector and the data in each dimension are subjected to max-min normalization.

[0016] Preferably, the deterministic action output by the strategy network in the current state is sent to the actuator as a control command, including: the strategy network outputs a two-dimensional deterministic action vector to the current normalized state vector; the first dimension of the action vector is denormalized to a specific raw material flow rate setpoint and sent to the frequency converter controlling the feeding auger to adjust the raw material flow rate; the second dimension of the action vector is denormalized to a steam valve opening setpoint and sent to the regulating valve controller to adjust the steam addition amount of the pellet mill.

[0017] Preferably, the exploration noise is Gaussian noise, and the noise amplitude is adaptively adjusted according to the confidence level of the Q-value network time-series difference error.

[0018] Preferably, calculating the instant reward value based on the state data and the preset production target includes: subtracting the normalized state vector from the preset production target vector to obtain the deviation of each state variable; weighting and summing the squares of each deviation, and taking the negative value of the sum as the instant reward value.

[0019] By calculating the deviation between the normalized state vector and the preset production target vector, and then taking the negative value of the weighted sum of the squares of each deviation as the immediate reward, the closer the actual state is to the target value, the smaller the deviation, and the higher the reward value; the larger the deviation, the lower the reward value. This provides a clear optimization direction for the policy network, guiding it to learn autonomously and enabling the granulation temperature and related variables to quickly approach and stabilize within the target range, thereby improving the adjustment accuracy and operational stability of the production process.

[0020] Secondly, the present invention provides a temperature control system for the feed pelleting production process, including a control module, a data acquisition module, and an execution module. The control module includes a processor and a memory, and the memory stores computer program instructions. When the computer program instructions are executed by the processor, the above-mentioned temperature control method for the feed pelleting production process is implemented.

[0021] By adopting the above technical solution, a computer program is generated from the temperature control method in the feed pelleting production process, and stored in a memory for loading and execution by a processor. This allows for the creation of a terminal device based on the memory and processor, making it convenient to use.

[0022] The technical solution of the present invention has the following beneficial technical effects: This invention employs a confidence-driven adaptive exploration noise, a composite priority sampling weighted by absolute temporal differential error and state sensitivity, and state-aware weighted action gradient regularization, combined with a dual Q-value network and a delayed update mechanism, to achieve closed-loop intelligent temperature control in the feed pelleting production process. This scheme dynamically adjusts the exploration intensity based on the agent's environmental awareness, focusing on key process state samples for learning, while automatically strengthening control smoothness constraints in regions where Q-values ​​are sensitive to state changes. It effectively solves the problems of slow learning convergence caused by indiscriminate random exploration, insufficient learning of key process boundaries due to lack of state sensitivity differentiation during experience playback, and severe control command jitter caused by the lack of smoothness constraints in strategy updates in existing technologies, ensuring stable production operation and consistent product quality. Attached Figure Description

[0023] Figure 1 A flowchart of a temperature control method in the feed pelleting production process; Figure 2 This is a schematic diagram comparing the superposition of two-dimensional motion noise. Figure 3 This is a schematic diagram illustrating how the regularization weights change with state sensitivity. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0025] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] This invention discloses a temperature control method in the feed pelleting production process, referring to... Figure 1 This includes the following steps: S1. Collect status data, calculate rewards, generate actions, and store them in the experience pool.

[0027] Acquire process status data of the feed pelleting production process. The process status data includes at least parameters reflecting the pellet mill temperature and control parameters affecting the pellet mill temperature regulation. Calculate the instant reward value based on the status data and preset production targets. Use the policy network to output actions and superimpose exploration noise to generate execution actions. Store the state transition tuple containing status data, execution actions, rewards, next state, and task end flag into the experience replay pool.

[0028] Raw state data is obtained by acquiring sensor data at a fixed sampling frequency of 0.1Hz using a programmable logic controller (PLC) in the industrial field. The parameter ranges for various data types are as follows: Raw material flow rate data: measured from 1 to 5 t / h, used to characterize the feed rate of the production line; Crusher current data: measured from 80 to 150 A, used to indirectly reflect the crushing load of the raw material; Pelletizer temperature data: measured from 75 to 95 ℃, which is the core target parameter for this control; Steam valve opening data: measured from 0 to 100%, used to adjust the amount of steam added to the pelletizer.

[0029] The four sets of raw data collected are used to construct the instantaneous state. The min-max normalization method is then used to eliminate the influence of dimensions, resulting in a four-dimensional state vector in which each component is in the interval between 0 and 1. The normalization relation is: ; in, For the normalized first 3D state components; For the first The original measurement value of the dimensional state; For the first The preset lower limit value of the dimension state corresponds to the lower limit of safe operation of the device; For the first The preset upper limit value of the dimension state corresponds to the upper limit of safe operation of the device.

[0030] A reward function is defined, which calculates the deviation of each state variable by subtracting the normalized state vector from the preset production target vector. The squares of these deviations are then weighted and summed, and the negative of the sum is taken as the immediate reward value. A larger reward value indicates that the current state is closer to the production target. For example, the weight coefficients of each state variable in the reward function are [0.1, 0.1, 0.7, 0.1], with granulation temperature having the highest weight, highlighting the core control target. The preset production target vector is [0.6, 0.5, 0.8, 0.5], corresponding to a raw material flow rate of 3.4 t / h, a crusher current of 115 A, a granulation temperature of 91℃, and a steam valve opening of 50% before normalization.

[0031] The normalized state vector is input into the policy network, which is a multilayer perceptron containing one input layer, two hidden layers, and one output layer. The hidden layers use the ReLU activation function, and the output layer uses the Tanh activation function. Forward propagation is implemented through the neural network construction and computation core module of the PyTorch framework, outputting a two-dimensional deterministic action vector. Each component's value range is limited to between -1 and 1. Specifically, the policy network has 4 neurons in the input layer, corresponding to a four-dimensional state vector; 64 neurons in each of the two hidden layers; and 2 neurons in the output layer, corresponding to a two-dimensional action vector. The Q-value network has 8 neurons in the input layer, which is a concatenation of the four-dimensional state vector and the two-dimensional action vector; 64 neurons in each of the two hidden layers; and 1 neuron in the output layer, corresponding to a single Q-value.

[0032] The process of generating execution actions by superimposing exploratory noise on deterministic actions is as follows: Maintain the exponential moving average of the absolute values ​​of the historical temporal difference errors of the two Q-value networks. and The update relation is: ; in, It is the first The exponential moving average of the absolute values ​​of the historical time-series difference errors of a Q-value network; This is a smoothing coefficient used to smooth out fluctuations in historical time-series difference errors and avoid the impact of single error mutations on confidence calculation. In this embodiment, the preferred value is 0.95. The number calculated during the previous update The absolute value of the temporal difference error of a Q-value network. In the initial training phase, two exponential moving averages... and It is initialized to a preset positive number, such as 1.

[0033] The confidence scores of the two Q-value networks were then calculated. and The minimum value was selected as the final confidence level. : ; ; in, This is a preset, extremely small positive number used to prevent calculation errors when the denominator is zero. In this embodiment, the value is set to [value missing]. Confidence score is used to characterize the reliability of a Q-value network's prediction of the value of the current state. The higher the confidence score, the more fully the agent understands the state.

[0034] Calculate the noise scaling factor And sample noise vectors from the standard normal distribution. Generate execution actions : ; ; in, This is a noise scaling factor with a value range of 0 to 1, used to dynamically adjust the amplitude of the exploration noise based on the confidence level. This is a noise vector with the same action dimension sampled from a standard normal distribution N(0,1); It is a two-dimensional deterministic action vector output by the policy network at time t, and is the core control command source of the entire temperature control system.

[0035] like Figure 2 As shown in the figure, this diagram visually illustrates the before-and-after comparison of the two-dimensional deterministic actions output by the policy network, after superimposing adaptively scaled Gaussian noise to generate the final executed action. Dimension 1 corresponds to the raw material flow control action, and dimension 2 corresponds to the steam valve opening control action. The diagram clearly demonstrates the disturbance characteristics of exploration noise on the original control action, and the amplitude of this disturbance is dynamically adjusted with the Q-value network confidence, achieving a balance between exploration and utilization.

[0036] The actions are clipped to the action boundary between -1 and 1. Then, the normalized current state, the clipped actions, the immediate reward value, the normalized next state, and the task end flag are encapsulated into a state transition tuple and stored in a double-ended queue implemented using the double-ended queue class of the Python collection module as an experience replay pool.

[0037] S2. Sample training samples from the experience pool according to composite priority.

[0038] Based on a composite priority index weighted by absolute temporal difference error and state sensitivity value, state transition tuples are sampled from the empirical replay pool, where the state sensitivity value is obtained by calculating the gradient norm of the first Q-value function with respect to the state data.

[0039] After storing the state transition tuples, high-value samples need to be selected from the experience replay pool based on a composite priority index to improve data utilization efficiency and network learning performance. Individual tuples, composite priority indicators The calculation formula is as follows: ; in, The weighting coefficient for the absolute temporal difference error is used to adjust the degree of influence of the prediction error on the sample priority. In this embodiment, the value is 1. For the first The absolute temporal difference error corresponding to each tuple is calculated and stored when the Q-value network was last updated using this tuple; This is the weighting coefficient for the state sensitivity value, used to adjust the degree of influence of state importance on sample priority. In this embodiment, the value is set to 0.1. For the first The state sensitivity values ​​corresponding to each tuple are used to characterize the sensitivity of the Q-value function to changes in the current state, and are calculated as follows: That is, the L2 norm of the gradient of the first Q-value function with respect to the state vector. and They respectively refer to the first from the experience replay pool The state vector and action vector extracted from each state transition tuple; It is the gradient operator in physics, representing the calculation of partial derivatives; A preset, extremely small positive number is used to ensure that the new tuple has a sampling probability greater than zero; in this embodiment, the value is set to [value missing]. In this embodiment, to improve sampling efficiency, the state sensitivity value can be recalculated for each tuple in the experience replay pool using the latest first Q-value network every preset number of update steps, for example, every 100 steps. and composite priority indicators .

[0040] The sampling probability of each tuple can be further calculated based on the composite priority index, and the relationship is as follows: ; in, For the first The probability that each state transition tuple is sampled; For the first Composite priority index for each tuple; This is a priority index used to adjust the degree of influence of priority differences on sampling probability; in this embodiment, it is set to 0.6. The index representing all state transition tuples in the experience replay pool; For the first The composite priority index of each state transition tuple is obtained by weighted summation of absolute time difference error and state sensitivity value; For all tuples in the experience replay pool sum.

[0041] This embodiment uses the SumTree data structure to store priorities and perform weighted sampling. Each time, 256 tuples are extracted from the experience replay pool to form a batch for subsequent network parameter updates.

[0042] S3, Update the dual-Q network with delayed updates to the regularized policy network.

[0043] The network is updated using state transition tuples: the next action is calculated based on the target policy network and the next state, the smaller value of the output of the two target Q-value networks is taken to construct the target Q-value, and then the two Q-value networks are updated; the policy network is updated by delaying the update, which is achieved by optimizing the loss function containing Q-value terms and weighted action gradient regularization terms, where the weight of the regularization term is positively correlated with the state sensitivity value.

[0044] After obtaining the sampling batch, the Q-value network and the policy network are updated sequentially. The two Q-value networks are updated iteratively first, and then the policy network is updated with a delay, so as to ensure training stability.

[0045] When updating the Q-value network, the next action is calculated through the target policy network based on the next state in the sampled tuples. A clipped noise is added to the next action to achieve target policy smoothing. The noise is sampled from a normal distribution with a mean of 0 and a standard deviation of 0.2, and clipped to the range of -0.5 to 0.5.

[0046] The Q-value of the next state is calculated using two objective Q-value networks. The smaller of the two objective Q-values ​​is selected and used together with the immediate reward value and the discount factor to calculate the objective Q-value, as shown in the following formula: ; in, For the first The target Q value corresponding to each tuple; For the first The instant reward value corresponding to each tuple; This is a discount factor used to weigh the importance of immediate rewards against future long-term rewards; in this example, it is set to 0.99. , These are the next-state Q-values ​​output by the networks for the two target Q-values, respectively.

[0047] The mean squared error loss function is used to calculate the error between the predicted value and the target Q value of the two online Q-value networks respectively. The two errors are summed to obtain the total Q-value loss. The learning rate is then used to calculate the loss. The Adam optimizer completes gradient descent updates, simultaneously optimizing the parameters of the two Q-valued networks.

[0048] The policy network adopts a delayed update method. After every two Q-value network updates are completed, the policy network and target network are updated once.

[0049] First, the state is obtained from the sampled tuples, and the corresponding action is calculated using the current policy network. Then, the gradient of the first Q-value function with respect to the state is calculated, and the L2 norm is obtained as the state sensitivity value. The state sensitivity value... The calculation method is the same as in step S2. Its value can be the latest result calculated and stored in step S2 to save computational overhead, or it can be recalculated using the current network parameters for higher accuracy.

[0050] The weight coefficients of the regularization term are calculated based on the state sensitivity value and the preset reference gradient value, and the relationship is as follows: ; in, For the first The regularization term weight coefficients corresponding to each tuple; The base weight coefficient is used to set the basic constraint strength of the regularization term; in this embodiment, the value is 0.01. The reference gradient value is set as the exponential moving average of the state sensitivity value over historical batches. It is used to normalize the state sensitivity value to a suitable range. For example, the smoothing coefficient of the exponential moving average of the reference gradient value is 0.9, and the initial value is set to 1. For the first The state sensitivity value corresponding to each tuple; It is the hyperbolic tangent function.

[0051] like Figure 3 As shown in the figure, the curve exhibits an S-shaped characteristic of first rising rapidly and then leveling off, consistent with the hyperbolic tangent function mapping relationship used in this invention. This figure visually demonstrates that when state sensitivity is low, the regularization constraint is weak, ensuring the policy's exploratory capability; as state sensitivity increases, the regularization weight increases rapidly, automatically strengthening the smoothness constraint on the policy's output actions in regions where the Q-value is sensitive to state changes, suppressing drastic fluctuations in control commands.

[0052] The square of the gradient norm of the policy network's output action with respect to the state is used as the regularization term. A loss function containing a Q-value term and a weighted regularization term is constructed, with the following relationship: ; in, The total loss of the policy network; This is the evaluation value of the first online Q-value network for the current state and the output action of the policy network; This is the regularization term, which is the square of the gradient norm of the policy network's output action with respect to the state. It is used to penalize drastic changes in the policy in the state space. These are the weight coefficients for the regularization term; To approximate the mathematical expectation, in this specific embodiment of the invention, since the complete probability distribution of the state-action space cannot be obtained, a Monte Carlo sampling approximation method is used. The arithmetic mean of all state transition tuples within a batch sampled from the experience replay pool based on a composite priority index is taken. In this embodiment, each sampling batch contains 256 state transition tuples.

[0053] Learning rate The Adam optimizer completes the policy network parameter update and simultaneously performs soft updates on the two target Q-value networks and one target policy network. The soft update relationship is as follows: ; in, These are the parameters of the target network; These are the parameters for the corresponding online network; This is the soft update coefficient, used to slowly update the target network parameters to avoid sudden changes in the target value that could lead to training instability. In this embodiment, the value is set to 0.005.

[0054] S4, after the output control action is denormalized, drives the actuator.

[0055] The deterministic actions output by the policy network in the current state are sent as control commands to the actuator.

[0056] After the network training is completed, it enters the actual control stage. In this stage, exploration noise is no longer added. The normalized state vector is directly input into the trained policy network to obtain a two-dimensional deterministic action vector.

[0057] To adapt to the physical range of the field actuator, the motion vector needs to be denormalized. In this embodiment, the flow motion component is mapped to a raw material flow rate setting value of 1.5 to 4.5 t / h, and the steam valve motion component is mapped to a steam valve opening setting value of 0 to 100%.

[0058] Finally, the flow rate setpoint is converted into a frequency signal of 20 to 50 Hz and sent to the feeding auger frequency converter; the steam valve opening setpoint is sent to the regulating valve controller to complete the closed-loop intelligent temperature control of the feed pelleting production process.

[0059] This embodiment achieves adaptive adjustment of exploration intensity according to the agent's cognitive level through the above steps, reducing ineffective trial and error; it improves the control accuracy of key process points by focusing on key state samples through composite priority sampling; and it suppresses drastic fluctuations in control commands through state-aware action gradient regularization, ensuring stable operation of the production process and consistency of product quality.

[0060] This invention also discloses a temperature control system for a feed pelleting production process, comprising a control module, a data acquisition module, and an execution module. The data acquisition module collects data on raw material flow rate, grinder current, pellet mill temperature, and steam valve opening status from the feed production line. The control module includes a processor and a memory, the memory storing computer program instructions. When the processor executes the computer program instructions, it implements a temperature control method for a feed pelleting production process according to this invention. The execution module receives control instructions output by the control module and adjusts the raw material flow rate and steam addition amount accordingly.

[0061] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A temperature control method in the feed pelleting production process, characterized in that, include: Obtain process status data of the feed pelleting production process. The process status data shall include at least the parameters reflecting the temperature of the pellet mill and the control parameters affecting the temperature regulation of the pellet mill. Calculate the instant reward value based on status data and preset production targets; The process involves using a policy network to output an action and superimposing exploration noise to generate and execute the action. This includes: outputting a deterministic action for the current state using the policy network; calculating the exponential moving average of the absolute values ​​of the historical time-series difference errors of the two Q-value networks; adding a preset minimum positive number to each of the two exponential moving averages, taking the reciprocal, calculating the confidence level, and selecting the smallest of these as the final confidence level; adding 1 to the final confidence level, taking the reciprocal to obtain a noise scaling factor, and sampling noise from a standard normal distribution; multiplying the sampled noise by the noise scaling factor and superimposing it onto the deterministic action to generate and execute the action. Store the state transition tuple containing state data, executed action, reward, next state, and task end flag into the experience replay pool; Based on a composite priority index weighted by absolute temporal difference error and state sensitivity value, state transition tuples are sampled from the empirical replay pool, and the state sensitivity value is obtained by calculating the gradient norm of the first Q-value function with respect to the state data. The network is updated using state transition tuples: The next action is calculated based on the target policy network and the next state. The smaller value of the outputs of the two target Q-value networks is used to construct the target Q-value, which is then used to update the two Q-value networks. The policy network is updated with a delay by optimizing a loss function that includes Q-value terms and weighted action gradient regularization terms. This includes: obtaining the state from the sampled tuples and calculating the corresponding action using the current policy network; calculating the gradient of the first Q-value function with respect to the state and obtaining the L2 norm as the state sensitivity value; calculating the weight coefficients of the regularization term based on the state sensitivity value and a preset reference gradient value; calculating the square of the gradient norm of the policy network output action with respect to the state as the regularization term; constructing a loss function that includes Q-value terms and weighted regularization terms, and updating the policy network parameters using gradient descent; the weights of the regularization term are positively correlated with the state sensitivity value. The deterministic actions output by the policy network in the current state are sent as control commands to the actuator.

2. The temperature control method in the feed pelleting process according to claim 1, characterized in that, The composite priority index, weighted by absolute temporal difference error and state sensitivity value, samples state transition tuples from the empirical replay pool, including: For each state transition tuple in the experience replay pool, calculate the absolute timing difference error; Calculate the gradient of the first Q-value function with respect to the state vector and obtain the L2 norm as the state sensitivity value; The composite priority index is obtained by multiplying the absolute time-series difference error and the state sensitivity value by preset first and second weighting coefficients, respectively; The probability of each tuple being sampled is calculated based on the composite priority index; Based on the stated probability, a batch of tuples is drawn non-repeatedly from the empirical replay pool for network updates.

3. The temperature control method in the feed pelleting process according to claim 1, characterized in that, The process of calculating the next action based on the target policy network and the next state, constructing the target Q-value by taking the smaller value from the outputs of the two target Q-value networks, and then updating the two Q-value networks includes: Based on the next state in the sampled tuple, the next action is calculated through the target policy network; Add a cropped noise to the next action to achieve smoothing of the target policy; The Q-value of the next state is calculated using two target Q-value networks; Select the smaller of the two target Q values ​​and calculate the target Q value together with the immediate reward value and the discount factor; The parameters of two Q-value networks are updated simultaneously using the mean squared error loss function and the gradient descent method.

4. The temperature control method in the feed pelleting process according to claim 1, characterized in that, The process status data includes raw material flow rate, crusher current, granulator temperature, and steam valve opening. The four sets of data collected are used to form a four-dimensional state vector, and the data in each dimension are subjected to maximum and minimum normalization.

5. The temperature control method in the feed pelleting process according to claim 1, characterized in that, Sending the deterministic action output by the policy network in the current state as a control command to the execution mechanism includes: the policy network outputting a two-dimensional deterministic action vector to the current normalized state vector; The first dimension of the motion vector is denormalized into a specific raw material flow rate setpoint and sent to the frequency converter controlling the feeding auger to adjust the raw material flow rate. The second dimension of the motion vector is denormalized to the steam valve opening setpoint and sent to the regulating valve controller to adjust the steam addition amount of the pellet mill.

6. The temperature control method in the feed pelleting process according to claim 1, characterized in that, The exploration noise is Gaussian noise, and the noise amplitude is adaptively adjusted according to the confidence level of the time-series difference error of the Q-value network.

7. The temperature control method in the feed pelleting process according to claim 1, characterized in that, The instant reward value is calculated based on the state data and the preset production target, including: subtracting the normalized state vector from the preset production target vector to obtain the deviation of each state variable; and weighted summing the squares of each deviation, taking the negative value of the sum as the instant reward value.

8. A temperature control system for feed pelleting production process, characterized in that, include: The system includes a control module, a data acquisition module, and an execution module. The control module includes a processor and a memory. The memory stores computer program instructions. When the computer program instructions are executed by the processor, the temperature control method in the feed pelleting production process according to any one of claims 1 to 7 is implemented.

Images