A coal yard dynamic blending optimization method and system
By optimizing the real-time detection and reinforcement learning framework, the problems of difficult quality control and reliance on manual experience in the blending of thermal coal were solved, achieving precise control of the calorific value of the product coal and equipment safety, and meeting the needs of multi-objective coordination.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI GOLD STAR INTELLIGENT CONTROL TECH CO LTD
- Filing Date
- 2026-06-17
- Publication Date
- 2026-07-14
AI Technical Summary
Existing thermal coal blending processes suffer from problems such as difficulty in controlling the quality of raw coal, reliance on manual experience for production regulation, difficulty in product quality management, slow response, static modeling that ignores dynamic process characteristics, and difficulty in online coordination of multiple objectives. These issues lead to problems such as large fluctuations in the calorific value of the product coal and potential safety hazards in equipment.
By acquiring real-time online coal quality monitoring data, conveyor belt operation status data, and stockpile inventory data, a state space and action space are constructed. Dynamic blending optimization is performed using a reinforcement learning framework and reward function. Parameter updates are then performed by combining a policy network and a value network, enabling real-time online monitoring and feedback control throughout the entire process.
It achieves precise control of the calorific value of the product coal, reduces calorific value fluctuations, meets stability requirements, automatically meets process constraints, avoids equipment overload, establishes a closed-loop feedback regulation mechanism, and achieves multi-objective balance.
Smart Images
Figure CN122386640A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of coal processing and energy utilization technology, specifically relating to a method and system for dynamic blending optimization in coal yards. Background Technology
[0002] In industries such as power generation, metallurgy, and chemicals, thermal coal is a primary fuel, and its combustion performance directly affects boiler operating efficiency, emission levels, and equipment safety. To meet the calorific value requirements of different users, it is often necessary to blend raw coal from various sources and of different qualities in a certain proportion to form finished coal that meets downstream production standards.
[0003] The current coal blending process for thermal power faces the following key problems in practical applications: (1) Difficulty in controlling the quality of raw coal: Insufficient representativeness of sampling, delayed testing cycle and unsound traceability mechanism for raw coal entering the site result in untimely and limited accuracy of raw coal quality information, which cannot provide accurate basis for the blending process.
[0004] (2) Production regulation relies on human experience: The production regulation process relies heavily on human experience and lacks unified and stable control standards. The regulation effect is easily affected by subjective factors such as the skill level and working status of operators, which restricts the standardization and stability of the production process.
[0005] (3) Product quality management is difficult: the coal blending process lacks effective monitoring methods, and the coal blending system is not flexible and responsive enough, which makes it difficult to control the blending results and the product quality is unstable, and cannot meet the customer's precise requirements for indicators such as calorific value.
[0006] Although various intelligent blending technologies have been proposed and applied, the following significant drawbacks still exist in practical industrial scenarios: (1) It relies heavily on offline data and has a serious delay in response. Laboratory testing cycles typically last 4–24 hours, which cannot reflect real-time fluctuations in incoming coal; coal blending strategies are out of sync with actual coal quality, resulting in large fluctuations in product calorific value and making it difficult to meet the stringent requirements for the stability of product coal calorific value.
[0007] (2) Static modeling, ignoring dynamic process characteristics Traditional optimization models treat stockpiles as an ideal state of "unlimited inventory" or "constant coal quality," without considering the dynamic consumption of stockpiles over time, changes in equipment status such as belt load, etc. Furthermore, they cannot adapt to the time-varying characteristics of continuous production processes, making blending strategies prone to failure.
[0008] (3) Multi-objective conflicts are difficult to coordinate online. While multi-objective evolutionary algorithms can generate Pareto fronts, they have high computational complexity and lack autonomous decision-making capabilities.
[0009] (4) The ability to handle hard constraints is weak, which can easily lead to infeasible solutions. Hard process constraints such as maximum belt load, minimum safety stockpile, and equipment start-up and shutdown restrictions are often simplified to soft constraints or ignored in traditional optimization. When the calorific value target conflicts with the equipment capacity (such as high-calorific-value coal requiring a large amount of material to be discharged but the belt is already overloaded), the system cannot automatically avoid risks, posing a safety production hazard.
[0010] Therefore, how to achieve dynamic coal blending that conforms to the current blending process and can be monitored and controlled online in real time throughout the entire process, in order to solve the quality and efficiency problems existing in the current thermal coal blending process and meet users' needs for precise control of the calorific value of the product coal, has become an urgent technical problem to be solved. Summary of the Invention
[0011] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method and system for dynamic blending optimization in coal yards, which can realize dynamic blending in coal yards with real-time online detection and feedback control throughout the entire process, as well as precise control of the calorific value of product coal.
[0012] This invention provides a method for dynamic blending optimization in coal yards, comprising: Real-time online coal quality monitoring data, belt operation status data, and stockpile inventory data are acquired from the coal conveyor belt. Fluctuation prediction is performed based on the online coal quality monitoring data to obtain estimated coal quality data. Based on the online coal quality monitoring data, the conveyor belt operation status data, the stockpile inventory data, and the estimated coal quality data, a state space, an action space, and a reward function are constructed. The predicted action space corresponding to the action space is calculated based on the state space using a pre-built policy network. The updated state space of the coal yard after executing the predicted action space is obtained, and the reward function is updated and calculated based on the updated state space to obtain the updated reward value; The first value estimate corresponding to the state space and the second value estimate corresponding to the updated state space are calculated using a pre-constructed value network. The policy network and the value network are updated with parameters based on the updated reward value, the first value estimate, and the second value estimate, respectively. The updated policy network is used to perform calculations based on the updated state space to obtain the optimal blending action.
[0013] In one embodiment of the present invention, the step of performing fluctuation prediction based on the online coal quality monitoring data to obtain estimated coal quality data includes: Based on the online coal quality detection data and the pre-acquired historical hidden states, ignore information is calculated to generate candidate states; Calculate the current hidden state based on the historical hidden states and the candidate states; A fully connected mapping is performed on the current hidden state to obtain the estimated coal quality data.
[0014] In one embodiment of the present invention, before performing fluctuation prediction based on the online coal quality monitoring data, the method further includes: Coal quality online monitoring data, conveyor belt operation status data, and stockpile inventory data are used as preprocessed data; Outlier removal is performed on the preprocessed data to obtain cleaning and testing data; The cleaning and testing data are normalized to obtain coal quality online testing data, conveyor belt operation status data, and stockpile inventory data after feature mapping.
[0015] In one embodiment of the present invention, the step of constructing a state space, an action space, and a reward function based on the online coal quality monitoring data, the conveyor belt operation status data, the stockpile inventory data, and the estimated coal quality data includes: A state space is generated based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the preset coal yard target data; The motion space is generated based on the preset stockpile discharge ratio and the preset conveyor belt material ratio; Based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the coal yard target data, respectively generate coal quality compliance rewards, belt load safety rewards, stockpile inventory balance rewards, feedforward correction rewards, and over-limit penalties. A reward function is constructed based on the coal quality compliance reward, the belt load safety reward, the stockpile inventory balance reward, the feedforward correction reward, and the over-limit penalty item.
[0016] In one embodiment of the present invention, the step of generating coal quality compliance rewards, belt load safety rewards, stockpile inventory balance rewards, feedforward correction rewards, and over-limit penalties based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the coal yard target data respectively includes: A coal quality compliance reward is generated based on the calorific value and sulfur content in the online coal quality monitoring data and the target calorific value and target sulfur content in the coal yard target data. A belt load safety reward is generated based on the current belt load in the online coal quality monitoring data and the maximum belt load in the coal yard target data. A stockpile inventory balance reward is generated based on the stockpile inventory data and the expected stockpile inventory in the coal yard target data. A feedforward correction reward is generated based on the estimated coal quality data and the calorific value in the online coal quality monitoring data. An over-limit penalty item is generated based on the current belt load and the maximum belt load, the stockpile inventory and the expected stockpile inventory, and the calorific value and the target calorific value.
[0017] In one embodiment of the present invention, updating the parameters of the policy network and the value network based on the updated reward value, the first value estimate, and the second value estimate respectively includes: The policy loss value is calculated using the objective function corresponding to the policy network based on the updated reward value, the first value estimate, and the second value estimate. The value loss value is calculated using the objective function corresponding to the value network based on the updated reward value, the first value estimate, and the second value estimate. The parameters of the policy network and the value network are updated based on the policy loss value and the value loss value, respectively.
[0018] In one embodiment of the present invention, the step of calculating the policy loss value using the objective function corresponding to the policy network based on the updated reward value, the first value estimate, and the second value estimate includes: Calculate the temporal difference residual based on the updated reward value, the first value estimate, and the second value estimate; The dominance function value is calculated based on the time-series difference residuals. The strategy probability ratio is calculated based on the first action probability density corresponding to the action space and the second action probability density corresponding to the predicted action space. The policy loss value is calculated using the objective function corresponding to the policy network, based on the advantage function value and the policy probability ratio.
[0019] In one embodiment of the present invention, the step of calculating the value loss value using the objective function corresponding to the value network based on the updated reward value, the first value estimate, and the second value estimate includes: Calculate the value objective based on the updated reward value and the second value estimate; The value loss value is calculated based on the first value estimate and the value objective using the objective function corresponding to the value network.
[0020] In one embodiment of the present invention, the pre-built policy network and the pre-built value network are obtained through the following steps: Acquire coal quality testing training data, belt operation status training data, and stockpile inventory training data on the coal conveyor belt; perform fluctuation prediction based on the coal quality testing training data to obtain estimated coal quality training data. Based on the coal quality testing training data, the stockpile inventory training data, the belt conveyor operation status training data, and the estimated coal quality training data, a training state space, a training action space, and a training reward function are constructed. The predicted training action space corresponding to the training action space is calculated using a preset initial policy network based on the training state space. The state transition probability is calculated based on the training state space and the predicted training action space to obtain the updated training state space; The training reward function is updated and calculated based on the updated training state space to obtain the updated training reward value; The first training value estimate corresponding to the training state space and the second training value estimate corresponding to the updated training state space are calculated using a preset initial value network. The parameters of the initial policy network and the initial value network are updated based on the updated training reward value, the first training value estimate, and the second training value estimate, respectively, to obtain the pre-constructed policy network and the pre-constructed value network.
[0021] In one embodiment of the present invention, the step of calculating the state transition probability based on the training state space and the predicted training action space to obtain an updated training state space includes: An initial state space is generated based on the training state space and the predicted training action space; The state transition probabilities of the initial state space are calculated based on the initial state space and the preset Gaussian noise to obtain the updated state space.
[0022] In one embodiment of the present invention, generating an initial state space based on the training state space and the predicted training action space includes: The stockpile inventory is updated based on the proportion of belt material in the predicted training action space and the current belt load in the training state space to generate the stockpile inventory in the initial state space. The coal quality is updated based on the stockpile discharge ratio in the predicted training action space and the intermediate calorific value and intermediate sulfur content in the training state space to generate the intermediate calorific value and intermediate sulfur content in the initial state space. The coal quality is updated based on the proportion of belt material in the predicted training action space and the current belt load, final calorific value, and final sulfur content in the training state space to generate the final calorific value and final sulfur content in the initial state space.
[0023] The present invention also provides a dynamic blending optimization system for coal yards, characterized in that it comprises: The data acquisition module is used to acquire in real time online coal quality detection data, belt operation status data, and stockpile inventory data on the coal conveyor belt, as well as the updated state space after the coal yard executes the predicted action space; The coal quality prediction module is used to predict fluctuations based on the online coal quality detection data to obtain estimated coal quality data. The reinforcement learning framework construction module is used to construct the state space, action space, and reward function based on the online coal quality detection data, the belt conveyor operation status data, the stockpile inventory data, and the estimated coal quality data. The dynamic blending calculation module is used to calculate the predicted action space corresponding to the action space based on the state space using a pre-built policy network, and to calculate based on the updated state space using the updated policy network to obtain the optimal blending action. The strategy optimization module is used to update the reward function according to the updated state space to obtain an updated reward value; calculate the first value estimate corresponding to the state space and the second value estimate corresponding to the updated state space using a pre-constructed value network; and update the parameters of the strategy network and the value network according to the updated reward value, the first value estimate, and the second value estimate.
[0024] The beneficial effects of this invention are as follows: This invention acquires coal quality and load data in real time through online detection instruments throughout the entire process, and combines this with feedforward prediction to dynamically match the coal blending strategy with the actual coal quality, reducing the fluctuation range of product calorific value to meet stability requirements; based on the reward function, it automatically meets the hard constraints of the process such as belt conveyor capacity and minimum safety stock in the stockpile during the blending process, avoiding equipment overload operation; a closed-loop feedback adjustment mechanism is established to dynamically adjust and optimize the update of strategy network and value network parameters, automatically correct the front-end coal blending strategy based on coal quality detection results, efficiently solve for the optimal strategy to obtain the optimal blending action, and achieve multi-objective balance. Attached Figure Description
[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0026] Figure 1 This is a process flow diagram of dynamic coal blending provided in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a dynamic blending optimization method for coal yards provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of the simulation stage of the dynamic blending optimization method for coal yards provided in one embodiment of the present invention; Figure 4 This is a functional block diagram of dynamic blending optimization in a coal yard provided in one embodiment of the present invention; Among them, 11 is 2A belt, 12 is 2B belt, 13 is 3A belt, 14 is 3B belt, 15 is 4A belt, 16 is 4B belt, 17 is 5A belt, 18 is 5B belt, 19 is belt number 6, 21 is the northwest storage yard, 22 is the southwest storage yard, 23 is the northeast storage yard, and 24 is the southeast storage yard. Detailed Implementation
[0027] Example 1 The present invention provides a dynamic blending optimization method for coal yards, which can be applied to the efficient, stable and precise blending of raw coal of different qualities in ports or coal transfer stations, so as to achieve precise control of the calorific value of coal products and automated management of the production process.
[0028] In industries such as power generation, metallurgy, and chemicals, thermal coal is a primary fuel, and its combustion performance directly affects boiler operating efficiency, emission levels, and equipment safety. To meet the calorific value requirements of different users, it is often necessary to blend raw coal from various sources and of different qualities in a certain proportion to form finished coal that meets downstream production standards.
[0029] See Figure 1 The image shows a practical application scenario of the dynamic blending optimization method for coal yards provided by this invention.
[0030] by Figure 1 Taking the coal yard blending process shown as an example, during the stockpiling process, the raw coal is crushed to below 250mm by a crusher and then transported via belt conveyor 13 (3A) and belt conveyor 14 (3B). The incoming coal on belt conveyor 13 (3A) and belt conveyor 14 is then stockpiled into four stockpiles: northwest, southwest, northeast, and southeast, by two stackers. Specifically, the western stacker stockpiles the material from belt conveyor 13 into the southwest stockpile 22 and northwest stockpile 21; the eastern stacker stockpiles the material from belt conveyor 14 into the southeast stockpile 24 and northeast stockpile 23.
[0031] During the blending process, belts 3A 13 and 3B 14 can directly connect to belts 5A 17 and 5B 18 for direct coal blending. At the same time, each stockpile discharges material according to a pre-set discharge ratio via a volumetric feeder at the bottom. Material from the northwest stockpile 21 and northeast stockpile 23 is discharged to belt 2A 11 and then transferred to belt 5A 17 via belt 4A 15. Material from the southwest stockpile 22 and southeast stockpile 24 is discharged to belt 2B 12 and then transferred to belt 5B 18 via belt 4B 16.
[0032] During the mixing process, belts 17 (5A) and 18 (5B) are subjected to secondary crushing to a size of less than 70mm by a crusher, and then mixed by belt 19 (6) before being loaded and delivered.
[0033] This invention addresses the technical problems existing in current thermal coal blending technologies, such as response lag, static modeling, weak constraint handling, and poor multi-path coordination capabilities, by providing a dynamic blending optimization method for coal yards based on the CMDP (Constrained Markov Decision Process) framework.
[0034] Please see Figure 2 As shown in the figure, an embodiment of the present invention provides a method for dynamic blending optimization in coal yards, comprising the following steps: S11. Real-time acquisition of online coal quality monitoring data, belt operation status data, and stockpile inventory data on the coal conveyor belt; based on the online coal quality monitoring data, fluctuation prediction is performed to obtain estimated coal quality data.
[0035] In this embodiment of the invention, the online coal quality monitoring data can be obtained by an online coal quality monitoring instrument installed on the conveyor belt, the stockpile inventory data can be obtained by a 3D radar level gauge installed on the coal conveyor belt, and the conveyor belt operation status data can be obtained by a belt scale installed on the conveyor belt. Furthermore, the online coal quality monitoring instrument can be replaced by a laser-induced breakdown spectrometer, a neutron activation coal quality analyzer, a microwave coal quality analyzer, etc., and the 3D radar level gauge in the stockpile can be replaced by an ultrasonic level gauge, etc., without further limitations.
[0036] In a practical application scenario of this invention, see [reference] Figure 1 As shown, the online coal quality monitoring instrument can be installed in the following locations: 1. Located at the front end of conveyor belt 19 of No. 6, it can provide timely feedback and adjustment of the coal yard front-end batching process and proportioning parameters; 2. It can be installed at the tail of 4A belt 15 and 4B belt 16, or at the tail of 2A belt 11 and 2B belt 12, to detect the calorific value of the products on 2A belt 11 and 2B belt 12 in real time and to provide timely feedback to adjust the batching process and proportioning parameters at the front end of the coal yard. 3. It is set at the front end of 3A belt 13 and 3B belt 14 to evaluate the calorific value of raw coal before it enters the stockpile and before it enters 5A belt 17 and 5B belt 18. The data is then used for feedforward correction to adjust the batching process and proportioning parameters.
[0037] The 3D radar level gauges for the stockpiles can be installed directly above the southwest stockpile 22, northwest stockpile 21, southeast stockpile 24, and northeast stockpile 23 to detect the current volume and coal inventory of the stockpiles.
[0038] The belt scale can be installed on belts 3A 13, 3B 14, 5A 17, 5B 18, and belt 6 19 to detect the load weight on the belts.
[0039] Specifically, the online coal quality monitoring data includes data from different times. Different belt positions Heat generation Sulfur content The real-time data from the storage yard is for different times. Different storage yard locations Stockpile inventory The belt conveyor operating status data includes data from different times. Different belt positions Current load of the belt Where i can be belt 11 (2A), belt 12 (2B), belt 13 (3A), belt 14 (3B), belt 15 (4A), belt 16 (4B), belt 17 (5A), belt 18 (5B), belt 19 (6), and belt j can be 1, 2, 3, or 4, representing the Northwest Yard 21, Southwest Yard 22, Northeast Yard 23, and Southeast Yard 24, respectively.
[0040] In this embodiment of the invention, before performing fluctuation prediction based on the online coal quality monitoring data, data preprocessing can be performed on the acquired real-time online coal quality monitoring data on the coal conveyor belt, belt operation status data, and stockpile inventory data, including: Coal quality online monitoring data, conveyor belt operation status data, and stockpile inventory data are used as preprocessed data; Outlier removal is performed on the preprocessed data to obtain cleaning and testing data; The cleaning and testing data are normalized to obtain coal quality online testing data, conveyor belt operation status data, and stockpile inventory data after feature mapping.
[0041] Specifically, online coal quality monitoring data, conveyor belt operation status data, and stockpile inventory data at different times form corresponding time-series data. Based on the corresponding time-series data, historical averages for different data types can be calculated. and historical standard deviation Judging the data at the current moment based on the historical mean and historical standard deviation. Does it meet the requirements? If the deviation is within 3 times the standard deviation, the data is considered normal and retained; if the deviation exceeds 3 times the standard deviation, the data is considered abnormal (possibly caused by sensor failure, communication interference, etc.).
[0042] Taking the calorific value in online coal quality monitoring data as an example, the calorific value at different times can form a time series of calorific values, and the historical average calorific value can be calculated based on the time series of calorific values. and historical standard deviation of calorific value The calorific value at the current moment is determined based on the historical mean and historical standard deviation of calorific value. Does it meet the requirements? If the deviation is within 3 times the standard deviation, it will be retained.
[0043] In addition, data that is identified as anomalous can be replaced instead of being discarded directly (to avoid data breakpoints), in order to maintain the continuity of time series data.
[0044] Furthermore, regarding the cleaned data Map to the [0,1] interval to eliminate dimensional differences between different physical quantities.
[0045] In this embodiment of the invention, the step of predicting fluctuations based on the online coal quality monitoring data to obtain estimated coal quality data includes: Based on the online coal quality detection data and the pre-acquired historical hidden states, ignore information is calculated to generate candidate states; Calculate the current hidden state based on the historical hidden states and the candidate states; A fully connected mapping is performed on the current hidden state to obtain the estimated coal quality data.
[0046] Specifically, this invention can capture long-term dependencies in coal quality data using an LSTM (Long Short-Term Memory) network, based on three gating structures (forget gate, input gate, and output gate) and two states (hidden state and cell state).
[0047] Alternatively, the present invention can also be based on GRU (Gated Recurrent Unit) to realize time-series prediction of coal quality data through two gating structures (update gate and reset gate) and one state (hidden state), which is simpler in structure and more efficient in computation.
[0048] In this embodiment of the invention, when predicting fluctuations in online coal quality monitoring data, a sliding time window is formed by sequentially sorting data such as calorific value and sulfur content as the input data sequence. Each historical data point within the sliding time window is then processed sequentially according to time order. At each time step, the gating structure outputs a hidden state based on the current input data and the historical hidden state. After processing all historical data within the sliding time window, the hidden state of the last time step is mapped to the estimated coal quality data for the next sampling period, including the next time step, through a fully connected layer. Estimated heat generation and estimated sulfur content .
[0049] This invention is based on a feedforward prediction model, which predicts the coal quality for the next sampling period based on historical coal quality data on the conveyor belt. It can be used as a model framework for subsequent CMDP framework feedforward correction.
[0050] S12. Construct a state space, action space, and reward function based on the online coal quality monitoring data, the belt conveyor operation status data, the stockpile inventory data, and the estimated coal quality data.
[0051] This invention can be based on the CMDP (Constrained Markov Decision Process) framework to ensure that, during the process of learning the optimal strategy in the dynamic blending of coal in the coal yard, the long-term cumulative reward is maximized while meeting process constraints and safety limitations.
[0052] In this embodiment of the invention, the step of constructing a state space, an action space, and a reward function based on the online coal quality monitoring data, the conveyor belt operation status data, the stockpile inventory data, and the estimated coal quality data includes: A state space is generated based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the preset coal yard target data; The motion space is generated based on the preset stockpile discharge ratio and the preset conveyor belt material ratio; Based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the coal yard target data, respectively generate coal quality compliance rewards, belt load safety rewards, stockpile inventory balance rewards, feedforward correction rewards, and over-limit penalties. A reward function is constructed based on the coal quality compliance reward, the belt load safety reward, the stockpile inventory balance reward, the feedforward correction reward, and the over-limit penalty item.
[0053] Furthermore, the generation of coal quality compliance rewards, conveyor belt load safety rewards, stockpile inventory balancing rewards, feedforward correction rewards, and over-limit penalties based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the coal yard target data, respectively, includes: A coal quality compliance reward is generated based on the calorific value and sulfur content in the online coal quality monitoring data and the target calorific value and target sulfur content in the coal yard target data. A belt load safety reward is generated based on the current belt load in the online coal quality monitoring data and the maximum belt load in the coal yard target data. A stockpile inventory balance reward is generated based on the stockpile inventory data and the expected stockpile inventory in the coal yard target data. A feedforward correction reward is generated based on the estimated coal quality data and the calorific value in the online coal quality monitoring data. An over-limit penalty item is generated based on the current belt load and the maximum belt load, the stockpile inventory and the expected stockpile inventory, and the calorific value and the target calorific value.
[0054] In a practical application scenario of this invention, CMDP is defined as a triple: Where Z(t) is the state space, A(t) is the action space, and R(t) is the reward function.
[0055] Specifically, the state space is represented as:
[0056] in, These represent the calorific value of the coal on belts 3A (13), 3B (14), 4A (15), 4B (16), and 6 (19), respectively. These represent the stockpiles in Northwest Stockpile 21, Southwest Stockpile 22, Northeast Stockpile 23, and Southeast Stockpile 24, respectively. These represent the sulfur content of the coal carried on conveyor belts 3A (13), 3B (14), 4A (15), 4B (16), and 6 (19), respectively. These represent the belt running status data for belts 2A11, 2B12, 3A13, 3B14, 5A17, 5B18, and belt 619, respectively. These represent the estimated calorific value and estimated sulfur content of the coal on conveyor belts 13 (3A) and 14 (3B) at the next moment, respectively. Represented as a state space, These are respectively represented as the preset target calorific value, maximum allowable sulfur content, maximum belt load, and maximum stockpile capacity.
[0057] Specifically, the action space is represented as:
[0058] in, These represent the stockpile discharge ratios for Northwest Stockpile 21, Southwest Stockpile 22, Northeast Stockpile 23, and Southeast Stockpile 24, respectively. ; ), The proportions of belt material for belts 5A (17) and 5B (18) are represented respectively. The proportion of stockpiled materials in Northwest Stockpile 21 and Southwest Stockpile 22. (The proportion of stockpiled materials in Northeast Stockpile 23 and Southeast Stockpile 24). This represents the percentage of material transported from conveyor belt 13 to the Northwest stockpile 21 ( ). (This refers to the percentage of materials transported by conveyor belt 13 to the southwest storage yard 22). This represents the percentage of material transported from conveyor belt 14 (3B) to the Northeast storage yard (23). (This refers to the percentage of materials transported from conveyor belt 14 to the southeast storage yard 24). This represents the percentage of belt material on belt 17 (5A) on belt 19 (6). The proportion of material on belt 18 of belt 5B. Represented as action space; Specifically, the reward function is expressed as:
[0059] in, Represented as the reward function value, This is a reward for meeting coal quality standards. This is indicated as a safety bonus for belt load. This is represented as a stockyard inventory leveling reward. This is represented as a feedforward correction reward. This is indicated as an over-limit penalty item. These are respectively represented as preset weighting coefficients;
[0060]
[0061]
[0062]
[0063]
[0064] in, These represent the preset reward coefficients. These are respectively represented as preset penalty constants.
[0065] It should be noted that the coal quality compliance reward is used to measure whether the calorific value and sulfur content of the final product coal (belt 19 of No. 6) are close to the target calorific value and the maximum allowable sulfur content; the belt load safety reward reflects whether the current load of each belt (such as belt 15 of 4A, belt 16 of 4B, belt 19 of No. 6, etc.) is within the safe operating range; the stockpile inventory balance reward is used to assess whether the inventory of the stockpiles (Northwest Stockpile 21, Southwest Stockpile 22, Northeast Stockpile 23, Southeast Stockpile 24) meets expectations; the feedforward correction reward is based on the prediction of future coal quality on belt 13 of 3A and belt 16 of 4B using networks such as LSTM, and assesses whether the blending ratio should be adjusted in advance to cope with raw material fluctuations; the over-limit penalty item avoids safety risks through hard constraint penalties (which can be adjusted according to the actual process conditions), and is a negative reward applied when the action violates the hard constraints of the process (such as belt overload, stockpile inventory below the safety threshold, etc.).
[0066] This invention quantifies and coordinates four major objectives—coal quality compliance, equipment safety, inventory balance, and feedforward correction—through a weighted reward function. By using load safety rewards and over-limit penalties in the reward function, it achieves an automatic avoidance mechanism for hard process constraints such as maximum belt load and minimum safety stock in the storage yard. Combined with the PPO algorithm, it efficiently solves the optimal strategy, achieving multi-objective balance without manual intervention.
[0067] S13. Calculate the predicted action space corresponding to the action space based on the state space using a pre-built policy network.
[0068] In a constrained industrial environment (CMDP), this invention enables a policy network to learn to make "good and stable" decisions in each state by continuously interacting with the environment through PPO (Proximal Policy Optimization). At the same time, a value network is used to evaluate the quality of the state, helping the policy to learn more efficiently.
[0069] Furthermore, the PPO policy optimization algorithm can be replaced by reinforcement learning algorithms suitable for continuous action spaces, such as DDPG (Deep Deterministic Policy Gradient) and SAC (Soft Actor-Critic), without further restrictions.
[0070] Specifically, the policy network generates actions based on the input state space and the hidden layers (3 fully connected layers and ReLU activation function), outputs the probability distribution parameters of the action space at the next time step, and thus obtains the optimal blending action that meets the process constraints.
[0071] Specifically, the value network calculates the state value estimate based on the long-term cumulative reward (state value) of the input state space using the hidden layers (3 fully connected layers and the ReLU activation function).
[0072] In this embodiment of the invention, the strategy network calculates based on the state space corresponding to the current time t to obtain the predicted action space corresponding to the optimal blending action at the next time. The coal yard adjusts the next time according to the predicted action space to obtain the updated state space after the coal yard executes the predicted action space.
[0073] S14. Obtain the updated state space after the coal yard executes the predicted action space, and update the reward function according to the updated state space to obtain the updated reward value.
[0074] In this embodiment of the invention, the reward function defines the optimization objective and is the sole criterion for evaluating the quality of the policy. The essence of reinforcement learning is to maximize long-term cumulative reward; therefore, the reward function must be used as the core objective during the policy network and value network update process. The reward function used during network update is the immediate reward obtained after executing the prediction action space at time t, corresponding to the reward from the action space... Transfer to the updated state space Rewards obtained during the process.
[0075] S15. Calculate the first value estimate corresponding to the state space and the second value estimate corresponding to the updated state space using the pre-constructed value network.
[0076] In this embodiment of the invention, the value network is used to evaluate the quality of a state, the advantage function uses the value network to judge the quality of an action, the policy network optimizes action selection based on the advantage function, the value network continuously approximates the true state value to make the advantage estimation more accurate, and the value network and policy network optimize alternately until convergence.
[0077] In this embodiment of the invention, the value network is based on the input state vector (i.e., the state space). and updating the state space and value network parameters Output the corresponding state value estimate (i.e., the first value estimate). Second value estimation ), of which, the first value estimate Representation to state space The expected cumulative reward that can be obtained, second value estimate Represents updating the state space The expected cumulative reward that can be obtained.
[0078] S16. Update the parameters of the policy network and the value network according to the updated reward value, the first value estimate and the second value estimate respectively.
[0079] In this embodiment of the invention, updating the parameters of the policy network and the value network based on the updated reward value, the first value estimate, and the second value estimate respectively includes: The policy loss value is calculated using the objective function corresponding to the policy network based on the updated reward value, the first value estimate, and the second value estimate. The value loss value is calculated using the objective function corresponding to the value network based on the updated reward value, the first value estimate, and the second value estimate. The parameters of the policy network and the value network are updated based on the policy loss value and the value loss value, respectively.
[0080] Specifically, the step of calculating the policy loss value using the objective function corresponding to the policy network based on the updated reward value, the first value estimate, and the second value estimate includes: Calculate the temporal difference residual based on the updated reward value, the first value estimate, and the second value estimate; The dominance function value is calculated based on the time-series difference residuals. The strategy probability ratio is calculated based on the first action probability density corresponding to the action space and the second action probability density corresponding to the predicted action space. The policy loss value is calculated using the objective function corresponding to the policy network, based on the advantage function value and the policy probability ratio.
[0081] Specifically, the step of calculating the value loss value using the objective function corresponding to the value network based on the updated reward value, the first value estimate, and the second value estimate includes: Calculate the value objective based on the updated reward value and the second value estimate; The value loss value is calculated based on the first value estimate and the value objective using the objective function corresponding to the value network.
[0082] In this embodiment of the invention, the time-series differential residual can be calculated using the following formula:
[0083] in, Represented as time-series difference residuals, This is represented as a preset discount factor. This indicates an update to the reward value. This is represented as the first value estimate. This is represented as the second value estimate.
[0084] when A value greater than 0 indicates that the actual reward plus the value of the next state is greater than the expected value, suggesting that the current state is underestimated. when <0 indicates that the actual reward plus the value of the next state is less than the expected value, suggesting that the current state is overestimated; when =0 indicates that the value estimate is accurate.
[0085] Furthermore, the dominance function is calculated based on the time-series difference error sequence formed by the time-series difference residuals at different times, as well as preset GAE parameters and preset discount factors, to obtain the dominance function value. , used to indicate the quality of the predicted action space relative to the average level; when >0 indicates that the predicted action space is better than the average level, and the probability of the corresponding action in the action space is increased; when <0 indicates that the predicted action space is worse than the average level, and the probability of the corresponding action in the action space is reduced; when =0 indicates that the predicted action space is comparable to the average level, and the probability of the corresponding action in the action space is not adjusted.
[0086] Furthermore, value objectives It can be calculated using the following formula:
[0087] The value objective is used to represent the target value that the value network wants to fit, i.e., the state space. The true value is conveyed through instant rewards. and the discount value of the next state This represents a supervised learning label for the value network.
[0088] In this embodiment of the invention, the value network minimizes the first value estimate. With value goals The error is used to update the value network. Specifically, the objective function corresponding to the value network is updated based on the first value estimate. With value goals By performing loss calculations, the value of the loss can be obtained. Then, the parameters in the value network are updated using the following parameter formula to achieve the update of the value network:
[0089] in, Represented as a value loss value, This is represented by the preset value network learning rate. Represented as the updated value network parameters, Represented as historical value network parameters.
[0090] In this embodiment of the invention, the policy network can be based on the action space and policy network parameters. Output the action mean, action standard deviation, and action probability density. The first action probability density corresponding to the action space is: The second action probability density corresponding to the predicted action space is: , For a state-action space pair, the probability ratio of the old and new policies can be calculated using the following formula. (i.e., the probability ratio of the strategies):
[0091] when =1 indicates that the probability of the new strategy on the corresponding action in the prediction action space is the same as that of the old strategy; when >1 indicates that the new strategy is more inclined to select the action corresponding to the predicted action space; when <1 indicates that the new strategy is less inclined to select actions corresponding to the predicted action space.
[0092] Furthermore, the objective function corresponding to the policy network is based on the policy probability ratio. Advantage function And the preset trust domain boundary (pruning boundary). Loss calculation is performed to obtain the pruned target. .
[0093] Then, the policy network parameters are updated by minimizing the loss function using the following formula:
[0094]
[0095] in, Represented as the policy loss value, This is represented by the preset policy network learning rate. This represents the updated policy network parameters. This is represented as the network parameters of the historical strategy.
[0096] This invention's alternating update strategy network (maximizing) ) and value network (minimize) For example, when the policy network loses... The fluctuation range is less than 1e 4. And value network loss When the value is less than 0.01, the network is considered to have converged. This invention achieves robust policy optimization by finding a balance between "maximizing rewards" and "maintaining policy stability" through pruning probability ratios.
[0097] S17. The updated policy network is used to calculate the optimal blending action based on the updated state space.
[0098] In this embodiment of the invention, the updated state space includes updated online coal quality monitoring data, conveyor belt operation status data, and stockpile inventory data. The updated strategy network is then used to predict based on the updated state space to obtain the updated action space corresponding to the updated state space, i.e., the optimal blending action. The state space after executing the updated action space is then obtained. The value network and strategy network are then iteratively optimized in real time by combining the updated action space and the updated state space, thereby achieving dynamic blending optimization.
[0099] In the coal yard dynamic blending optimization method of the present invention, the strategy network and value network can be optimized through online learning or periodic updates, and the strategy can be updated to achieve dynamic closed-loop feedback control throughout the entire process.
[0100] Example 2 In the simulation process of the dynamic blending optimization method for coal yards provided by this invention, the updated state space after predicting the action space can be obtained through the state transition probability. The specific simulation process, or the pre-constructed policy network and pre-constructed value network, can be obtained through the following steps: S21. Obtain coal quality testing training data, belt operation status training data, and stockpile inventory training data on the coal conveyor belt. Based on the coal quality testing training data, perform fluctuation prediction to obtain estimated coal quality training data. S22. Construct a training state space, a training action space, and a training reward function based on the coal quality testing training data, the stockpile inventory training data, the belt conveyor operation status training data, and the estimated coal quality training data. S23. Calculate the predicted training action space corresponding to the training action space based on the training state space using a preset initial policy network. S24. Calculate the state transition probability based on the training state space and the predicted training action space to obtain the updated training state space. S25. The training reward function is updated and calculated according to the updated training state space to obtain the updated training reward value; S26. Calculate the first training value estimate corresponding to the training state space and the second training value estimate corresponding to the updated training state space using a preset initial value network. S27. Update the parameters of the initial policy network and the initial value network according to the updated training reward value, the first training value estimate and the second training value estimate, respectively, to obtain the pre-constructed policy network and the pre-constructed value network.
[0101] In a practical application scenario of this invention, CMDP is defined as a quadruple: Where Z(t) is the state space, A(t) is the action space, and R(t) is the reward function. Let be the state transition probability.
[0102] Further, the step of calculating the state transition probability based on the training state space and the predicted training action space to obtain the updated training state space includes: An initial state space is generated based on the training state space and the predicted training action space; The state transition probabilities of the initial state space are calculated based on the initial state space and the preset Gaussian noise to obtain the updated state space.
[0103] Specifically, generating the initial state space based on the training state space and the predicted training action space includes: The stockpile inventory is updated based on the proportion of belt material in the predicted training action space and the current belt load in the training state space to generate the stockpile inventory in the initial state space. The coal quality is updated based on the stockpile discharge ratio in the predicted training action space and the intermediate calorific value and intermediate sulfur content in the training state space to generate the intermediate calorific value and intermediate sulfur content in the initial state space. The coal quality is updated based on the proportion of belt material in the predicted training action space and the current belt load, final calorific value, and final sulfur content in the training state space to generate the final calorific value and final sulfur content in the initial state space.
[0104] In a practical application scenario of this invention, the stockpile inventory update is represented as follows:
[0105] in, This represents the stockpile inventory in the initial state space. This is represented as the stockpile inventory in the training state space. Expressed as the amount of material added. This is expressed as the amount of material discharged. ; Specifically, the amount of material added This can be expressed by the following formula:
[0106]
[0107]
[0108]
[0109] Feeding amount This can be expressed by the following formula:
[0110]
[0111]
[0112]
[0113] in, , , , These represent the stockpiling volumes for Northwest Stockpile 21, Southwest Stockpile 22, Northeast Stockpile 23, and Southeast Stockpile 24, respectively. , , These represent the discharge volumes for Northwest Yard 21, Southwest Yard 22, Northeast Yard 23, and Southeast Yard 24, respectively.
[0114] Furthermore, when updating coal quality, including intermediate coal quality updating and final coal quality updating, the intermediate calorific value and intermediate sulfur content refer to updating the calorific value and sulfur content of the coal on conveyor belts 11 of 2A and 12 of 2B.
[0115]
[0116]
[0117]
[0118] in, This is expressed as the calorific value and sulfur content of the coal on conveyor belt 11 of belt 2A. This is expressed as the calorific value and sulfur content of the coal on conveyor belt 12 of belt 2B; When performing the final coal quality update, the final calorific value and final sulfur content in the initial state space refer to the update of the calorific value and sulfur content of the coal on conveyor belt 19 of No. 6:
[0119]
[0120] in:
[0121]
[0122]
[0123]
[0124] in, This is expressed as the calorific value and sulfur content of the coal on conveyor belt 19 of belt #6.
[0125] This invention incorporates time-varying characteristics such as dynamic changes in stockpile inventory and belt load fluctuations into the CMDP framework, and considers random noise in the state transition probability, enabling the blending strategy to adapt to dynamic changes in continuous production and avoid strategy failure.
[0126] Example 3 like Figure 3 The diagram shown is a functional block diagram of a dynamic blending optimization system for coal yards provided in an embodiment of the present invention.
[0127] The coal yard dynamic blending optimization system 300 of this invention can be installed in an electronic device. Depending on the functions implemented, the coal yard dynamic blending optimization system 300 may include a data acquisition module 301, a coal quality prediction module 302, a reinforcement learning framework construction module 303, a dynamic blending calculation module 304, and a strategy optimization module 305. A module of this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.
[0128] In this embodiment, the functions of each module / unit are as follows: The data acquisition module 301 is used to acquire in real time online coal quality detection data, belt operation status data and stockpile inventory data on the coal conveyor belt, as well as the updated state space after the coal yard executes the prediction action space; The coal quality prediction module 302 is used to predict fluctuations based on the online coal quality detection data to obtain estimated coal quality data. The reinforcement learning framework construction module 303 is used to construct a state space, an action space, and a reward function based on the online coal quality detection data, the belt conveyor operation status data, the stockpile inventory data, and the estimated coal quality data. The dynamic blending calculation module 304 is used to calculate the predicted action space corresponding to the action space based on the state space using a pre-constructed policy network, and to perform calculations based on the updated state space using the updated policy network with updated parameters to obtain the optimal blending action.
[0129] The strategy optimization module 305 is used to update the reward function according to the updated state space to obtain an updated reward value; calculate the first value estimate corresponding to the state space and the second value estimate corresponding to the updated state space using a pre-constructed value network; and update the parameters of the strategy network and the value network according to the updated reward value, the first value estimate, and the second value estimate.
[0130] In detail, each module in the coal yard dynamic blending optimization system 300 of this embodiment of the invention adopts the same technical means as the audio-driven real-time action generation method in the attached figure when in use, and can produce the same technical effect, which will not be repeated here.
[0131] In summary, this invention acquires real-time coal quality and load data through online monitoring instruments throughout the entire process. Combined with feedforward prediction, it dynamically matches the coal blending strategy with the actual coal quality, reducing the fluctuation range of product calorific value to meet stability requirements. Based on the reward function, it automatically meets process constraints such as belt conveyor capacity and minimum safety stock in the stockpile during the blending process, avoiding equipment overload operation. A closed-loop feedback adjustment mechanism is established to dynamically adjust and optimize the update of strategy network and value network parameters. Based on the coal quality testing results, the front-end coal blending strategy is automatically corrected, efficiently solving for the optimal strategy to obtain the optimal blending action and achieving multi-objective balance.
[0132] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0133] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0134] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0135] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0136] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for dynamic blending optimization in coal yards, characterized in that, include: Real-time online coal quality monitoring data, belt operation status data, and stockpile inventory data are acquired from the coal conveyor belt. Fluctuation prediction is performed based on the online coal quality monitoring data to obtain estimated coal quality data. Based on the online coal quality monitoring data, the conveyor belt operation status data, the stockpile inventory data, and the estimated coal quality data, a state space, an action space, and a reward function are constructed. The predicted action space corresponding to the action space is calculated based on the state space using a pre-built policy network. The updated state space of the coal yard after executing the predicted action space is obtained, and the reward function is updated and calculated based on the updated state space to obtain the updated reward value; The first value estimate corresponding to the state space and the second value estimate corresponding to the updated state space are calculated using a pre-constructed value network. The policy network and the value network are updated with parameters based on the updated reward value, the first value estimate, and the second value estimate, respectively. The updated policy network is used to perform calculations based on the updated state space to obtain the optimal blending action.
2. The coal yard dynamic blending optimization method according to claim 1, characterized in that, The step of predicting fluctuations based on the online coal quality monitoring data to obtain estimated coal quality data includes: Based on the online coal quality detection data and the pre-acquired historical hidden states, ignore information is calculated to generate candidate states; Calculate the current hidden state based on the historical hidden states and the candidate states; A fully connected mapping is performed on the current hidden state to obtain the estimated coal quality data.
3. The coal yard dynamic blending optimization method according to claim 1, characterized in that, Before performing fluctuation prediction based on the online coal quality monitoring data, the method further includes: Coal quality online monitoring data, conveyor belt operation status data, and stockpile inventory data are used as preprocessed data; Outlier removal is performed on the preprocessed data to obtain cleaning and testing data; The cleaning and testing data are normalized to obtain coal quality online testing data, conveyor belt operation status data, and stockpile inventory data after feature mapping.
4. The coal yard dynamic blending optimization method according to claim 1, characterized in that, The process of constructing a state space, action space, and reward function based on the online coal quality monitoring data, the conveyor belt operation status data, the stockpile inventory data, and the estimated coal quality data includes: A state space is generated based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the preset coal yard target data; The motion space is generated based on the preset stockpile discharge ratio and the preset conveyor belt material ratio; Based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the coal yard target data, respectively generate coal quality compliance rewards, belt load safety rewards, stockpile inventory balance rewards, feedforward correction rewards, and over-limit penalties. A reward function is constructed based on the coal quality compliance reward, the belt load safety reward, the stockpile inventory balance reward, the feedforward correction reward, and the over-limit penalty item.
5. The coal yard dynamic blending optimization method according to claim 4, characterized in that, The process generates coal quality compliance rewards, conveyor belt load safety rewards, stockpile inventory balancing rewards, feedforward correction rewards, and over-limit penalties based on the online coal quality monitoring data, the stockpile inventory data, the estimated coal quality data, and the coal yard target data, including: A coal quality compliance reward is generated based on the calorific value and sulfur content in the online coal quality monitoring data and the target calorific value and target sulfur content in the coal yard target data. A belt load safety reward is generated based on the current belt load in the online coal quality monitoring data and the maximum belt load in the coal yard target data. A stockpile inventory balance reward is generated based on the stockpile inventory data and the expected stockpile inventory in the coal yard target data. A feedforward correction reward is generated based on the estimated coal quality data and the calorific value in the online coal quality monitoring data. An over-limit penalty item is generated based on the current belt load and the maximum belt load, the stockpile inventory and the expected stockpile inventory, and the calorific value and the target calorific value.
6. The coal yard dynamic blending optimization method according to claim 1, characterized in that, The step of updating the parameters of the policy network and the value network based on the updated reward value, the first value estimate, and the second value estimate includes: The policy loss value is calculated using the objective function corresponding to the policy network based on the updated reward value, the first value estimate, and the second value estimate. The value loss value is calculated using the objective function corresponding to the value network based on the updated reward value, the first value estimate, and the second value estimate. The parameters of the policy network and the value network are updated based on the policy loss value and the value loss value, respectively.
7. The coal yard dynamic blending optimization method according to claim 6, characterized in that, The step of calculating the policy loss value using the objective function corresponding to the policy network based on the updated reward value, the first value estimate, and the second value estimate includes: Calculate the temporal difference residual based on the updated reward value, the first value estimate, and the second value estimate; The dominance function value is calculated based on the time-series difference residuals. The strategy probability ratio is calculated based on the first action probability density corresponding to the action space and the second action probability density corresponding to the predicted action space. The policy loss value is calculated using the objective function corresponding to the policy network, based on the advantage function value and the policy probability ratio.
8. The coal yard dynamic blending optimization method according to claim 6, characterized in that, The step of calculating the value loss value using the objective function corresponding to the value network based on the updated reward value, the first value estimate, and the second value estimate includes: Calculate the value objective based on the updated reward value and the second value estimate; The value loss value is calculated based on the first value estimate and the value objective using the objective function corresponding to the value network.
9. The coal yard dynamic blending optimization method according to claim 1, characterized in that, The pre-built policy network and pre-built value network are obtained through the following steps: Acquire coal quality testing training data, belt operation status training data, and stockpile inventory training data on the coal conveyor belt; perform fluctuation prediction based on the coal quality testing training data to obtain estimated coal quality training data. Based on the coal quality testing training data, the stockpile inventory training data, the belt conveyor operation status training data, and the estimated coal quality training data, a training state space, a training action space, and a training reward function are constructed. The predicted training action space corresponding to the training action space is calculated using a preset initial policy network based on the training state space. The state transition probability is calculated based on the training state space and the predicted training action space to obtain the updated training state space; The training reward function is updated and calculated based on the updated training state space to obtain the updated training reward value; The first training value estimate corresponding to the training state space and the second training value estimate corresponding to the updated training state space are calculated using a preset initial value network. The parameters of the initial policy network and the initial value network are updated based on the updated training reward value, the first training value estimate, and the second training value estimate, respectively, to obtain the pre-constructed policy network and the pre-constructed value network.
10. The coal yard dynamic blending optimization method according to claim 9, characterized in that, The step of calculating the state transition probability based on the training state space and the predicted training action space to obtain the updated training state space includes: An initial state space is generated based on the training state space and the predicted training action space; The state transition probabilities of the initial state space are calculated based on the initial state space and the preset Gaussian noise to obtain the updated state space.
11. The coal yard dynamic blending optimization method according to claim 10, characterized in that, The step of generating an initial state space based on the training state space and the predicted training action space includes: The stockpile inventory is updated based on the proportion of belt material in the predicted training action space and the current belt load in the training state space to generate the stockpile inventory in the initial state space. The coal quality is updated based on the stockpile discharge ratio in the predicted training action space and the intermediate calorific value and intermediate sulfur content in the training state space to generate the intermediate calorific value and intermediate sulfur content in the initial state space. The coal quality is updated based on the proportion of belt material in the predicted training action space and the current belt load, final calorific value, and final sulfur content in the training state space to generate the final calorific value and final sulfur content in the initial state space.
12. A dynamic blending optimization system for coal yards, characterized in that, include: The data acquisition module is used to acquire in real time online coal quality detection data, belt operation status data, and stockpile inventory data on the coal conveyor belt, as well as the updated state space after the coal yard executes the predicted action space; The coal quality prediction module is used to predict fluctuations based on the online coal quality detection data to obtain estimated coal quality data. The reinforcement learning framework construction module is used to construct the state space, action space, and reward function based on the online coal quality detection data, the belt conveyor operation status data, the stockpile inventory data, and the estimated coal quality data. The dynamic blending calculation module is used to calculate the predicted action space corresponding to the action space based on the state space using a pre-built policy network, and to calculate based on the updated state space using the updated policy network to obtain the optimal blending action. The strategy optimization module is used to update the reward function according to the updated state space to obtain an updated reward value; calculate the first value estimate corresponding to the state space and the second value estimate corresponding to the updated state space using a pre-constructed value network; and update the parameters of the strategy network and the value network according to the updated reward value, the first value estimate, and the second value estimate.