Process parameter optimization method based on deep reinforcement learning

Abstract

The invention provides a process parameter optimization method based on deep reinforcement learning, and relates to the technical field of intelligent manufacturing, and the method comprises the steps: collecting the state, process parameters and output data of a production system in real time, carrying out the preprocessing, and storing, and constructing a high-quality data set; training a process parameter optimization model based on a deep reinforcement learning algorithm by using the data, so that the process parameter optimization model can output a process parameter compensation value according to a current state, and realizing dynamic updating and seamless switching of the model by maintaining a model library comprising an original model and an updated model; in the application stage, the latest model is called to generate a compensation value, target parameters are formed to guide production after amplitude limitation and upper and lower bound constraint processing, new data are fed back to a data layer, and closed-loop optimization is formed. According to the method, dynamic, safe and continuous optimization of process parameters is realized, and the self-adaptive capability and the operation efficiency of a production system are improved.

Description

Technical Field

[0001] This application belongs to the field of intelligent manufacturing technology, and specifically relates to a process parameter optimization method based on deep reinforcement learning. Background Art

[0002] In process manufacturing companies, process parameters are key input variables that influence product quality, production efficiency, and energy consumption. Optimizing them is crucial for improving overall production efficiency. Traditional process parameter optimization methods primarily include manual experience-based adjustments, optimization based on numerical simulation, and offline solutions using static optimization algorithms or artificial intelligence models.

[0003] However, these methods have obvious limitations in practical applications: manual experience relies on the skills of operators, and the adjustment efficiency is low and the consistency is poor when faced with complex production systems with multiple parameters and high coupling; although numerical simulation methods can simulate the production process, it is difficult to accurately characterize the dynamic disturbance and nonlinear coupling relationship in the real production environment, resulting in a large deviation between the simulation results and the actual operation; and methods based on static optimization algorithms or traditional machine learning usually perform offline modeling under fixed working conditions, and the resulting process parameters do not have real-time adaptability and cannot effectively deal with the state fluctuations and external interference that frequently occur in the production process.

[0004] In recent years, reinforcement learning technology has been explored for application in process parameter optimization due to its ability to learn online and make dynamic decisions without requiring precise system models. However, existing approaches often remain limited to training in simulation environments or single-shot model deployments, lacking mechanisms for continuous model updates and insufficiently considering resource conflicts and system stability issues associated with online model application and retraining. Furthermore, direct application of model outputs to production systems without appropriate constraints can pose control risks. Summary of the Invention

[0005] This application provides a process parameter optimization method based on deep reinforcement learning to solve one of the above technical problems.

[0006] The technical solutions adopted in this application are: The present invention provides a method for optimizing process parameters based on deep reinforcement learning, including: S1. Collect status data, process parameter data and output data from the production system in real time, and pre-process and store the collected data; S2. Based on a deep reinforcement learning algorithm, using the data to train and dynamically update a process parameter optimization model; the process parameter optimization model is used to output compensation values ​​for the process parameters based on the current state data of the production system; S3. Calling the process parameter optimization model to generate process parameter compensation values ​​based on the real-time acquired production system status data. After constraint processing, target process parameters are obtained and input into the production system to guide production. Meanwhile, new data generated during the production process is fed back to step S1. In step S2, the dynamic update of the process parameter optimization model is achieved by maintaining a model library, which includes at least an original model for current production and an updated model trained based on new data, and the model used for production is switched according to a predetermined update strategy.

[0007] According to one embodiment of the present application, the step S1 includes: Collecting status data reflecting the operating status of the production system, currently input process parameter data, and output data representing production results from the control system and sensor network of the production system; Standardize the collected data, remove outliers, delete duplicates, and fill in missing values; The processed data is stored in the database to form a historical data set.

[0008] According to one embodiment of the present application, the step S2 includes an offline training phase and an online update phase: The offline training phase obtains an initial process parameter optimization model based on historical data training; the online updating phase continuously corrects and updates the initial process parameter optimization model based on new data generated during the real-time operation of the production system.

[0009] According to one embodiment of the present application, during the offline training phase, a deterministic policy gradient algorithm based on an Actor-Critic architecture is used to train the initial process parameter optimization model.

[0010] According to one embodiment of the present application, the online update stage specifically includes: In the process of guiding production by the initial process parameter optimization model, the state data, process parameter data and output data of the production system are collected and stored in an experience replay pool after processing; When the number of samples stored in the experience replay pool reaches a preset capacity C, all the data in the pool are used to update the model and obtain a revised updated model; A copy of the updated model is made as a transit model and stored together with the current original model in the model library; the model with the latest generation time is selected from the model library as the original model for guiding the next production, and the model with the earliest generation time is eliminated.

[0011] According to one embodiment of the present application, the preset capacity C of the experience replay pool is a positive integer.

[0012] According to one embodiment of the present application, the step S3 includes: Obtaining current state data of the production system and inputting it into the process parameter optimization model to obtain process parameter compensation values; Limiting the compensation value of the process parameter to obtain a constrained compensation value; Calculate the current target process parameters based on the constrained compensation value and the process parameters of the previous time unit, and perform upper and lower bound truncation processing on the target process parameters to ensure that they are within a preset safety range; The target process parameters after truncation are input into the production system, and the new status data, process parameter data and output data generated in this process are collected and fed back to step S1.

[0013] According to one embodiment of the present application, the amplitude limitation is implemented by a constraint coefficient, and the constraint coefficient is adjusted according to the control capability of the production system control system; The minimum and maximum values ​​of the process parameters on which the upper and lower bound truncation processes are based are preset based on experience or expert knowledge.

[0014] A second aspect of the present application provides a computer-readable storage medium having a program stored thereon, which implements the steps in the method when executed by a processor.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the method described above when executing the program.

[0016] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows: This application ensures the timeliness and accuracy of model training data through real-time collection and preprocessing of production system status data, process parameters and output data in step S1; combined with the model training mechanism based on deep reinforcement learning in step S2, the process parameter optimization model can learn the nonlinear dynamic characteristics of the production process, output the optimal compensation value according to the current state, and significantly enhance the response ability to production fluctuations.

[0017] In step S2, a model library mechanism containing both the original model and the updated model is introduced, and a clear model switching strategy (such as based on timestamps or performance evaluation) is set. This allows one model to be used for current production guidance while the other model can be independently trained and validated based on new data. This design effectively avoids service interruptions during the model update process and ensures the continuous and stable operation of the production system.

[0018] In step S3, the process parameter compensation values ​​output by the model are constrained (such as limiting the adjustment range and setting upper and lower bounds) to ensure that the generated target process parameters are within a safe and controllable range, preventing production equipment from operating beyond its limits due to model misjudgment or abnormal output, and improving the system's robustness and engineering practicality.

[0019] In step S3, new data generated during the production process is continuously fed back to step S1, forming a complete "perception-decision-execution-learning" closed loop. This mechanism enables the process parameter optimization model to continuously absorb new knowledge over the long term, gradually approaching a more optimal control strategy, and possessing excellent self-adaptation and self-evolution capabilities.

[0020] It can realize dynamic optimization and safe application of process parameters without interrupting production, effectively improve product qualification rate, reduce energy consumption and raw material loss, and meet the urgent needs of process manufacturing companies for high-quality, high-efficiency and low-cost production. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a process parameter optimization method based on deep reinforcement learning provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Figure 3 This is an architectural diagram of a process parameter optimization method based on deep reinforcement learning provided in an embodiment of the present application; Figure 4 A schematic diagram of various elements of a process parameter optimization model based on deep reinforcement learning training in a process parameter optimization method based on deep reinforcement learning provided in an embodiment of the present application; Figure 5 A flowchart of a process parameter optimization model update for a process parameter optimization method based on deep reinforcement learning provided in an embodiment of the present application; Figure 6 This is an operating logic diagram of the application layer software for a process parameter optimization method based on deep reinforcement learning provided in an embodiment of the present application.

[0022] Reference numerals: 810 , processor; 820 , communication interface; 830 , memory; 840 , communication bus. DETAILED DESCRIPTION

[0023] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0024] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application may also be implemented in other ways than those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features of each embodiment may be combined with each other unless there is a conflict.

[0025] In this application, unless otherwise expressly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0026] Example 1 like Figure 1 As shown, a process parameter optimization method based on deep reinforcement learning includes: S1. Collect status data, process parameter data and output data from the production system in real time, and pre-process the collected data before storage.

[0027] As mentioned above, real-time acquisition: by establishing a communication connection with the control system (such as PLC, DCS) and sensor network of the production system, status data reflecting the production operation status, currently executed process parameter data and output data related to production results are obtained periodically or event-triggered.

[0028] Data preprocessing: Clean and standardize the raw collected data to eliminate noise, outliers, and format inconsistencies, improve data quality, and ensure its suitability for the training needs of deep reinforcement learning models.

[0029] Data storage: Store the processed data in a local or cloud database in a time series or other structured manner to form a dataset that can be used for offline training, online updates, and historical tracing.

[0030] This step not only ensures the integrity and timeliness of the data, but also provides continuous data flow support for dynamic learning at the model layer and real-time feedback at the application layer. It is a prerequisite for achieving closed-loop optimization.

[0031] For example, let’s take the continuous casting production line of a steel company as an example: Status data include: molten steel temperature in the tundish, vibration frequency of the crystallizer, cooling water flow distribution, real-time value of the casting speed, ambient humidity, etc. These data are uploaded in real time by temperature sensors, flow meters, vibration detectors and other equipment distributed at key nodes of the production line.

[0032] Process parameter data include: set casting speed target value, water spray volume setting value of each section in the secondary cooling zone, electromagnetic stirring intensity, etc. These parameters are usually sent to the actuator by the production control system (such as HMI or MES system).

[0033] Output data includes: the number of cracks on the surface of the ingot, the rating of internal inclusions, the results of dimensional tolerance inspection, the output per unit time, etc. Some of them are automatically collected through online inspection equipment (such as visual recognition systems), and some are entered by quality inspectors.

[0034] This data is connected to the data acquisition module via industrial Ethernet or serial communication protocols (such as Modbus and OPC UA). During the preprocessing phase, the system identifies a temperature reading jump due to a transient sensor failure (e.g., a sudden change from 1520°C to 3000°C). This is marked as an anomaly and interpolated from the previous and next time points. All numerical data is normalized to the range [0, 1], and redundant records are removed. Finally, the cleaned data is stored in a time series database by timestamp for subsequent model invocation.

[0035] It should be noted that in specific implementation scenarios, in addition to conventional sensor data, the above solution can also be used to access equipment health monitoring data (such as motor vibration spectrum, bearing temperature), energy consumption data (such as electricity consumption, gas usage) or external environmental information (such as workshop temperature and humidity, raw material batch quality reports) to enrich the state representation dimension and enhance the model's perception of complex working conditions.

[0036] In a specific implementation scenario, based on the above solution, a lightweight collection and preprocessing module can be deployed on the edge computing node close to the production site to achieve local caching and preliminary cleaning of data, and then upload key data to the central server, reducing network transmission pressure and improving system response speed.

[0037] In specific implementation scenarios, data integrity verification rules (such as breakpoint detection and sampling frequency deviation warning) can be introduced on the basis of the above solution. When it is found that a certain type of data has not been updated for a long time or the missing rate exceeds the threshold, an alarm will be automatically triggered and a log will be recorded to facilitate operation and maintenance personnel to troubleshoot communication failures or equipment abnormalities.

[0038] In specific implementation scenarios, the above solution can also be used to adaptively adjust the data collection frequency based on different production phases (such as startup, steady-state operation, and model changeover). For example, during periods of rapid system state changes, the sampling density can be increased to capture more dynamic features, while the frequency can be appropriately reduced during steady-state periods to conserve storage resources.

[0039] In specific implementation scenarios, based on the above solution, it is also possible to perform version control on the stored data sets and add semantic labels in combination with production events (such as process changes and equipment maintenance) to facilitate classification by working conditions during subsequent model training, supporting multi-scenario modeling and transfer learning.

[0040] In specific implementation scenarios, based on the above solution, data involving the core production parameters of the enterprise can be desensitized or encrypted before storage to ensure the security of the data during transmission and storage, in compliance with industrial information security specifications.

[0041] S2. Based on a deep reinforcement learning algorithm, use the data to train and dynamically update a process parameter optimization model; the process parameter optimization model is used to output compensation values ​​of the process parameters based on the status data of the current production system.

[0042] As described above, the initial training phase involves pre-training the process parameter optimization model offline using the historical data accumulated in step S1. This model uses the production system's state data as input, the compensation values ​​of the process parameters relative to the previous moment as output actions, and the production system's output indicators (such as output, quality score, and energy efficiency) as reward signals. This model constructs a Markov decision process based on reinforcement learning. By learning from a large number of historical state-action-reward sequences, the model initially grasps the appropriate adjustment strategies for different operating conditions.

[0043] During the online dynamic update phase, the trained initial model is deployed in the actual production environment, where it continuously interacts with the production system (i.e., the "environment") as an "agent." During operation, the system continuously collects new states, actions (i.e., actual application compensation values), and corresponding reward data, storing this data in an experience replay pool. Once a sufficient amount of new data has accumulated, the model update mechanism is activated, using this new data to retrain or fine-tune the model, gradually adapting it to the current operational characteristics and overcoming any discrepancies between historical data and real-world operating conditions.

[0044] The final process parameter optimization model has the ability to quickly output reasonable compensation values ​​based on real-time status data, thereby realizing dynamic and precise control of process parameters and improving the overall performance of the production system.

[0045] For example, let’s take the continuous polymerization reaction device of a chemical company as an example: State data include: reactor temperature, pressure, raw material feed flow rate, catalyst concentration, stirring rate, etc. The process parameters are the set temperature control target value, feed ratio, etc. The output data include quality indicators such as polymer molecular weight distribution, conversion rate, and by-product content.

[0046] Suppose the system is currently operating in steady state, but a change in raw material batches causes a decrease in reaction activity. Traditional static models may not be able to promptly identify this change. However, the process parameter optimization model in this invention, upon sensing state characteristics such as a slowing temperature rise rate and a decrease in conversion rate, outputs a compensation value combination of increasing the feed flow rate and raising the reaction temperature. This compensation value, after being constrained, is applied to the control system, prompting it to resume efficient operation and continuously collect feedback data during subsequent operations.

[0047] This new data was used to update the model, enabling it to learn the implicit rule that the temperature setting should be raised in advance when certain types of raw materials enter the system. After multiple rounds of iteration, the model gradually developed the ability to adapt to the characteristics of different raw materials, significantly outperforming fixed rules or static optimization models.

[0048] In addition, the model outputs a "compensation value" rather than an absolute parameter value. This design makes the adjustment smoother and more controllable, avoids system oscillations caused by large jumps, and facilitates compatibility with existing control systems.

[0049] It should be noted that in specific implementation scenarios, based on the above scheme, the reward function can also comprehensively consider multiple production objectives, such as optimizing product quality, unit energy consumption and equipment loss at the same time, and constructing a composite reward signal through weighted or layered methods to enable the model to achieve balanced decision-making among multiple objectives.

[0050] In specific implementation scenarios, based on the above solution, for systems with newly built production lines or replaced equipment, you can use models trained on existing similar production lines as the initial model, and quickly fine-tune it with a small amount of new production line data to shorten model convergence time and improve deployment efficiency.

[0051] In specific implementation scenarios, based on the above scheme, in addition to the feedforward neural network, a network structure with memory capabilities (such as recurrent neural network RNN, long short-term memory network LSTM) can be used as the model backbone according to the time series characteristics of the production process to enhance the modeling ability of historical state dependencies.

[0052] In specific implementation scenarios, the above solution can be used to add decision confidence or prediction variance information to the model output compensation value. When the confidence falls below a threshold, the system can automatically switch to manual intervention mode or a conservative strategy to improve operational safety.

[0053] In a specific implementation scenario, based on the above solution, you can first use a simulation environment or historical data for large-scale pre-training, and then perform small-step, high-frequency online fine-tuning in a real environment to form an "offline + online" collaborative training mechanism that takes into account both training efficiency and actual adaptability.

[0054] In specific implementation scenarios, in addition to triggering updates based on a fixed amount of data, the above solution can also dynamically trigger model retraining based on production events (such as shift changes, material changes, and equipment restarts) or performance degradation detection (such as a decrease in rewards for multiple consecutive cycles) to improve the targetedness and timeliness of updates.

[0055] In a specific implementation scenario, based on the above solution, the status data of multiple related processes can be uniformly input into the model, and the coordinated compensation values ​​of multiple process parameters can be output to achieve global optimization across processes rather than single-point local optimization.

[0056] S3. Calling the process parameter optimization model to generate process parameter compensation values ​​based on the real-time acquired production system status data. After constraint processing, target process parameters are obtained and input into the production system to guide production. Meanwhile, new data generated during the production process is fed back to step S1. In step S2, the dynamic update of the process parameter optimization model is achieved by maintaining a model library, wherein the model library includes at least an original model for current production and an updated model trained based on new data, and the model used for production is switched according to a predetermined update strategy.

[0057] As described above, model invocation and compensation value generation: During production system operation, current state data (such as temperature, pressure, and flow) is acquired in real time and fed into the currently active process parameter optimization model. This model, trained using deep reinforcement learning, determines the appropriate adjustment strategy based on the current system state and outputs a set of process parameter compensation values—fine-tuning adjustments to the currently set parameters (e.g., "+5°C" or "-0.3MPa").

[0058] Constraint processing: To ensure production safety and equipment stability, the compensation value output by the model must be constrained. This processing includes two levels: Compensation range limit: Set the maximum range of change for a single adjustment to prevent drastic parameter jumps due to model misjudgment; Boundary control: Ensure that the target process parameters after compensation do not exceed the upper and lower limits allowed by the equipment (such as maximum temperature resistance, maximum flow rate) to avoid over-limit operation.

[0059] The target process parameters obtained after processing are converted into standard control instructions and sent to the control layer of the production system (such as PLC or DCS system) through the communication interface to guide the next stage of production operation.

[0060] Data Feedback Closed Loop: After the target parameters are achieved, the system continues to collect new status data, actual process parameters, and corresponding output results, and transmits this new data back to the data acquisition and storage module in step S1. This mechanism forms a complete closed loop of "perception-decision-execution-learning", providing continuous data support for dynamic model updates.

[0061] Model library mechanism (dynamic update implementation in S2) In order to solve the conflict problem between online application and retraining of the model, the present invention introduces a model library mechanism in step S2 to achieve seamless switching and continuous evolution of the model.

[0062] The model library maintains at least two model instances: Original Model: The master model currently being used for production guidance; Updated model: A new version of the model that is independently trained based on the latest production data.

[0063] When the updated model training is completed, the system replaces the updated model with the new original model according to the predetermined update strategy (such as scheduled switching, switching after performance evaluation meets the standards, switching when production batches switch, etc.). The original model can be retained as a backup or for comparative analysis.

[0064] This mechanism ensures that there is always a stably running model serving production, while another model can be updated silently in the background, avoiding interference of the model training process on production control and improving the reliability and continuity of the system.

[0065] For example, let's take a cement clinker burning system as an example: During the current operation of the rotary kiln, status data indicates that the temperature at the kiln tail is low and the raw material feed rate fluctuates greatly. The system calls the currently activated process parameter optimization model. After inputting these status data, the model outputs the following compensation values: Pulverized coal feeding amount at kiln head: +8% Secondary air temperature setting value: +15℃ Speed ​​fine adjustment: +0.2 rpm During the constraint processing phase, the system detected a high load on the "pulverized coal feed rate." The single compensation limit was set at +6%, so it was adjusted to +6%. The remaining parameters were within the permitted range and passed without issue. The resulting target process parameters were then sent to the kiln system control system for combustion optimization.

[0066] During the next hour of operation, the system continuously collects new temperature curves, energy consumption data, clinker f-CaO content and other information, and uploads this data to the data layer for subsequent model updates.

[0067] Meanwhile, in the background, a new, updated model is being retrained based on new data accumulated over the past 24 hours. Once training is complete and stability verified, the system automatically switches to the new, original model at the start of the next production shift, placing the original model in a pending replacement state. This entire process requires no downtime or interruption to control, achieving a "zero-perception" model upgrade.

[0068] It should be noted that in a specific implementation scenario, the model library can be expanded to a set containing multiple candidate models based on the above solution. For example, special models under different working conditions (such as furnace startup model, steady-state model, load reduction model) can be maintained at the same time. The system automatically selects the most suitable model for use according to the current production stage to improve decision-making accuracy.

[0069] In specific implementation scenarios, based on the above solution, a switching mechanism based on performance evaluation can be set up. For example, switching is allowed only when the new model performs better than the current model by a certain threshold in simulation playback; or a grayscale release method is adopted to first test-run the new model on some production lines to verify its stability before full promotion.

[0070] In specific implementation scenarios, based on the above solution, when switching models or updating parameters, a gradual adjustment strategy (such as exponential weighted average) can be adopted to make the changes in process parameters smoother and reduce the impact on the production system. This is especially suitable for continuous process industries with extremely high stability requirements.

[0071] In specific implementation scenarios, based on the above solution, in critical processes or high-risk operations, the system can prompt the operator to confirm before outputting the target process parameters, or present the model suggestions as auxiliary decision-making options on the human-computer interface, realizing the integration of intelligence and human experience.

[0072] In a specific implementation scenario, based on the above solution, when a sudden fault is detected (such as sensor failure or actuator jamming), the system can automatically suspend model output, switch to preset safety parameters or manual mode, and record data during the abnormal period for subsequent model robustness training.

[0073] In specific implementation scenarios, based on the above solution, for multiple similar production lines within the same enterprise, a shared model library can be built on a unified platform to support model migration and adaptation between different production lines, reduce repeated training costs, and improve overall intelligent deployment efficiency.

[0074] In specific implementation scenarios, building on the above solution, a unique identifier and training log can be generated for each model update, supporting historical version queries and rapid rollbacks. If a new model causes performance degradation after launch, the previous stable version can be immediately restored to ensure production safety.

[0075] In specific implementation scenarios, based on the above solution, the target process parameters output by the model can be synchronized to the manufacturing execution system (MES) for production plan adjustment, energy consumption forecasting and quality traceability, thereby achieving a deep integration of process optimization and operation management.

[0076] According to one embodiment of the present application, the step S1 includes: Collecting status data reflecting the operating status of the production system, currently input process parameter data, and output data representing production results from the control system and sensor network of the production system; Standardize the collected data, remove outliers, delete duplicates, and fill in missing values; The processed data is stored in the database to form a historical data set.

[0077] As mentioned above, first, during the data acquisition phase, the system acquires three types of key data in real time from the production system's control system and sensor network: First, status data reflecting the current operating state of the production system, such as physical quantities like temperature, pressure, flow, speed, and vibration; second, process parameter data currently input into the production system, namely, the setpoints or operating instructions currently being executed by the control system; and third, output data representing production results, such as product quality indicators, output, energy consumption levels, and scrap rates. This data is read periodically or event-triggered from control devices such as PLCs, DCSs, and SCADA, or various sensors, using industrial communication protocols to ensure data synchronization and integrity.

[0078] Secondly, during the data preprocessing phase, the collected raw data undergoes a series of processing operations to improve data quality. These include: standardization, which unifies data of varying dimensions and magnitudes to the same scale, eliminating the impact of unit differences; outlier removal, which identifies and removes erroneous data that significantly deviates from the normal range, such as extreme values ​​caused by sensor failures; deduplication, which eliminates redundant records caused by communication retransmissions or excessively high acquisition frequencies; and missing value filling, which uses methods such as interpolation, previous and next value substitution, or statistical averaging to fill in gaps in data caused by brief communication interruptions or equipment failures. Through these processes, data accuracy, consistency, and availability are guaranteed.

[0079] Finally, during the data storage phase, the processed data is stored in a database in an orderly fashion, using timestamps or other logical structures. This creates a historical dataset that can be used for model training and analysis. This dataset not only supports offline training of the initial model but also provides continuous data input for subsequent online updates, forming the foundation for a closed-loop data loop for the entire optimization system.

[0080] According to one embodiment of the present application, the step S2 includes an offline training phase and an online update phase: The offline training stage obtains an initial process parameter optimization model based on historical data training; The online updating stage continuously corrects and updates the initial process parameter optimization model based on new data generated during the real-time operation of the production system.

[0081] As mentioned above, during the offline training phase, the system uses the historical data accumulated in step S1 as training samples to construct an initial process parameter optimization model. This historical data includes state data, process parameter data, and output data collected during past production processes. The model uses state data as input, the compensation value of the process parameter relative to the previous moment as the output action, and the production efficiency reflected by the output data as the reward signal, constructing the Markov decision process required for deep reinforcement learning. By simulating the interaction between the intelligent agent and the production environment in an offline environment, the model continuously adjusts its internal parameters, learns the optimal adjustment strategy to be adopted under different working conditions, and ultimately forms an initial process parameter optimization model with basic decision-making capabilities. After training, this model is deployed in the actual production environment and serves as the basis for subsequent online optimization.

[0082] During the online update phase, the initial process parameter optimization model has been put into actual operation and is involved in guiding the control decisions of the production system. During this process, the system continuously collects new data generated after the model is applied, including new status information, compensation values ​​output by the model, actual process parameters implemented, and corresponding output results. These new data are used to continuously revise and update the model after undergoing the same preprocessing process as step S1. The update process can be carried out independently in the background. By adding new data to the training set, the model is fine-tuned or retrained to gradually adapt it to the actual characteristics of the current production environment and correct model deviations caused by changes in working conditions, equipment aging, or differences in raw materials. This stage ensures that the model is not solidified on historical knowledge, but can continue to evolve with the evolution of the production process, maintaining the timeliness and accuracy of the optimization capabilities.

[0083] The offline training phase provides the model with initial capabilities, while the online update phase gives the model the ability to continue learning. The combination of the two enables the model to transition from static deployment to dynamic optimization, providing a guarantee for long-term and efficient regulation of process parameters.

[0084] According to one embodiment of the present application, during the offline training phase, a deterministic policy gradient algorithm based on an Actor-Critic architecture is used to train the initial process parameter optimization model.

[0085] According to one embodiment of the present application, the online update stage specifically includes: In the process of guiding production by the initial process parameter optimization model, the state data, process parameter data and output data of the production system are collected and stored in an experience replay pool after processing; When the number of samples stored in the experience replay pool reaches a preset capacity C, all the data in the pool are used to update the model and obtain a revised updated model; A copy of the updated model is made as a transit model and stored together with the current original model in the model library; the model with the latest generation time is selected from the model library as the original model for guiding the next production, and the model with the earliest generation time is eliminated.

[0086] As described above, while the initial process parameter optimization model is in operation and guiding the production system, the system continuously collects new state data, actual process parameter data, and corresponding output data generated during the model's control period. After undergoing the same preprocessing as step S1, this data is stored in a temporary storage area called the "experience replay pool." The experience replay pool is used to centrally manage recently accumulated valid training samples, ensuring that the data used for model updates is sufficiently representative and timely.

[0087] When the number of samples stored in the experience replay pool reaches the preset capacity threshold C, a model update operation is triggered. The system uses all the sample data in the pool to retrain or fine-tune the current initial process parameter optimization model. By reverse learning these actual operational decision-making experiences, the model parameters are modified to obtain a revised and updated model that adapts to the current working conditions.

[0088] To seamlessly integrate model updates with online applications, the system then copies the updated model to create a transition model. This transition model, along with the original model currently in use, is then integrated into the model library for unified management. The library maintains multiple versions of the process parameter optimization model, each with a time stamp.

[0089] When the next production decision cycle arrives, the system selects the model with the latest generation time from the model library as the new original model for subsequent process parameter optimization decisions. At the same time, the model with the earliest generation time in the model library is removed to keep the scale of the model library stable and ensure that the models used always reflect the latest production knowledge.

[0090] This mechanism avoids possible service interruptions or performance fluctuations during model updates, and enables continuous evolution and stable output of process parameter optimization capabilities.

[0091] According to one embodiment of the present application, the preset capacity C of the experience replay pool is a positive integer.

[0092] As mentioned above, the preset capacity C of the experience replay pool is a positive integer, which is used to set the minimum number of samples required to accumulate to trigger a model update. This value is reasonably configured based on the data acquisition frequency, operating condition change speed, and computing resource conditions of the actual production system. When the number of samples stored in the pool reaches the preset value, the system starts the model update process to ensure that the amount of data used for training is sufficient and representative, while avoiding unstable model updates due to insufficient data accumulation, or update delays due to excessive accumulation. The specific value of capacity C can be flexibly set according to different process scenarios to balance the timeliness of model updates and training efficiency.

[0093] According to one embodiment of the present application, the step S3 includes: Obtaining current state data of the production system and inputting it into the process parameter optimization model to obtain process parameter compensation values; Limiting the compensation value of the process parameter to obtain a constrained compensation value; Calculate the current target process parameters based on the constrained compensation value and the process parameters of the previous time unit, and perform upper and lower bound truncation processing on the target process parameters to ensure that they are within a preset safety range; The target process parameters after truncation are input into the production system, and the new status data, process parameter data and output data generated in this process are collected and fed back to step S1.

[0094] As mentioned above, the first step is to obtain the current state data of the production system, including real-time information reflecting the system's operating status, such as temperature, pressure, and flow rate. This state data is then fed into the currently active process parameter optimization model. The model, based on its internal strategy, outputs a set of process parameter compensation values—the adjustments that should be made to the current process parameters.

[0095] Next, to prevent excessive adjustments to the model output, which could cause system fluctuations or equipment shocks, the generated process parameter compensation values ​​are subject to range limits. Each compensation value is assigned a permissible range, and any values ​​outside this range are compressed to the upper or lower limit. This constrained compensation value ensures smooth and controllable adjustments.

[0096] The target process parameter for the current time unit is then calculated based on the constrained compensation value and the actual process parameter value from the previous time unit. To ensure production safety and stable equipment operation, the target process parameter is further truncated to its upper and lower bounds so that its final value falls within the preset safe operating range, avoiding anomalies or failures caused by parameter exceeding the limit.

[0097] Finally, the fully constrained and truncated target process parameters are converted into standard control instructions and input into the production system's control layer to guide the next stage of production operations. Meanwhile, during the parameter execution process, the system continuously collects new state data, actual process parameter data, and corresponding output data, and feeds this new data back to step S1 for subsequent data storage and model updates, forming a complete optimization closed loop.

[0098] According to one embodiment of the present application, the amplitude limitation is implemented by a constraint coefficient, and the constraint coefficient is adjusted according to the control capability of the production system control system; The minimum and maximum values ​​of the process parameters on which the upper and lower bound truncation processes are based are preset based on experience or expert knowledge.

[0099] As mentioned above, the amplitude limitation is achieved through a constraint coefficient, which is used to limit the maximum range of variation of the process parameter compensation value in a single adjustment to prevent fluctuations in the production system operation or shocks to the equipment due to excessive adjustment amplitude. The specific value of the constraint coefficient is set according to the responsiveness of the production system control system, the adjustment accuracy of the actuator, and the stability requirements of the process. For example, for systems with slow response or large inertia, a smaller constraint coefficient can be set to achieve smooth adjustment; for systems with rapid response, the constraint coefficient can be appropriately relaxed to improve adjustment efficiency. This coefficient can be dynamically adjusted or configured in stages according to actual operating conditions to adapt to control requirements under different working conditions.

[0100] The minimum and maximum values ​​of the process parameters used for the upper and lower bound truncation processing are pre-set according to the production process requirements, the safe operation boundaries of the equipment, and long-term accumulated experience or expert knowledge. These boundary values ​​represent the minimum and maximum operating limits allowed for each process parameter in actual production. For example, the temperature cannot exceed the heat resistance limit of the equipment, and the pressure cannot be lower than the threshold required for the system to maintain stability. After the target process parameters are generated, if their values ​​exceed the preset minimum or maximum value range, they will be forcibly adjusted to the corresponding boundary values ​​to ensure that the parameters finally issued are always within a safe and compliant operating range, thereby ensuring the stability and safety of the production process.

[0101] An embodiment of a second aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of the embodiments of the first aspect when executing the program.

[0102] Figure 2 An example of a physical structure diagram of an electronic device is shown below. Figure 2 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the method in any embodiment of the first aspect above, the method including: S1. Collect status data, process parameter data and output data from the production system in real time, and pre-process and store the collected data; S2. Based on a deep reinforcement learning algorithm, using the data to train and dynamically update a process parameter optimization model; the process parameter optimization model is used to output compensation values ​​for the process parameters based on the current state data of the production system; S3. Calling the process parameter optimization model to generate process parameter compensation values ​​based on the real-time acquired production system status data. After constraint processing, target process parameters are obtained and input into the production system to guide production. Meanwhile, new data generated during the production process is fed back to step S1. In step S2, the dynamic update of the process parameter optimization model is achieved by maintaining a model library, wherein the model library includes at least an original model for current production and an updated model trained based on new data, and the model used for production is switched according to a predetermined update strategy.

[0103] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as standalone products, stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories, random access memories, magnetic disks, or optical disks.

[0104] On the other hand, the present invention further provides a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the method provided by each of the above methods, including: S1. Collect status data, process parameter data and output data from the production system in real time, and pre-process and store the collected data; S2. Based on a deep reinforcement learning algorithm, using the data to train and dynamically update a process parameter optimization model; the process parameter optimization model is used to output compensation values ​​for the process parameters based on the current state data of the production system; S3. Calling the process parameter optimization model to generate process parameter compensation values ​​based on the real-time acquired production system status data. After constraint processing, target process parameters are obtained and input into the production system to guide production. Meanwhile, new data generated during the production process is fed back to step S1. In step S2, the dynamic update of the process parameter optimization model is achieved by maintaining a model library, which includes at least an original model for current production and an updated model trained based on new data, and the model used for production is switched according to a predetermined update strategy.

[0105] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for cigarette box image recognition provided by the above methods is implemented, and the method includes: S1. Collect status data, process parameter data and output data from the production system in real time, and pre-process and store the collected data; S2. Based on a deep reinforcement learning algorithm, using the data to train and dynamically update a process parameter optimization model; the process parameter optimization model is used to output compensation values ​​for the process parameters based on the current state data of the production system; S3. Calling the process parameter optimization model to generate process parameter compensation values ​​based on the real-time acquired production system status data. After constraint processing, target process parameters are obtained and input into the production system to guide production. Meanwhile, new data generated during the production process is fed back to step S1. In step S2, the dynamic update of the process parameter optimization model is achieved by maintaining a model library, which includes at least an original model for current production and an updated model trained based on new data, and the model used for production is switched according to a predetermined update strategy.

[0106] Example 2 The data layer collects, processes and stores data from the production system, providing data support for the process parameter optimization model training of the model layer; the model layer is mainly responsible for the training and correction and update of the initial process parameter optimization model, and provides the application layer with a highly adaptable process parameter optimization model; the application layer includes software that interacts with the production system control layer, and is responsible for applying the trained process parameter optimization model and displaying the operation process of the production system.

[0107] A process parameter optimization model library is constructed in the model layer. Through the continuous updating and exchange of three models, the conflicts between model application and model correction and update are overcome, thereby ensuring that the process parameter optimization model can always adapt to the production process of the production system. Among them, Model 1 is the original process parameter optimization model (it is the initial process parameter optimization model at the initial moment when the production system starts running), which is responsible for outputting process parameters in real time according to the operating status of the production system to guide the operation of the production system. Model 2, as a transit model, is copied from the updated and revised process parameter optimization model. After the copy is completed, it replaces the original process parameter optimization model. Model 3 is a process parameter optimization model that is continuously updated and revised through the production system operation data. It can continuously learn knowledge in the production system to ensure the continued efficiency of the process parameter optimization model. The specific update and exchange process of the three models is described as follows: (1) When the original process parameter optimization model (the model is the above-mentioned model 1) guides the production process, the state data, process parameter data and output data can be collected from the production system. Including the current time unit Status data, forward The data of time units form a time window size of Status data , the data of the process parameters to be compensated in the current time unit , and the output data of the current time unit , the time window size of the next time unit is Status data .

[0108] (2) As the production process progresses, the data collected Deposit into the experience replay pool In [1], the sample capacity of the experience replay pool is set to C. When the capacity of the experience replay pool reaches the upper limit, all sample data is removed to update the process parameter optimization model and the experience replay pool capacity is cleared. The updated model is Model 3, which is copied to form Model 2. Each time, the process parameter optimization model with the most recent generation time is selected from the model library as the original process parameter optimization model to guide the production process. The model with the most recent generation time is replaced by the model with the most recent generation time.

[0109] The application layer includes software that interacts with the production control system, connected via a serial port. Software functions include setting upper and lower bounds for process parameters, setting limits on process parameter compensation, dynamically displaying status data and process parameters, dynamically displaying production system output data, and training an initial process parameter optimization model.

[0110] Among them, when applying the process parameter optimization model, in order to avoid the adverse effects of the model's performance fluctuations on the production system's operating performance and to ensure the stability of the production system, the process parameter compensation and the process parameters are restricted by upper and lower limits. The restriction formulas are shown in (1)-(3). The restriction processing formula for process parameter compensation is shown in formula (1). Indicates the Process parameters output by the time unit process parameter optimization model Compensation value; Indicates the Time unit process parameters The value of It represents the constraint coefficient, which is a hyperparameter and can be adjusted according to the control system capability in the production system. If the control system has strong control capability, a larger value can be set. Otherwise, set a smaller value value; After the constraint Time unit process parameters To ensure the stability of the production system, the process parameters are truncated. The truncation range can be confirmed in advance by experienced employees or experts. The truncation operation is shown in formula (2). Indicates the The value of the process parameter in time unit; Indicates the The value of the process parameter after the time unit is processed by formula (1) is solved as shown in formula (3). In formula (3), Indicates the number of process parameters; Indicates the minimum value to which the process parameter can be set; Indicates the maximum value that the process parameter can be set to.

[0111] (1) (2) (3) Example 3 like Figure 3As shown, the construction of the data layer includes three modules: data acquisition, data storage, and data processing. The data acquisition module collects status data, process parameter data, and output data of the production system from the control system, sensor network and other devices of the production system; and then stores the above data in the database to form a data storage module.

[0112] One of the purposes of the data collected by the data layer is to train the process parameter optimization model. Therefore, the collected data is processed. The processing operations include data standardization, outlier removal, duplicate value deletion, and missing value filling. The processed data is also stored in the data storage module.

[0113] The construction of the model layer includes the training of the initial process parameter optimization model and its online update and correction.

[0114] The process of training the initial process parameter optimization model includes: (1) Mining the operating logic of the production system based on data from the data layer. This includes training a relationship model between state and process parameter data with a time window size of and future unit time state data based on a deep learning algorithm. This model is defined as a state transition model; and training a relationship model between process parameters and output data. This model is defined as an evaluation model.

[0115] (2) Build a feedforward neural network to represent the process parameter optimization model. The input is the state data with a time window size of K, and the output is the compensation value of the process parameter data in the future time unit.

[0116] (3) The compensation of process parameters is defined as action, the state transition model and the evaluation model constitute the environment, the output value of the evaluation model is defined as reward, the state data within the time window of size K is defined as state, the state data within the time window after the time window slides for one time unit is defined as the next state, and the process parameter optimization model is defined as the agent.

[0117] (4) Based on the state, action, reward, environment and agent defined in (3), the process parameter optimization model is trained based on deep reinforcement learning. The elements required for training the initial process parameter optimization model based on the deep reinforcement learning algorithm are as follows: Figure 4 When training the process parameter optimization model, the number of state transitions should not be too long, and the time span between the initial and final states should not be too large (this application provides a maximum reference value of 64). This setting is defined as the number of rounds of interaction between the agent and the environment. The reason for the above setting is to reduce the problem of excessive error accumulation in the state transition model, which leads to poor adaptability of the trained initial process parameter optimization model when guiding the operation of the production system.

[0118] After the initial process parameter optimization model is trained, there is a discrepancy between the state transition model trained using historical data and the state transition of the production system in the real production environment. This will cause the state data generated by the continuous interaction between the agent and the environment to be used to train the initial process parameter optimization model to deviate from the state data generated by the actual operation of the production system. Therefore, it is necessary to continuously update and revise the model in the real production environment. The update and revision flowchart is as follows: Figure 5 The specific contents are described as follows: (1) The initial process parameter model is used as the original process parameter optimization model, and the process parameter compensation value is generated according to the state data, and then the process parameters of the next time unit are determined, including the current time unit Status data, forward The data of time units form a time window size of Status data In order to avoid the adverse effects of model performance fluctuations on the performance of the production system and to ensure the stability of the production system, the upper and lower bounds of process parameter compensation and process parameter setting are restricted. The restriction formulas are shown in Equations (1), (2) and (3).

[0119] (2) During the production process guided by the original process parameter optimization model, status, process parameters, and output data can be collected from the production system. The collected data is processed according to the data processing method in the data layer and then stored in the experience replay pool. The sample capacity of the experience replay pool is set to a positive integer C. When the sample capacity in the experience replay pool reaches the upper limit C, all data is used to update the process parameter model and form an updated process parameter optimization model. The sample data in the experience replay pool is transferred to the data storage module in the data layer.

[0120] (3) The updated process parameter model is copied to form a model including the original process parameter optimization model, the copied and updated process parameter optimization model, and the updated process parameter optimization model. The process parameter optimization model used next time is the process parameter optimization model with the closest time tag in the model library, and the process parameter optimization model with the furthest time tag is eliminated.

[0121] The construction of the application layer includes the development of software and the connection with the control system and sensor network, with the purpose of intuitive use for customers and improving the intelligence level of the production system. The software can be connected to the control system and sensor network through the serial port. The functions of the software include setting the upper and lower limits of process parameters, setting the limit of process parameter compensation size, dynamic display of status data and process parameters, dynamic display of production system output data, and setting the initial process parameter optimization model training. The operating logic of the application layer software is as follows: Figure 6The specific process is as follows. The compensation of the process parameters output by the process parameter optimization model and the processing method of the compensated process parameters refer to formulas (1)-(3).

[0122] (1) During the operation of the production system, status data is output per unit time and fed back to the application layer. If the initial process parameter optimization model needs to be updated, the parameter values ​​for training the process parameter optimization model are set and the training process parameter model is started. After the model training is completed, the status data is output to the process parameter optimization model and the compensation value of each process parameter is output.

[0123] (2) Then, set the constraints on the process parameter compensation value at the application layer, that is, value; and set the process parameter constraints, that is, Value and The above constraint values ​​can also be set in advance.

[0124] (3) Output the process parameters after the decision and input them into the production system to guide its operation. In addition, the application layer dynamically monitors the production system status data, process parameter data, and production system output data to prevent the model from having adverse effects on the production system when applied.

[0125] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0126] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0127] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A process parameter optimization method based on deep reinforcement learning, characterized in that: include: S1. Collect status data, process parameter data and output data from the production system in real time, and pre-process and store the collected data; S2. Based on a deep reinforcement learning algorithm, using the data to train and dynamically update a process parameter optimization model; The process parameter optimization model is used to output compensation values ​​of process parameters according to the status data of the current production system; S3. Calling the process parameter optimization model to generate process parameter compensation values ​​based on the real-time acquired production system status data. After constraint processing, target process parameters are obtained and input into the production system to guide production. Meanwhile, new data generated during the production process is fed back to step S1. In step S2, the dynamic update of the process parameter optimization model is achieved by maintaining a model library, which includes at least an original model for current production and an updated model trained based on new data, and the model used for production is switched according to a predetermined update strategy.

2. The method according to claim 1, characterized in that The S1 step includes: Collecting status data reflecting the operating status of the production system, currently input process parameter data, and output data representing production results from the control system and sensor network of the production system; Standardize the collected data, remove outliers, delete duplicates, and fill in missing values; The processed data is stored in the database to form a historical data set.

3. The method according to claim 1, characterized in that The S2 step includes an offline training phase and an online update phase: The offline training stage obtains an initial process parameter optimization model based on historical data training; The online updating stage continuously corrects and updates the initial process parameter optimization model based on new data generated during the real-time operation of the production system.

4. The method according to claim 3, characterized in that During the offline training phase, a deterministic policy gradient algorithm based on an Actor-Critic architecture is used to train the initial process parameter optimization model.

5. The method according to claim 3, characterized in that The online update phase specifically includes: In the process of guiding production by the initial process parameter optimization model, the state data, process parameter data and output data of the production system are collected and stored in an experience replay pool after processing; When the number of samples stored in the experience replay pool reaches a preset capacity C, all the data in the pool are used to update the model and obtain a revised updated model; A copy of the updated model is made as a transit model and stored together with the current original model in the model library; the model with the latest generation time is selected from the model library as the original model for guiding the next production, and the model with the earliest generation time is eliminated.

6. The method according to claim 5, characterized in that The preset capacity C of the experience replay pool is a positive integer.

7. The method according to claim 1, characterized in that The S3 step includes: Obtaining current state data of the production system and inputting it into the process parameter optimization model to obtain process parameter compensation values; Limiting the compensation value of the process parameter to obtain a constrained compensation value; Calculate the current target process parameters based on the constrained compensation value and the process parameters of the previous time unit, and perform upper and lower bound truncation processing on the target process parameters to ensure that they are within a preset safety range; The target process parameters after truncation are input into the production system, and the new status data, process parameter data and output data generated in this process are collected and fed back to step S1.

8. The method according to claim 7, characterized in that The amplitude limitation is achieved by a constraint coefficient, which is adjusted according to the control capability of the production system control system; The minimum and maximum values ​​of the process parameters on which the upper and lower bound truncation processes are based are preset based on experience or expert knowledge.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method according to any one of claims 1 to 8 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 8 are implemented.

Images