A papermaking process optimization system and method based on deep reinforcement learning
Through the deep reinforcement learning system combining physical and data models, the control parameters of the papermaking process are automatically optimized, which solves the problem of insufficient scientificity and flexibility of papermaking process optimization in the existing technology, and realizes multi-objective collaborative optimization of the papermaking process, improving the Paper quality and efficiency.
Patent Information
- Application Number
- CN202311806960.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-12-26
AI Technical Summary
The existing papermaking process optimization methods lack scientificity and flexibility, and are difficult to adapt to the complexity and dynamicity of the papermaking process. Traditional mathematical models are difficult to effectively deal with nonlinearity, time-varying and uncertainty, and ignore the multi-objective and multi-partial relationships. mutual influence.
The papermaking process optimization system based on deep reinforcement learning is adopted, and the deep neural network model is automatically learned and generated control parameters, combined with physical and data models, and multi-objective and multi-part collaborative optimization of the papermaking process is achieved.
It realizes automated and intelligent optimization control of the papermaking process, improves paper quality, reduces energy consumption and costs, and improves the efficiency and competitiveness of the papermaking industry.
Smart Images

Figure CN117742273B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of papermaking process optimization, and specifically relates to a papermaking process optimization system and method based on deep reinforcement learning. Background Art
[0002] Papermaking is an industrial process that converts fiber raw materials into paper, involving multiple steps and parts, such as pulp preparation, paper forming, paper pressing, paper drying and paper finishing. The quality, efficiency and sustainability of the papermaking process are affected by many factors, such as the nature of the raw materials, process parameters, equipment status, environmental conditions, etc. Therefore, the optimization of the papermaking process is a challenging and important task, which aims to improve the quality of paper, reduce the energy consumption and cost of papermaking, and reduce the environmental impact of papermaking.
[0003] At present, the optimization of papermaking process mainly relies on experience and rules, or adopts traditional mathematical models and control methods. However, these methods have some limitations, such as:
[0004] The empirical and rule-based methods lack scientificity and flexibility, cannot adapt to the complexity and dynamics of the papermaking process, and cannot make full use of the large amount of data generated in the papermaking process;
[0005] Traditional mathematical models and control methods require accurate modeling and parameter estimation of the papermaking process, but the nonlinearity, time-varying and uncertainty of the papermaking process make this process difficult and time-consuming, and the errors and incompleteness of the model will affect the optimization effect;
[0006] Traditional mathematical models and control methods usually only optimize a single goal or a single part of the papermaking process, while ignoring the mutual influence and synergistic effects between multiple goals and multiple parts of the papermaking process.
[0007] Therefore, it is necessary to develop a new papermaking process optimization system that can overcome the above limitations, achieve multi-objective and multi-part collaborative optimization of the papermaking process, and improve the intelligence level of the papermaking process. Summary of the invention
[0008] This embodiment discloses a papermaking process optimization system based on deep reinforcement learning, which can automatically learn and generate control parameters of the papermaking process based on the data of the papermaking process through a deep neural network model, thereby improving the quality of paper and reducing the energy consumption and cost of the papermaking process. The system includes the following modules:
[0009] Papermaking process module, used to simulate the physical and chemical changes in the papermaking process, including pulp preparation, paper forming, paper pressing, paper drying and paper finishing;
[0010] Data collection module, used to collect various data in the papermaking process module, including paper quality data and energy consumption and cost data of the papermaking process;
[0011] The deep reinforcement learning module is used to learn the dynamic characteristics and laws of the papermaking process through a deep neural network model based on the data collected by the data acquisition module, thereby generating action selections for the papermaking process;
[0012] The optimization control module is used to obtain the control parameters of the papermaking process according to the action selection generated by the deep reinforcement learning module and the predefined optimization objective function and constraint condition function, and optimize the control of each step and part in the papermaking process module based on the control parameters.
[0013] The beneficial effect of this implementation scheme is that, through the coordinated work of the above modules, it is possible to achieve automated and intelligent optimization control of the papermaking process, improve the quality of paper, reduce the energy consumption and cost of the papermaking process, and improve the efficiency and competitiveness of the papermaking industry.
[0014] In a preferred implementation, the papermaking process module specifically includes:
[0015] The physical model submodule is used to establish a mathematical model of the papermaking process based on the physical principles and chemical reactions of the papermaking process, and to describe the input-output relationship and internal state changes of the papermaking process;
[0016] The data-driven submodule is used to establish a data model of the papermaking process based on the actual data of the papermaking process through machine learning or data mining methods to supplement the nonlinear and uncertain characteristics of the papermaking process;
[0017] The model fusion submodule is used to generate a hybrid model of the papermaking process through a model fusion method based on the mathematical model and data model of the papermaking process obtained by the physical model submodule and the data driven submodule.
[0018] The beneficial effect of this implementation scheme is that, through the coordinated work of the above sub-modules, it can effectively simulate the physical and chemical changes of the papermaking process, improve the accuracy and adaptability of the papermaking process model, provide high-quality model input for the subsequent deep reinforcement learning module, and improve the learning and optimization capabilities of the deep reinforcement learning module.
[0019] In a preferred implementation, the deep reinforcement learning module specifically includes:
[0020] The state representation submodule is used to preprocess and extract features of the data collected by the data acquisition module to obtain the state representation of the papermaking process, including the quality state of the paper, the energy consumption state of the papermaking process and the cost state of the papermaking process;
[0021] The action selection submodule is used to generate action selection of the papermaking process according to the state representation of the papermaking process through a deep neural network model, including the actions of pulp preparation, paper forming, paper pressing, paper drying and paper finishing;
[0022] The reward function submodule is used to calculate the reward value of the papermaking process according to the state representation and action selection of the papermaking process through a predefined reward function, including the quality reward of the paper, the energy consumption reward of the papermaking process, and the cost reward of the papermaking process;
[0023] The strategy updating submodule is used to update the strategy parameters of the papermaking process through a deep neural network model according to the state representation, action selection and reward value of the papermaking process.
[0024] The beneficial effect of this implementation scheme is that, through the collaborative work of the above sub-modules, the deep neural network model can be effectively utilized to automatically learn and optimize the control strategy of the papermaking process, improve the degree to which the optimization objectives and constraints of the papermaking process are satisfied, provide high-quality strategies and parameter outputs for subsequent optimization control modules, and improve the control execution capabilities of the optimization control modules.
[0025] In a preferred implementation, the optimization control module specifically includes:
[0026] The model prediction submodule is used to predict the future state of the papermaking process, including the quality state of the paper, the energy consumption state of the papermaking process, and the cost state of the papermaking process through the mathematical model in the papermaking process module according to the action selection generated by the deep reinforcement learning module;
[0027] The optimization target submodule is used to calculate the optimization target value of the papermaking process according to the future state of the papermaking process through a predefined optimization target function, including the quality target of the paper, the energy consumption target of the papermaking process and the cost target of the papermaking process;
[0028] The constraint condition submodule is used to calculate the constraint condition values of the papermaking process according to the future state of the papermaking process through the predefined constraint condition function, including the quality constraint of the paper, the energy consumption constraint of the papermaking process and the cost constraint of the papermaking process;
[0029] The control parameter submodule is used to obtain the control parameters of the papermaking process by solving the optimization problem according to the optimization target value and constraint condition value of the papermaking process, including the control parameters of pulp preparation, paper forming, paper pressing, paper drying and paper finishing;
[0030] The control execution submodule is used to control and execute various steps and parts in the papermaking process module according to the control parameters obtained by the optimization control module, so as to achieve optimized control of the papermaking process.
[0031] The beneficial effects of this implementation scheme are that it can predict and analyze the future state of the papermaking process, providing a basis for setting optimization goals and constraints; it can obtain the optimal control parameters of the papermaking process by solving the optimization problem, thereby improving the quality of the papermaking process and reducing energy consumption, and improving the efficiency and economy of the papermaking process; it can optimize the control of each step and part of the papermaking process through the control execution method, thereby realizing the automation and intelligence of the papermaking process.
[0032] In a preferred implementation, the control parameter submodule uses a genetic algorithm to solve the optimization problem.
[0033] The beneficial effects of this implementation scheme are that the genetic algorithm is a global optimization method based on natural selection and genetic mechanisms, which can find multiple optimization solutions in the search space, thereby achieving multi-objective optimization of the papermaking process; the genetic algorithm has parallelism, robustness and adaptability, and can effectively handle the nonlinear, multi-peak and dynamic characteristics of the papermaking process, and improve the efficiency and quality of solving optimization problems.
[0034] In a preferred implementation, the system also includes an optimization evaluation module, which is used to evaluate and provide feedback on the execution effect of the control parameters in the optimization control module based on the data collected by the data acquisition module and the optimization objectives of the papermaking process, thereby providing a basis for learning and updating for the deep reinforcement learning module.
[0035] The beneficial effect of this implementation scheme is that it realizes effective evaluation and feedback of the execution effect of the control parameters of the papermaking process, thereby providing a basis for learning and updating the deep reinforcement learning module, and realizing the adaptive and continuous improvement of the optimized control of the papermaking process.
[0036] In a preferred implementation, the optimization assessment module specifically includes:
[0037] The optimization definition submodule is used to define the optimization indicators of the papermaking process according to the optimization objectives of the papermaking process, including the quality indicators of the paper and the energy consumption indicators and cost indicators of the papermaking process;
[0038] The optimization evaluation submodule is used to calculate the execution effect of the control parameters in the optimization control module according to the data collected by the data acquisition module and the optimization indicators defined by the optimization definition submodule, including the improvement degree and weight of the control parameters on each optimization indicator. The execution effect of the control parameters is calculated using the following formula:
[0039]
[0040] Among them, E(s,a) is a measure of the execution effect of the control parameters of the papermaking process, w i is the weight coefficient of the optimization index, I i (s,a) is the current value of the i-th optimization index of the papermaking process, is the optimal value of the i-th optimization index of the papermaking process, n is the number of optimization indexes of the papermaking process, a is the action selection vector, and s is the state representation vector;
[0041] The optimization feedback submodule is used to generate evaluation results of the control parameters, including the overall score and optimization suggestions of the control parameters, according to the execution effects of the control parameters calculated by the optimization evaluation submodule, and transmit them to the deep reinforcement learning module and the user.
[0042] The beneficial effect of this embodiment is to achieve effective definition and calculation of optimization indicators of the papermaking process, thereby providing users with scores and suggestions for optimization control of the papermaking process, thereby improving user satisfaction and trust.
[0043] In a preferred embodiment, the system further includes a fault diagnosis module for detecting and locating abnormal conditions in the papermaking process based on the data collected by the data collection module and the optimization target of the papermaking process.
[0044] The beneficial effect of this embodiment is that the fault diagnosis module can timely discover faults and abnormalities in the papermaking process, thereby improving the safety and reliability of the papermaking process.
[0045] In a preferred implementation, the fault diagnosis module specifically includes:
[0046] A fault detection submodule is used to determine whether there is a fault in the papermaking process, as well as the type and severity of the fault through an abnormality detection method based on the data collected by the data acquisition module;
[0047] The fault location submodule is used to determine the location and cause of the fault in the papermaking process through a fault tree analysis method according to the fault type and severity determined by the fault detection submodule;
[0048] The fault handling submodule is used to generate fault handling measures and suggestions based on the fault location and cause determined by the fault locating submodule through a fault recovery method, and transmit them to the optimization control module and the user.
[0049] The beneficial effects of this implementation scheme are that the fault diagnosis module can accurately locate the location and cause of the fault, providing a basis for fault handling; the fault diagnosis module can generate measures and suggestions for fault handling, providing guidance for fault recovery; the fault diagnosis module can work in conjunction with the optimization control module to achieve fault tolerance and self-healing of the papermaking process.
[0050] The present invention also provides a papermaking process optimization method based on deep reinforcement learning, which comprises the following steps:
[0051] Step 1: Simulate the physical and chemical changes of the papermaking process, including pulp preparation, paper forming, paper pressing, paper drying and paper finishing;
[0052] Step 2: Collect various data in the papermaking process, including paper quality data and energy consumption and cost data of the papermaking process;
[0053] Step 3: Based on the collected data, the dynamic characteristics and rules of the papermaking process are learned through a deep neural network model to generate action selections for the papermaking process;
[0054] Step 4: according to the generated action selection and the predefined optimization objective function and constraint condition function, the control parameters of the papermaking process are obtained, and each step and part of the papermaking process is optimized and controlled based on the control parameters;
[0055] Step 5: Based on the collected data and the optimization objectives of the papermaking process, the execution effects of the control parameters in the optimization control are evaluated and feedback is provided, thereby providing a basis for learning and updating the deep reinforcement learning model;
[0056] Step 6: Repeat steps 2 to 5 until the preset optimization termination condition is reached or the user's termination instruction is issued.
[0057] The beneficial effects of this implementation scheme are as follows: through the optimization method of this implementation scheme, intelligent analysis and processing of papermaking process data are achieved, thereby improving the optimization effect and control accuracy of the papermaking process; by optimizing and controlling each step and part of the papermaking process, the quality of the papermaking process is improved and the energy consumption is reduced, thereby improving the efficiency and economy of the papermaking process; by evaluating and providing feedback on the execution effect of the control parameters in the optimization control, continuous learning and updating of the control parameters of the papermaking process are achieved, thereby improving the optimization capability and adaptability of the papermaking process. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 This is a schematic diagram of the system structure in Embodiment 1 of the present invention;
[0059] Figure 2 Schematic diagram of the system structure in Embodiment 2-4 of the present invention;
[0060] Figure 3 is a schematic diagram of the system structure in Embodiment 5 of the present invention;
[0061] Figure 4 Schematic diagram of the system structure in Embodiment 6 of the present invention;
[0062] Figure 5 This is a flow chart of the method in Example 7 of the present invention. DETAILED DESCRIPTION
[0063] In order to enable those skilled in the art to better understand the technical solution, the technical solution is described in detail below in conjunction with the embodiments. The description in this section is only exemplary and explanatory and should not have any limiting effect on the scope of protection of the present invention.
[0064] Embodiment 1:
[0065] This embodiment provides a papermaking process optimization system based on deep reinforcement learning, and its structural diagram is shown in FIG. Figure 1 As shown, it includes the following modules:
[0066] The papermaking process module is used to simulate the physical and chemical changes in the papermaking process, including pulp preparation, paper forming, paper pressing, paper drying and paper finishing.
[0067] Specifically, this module uses a mathematical model based on physical equations and empirical formulas to simulate various stages of the papermaking process, including pulp preparation, paper forming, paper pressing, paper drying and paper finishing. The input of this module is the raw material, proportion, quality and other parameters of the pulp, as well as the speed, temperature, humidity, pressure and other parameters of the paper machine. The output of this module is indicators such as paper quality, energy consumption and cost.
[0068] The data acquisition module is used to collect various data in the papermaking process module, including paper quality data and energy consumption and cost data of the papermaking process.
[0069] Specifically, the data acquisition module is used to collect various data in the papermaking process module, including quality data such as paper gloss, smoothness, thickness, strength, whiteness, and energy consumption and cost data such as power consumption, water consumption, steam consumption, and chemical consumption in the papermaking process. The data acquisition module can realize real-time, continuous, and accurate monitoring and collection of the papermaking process, obtain the operating status and effect of the papermaking process, and provide basic data for subsequent deep reinforcement learning and optimization control.
[0070] The deep reinforcement learning module is used to learn the dynamic characteristics and laws of the papermaking process through a deep neural network model based on the data collected by the data acquisition module, thereby generating action selections for the papermaking process.
[0071] Specifically, the deep reinforcement learning module includes a deep neural network and a reinforcement learning algorithm. The deep neural network is used to perform nonlinear mapping and feature extraction on the data of the papermaking process. The reinforcement learning algorithm is used to update and optimize the parameters of the deep neural network according to the optimization objectives and constraints of the papermaking process, as well as the reward and penalty mechanism of the papermaking process, so as to generate action selections for the papermaking process. The deep reinforcement learning module can realize intelligent and adaptive learning and optimization of the papermaking process, thereby improving the efficiency and quality of the papermaking process.
[0072] The optimization control module is used to obtain the control parameters of the papermaking process according to the action selection generated by the deep reinforcement learning module and the predefined optimization objective function and constraint condition function, and optimize the control of each step and part in the papermaking process module based on the control parameters.
[0073] Specifically, the optimization control module includes an optimization controller and an actuator. The optimization controller is used to obtain the control parameters of the papermaking process, such as the raw materials, proportions, quality and other parameters of the pulp, and the speed, temperature, humidity, pressure and other parameters of the papermaking machine, based on the action selection generated by the deep reinforcement learning module and the predefined optimization objective function and constraint condition function. The actuator is used to optimize the various steps and parts in the papermaking process module based on the control parameters, such as adjusting the preparation of pulp, the forming of paper, the pressing of paper, the drying of paper and the finishing of paper, so as to achieve the purpose of improving the quality of paper and reducing the energy consumption and cost of papermaking. The optimization control module can realize comprehensive and accurate optimization control of the papermaking process, thereby improving the performance and efficiency of the papermaking process.
[0074] This embodiment adopts a papermaking process optimization system based on deep reinforcement learning, which can realize comprehensive, automated and intelligent optimization and control of the papermaking process, thereby improving the efficiency and quality of the papermaking process.
[0075] Embodiment 2:
[0076] Based on Example 1, this example implements a specific papermaking process module to improve the modeling and optimization capabilities of the papermaking process.
[0077] like Figure 2 As shown in the figure, the papermaking process module specifically includes the following three sub-modules:
[0078] The physical model submodule is used to establish a mathematical model of the papermaking process based on the physical principles and chemical reactions of the papermaking process, and to describe the input-output relationship and internal state changes of the papermaking process. For example, the physical model submodule can use basic equations such as conservation of mass, conservation of energy, conservation of momentum, and empirical relationships such as the rheology of pulp, water absorption of paper, and shrinkage of paper to establish a differential equation group of the papermaking process, thereby describing the dynamic behavior and steady-state characteristics of the papermaking process.
[0079] The data-driven submodule is used to establish a data model of the papermaking process based on actual data in the papermaking process through machine learning or data mining methods to supplement the nonlinear and uncertain characteristics of the papermaking process. For example, the data-driven submodule can use machine learning algorithms such as neural networks, support vector machines, random forests, or data mining methods such as clustering, association, and classification to extract data features of the papermaking process from historical data of the papermaking process, thereby establishing a data model of the papermaking process for predicting, estimating, or compensating for unknown or unmeasurable variables or parameters of the papermaking process.
[0080] The model fusion submodule is used to generate a hybrid model of the papermaking process by a model fusion method based on the mathematical model and data model of the papermaking process obtained by the physical model submodule and the data-driven submodule. For example, the model fusion submodule can combine or integrate the mathematical model and data model of the papermaking process by using a model fusion method such as weighted averaging, weighted voting, stacking, fusion, etc., so as to obtain a hybrid model of the papermaking process, which is used to improve the model accuracy and generalization ability of the papermaking process.
[0081] In this embodiment, the model fusion submodule adopts a papermaking process model fusion technology based on weighted average, which includes the following steps:
[0082] According to the mathematical model and data model of the papermaking process obtained by the physical model submodule and the data driven submodule, the predicted values of the output variables of the papermaking process, such as the quality of paper, the energy consumption of the papermaking process and the cost of the papermaking process, are calculated respectively;
[0083] According to the fitting degree and prediction ability of the mathematical model and data model of the papermaking process obtained by the physical model submodule and the data driven submodule, the weights of the mathematical model and data model of the papermaking process are determined respectively, for example, the weight of the physical model is 0.6, and the weight of the data model is 0.4;
[0084] Based on the predicted values and weights of the mathematical model and data model of the papermaking process, the weighted average method is used to calculate the comprehensive predicted values of the output variables of the papermaking process, such as the quality of paper, the energy consumption of the papermaking process, and the cost of the papermaking process;
[0085] According to the comprehensive predicted values of the output variables of the papermaking process, a hybrid model of the papermaking process is generated as the final model of the papermaking process.
[0086] This embodiment adopts a papermaking process optimization system based on deep reinforcement learning. The system implements the papermaking process module in detail. According to the physical principles and chemical reactions of the papermaking process and the actual data in the papermaking process, a mathematical model, a data model and a hybrid model of the papermaking process are established, thereby achieving efficient and accurate simulation and description of the papermaking process.
[0087] Embodiment 3:
[0088] Based on Example 1, this example specifically implements the deep reinforcement learning module to improve the optimization capability and effect of the papermaking process.
[0089] like Figure 2 As shown in the figure, the deep reinforcement learning module specifically includes the following four sub-modules:
[0090] The state representation submodule is used to preprocess and extract features of the data collected by the data acquisition module to obtain the state representation of the papermaking process, including the quality status of the paper, the energy consumption status of the papermaking process and the cost status of the papermaking process. The state representation submodule can use standardization, normalization, dimensionality reduction and other methods to convert the data collected by the data acquisition module into a data format suitable for the input of the deep neural network model, such as a vector or matrix, so as to obtain the state representation of the papermaking process.
[0091] The action selection submodule is used to generate action selection of the papermaking process according to the state representation of the papermaking process through a deep neural network model, including the action of pulp preparation, paper forming, paper pressing, paper drying and paper finishing. The function of this submodule is to select an action that is most conducive to the optimization target according to the current state of the papermaking process, such as adjusting the proportion of pulp, the thickness of paper, the humidity of paper, etc. The action selection submodule can be implemented by methods such as multi-layer perceptron or convolutional neural network to improve the selectivity and adaptability of the action.
[0092] In this embodiment, the action selection submodule adopts an action selection method based on a multi-layer perceptron, which includes the following steps:
[0093] According to the state representation of the papermaking process, the input layer of the multilayer perceptron is constructed, which includes three input nodes, corresponding to the quality state of the paper, the energy consumption state of the papermaking process, and the cost state of the papermaking process;
[0094] According to the action selection of the papermaking process, the output layer of the multilayer perceptron is constructed, which includes five output nodes, corresponding to the actions of pulp preparation, paper forming, paper pressing, paper drying and paper finishing.
[0095] According to the nonlinear relationship between the state representation and action selection of the papermaking process, a hidden layer of a multilayer perceptron is constructed, including several hidden nodes, each of which is connected to all nodes of the input layer and the output layer and has different weights and biases;
[0096] According to the state representation of the papermaking process and the historical data of action selection, the back propagation algorithm is used to train the parameters of the multilayer perceptron so that the error between the output value of the multilayer perceptron and the true value is minimized, and the action selection model is obtained;
[0097] According to the current value represented by the state of the papermaking process, the action selection model is used to predict the current value of the action selection of the papermaking process, including the action of pulp preparation, the action of paper forming, the action of paper pressing, the action of paper drying and the action of paper finishing.
[0098] The reward function submodule is used to calculate the reward value of the papermaking process, including the quality reward of the paper, the energy consumption reward of the papermaking process, and the cost reward of the papermaking process, according to the state representation and action selection of the papermaking process through a predefined reward function. The role of this submodule is to evaluate the quality of each action according to the optimization goal of the papermaking process, give a positive or negative feedback, and guide the subsequent strategy update. The reward function submodule can be implemented using methods such as linear functions or nonlinear functions to improve the rationality and effectiveness of the reward.
[0099] In this embodiment, the reward function submodule adopts a reward function method based on a linear function, which includes the following steps:
[0100] According to the optimization goal of the papermaking process, the reward function of the papermaking process is defined to describe the relationship between the reward value of the papermaking process and the state representation and action selection of the papermaking process;
[0101] According to the optimization goal of the papermaking process, determine the weight coefficients of the state representation and action selection of the papermaking process, such as the weight coefficients of the quality of the paper, the energy consumption of the papermaking process, and the cost of the papermaking process, so that the reward value of the papermaking process is consistent with the optimization goal of the papermaking process. For example, the higher the quality of the paper, the lower the energy consumption and cost of the papermaking process, and the higher the reward value of the papermaking process. Conversely, the lower the reward value of the papermaking process;
[0102] According to the current value of the state representation and action selection of the papermaking process, the reward function is used to calculate the current value of the reward value of the papermaking process, including the quality reward of the paper, the energy consumption reward of the papermaking process and the cost reward of the papermaking process.
[0103] The strategy update submodule is used to update the strategy parameters of the papermaking process through the deep neural network model according to the state representation, action selection and reward value of the papermaking process, so as to improve the optimization effect of the papermaking process. The role of this submodule is to adjust the weights and biases of the deep neural network model according to the historical experience of the papermaking process, so that it can better learn the dynamic characteristics and laws of the papermaking process and generate better action selection. The strategy update submodule can be implemented by reinforcement learning algorithm or meta-learning algorithm to improve the learning speed and adaptability of the strategy.
[0104] In this embodiment, the strategy update submodule adopts a strategy update method based on a reinforcement learning algorithm, which includes the following steps:
[0105] According to the state representation, action selection and reward value of the papermaking process, a deep neural network model is constructed to describe the policy function of the papermaking process and output the policy parameters of the papermaking process, such as the parameters of pulp preparation, paper forming, paper pressing, paper drying and paper finishing.
[0106] According to the state representation, action selection and reward value of the papermaking process, a deep neural network model is constructed to describe the value function of the papermaking process and output the value estimation of the papermaking process, such as the quality value of paper, the energy consumption value of the papermaking process and the cost value of the papermaking process.
[0107] According to the policy function and value function of the papermaking process, a reinforcement learning algorithm, such as a policy gradient algorithm or an actor-critic algorithm, is used to update the parameters of the deep neural network model so that the error between the output value of the policy function and the value function of the papermaking process and the true value is minimized, thereby obtaining a policy update model.
[0108] According to the current value represented by the state of the papermaking process, the strategy updating model is used to predict the current values of the strategy parameters of the papermaking process, including the parameters of pulp preparation, paper forming, paper pressing, paper drying and paper finishing.
[0109] This embodiment provides a papermaking process optimization system based on deep reinforcement learning, which specifically implements the deep reinforcement learning module. According to the state representation, action selection and reward value of the papermaking process, the strategy parameters of the papermaking process are updated through a deep neural network model, thereby improving the optimization effect of the papermaking process. The system can use reinforcement learning algorithms and deep neural network models to dynamically learn and update the strategies of the papermaking process, so as to timely and effectively control the output variables of the papermaking process.
[0110] Embodiment 4:
[0111] Based on Example 2, this example specifically implements the optimization control module to improve the optimization capability and effect of the papermaking process.
[0112] like Figure 2 As shown, the optimization control module specifically includes the following five sub-modules:
[0113] The model prediction submodule is used to predict the future state of the papermaking process, including the quality state of the paper, the energy consumption state of the papermaking process, and the cost state of the papermaking process, based on the action selection generated by the deep reinforcement learning module and the mathematical model in the papermaking process module. The function of this submodule is to simulate and predict the future state of the papermaking process based on the current state of the papermaking process and the action selection, using the mathematical model, so as to provide a basis for the calculation of subsequent optimization objectives and constraints. The model prediction submodule can be implemented using methods such as linear programming or nonlinear programming to improve the accuracy and sensitivity of the prediction.
[0114] In this embodiment, the model prediction submodule adopts a papermaking process state prediction technology based on linear programming, which includes the following steps:
[0115] Determine the control variables of the papermaking process based on the action selections generated by the deep reinforcement learning module, including the control variables of pulp preparation, paper forming, paper pressing, paper drying, and paper finishing;
[0116] According to the mathematical model in the papermaking process module, the state equation of the papermaking process is established to describe the relationship between the state variables and the control variables of the papermaking process. The state variables include the quality state of the paper, the energy consumption state of the papermaking process and the cost state of the papermaking process.
[0117] According to the state equation of the papermaking process, the future state of the papermaking process is calculated, that is, the expected value of the state variable of the papermaking process under given control variables, and the confidence interval of the state variable of the papermaking process, that is, the possible fluctuation range of the state variable of the papermaking process;
[0118] Output the future state of the papermaking process, including the expected values and confidence intervals of the quality state of paper, the energy consumption state of the papermaking process, and the cost state of the papermaking process.
[0119] The optimization target submodule is used to calculate the optimization target value of the papermaking process according to the future state of the papermaking process through the predefined optimization target function, including the quality target of the paper, the energy consumption target of the papermaking process and the cost target of the papermaking process. The function of this submodule is to evaluate and calculate the optimization target value of the papermaking process according to the future state of the papermaking process using the optimization target function, thereby providing the target function for solving the subsequent optimization problem. The optimization target submodule can be implemented by weighted sum or weighted product methods to improve the rationality and feasibility of the optimization target.
[0120] In this embodiment, the optimization target submodule adopts a papermaking process optimization target calculation technology based on weighted sum, which includes the following steps:
[0121] According to the future state of the papermaking process, that is, the expected values and confidence intervals of the quality state of the paper, the energy consumption state of the papermaking process, and the cost state of the papermaking process, the optimization objective function of the papermaking process is determined. The optimization objective function is the weighted sum of the quality target of the paper, the energy consumption target of the papermaking process, and the cost target of the papermaking process, wherein the weight of each target is set by the user according to his own needs and preferences;
[0122] According to the optimization objective function of the papermaking process, the optimization objective value of the papermaking process, that is, the maximum value or minimum value of the optimization objective function of the papermaking process, and the confidence interval of the optimization objective value of the papermaking process, that is, the possible fluctuation range of the optimization objective function of the papermaking process are calculated;
[0123] Output the optimization target values of the papermaking process, including the maximum or minimum values and confidence intervals of the paper quality target, the energy consumption target of the papermaking process, and the cost target of the papermaking process.
[0124] The constraint condition submodule is used to calculate the constraint condition values of the papermaking process according to the future state of the papermaking process through the predefined constraint condition function, including the quality constraint of the paper, the energy consumption constraint of the papermaking process and the cost constraint of the papermaking process. The constraint condition submodule can be implemented by using methods such as equality or inequality to improve the completeness and effectiveness of the constraint condition.
[0125] In this embodiment, the constraint condition submodule adopts a papermaking process constraint condition calculation technology based on inequality, which includes the following steps:
[0126] According to the future state of the papermaking process, that is, the expected value and confidence interval of the quality state of the paper, the energy consumption state of the papermaking process, and the cost state of the papermaking process, the constraint condition function of the papermaking process is determined, and the constraint condition function is an inequality of the quality constraint of the paper, the energy consumption constraint of the papermaking process, and the cost constraint of the papermaking process, wherein the upper limit or lower limit of each constraint is set by the user according to his own needs and preferences;
[0127] According to the constraint condition function of the papermaking process, the constraint condition value of the papermaking process, that is, the feasible solution set of the constraint condition function of the papermaking process, and the confidence interval of the constraint condition value of the papermaking process, that is, the possible fluctuation range of the constraint condition function of the papermaking process are calculated;
[0128] Output the constraint values of the papermaking process, including the feasible solution set and confidence interval of the paper quality constraint, the energy consumption constraint of the papermaking process, and the cost constraint of the papermaking process.
[0129] The control parameter submodule is used to obtain the control parameters of the papermaking process by solving the optimization problem according to the optimization target value and constraint condition value of the papermaking process, including the control parameters of pulp preparation, paper forming, paper pressing, paper drying and paper finishing. For example, the control parameter submodule can use a genetic algorithm to solve the optimization problem, and obtain the control parameters of the papermaking process, such as the temperature of pulp preparation, the pressure of paper forming, the humidity of paper pressing, the speed of paper drying and the tension of paper finishing, according to the optimization target value and constraint condition value of the papermaking process, such as the quality target of the paper, the energy consumption target of the papermaking process, the cost target of the papermaking process, the quality constraint of the paper, the energy consumption constraint of the papermaking process and the cost constraint of the papermaking process.
[0130] The control execution submodule is used to control and execute the various steps and parts in the papermaking process module according to the control parameters obtained by the optimization control module, so as to achieve the optimization control of the papermaking process. The function of this submodule is to control and operate and adjust the various steps and parts in the papermaking process module according to the control parameters obtained by the optimization control module using the actuator, so as to make the papermaking process reach the optimal state. For example, the control execution submodule can use controllers such as PID controllers, fuzzy controllers, neural network controllers, etc., and control and execute the various steps and parts in the papermaking process module, such as the heater for pulp preparation, the wire part for paper forming, the pressure roller for paper pressing, the fan for paper drying, and the reel for paper finishing, according to the control parameters obtained by the optimization control module, such as the temperature of pulp preparation, the pressure of paper forming, the humidity of paper pressing, the speed of paper drying, and the tension of paper finishing, so as to achieve the optimization control of the papermaking process.
[0131] This embodiment provides a papermaking process optimization system based on deep reinforcement learning, which specifically implements the optimization control module. According to the abnormal diagnosis results and the set optimization target values, the future state of the papermaking process is predicted, and the control parameters of the papermaking process are determined. The system can use linear programming and genetic algorithm methods to dynamically predict and optimize the state of the papermaking process, so as to timely and effectively adjust the control parameters of the papermaking process.
[0132] The system has the following advantages: the system can automatically generate control parameters of the papermaking process according to the abnormal diagnosis results of the papermaking process and the set optimization target values, without human intervention, thereby improving the automation and intelligence level of the papermaking process; the system can adjust the control parameters of the papermaking process in real time according to the future state of the papermaking process to achieve the optimization goals and constraints of the papermaking process, thereby improving the stability and controllability of the papermaking process; the system can optimize and solve the control parameters of the papermaking process according to the optimization objective function and constraint function of the papermaking process, and improve the speed and quality of the solution by using linear programming and genetic algorithm methods; the system can decide whether to continue to execute the optimization control according to the preset evolution termination condition or the user's termination instruction, thereby realizing the continuity and adaptability of the optimization control of the papermaking process to adapt to the changes and uncertainties of the papermaking process.
[0133] Embodiment 5:
[0134] This embodiment adds an optimization evaluation module based on the embodiment 3, and implements the optimization evaluation module in detail to improve the optimization effect and evaluation capability of the papermaking process.
[0135] The optimization evaluation module is used to evaluate and provide feedback on the execution effect of the control parameters in the optimization control module based on the data collected by the data acquisition module and the optimization objectives of the papermaking process, thereby providing a basis for learning and updating for the deep reinforcement learning module.
[0136] like Figure 3 As shown in the figure, the optimization evaluation module specifically includes the following three sub-modules:
[0137] The optimization definition submodule is used to define the optimization indicators of the papermaking process according to the optimization objectives of the papermaking process, including the quality indicators of the paper and the energy consumption indicators and cost indicators of the papermaking process.
[0138] In this embodiment, the optimization definition submodule adopts the following method:
[0139] (1) According to the optimization goal of the papermaking process, select appropriate optimization indicators, such as quality indicators such as paper strength, glossiness, smoothness, and whiteness, as well as energy consumption indicators and cost indicators such as water consumption, power consumption, and pulp consumption in the papermaking process;
[0140] (2) According to the actual situation of the papermaking process, determine the weight coefficient of each optimization index. For example, according to factors such as the purpose of paper, customer demand, and market competition, give different optimization indexes different weights to reflect their importance in the papermaking process;
[0141] (3) Determine the optimal value of each optimization index based on the historical data of the papermaking process. For example, based on the data of the past papermaking process, statistically analyze the maximum value, minimum value, average value, standard deviation and other statistical quantities of each optimization index to determine its optimal level in the papermaking process.
[0142] The optimization evaluation submodule is used to calculate the execution effect of the control parameters in the optimization control module according to the data collected by the data acquisition module and the optimization indicators defined by the optimization definition submodule, including the improvement degree and weight of the control parameters on each optimization indicator. The execution effect of the control parameters is calculated using the following formula:
[0143]
[0144] Among them, E(s,a) is a measure of the execution effect of the control parameters of the papermaking process, w i is the weight coefficient of the optimization index, I i (s,a) is the current value of the i-th optimization index of the papermaking process, is the optimal value of the i-th optimization index of the papermaking process, n is the number of optimization indexes of the papermaking process, a is the action selection vector, and s is the state representation vector.
[0145] In this embodiment, the optimization evaluation submodule adopts the following method:
[0146] (1) Calculate the current state representation vector s of the papermaking process based on the data collected by the data acquisition module, including the values of various parameters and variables of the papermaking process, such as pulp concentration, temperature, flow rate, etc.;
[0147] (2) Calculating the action selection vector a of the papermaking process according to the control parameters selected by the optimization control module, including the set values of various controllers of the papermaking process, such as the switch state, power, frequency, etc. of the heater, agitator, and sprinkler of the pulp;
[0148] (3) According to the optimization index defined by the optimization definition submodule, calculate the current value I of each optimization index of the papermaking process i (s,a), including the quality index of paper and the energy consumption index and cost index of papermaking process, such as the strength, glossiness, smoothness, whiteness, etc. of paper, as well as the water consumption, electricity consumption, pulp consumption, etc. of papermaking process;
[0149] (4) According to the optimal value of each optimization index determined by the optimization definition submodule and weight coefficient w i , substitute into the formula The execution effect E(s,a) of the control parameters of the papermaking process is calculated. The larger the value, the better the execution effect of the control parameters. Conversely, the worse the execution effect of the control parameters.
[0150] The optimization feedback submodule is used to generate evaluation results of the control parameters, including the overall score and optimization suggestions of the control parameters, according to the execution effects of the control parameters calculated by the optimization evaluation submodule, and transmit them to the deep reinforcement learning module and the user.
[0151] In this embodiment, the optimization feedback submodule adopts the following method:
[0152] (1) According to the execution effect E(s,a) of the control parameter calculated by the optimization evaluation submodule, the control parameter is scored overall, such as using a five-point or ten-point system to give the score of the control parameter. If E(s,a) is greater than 0.8, 5 points are given, indicating that the execution effect of the control parameter is very good. If E(s,a) is less than 0.2, 1 point is given, indicating that the execution effect of the control parameter is very poor.
[0153] (2) According to the improvement degree and weight of each optimization index by the control parameters calculated by the optimization evaluation submodule, optimization suggestions are made for the control parameters, such as giving the advantages and disadvantages of the control parameters and possible improvement directions. For example, the control parameters can improve the strength and glossiness of paper, but will increase the water consumption and power consumption of the papermaking process. It is recommended to adjust the set values of the pulp heater and sprinkler to reduce energy consumption and cost;
[0154] (3) The evaluation results of the control parameters, including the overall scores and optimization suggestions of the control parameters, are transmitted to the deep reinforcement learning module and the user, so that the deep reinforcement learning module updates the control parameters according to the evaluation results, and the user adjusts the optimization objectives and parameters of the papermaking process according to the evaluation results.
[0155] This embodiment provides a papermaking process optimization system based on deep reinforcement learning, which specifically implements the optimization evaluation module. According to the data collected by the data acquisition module and the optimization target of the papermaking process, the execution effect of the control parameters in the optimization control module is evaluated and feedback is provided, thereby providing a basis for learning and updating for the deep reinforcement learning module. The system can use formulas and scoring methods to dynamically evaluate and feedback the control parameters of the papermaking process, so as to optimize the papermaking process in a timely and effective manner.
[0156] Embodiment 6:
[0157] Based on Example 1, this embodiment adds a fault diagnosis module and implements the fault diagnosis module in detail to improve the detection and location capabilities of abnormal situations in the papermaking process.
[0158] The fault diagnosis module is used to detect and locate abnormal conditions in the papermaking process based on the data collected by the data acquisition module and the optimization target of the papermaking process.
[0159] like Figure 4 As shown in the figure, the fault diagnosis module specifically includes the following three sub-modules:
[0160] The fault detection submodule is used to determine whether there is a fault in the papermaking process, as well as the type and severity of the fault through an abnormal detection method based on the data collected by the data acquisition module. The fault detection submodule can be implemented using methods such as statistical analysis or machine learning to improve the accuracy and sensitivity of detection.
[0161] In this embodiment, the fault detection submodule adopts a fault detection technology based on machine learning, which includes the following steps:
[0162] According to the data collected by the data acquisition module, the input layer of machine learning is constructed, including several input nodes, which correspond to various parameters in the papermaking process, such as temperature, humidity, pressure, flow rate, concentration, etc.
[0163] According to the type and severity of the fault, a machine learning output layer is constructed, including several output nodes, corresponding to the fault category, such as mechanical fault, electrical fault, chemical fault, etc., and the fault level, such as minor, moderate, severe, etc.;
[0164] According to the complex relationship between the data collected by the data acquisition module and the type and severity of the fault, a hidden layer of machine learning is constructed, including several hidden nodes, each of which is connected to all nodes of the input layer and the output layer and has different weights and biases;
[0165] According to the data collected by the data acquisition module and the historical data of the fault type and severity, the gradient descent algorithm is used to train the parameters of machine learning so that the error between the output value of machine learning and the true value is minimized, and the fault detection model is obtained.
[0166] The fault location submodule is used to determine the location and cause of the fault in the papermaking process through the fault tree analysis method according to the fault type and severity determined by the fault detection submodule. The fault location submodule can be implemented by logical reasoning or expert system to improve the accuracy and efficiency of positioning.
[0167] In this embodiment, the fault location submodule adopts a fault location technology based on an expert system, which includes the following steps:
[0168] According to the fault type and severity determined by the fault detection submodule, a corresponding fault tree is selected. A fault tree is a tree structure with the fault as the root node and the fault cause as the child node, which is used to represent the logical relationship between the fault and the fault cause;
[0169] According to the data collected by the data acquisition module, starting from the root node of the fault tree, searching downward along the branches of the fault tree, judging whether the conditions of each branch are established, until one or more possible fault causes are found;
[0170] According to the fault cause found, the location where the fault occurs in the papermaking process is determined. For example, if the fault cause is the failure of a certain sensor, then the location where the fault occurs is the location of the sensor.
[0171] The fault handling submodule is used to generate fault handling measures and suggestions based on the fault location and cause determined by the fault locating submodule through a fault recovery method, and transmit them to the optimization control module and the user. The fault handling submodule can be implemented using methods such as a rule base or a case base to improve the effectiveness and intelligence of the processing.
[0172] In this embodiment, the fault handling submodule adopts a fault handling technology based on a case library, which includes the following steps:
[0173] According to the fault location and cause determined by the fault location submodule, similar historical fault cases are retrieved from the case library. The case library is a database that stores historical fault cases. Each case includes information such as the location, cause, treatment measures, and treatment effects of the fault.
[0174] According to the retrieved historical fault cases, one or more best matching cases are selected, and fault handling measures and suggestions are generated according to the handling measures and handling effects in the cases. For example, if the handling measure in the case is to replace a certain sensor, then the generated fault handling measure is to recommend the user to replace the sensor.
[0175] The generated fault handling measures and suggestions are transmitted to the optimization control module and the user. The optimization control module can automatically or semi-automatically adjust the parameters of the papermaking process according to the fault handling measures and suggestions to restore the normal operation of the papermaking process. The user can manually or assistedly perform fault handling operations according to the fault handling measures and suggestions to eliminate the impact of the fault.
[0176] The present embodiment provides a papermaking process optimization system, which specifically implements a fault diagnosis module. Based on the data collected by the data acquisition module and the optimization target of the papermaking process, the system detects and locates abnormal conditions in the papermaking process, and generates fault handling measures and suggestions based on the location and cause of the fault. The system can use machine learning, expert system and case library methods to effectively diagnose and handle faults in the papermaking process to improve the stability and quality of the papermaking process.
[0177] Embodiment 7:
[0178] This embodiment provides a papermaking process optimization method based on deep reinforcement learning. The flowchart of the method is as follows: Figure 5 As shown, the following steps are included:
[0179] Step 1: Simulate the physical and chemical changes in the papermaking process, including pulp preparation, paper forming, paper pressing, paper drying and paper finishing. The company used a papermaking process simulation system based on mathematical models and simulation software, which can simulate various physical and chemical changes in the papermaking process, as well as indicators such as paper quality, energy consumption and cost based on different input parameters and control variables. The system can be used to test and verify different control parameters and evaluate their impact on the papermaking process.
[0180] Step 2: Collect various data in the papermaking process, including paper quality data and energy consumption and cost data of the papermaking process. Real-time monitoring and recording of various data in the papermaking process, such as pulp concentration, temperature, pH value, fiber length, fiber strength, fiber distribution, etc.; paper thickness, humidity, dryness, gloss, softness, water absorption, tensile strength, breaking length, etc.; as well as papermaking process energy consumption, cost, output, wastewater discharge, etc. These data are transmitted to a data center for training and updating of deep reinforcement learning models.
[0181] Step 3: Based on the collected data, the dynamic characteristics and laws of the papermaking process are learned through a deep neural network model to generate action selection for the papermaking process. The company used an optimization model based on multi-agent deep reinforcement learning, which consists of multiple deep neural networks. Each network corresponds to a step or part in the papermaking process, such as pulp preparation, paper forming, paper pressing, paper drying and paper finishing. The input of each network is the current state and historical data of the step or part, and the output is the control parameters of the step or part, such as pulp concentration, temperature, pH value, fiber length, fiber strength, fiber distribution, etc.; paper thickness, humidity, dryness, gloss, softness, water absorption, tensile strength, breaking length, etc.; as well as papermaking process energy consumption, cost, output, wastewater discharge, etc. The goal of each network is to maximize the reward function of the step or part, which is designed according to the optimization objectives and constraints of the papermaking process, such as paper quality, energy consumption and cost. Each network continuously learns and updates its own strategies and parameters by interacting with the papermaking process simulation system or the actual papermaking process to improve its own reward function value. At the same time, each network also coordinates and communicates with other networks to achieve overall optimization of the papermaking process.
[0182] Step 4: According to the generated action selection and the predefined optimization objective function and constraint condition function, the control parameters of the papermaking process are obtained, and each step and part of the papermaking process is optimized and controlled based on the control parameters. The control parameters of the papermaking process are obtained by using the generated action selection and the predefined optimization objective function and constraint condition function, and the obtained control parameters are transmitted to the controllers of each step and part of the papermaking process through an automated control system, such as the agitator, heater, acid-base regulator, fiber separator, etc. for pulp preparation; the wire part, water absorption box, vacuum box, press roller, etc. for paper forming; the press roller, steam sprayer, vacuum water absorber, etc. for paper pressing; the drying cylinder, fan, humidifier, etc. for paper drying; the cutting machine, winder, packaging machine, etc. for paper finishing. These controllers automatically adjust various variables in the papermaking process according to the control parameters, such as pulp concentration, temperature, pH value, fiber length, fiber strength, fiber distribution, etc.; paper thickness, humidity, dryness, glossiness, softness, water absorption, tensile strength, breaking length, etc.; as well as papermaking process energy consumption, cost, output, wastewater discharge, etc. The adjustment of these variables makes the papermaking process reach the optimal state, thereby improving the quality of paper, reducing energy consumption and cost, and increasing profits.
[0183] Step 5: Based on the collected data and the optimization goals of the papermaking process, evaluate and provide feedback on the execution effect of the control parameters in the optimization control, thereby providing a basis for learning and updating the deep reinforcement learning model. By comparing various data in the papermaking process before and after the optimization control, such as paper quality data and energy consumption and cost data of the papermaking process, the company evaluates the execution effect of the optimization control, such as the degree of improvement in paper quality, energy consumption and cost, as well as the stability and reliability of the papermaking process. These evaluation results, as feedback information, are transmitted to the deep reinforcement learning model to adjust and update the strategies and parameters of each network in the model to improve the performance and accuracy of the model.
[0184] Step 6: Repeat steps 2 to 5 until the preset optimization termination condition or the user's termination instruction is reached. According to the optimization goal of the papermaking process, the enterprise sets some optimization termination conditions, such as the quality, energy consumption and cost of paper reaching the expected level, or the stability and reliability of the papermaking process reaching a satisfactory level. When the optimization termination condition is met, or the user issues a termination instruction, the optimization process ends, and the deep reinforcement learning model saves the optimal strategy and parameters for the next optimization process.
[0185] This embodiment provides a papermaking process optimization method based on deep reinforcement learning. The method utilizes deep neural network models, data acquisition and analysis techniques, optimization control and evaluation techniques, etc. to intelligently analyze and optimize various steps and parts of the papermaking process, thereby improving the quality and efficiency of the papermaking process, and reducing the energy consumption and cost of the papermaking process.
[0186] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. Specific examples are used in this article to illustrate the principle and implementation of the technical solution of the present invention. The above examples are only used to help understand the method of the present invention and its core idea. The above is only a preferred implementation of the present invention. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements, modifications or changes can be made, and the above technical features can also be combined in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the concept and technical solution of the invention to other occasions without improvement, should be regarded as the protection scope of the present invention.
Claims
1. A papermaking process optimization system based on deep reinforcement learning, characterized in that: include: Papermaking process module, used to simulate the physical and chemical changes in the papermaking process, including pulp preparation, paper forming, paper pressing, paper drying and paper finishing; Data collection module, used to collect various data in the papermaking process module, including paper quality data and energy consumption and cost data of the papermaking process; The deep reinforcement learning module is used to learn the dynamic characteristics and laws of the papermaking process through a deep neural network model based on the data collected by the data acquisition module, thereby generating action selections for the papermaking process; An optimization control module, used to obtain control parameters of the papermaking process according to the action selection generated by the deep reinforcement learning module and the predefined optimization objective function and constraint condition function, and optimize and control each step and part in the papermaking process module based on the control parameters; The papermaking process module specifically includes: The physical model submodule is used to establish a mathematical model of the papermaking process based on the physical principles and chemical reactions of the papermaking process, and to describe the input-output relationship and internal state changes of the papermaking process; The data-driven submodule is used to establish a data model of the papermaking process based on the actual data of the papermaking process through machine learning or data mining methods to supplement the nonlinear and uncertain characteristics of the papermaking process; The model fusion submodule is used to generate a hybrid model of the papermaking process by using the model fusion method based on the mathematical model and data model of the papermaking process obtained by the physical model submodule and the data driven submodule. The deep reinforcement learning module specifically includes: The state representation submodule is used to preprocess and extract features of the data collected by the data acquisition module to obtain the state representation of the papermaking process, including the quality state of the paper, the energy consumption state of the papermaking process and the cost state of the papermaking process; The action selection submodule is used to generate action selection of the papermaking process according to the state representation of the papermaking process through a deep neural network model, including the actions of pulp preparation, paper forming, paper pressing, paper drying and paper finishing; The reward function submodule is used to calculate the reward value of the papermaking process according to the state representation and action selection of the papermaking process through a predefined reward function, including the quality reward of the paper, the energy consumption reward of the papermaking process, and the cost reward of the papermaking process; The strategy update submodule is used to update the strategy parameters of the papermaking process through a deep neural network model according to the state representation, action selection and reward value of the papermaking process; The system also includes an optimization evaluation module, which is used to evaluate and provide feedback on the execution effect of the control parameters in the optimization control module based on the data collected by the data acquisition module and the optimization target of the papermaking process, so as to provide a basis for learning and updating for the deep reinforcement learning module; The optimization evaluation module specifically includes: The optimization definition submodule is used to define the optimization indicators of the papermaking process according to the optimization objectives of the papermaking process, including the quality indicators of the paper and the energy consumption indicators and cost indicators of the papermaking process; The optimization evaluation submodule is used to calculate the execution effect of the control parameters in the optimization control module according to the data collected by the data acquisition module and the optimization indicators defined by the optimization definition submodule, including the improvement degree and weight of the control parameters on each optimization indicator. The execution effect of the control parameters is calculated using the following formula: Among them, E(s,a) is a measure of the execution effect of the control parameters of the papermaking process, w i is the weight coefficient of the optimization index, I i (s,a) is the current value of the i-th optimization index of the papermaking process, is the optimal value of the i-th optimization index of the papermaking process, n is the number of optimization indexes of the papermaking process, a is the action selection vector, and s is the state representation vector; The optimization feedback submodule is used to generate evaluation results of the control parameters, including the overall score and optimization suggestions of the control parameters, according to the execution effects of the control parameters calculated by the optimization evaluation submodule, and transmit them to the deep reinforcement learning module and the user.
2. The papermaking process optimization system based on deep reinforcement learning according to claim 1, characterized in that: The optimization control module specifically includes: The model prediction submodule is used to predict the future state of the papermaking process, including the quality state of the paper, the energy consumption state of the papermaking process, and the cost state of the papermaking process through the mathematical model in the papermaking process module according to the action selection generated by the deep reinforcement learning module; The optimization target submodule is used to calculate the optimization target value of the papermaking process according to the future state of the papermaking process through a predefined optimization target function, including the quality target of the paper, the energy consumption target of the papermaking process and the cost target of the papermaking process; The constraint condition submodule is used to calculate the constraint condition values of the papermaking process according to the future state of the papermaking process through the predefined constraint condition function, including the quality constraint of the paper, the energy consumption constraint of the papermaking process and the cost constraint of the papermaking process; The control parameter submodule is used to obtain the control parameters of the papermaking process by solving the optimization problem according to the optimization target value and constraint condition value of the papermaking process, including the control parameters of pulp preparation, paper forming, paper pressing, paper drying and paper finishing; The control execution submodule is used to control and execute various steps and parts in the papermaking process module according to the control parameters obtained by the optimization control module, so as to achieve optimized control of the papermaking process.
3. The papermaking process optimization system based on deep reinforcement learning according to claim 2 is characterized in that: The control parameter submodule adopts a genetic algorithm to solve the optimization problem.
4. The papermaking process optimization system based on deep reinforcement learning according to claim 1, characterized in that: The system also includes a fault diagnosis module for detecting and locating abnormal conditions in the papermaking process based on the data collected by the data collection module and the optimization target of the papermaking process.
5. The papermaking process optimization system based on deep reinforcement learning according to claim 4 is characterized in that: The fault diagnosis module specifically includes: The fault detection submodule is used to determine whether there is a fault in the papermaking process, as well as the type and severity of the fault through an abnormality detection method based on the data collected by the data acquisition module; The fault location submodule is used to determine the location and cause of the fault in the papermaking process through a fault tree analysis method according to the fault type and severity determined by the fault detection submodule; The fault handling submodule is used to generate fault handling measures and suggestions based on the fault location and cause determined by the fault locating submodule through a fault recovery method, and transmit them to the optimization control module and the user.
6. A papermaking process optimization method based on deep reinforcement learning based on the papermaking process optimization system in any one of claims 1 to 5, characterized in that: The following steps are involved: Step 1: Simulate the physical and chemical changes of the papermaking process, including pulp preparation, paper forming, paper pressing, paper drying and paper finishing; Step 2: Collect various data in the papermaking process, including paper quality data and energy consumption and cost data of the papermaking process; Step 3: Based on the collected data, the dynamic characteristics and rules of the papermaking process are learned through a deep neural network model to generate action selections for the papermaking process; Step 4: according to the generated action selection and the predefined optimization objective function and constraint condition function, the control parameters of the papermaking process are obtained, and each step and part of the papermaking process is optimized and controlled based on the control parameters; Step 5: Based on the collected data and the optimization objectives of the papermaking process, the execution effects of the control parameters in the optimization control are evaluated and feedback is provided, thereby providing a basis for learning and updating the deep reinforcement learning model; Step 6: Repeat steps 2 to 5 until the preset optimization termination condition is reached or the user's termination instruction is issued.
