Data processing method and device, storage medium and electronic device

Abstract

The present invention provides a data processing method and device, a storage medium and an electronic device, the method comprising: obtaining each offline data, constructing a sample pool corresponding to each offline data; initializing a value model and a policy model, and executing a model training process; the model training process comprises: extracting offline data in the sample pool as training data; performing data enhancement processing on the training data to obtain multiple extended data; updating the value model based on the policy model and each extended data to obtain an updated value model, and updating the policy model based on the updated value model and the training data to obtain an updated policy model; checking whether the updated policy model and the updated value model have both reached a convergence state; if not, continuing to execute the model training process; if so, completing the model training process. Applying this method, the model is trained through offline data, and the data is amplified by data enhancement, so as to improve the effectiveness and stability of model training.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning technology, and in particular to a data processing method and device, a storage medium and an electronic device. Background Art

[0002] With the rapid development of the industrial control industry, there are a large number of production control links in industrial control systems. In most cases, the production control links in industrial control systems need to rely on manual, semi-automatic and automatic methods to control the operation of the system. In order to improve the operating efficiency of the system, deep reinforcement learning is usually used to train value models and policy models to explore the control strategies of industrial control systems in different environments.

[0003] The value model and the policy model belong to the actor-critic network model, in which the policy model, as the actor in the actor-critic network model, is used to generate corresponding execution actions according to the environmental state in the application scenario of the industrial control system, and the value model, as the critic in the actor-critic network model, is used to evaluate the performance of the actions executed by the value model under the environmental state. In order to save training costs, offline reinforcement learning is used in the prior art to train the model. However, in large-scale industrial control scenarios, the data quality is poor, and there is less data available for training, which cannot cover the state-action space in the control scenario well, making the model prone to overfitting and lacking good generalization capabilities in the production deployment environment. Summary of the invention

[0004] In view of this, the present invention provides a data processing method, through which the model is trained through offline data, and the data is amplified by data enhancement to improve the effectiveness and stability of model training.

[0005] The present invention also provides a data processing device, a storage medium and an electronic device to ensure the implementation and application of the above method in practice.

[0006] A data processing method, comprising:

[0007] Obtain various offline data in the industrial control system, and construct a sample pool corresponding to each offline data, wherein the offline data includes multiple parameters, and each parameter included in the offline data is a state parameter, an action parameter, a reward parameter, a loss parameter at the current moment, and a state parameter at the next moment;

[0008] Initializing a preset value model and a strategy model, and executing a preset model training process on the value model and the strategy model;

[0009] Among them, the model training process includes: extracting any offline data in the sample pool as training data; applying a pre-set data enhancement mode to perform data enhancement processing on the training data to obtain multiple extended data corresponding to the training data; updating the value model based on the strategy model and each of the extended data to obtain an updated value model, and updating the strategy model based on the updated value model and the training data to obtain an updated strategy model; checking whether the updated strategy model and the updated value model have reached a convergence state; if not, continuing to execute the model training process based on the updated value model and the updated strategy model; if so, completing the model training process.

[0010] In the above method, optionally, the initialization of the preset value model and strategy model includes:

[0011] Determining each first model parameter in the value model and each second model parameter in the strategy model;

[0012] Determine a preset first parameter value range of each of the first model parameters, and a preset second parameter value range of each of the second model parameters;

[0013] Randomly select any first parameter value from each of the first parameter value ranges, and randomly select any second parameter value from each of the second parameter value ranges;

[0014] The selected first parameter values ​​and second parameter values ​​are applied to fill the value model and the strategy model, thereby completing the initialization of the value model and the strategy model.

[0015] In the above method, optionally, applying a preset data enhancement mode to perform data enhancement processing on the training data includes:

[0016] Determine each data enhancement method included in the data enhancement mode, and a data enhancement algorithm corresponding to each data enhancement method;

[0017] Determine the data enhancement mode currently supported by the industrial control system;

[0018] The data enhancement algorithm corresponding to the data enhancement method currently supported by the industrial control system is applied to perform data enhancement processing on the training data.

[0019] Optionally, in the above method, updating the value model based on the extended data and the strategy model includes:

[0020] Applying the strategy model to predict action parameters at the next moment;

[0021] Based on the action parameter at the next moment, the state parameter at the current moment, the action parameter, the reward parameter and the state parameter at the next moment in each of the extended data, obtaining the minimum Bellman error of the value model;

[0022] An expected value of the minimized Bellman error is obtained to update the value model based on the expected value.

[0023] Optionally, in the above method, updating the policy model based on the updated value model and the training data to obtain the updated policy model includes:

[0024] Extracting the state parameters and action parameters at the current moment in the training data;

[0025] Based on the state parameters and action parameters at the current moment in the training data, the output of the updated value model is maximized, and based on the maximization of the output of the updated value model, the policy model is updated.

[0026] A data processing device, comprising:

[0027] An acquisition unit is used to acquire various offline data in the industrial control system and construct a sample pool corresponding to each offline data, wherein the offline data includes multiple parameters, and each parameter included in the offline data is a state parameter, an action parameter, a reward parameter, a loss parameter at a current moment, and a state parameter at a next moment;

[0028] An execution unit, used for initializing a preset value model and a strategy model, and executing a preset model training process on the value model and the strategy model;

[0029] Among them, the model training process includes: extracting any offline data in the sample pool as training data; applying a pre-set data enhancement mode to perform data enhancement processing on the training data to obtain multiple extended data corresponding to the training data; updating the value model based on the strategy model and each of the extended data to obtain an updated value model, and updating the strategy model based on the updated value model and the training data to obtain an updated strategy model; checking whether the updated strategy model and the updated value model have reached a convergence state; if not, continuing to execute the model training process based on the updated value model and the updated strategy model; if so, completing the model training process.

[0030] In the above device, optionally, the execution unit includes:

[0031] A first determination subunit, used to determine each first model parameter in the value model and each second model parameter in the strategy model;

[0032] A second determining subunit, used to determine a preset first parameter value range of each of the first model parameters, and a second parameter value range of each of the second model parameters;

[0033] A selection subunit, used to randomly select any first parameter value from each of the first parameter value ranges, and randomly select any second parameter value from each of the second parameter value ranges;

[0034] The initialization subunit is used to apply the selected first parameter values ​​and second parameter values ​​to fill the value model and the strategy model, and complete the initialization of the value model and the strategy model.

[0035] In the above device, optionally, the execution unit includes:

[0036] A third determination subunit is used to determine each data enhancement method included in the data enhancement mode, and a data enhancement algorithm corresponding to each data enhancement method;

[0037] A fourth determining subunit, used to determine the data enhancement mode currently supported by the industrial control system;

[0038] The processing subunit is used to apply the data enhancement algorithm corresponding to the data enhancement method currently supported by the industrial control system to perform data enhancement processing on the training data.

[0039] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned data processing method.

[0040] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform the above data processing method.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The present invention provides a data processing method, comprising: acquiring various offline data in an industrial control system, and constructing a sample pool corresponding to each of the offline data, wherein the offline data includes multiple parameters, and the various parameters included in the offline data are state parameters, action parameters, reward parameters, loss parameters at the current moment, and state parameters at the next moment; initializing a preset value model and a strategy model, and executing a preset model training process for the value model and the strategy model; wherein the model training process comprises: extracting any offline data in the sample pool as training data; applying a preset data enhancement mode to perform data enhancement processing on the training data to obtain multiple extended data corresponding to the training data; updating the value model based on the strategy model and each of the extended data to obtain an updated value model, and updating the strategy model based on the updated value model and the training data to obtain an updated strategy model; checking whether the updated strategy model and the updated value model have both reached a convergence state; if not, continuing to execute the model training process based on the updated value model and the updated strategy model; if so, completing the model training process. By using the method provided by the present invention, the model is trained using offline data, and the data is amplified by data enhancement, thereby improving the effectiveness and stability of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A method flow chart of a data processing method provided by an embodiment of the present invention;

[0045] Figure 2 A method example diagram of a data processing method provided by an embodiment of the present invention;

[0046] Figure 3 A flowchart of another method of a data processing method provided by an embodiment of the present invention;

[0047] Figure 4 A device structure diagram of a data processing device provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0051] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0052] The embodiment of the present invention provides a data processing method, which can be applied to various system platforms, and the execution subject can be a processor of a computer terminal or various mobile devices. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0053] S101: Acquire various offline data in the industrial control system, and construct a sample pool corresponding to each offline data.

[0054] The offline data includes multiple parameters, and the parameters included in the offline data are the state parameter s, action parameter a, reward parameter r, loss parameter c and state parameter s' at the next moment.

[0055] In the embodiment of the present invention, each offline data is offline data in the same application scenario in the industrial control system. The sample pool is a memory or container for storing each offline data.

[0056] S102: Initialize the preset value model and strategy model.

[0057] Specifically, after the value model and the policy model are initialized, a preset model training process is performed on the value model and the policy model, that is, the following process S103 to S105 is performed.

[0058] It should be noted that initializing the value model and the strategy model refers to assigning an initial parameter value to each model parameter in the value model and assigning an initial parameter value to each model parameter in the strategy model.

[0059] Among them, the relationship between the value model and the strategy model is: Q π (s,a), specifically represents the expected future return of the value model Q under the strategy π, when the environment is in state s, taking action a. The optimal strategy to be found is the strategy that maximizes the expected value of future returns at any time. The state parameter s can be a sensor representing the state of the environment in the industrial control scenario, taking a thermal power plant as an example, such as load, steam, flue gas temperature, steam, flue gas pressure, steam, flue gas flow, etc.; the action parameter a can be a variable that needs to be controlled in the current scenario, such as primary air and secondary air volume valves, main reheat flue gas damper valves, first and second level cooling water valve openings, etc.; r and c represent the returns and losses generated by different operations under different states, such as thermal efficiency of thermal power generation, nitrogen oxide emissions as loss indicators, etc.; s' is the state change after the corresponding action is performed, and its specific representation is consistent with s.

[0060] Furthermore, the initialization of the preset value model and strategy model includes:

[0061] Determining each first model parameter in the value model and each second model parameter in the strategy model;

[0062] Determine a preset first parameter value range of each of the first model parameters, and a preset second parameter value range of each of the second model parameters;

[0063] Randomly select any first parameter value from each of the first parameter value ranges, and randomly select any second parameter value from each of the second parameter value ranges;

[0064] The selected first parameter values ​​and second parameter values ​​are applied to fill the value model and the strategy model, thereby completing the initialization of the value model and the strategy model.

[0065] It can be understood that for each first model parameter in the value model, a corresponding first parameter value range is pre-set. In order to improve the training accuracy in subsequent model training, a parameter value is randomly selected in each first parameter value range as the parameter value of the first model parameter corresponding to the first parameter value range, and each selected first parameter value is filled into the value model to complete the initialization of the value model. Similarly, for each second model parameter in the strategy model, a corresponding second parameter value range is pre-set, and a second parameter value is selected in each second parameter value range as the parameter value of the second model parameter corresponding to the second parameter value range, and each second parameter value is filled into the strategy model to complete the initialization of the strategy model.

[0066] S103: extracting any offline data in the sample pool as training data.

[0067] In the embodiment of the present invention, when the value model and the policy model need to be trained, at least one offline data is randomly extracted from the sample pool as training data for training the value model and the policy model in the current model training process.

[0068] Optionally, before extracting offline data from the sample pool, each offline data in the sample pool may be sorted, and each time the model training process is executed, the offline data is extracted in the sorted order as the training data for the current model training process.

[0069] S104: Apply a preset data enhancement mode to perform data enhancement processing on the training data to obtain a plurality of expanded data corresponding to the training data.

[0070] In an embodiment of the present invention, when the value model needs to be updated, data enhancement is introduced to improve the stability and effectiveness of the value model in an industrial control environment. By performing data enhancement on the data, the original offline data is converted to obtain multiple expanded data.

[0071] It should be noted that the data enhancement mode includes multiple data enhancement methods. At least one data enhancement method can be applied to perform multiple data enhancement processing on the training data. Each data enhancement processing can obtain corresponding expanded data, and each expanded data is also the training data for training the model.

[0072] Specifically, data enhancement methods include but are not limited to zero-mean Gaussian noise, zero-mean uniform noise, random amplitude, random loss state, and mixed state.

[0073] S105: updating the value model based on the policy model and each of the expanded data to obtain an updated value model, and updating the policy model based on the updated value model and the training data to obtain an updated policy model.

[0074] In the embodiment of the present invention, the actor-critic method is applied in reinforcement learning to realize the learning and training of the value model and the policy model. When the value model is updated, multiple extended data are introduced to improve the robustness of the system, and each extended data is applied to minimize the Bellman errors, so as to achieve the expected value of the updated value model. At the same time, after the value model is updated, the policy model is updated by combining the training data and maximizing the value model data.

[0075] S106: Check whether the updated policy model and the updated value model have reached a convergence state.

[0076] In the present invention, if not, based on the updated value model and the updated policy model, the process returns to execute S103 ; if yes, the process executes S107 .

[0077] It should be noted that Actor-critic is a reinforcement learning algorithm framework that includes a policy model (Actor) and a value model (Critic). The Actor generates actions based on the state of the environment, and the Critic is responsible for evaluating the Actor's performance under the current state of the environment. The Actor adjusts its own strategy, that is, the model parameters of the policy model, based on the Critic's score, and strives to do better next time. The Critic adjusts its scoring strategy, that is, the model parameters of the value model, based on the rewards given by the system, which are equivalent to the scores of groundtruth and other judges. At the beginning, the policy model randomly generates actions and the value model randomly scores. However, due to the existence of rewards, the value model scores more and more accurately, and the policy model performs better and better. Finally, an optimized policy model can be obtained.

[0078] Specifically, during the model training process, the value model and the strategy model need to be trained multiple times. Each time the strategy model is trained, it performs corresponding actions according to the state and is scored by the value model. When it is determined that both models have not converged, the strategy model adjusts the model parameters, and the value model also adjusts the model parameters and repeats the process from S103 to S106 until both models converge and S107 is executed.

[0079] S107: Complete the model training process.

[0080] It should be noted that the trained value model and strategy model can be applied in the industrial control system to realize the operation and control process of the system in the corresponding application scenario.

[0081] refer to Figure 2 , Figure 2The data processing process corresponding to S103 to S107 is described above. In which, the offline data (s, a, r, c, s′) is obtained through the sample pool, and the offline data is enhanced to obtain multiple extended data. Using each extended data and (s i ,a,r,c,s′) i＝1,2,… Update the value model with the policy model; Update the policy model using the real offline data and the policy gradient provided by the value model.

[0082] In the data processing method provided by the embodiment of the present invention, when it is necessary to train the corresponding value model and policy model in any application scenario of the industrial control system, each offline data corresponding to the application scenario in the industrial control system is obtained, and a sample pool corresponding to each offline data is constructed, and each offline data is stored in the sample pool. The value model and the policy model are initialized, and the model training process is performed on the value model and the policy model at least once until the training is completed. In each training process, an offline data is extracted from the resource pool as training data, and a data enhancement mode is applied to the training data to perform data enhancement to obtain multiple expanded data. The value model is updated by applying each expanded data and policy model, and the policy model is updated by the training data and the updated value model. The convergence state of the two models is used to determine whether the training is completed. If not, the model training process continues.

[0083] By applying the method provided in the embodiment of the present invention, the model is trained using offline data, and the data is amplified by data enhancement, thereby improving the effectiveness and stability of the model training.

[0084] refer to Figure 3 Based on the content of S104, the application performs data enhancement processing on the training data in a pre-set data enhancement mode, which may specifically include:

[0085] S301: Determine each data enhancement method included in the data enhancement mode, and the data enhancement algorithm corresponding to each data enhancement method.

[0086] In the embodiment of the present invention, the data enhancement mode includes multiple data enhancement methods, each of which includes but is not limited to zero-mean Gaussian noise, zero-mean uniform noise, random amplitude, random loss state, and mixed state. Each data enhancement method is set with a corresponding data enhancement algorithm, and data can be amplified by the data enhancement algorithm.

[0087] Specifically, for the state parameter s in the offline data, the data enhancement algorithm corresponding to the zero-mean Gaussian noise is:

[0088]

[0089] Among them, ∈∈Ν(0,σ1), Ν(0,σ1) is a Gaussian distribution, σ is a hyperparameter, which is generally set to 3e-3 and has a value range of [10 -4 ,10 -2 ],s t is the state parameter at the current moment, It is the status parameter of the extended data at the current moment.

[0090] For the state parameter s in the offline data, the data enhancement algorithm corresponding to the zero-mean uniform noise is:

[0091]

[0092] Among them, ∈∈μ(-α,α), μ(-α,α) is a uniform distribution, α is a hyperparameter, generally set to 1e-3, and the value range is [10 -4 ,10 -2 ].

[0093] For the state parameter s in the offline data, the data enhancement algorithm corresponding to the random amplitude is:

[0094]

[0095] Among them, ∈∈μ(α,β), μ(α,β) is a random distribution, and α and β are hyperparameters.

[0096] For the state parameter s in the offline data, the data enhancement algorithm corresponding to the random loss state is:

[0097]

[0098] Among them, Ι is a 1 matrix of the same dimension as the state quantity, and is filled with 0 by random sampling from the Bernoulli distribution. This transformation can maximize the retention of information about the current state.

[0099] For the state parameter s in the offline data, the data enhancement algorithm corresponding to the mixed state is:

[0100]

[0101] Among them, λ∈Beta(α,α), α=0.4.

[0102] S302: Determine the data enhancement method currently supported by the industrial control system.

[0103] In an embodiment of the present invention, a user can select any data enhancement method in the data enhancement mode in the industrial control system according to needs, and the industrial control system changes the data enhancement selection parameters according to the user's selection. When the model training process is executed, the data enhancement method currently supported by the industrial control system is determined according to the data enhancement selection parameters in the industrial control system.

[0104] S303: Apply a data enhancement algorithm corresponding to the data enhancement method currently supported by the industrial control system to perform data enhancement processing on the training data.

[0105] In an embodiment of the present invention, a data enhancement algorithm corresponding to a data enhancement method currently supported by the industrial control system is obtained, and the data enhancement algorithm is applied to perform data enhancement processing on the training data.

[0106] For example, the data enhancement method currently supported by the industrial control system is zero-mean Gaussian noise. In this case, the state parameter s in the original training data is t Add Gaussian noise ∈, that is, s t +∈, through the data enhancement algorithm corresponding to zero-mean Gaussian noise, different Gaussian noises are added multiple times, and each time Gaussian noise is added, the corresponding augmented data is generated.

[0107] In the data processing method provided by the embodiment of the present invention, when the model training process is executed each time, any offline data is extracted from the sample pool as training data, and each data enhancement method included in the data enhancement mode in the industrial control system and the data enhancement algorithm corresponding to each data enhancement method are determined. The data enhancement method currently supported by the industrial control system is determined, and the data enhancement algorithm corresponding to the data enhancement method is applied to perform data enhancement processing on the training data to obtain multiple expanded data.

[0108] It should be noted that in offline reinforcement learning, the policy model cannot increase effective data by interacting with the environment. Therefore, data augmentation, as a simple technique, allows the policy model to locally explore the state of the data set. For data augmentation of states, if the state of the transformed data is within the domain of the state space, we consider it to be a valid transformation. By executing an effective enhancement strategy, we can artificially increase the amount of available training data. At the same time, for the above transformations, even if the transformation is physically feasible in the environment, an overly aggressive data augmentation strategy may eventually cause the policy model to converge in the wrong direction, because the reward of the original state may be inconsistent with the reward obtained from the transformed state.

[0109] By applying the method provided in the embodiment of the present invention, the original offline data is expanded through data enhancement, thereby increasing the training data in the model training process and ensuring the effectiveness and stability of the model after training.

[0110] In the method provided by the embodiment of the present invention, the updating of the value model based on the extended data and the strategy model includes:

[0111] Applying the strategy model to predict action parameters at the next moment;

[0112] Based on the action parameter at the next moment, the state parameter at the current moment, the action parameter, the reward parameter and the state parameter at the next moment in each of the extended data, obtaining the minimum Bellman error of the value model;

[0113] An expected value of the minimized Bellman error is obtained to update the value model based on the expected value.

[0114] Specifically, the expression formula for updating the value model is:

[0115]

[0116] Among them, t represents the current time, t+1 represents the next time, [r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t )] is to minimize the Bellman error, a t+1 The policy model predicts the action parameters for the next moment, γ is a hyperparameter with a value of (0,1), and E is the expectation.

[0117] In the embodiment of the present invention, the training data is augmented by combining the data enhancement algorithm, and the value model is updated by applying the augmented data and the strategy model. Therefore, the expression formula of the value model update combined with the data enhancement algorithm is combined to obtain the final expression formula of the value model update combined with the data enhancement process:

[0118]

[0119] in, and Represents data enhancement at time t+1 and time t.

[0120] Furthermore, in the model training process, after the value model is updated, the value model and the training data are applied to update the policy model, and the updating of the policy model based on the updated value model and the training data to obtain the updated policy model includes:

[0121] Extracting the state parameters and action parameters at the current moment in the training data;

[0122] Based on the state parameters and action parameters at the current moment in the training data, the output of the updated value model is maximized, and based on the maximization of the output of the updated value model, the policy model is updated.

[0123] In the method provided by the embodiment of the present invention, the policy model is updated by maximizing the value model output, and the update formula of the policy model is as follows:

[0124]

[0125] Among them, the current state parameter s in the training data t , and the action parameter α at the current moment t The input is sent to the policy model, which evaluates the action parameters of the state parameters and maximizes the data of the updated value model, and updates the policy model according to the output of the updated value model. According to the output of the updated value model, the expected value is obtained, and the policy model is updated according to the expected value.

[0126] Furthermore, in the above-mentioned process of updating the value model and the strategy model, each model parameter in the value model and the model parameter of the strategy model are actually updated according to the corresponding update formula, so that the strategy model can improve the action accuracy of the state and the accuracy of the value model's evaluation of the strategy model action during the next training.

[0127] The method provided in the embodiment of the present invention is applied to train the model in combination with data enhancement to ensure the effectiveness and stability of the model after training.

[0128] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0129] and Figure 1 Corresponding to the method described above, the embodiment of the present invention further provides a data processing device for Figure 1 In order to implement the method in the embodiment of the present invention, the data processing device provided in the embodiment of the present invention can be applied to a computer terminal or various mobile devices, and its structural diagram is as follows: Figure 4 As shown, specifically including:

[0130] The acquisition unit 401 is used to acquire various offline data in the industrial control system and construct a sample pool corresponding to each offline data, wherein the offline data includes multiple parameters, and each parameter included in the offline data is a state parameter, an action parameter, a reward parameter, a loss parameter at a current moment, and a state parameter at a next moment;

[0131] An execution unit 402 is used to initialize a preset value model and a strategy model, and execute a preset model training process on the value model and the strategy model;

[0132] Among them, the model training process includes: extracting any offline data in the sample pool as training data; applying a pre-set data enhancement mode to perform data enhancement processing on the training data to obtain multiple extended data corresponding to the training data; updating the value model based on the strategy model and each of the extended data to obtain an updated value model, and updating the strategy model based on the updated value model and the training data to obtain an updated strategy model; checking whether the updated strategy model and the updated value model have reached a convergence state; if not, continuing to execute the model training process based on the updated value model and the updated strategy model; if so, completing the model training process.

[0133] In the data processing device provided by the implementation of the present invention, when it is necessary to train the corresponding value model and policy model in any application scenario of the industrial control system, each offline data corresponding to the application scenario in the industrial control system is obtained, and a sample pool corresponding to each offline data is constructed, and each offline data is stored in the sample pool. The value model and the policy model are initialized, and the model training process is performed on the value model and the policy model at least once until the training is completed. In each training process, an offline data is extracted from the resource pool as training data, and the training data is enhanced by applying the data enhancement mode to obtain multiple expanded data. The value model is updated by applying each expanded data and policy model, and the policy model is updated by the training data and the updated value model. The convergence state of the two models is used to determine whether the training is completed. If not, the model training process continues.

[0134] By using the device provided in the embodiment of the present invention, the model is trained using offline data, and the data is amplified by data enhancement, thereby improving the effectiveness and stability of the model training.

[0135] In the device provided by the embodiment of the present invention, the execution unit includes:

[0136] A first determination subunit, used to determine each first model parameter in the value model and each second model parameter in the strategy model;

[0137] A second determining subunit, used to determine a preset first parameter value range of each of the first model parameters, and a second parameter value range of each of the second model parameters;

[0138] A selection subunit, used to randomly select any first parameter value from each of the first parameter value ranges, and randomly select any second parameter value from each of the second parameter value ranges;

[0139] The initialization subunit is used to apply the selected first parameter values ​​and second parameter values ​​to fill the value model and the strategy model, and complete the initialization of the value model and the strategy model.

[0140] In the device provided by the embodiment of the present invention, the execution unit includes:

[0141] A third determination subunit is used to determine each data enhancement method included in the data enhancement mode, and a data enhancement algorithm corresponding to each data enhancement method;

[0142] A fourth determining subunit, used to determine the data enhancement mode currently supported by the industrial control system;

[0143] The processing subunit is used to apply the data enhancement algorithm corresponding to the data enhancement method currently supported by the industrial control system to perform data enhancement processing on the training data.

[0144] In the device provided by the embodiment of the present invention, the execution unit includes:

[0145] The first training subunit is used to apply the strategy model to predict the action parameters at the next moment; based on the action parameters at the next moment, the state parameters at the current moment in each of the expanded data, the action parameters, the reward parameters and the state parameters at the next moment, obtain the minimized Bellman error of the value model; obtain the expected value of the minimized Bellman error to update the value model based on the expected value.

[0146] In the device provided by the embodiment of the present invention, the execution unit includes:

[0147] The second training sub-unit is used to extract the state parameters and action parameters at the current moment in the training data; maximize the output of the updated value model based on the state parameters and action parameters at the current moment in the training data, and update the strategy model based on the maximization of the output of the updated value model.

[0148] For the specific working process of each unit and sub-unit in the data processing device disclosed in the above embodiment of the present invention, reference can be made to the corresponding content in the data processing method disclosed in the above embodiment of the present invention, and will not be repeated here.

[0149] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned data processing method.

[0150] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 5As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to perform the following operations:

[0151] Obtain various offline data in the industrial control system, and construct a sample pool corresponding to each offline data, wherein the offline data includes multiple parameters, and each parameter included in the offline data is a state parameter, an action parameter, a reward parameter, a loss parameter at the current moment, and a state parameter at the next moment;

[0152] Initializing a preset value model and a strategy model, and executing a preset model training process on the value model and the strategy model;

[0153] Among them, the model training process includes: extracting any offline data in the sample pool as training data; applying a pre-set data enhancement mode to perform data enhancement processing on the training data to obtain multiple extended data corresponding to the training data; updating the value model based on the strategy model and each of the extended data to obtain an updated value model, and updating the strategy model based on the updated value model and the training data to obtain an updated strategy model; checking whether the updated strategy model and the updated value model have reached a convergence state; if not, continuing to execute the model training process based on the updated value model and the updated strategy model; if so, completing the model training process.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0155] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0156] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0157] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: include: Obtain various offline data in the industrial control system, and construct a sample pool corresponding to each offline data, wherein the offline data includes multiple parameters, and each parameter included in the offline data is a state parameter, an action parameter, a reward parameter, a loss parameter at the current moment, and a state parameter at the next moment; Initializing a preset value model and a strategy model, and executing a preset model training process on the value model and the strategy model; The model training process includes: extracting any offline data in the sample pool as training data; applying a preset data enhancement mode to perform data enhancement processing on the training data to obtain a plurality of extended data corresponding to the training data; updating the value model based on the strategy model and each of the extended data to obtain an updated value model, and updating the strategy model based on the updated value model and the training data to obtain an updated strategy model; checking whether the updated strategy model and the updated value model have reached a convergence state; if not, continuing to execute the model training process based on the updated value model and the updated strategy model; if so, completing the model training process; The application of a pre-set data enhancement mode to perform data enhancement processing on the training data includes: determining each data enhancement method included in the data enhancement mode, and a data enhancement algorithm corresponding to each data enhancement method; the data enhancement method at least includes zero-mean Gaussian noise, zero-mean uniform noise, random amplitude, random loss state and mixed state; determining the data enhancement method currently supported by the industrial control system; and applying the data enhancement algorithm corresponding to the data enhancement method currently supported by the industrial control system to perform data enhancement processing on the training data.

2. The method according to claim 1, characterized in that The initialization of the preset value model and strategy model includes: Determining each first model parameter in the value model and each second model parameter in the strategy model; Determine a preset first parameter value range of each of the first model parameters, and a preset second parameter value range of each of the second model parameters; Randomly select any first parameter value from each of the first parameter value ranges, and randomly select any second parameter value from each of the second parameter value ranges; The selected first parameter values ​​and second parameter values ​​are applied to fill the value model and the strategy model, thereby completing the initialization of the value model and the strategy model.

3. The method according to claim 1, characterized in that Updating the value model based on the expanded data and the strategy model includes: Applying the strategy model to predict action parameters at the next moment; Based on the action parameter at the next moment, the state parameter at the current moment, the action parameter, the reward parameter and the state parameter at the next moment in each of the extended data, obtaining the minimum Bellman error of the value model; An expected value of the minimized Bellman error is obtained to update the value model based on the expected value.

4. The method according to claim 3, characterized in that The updating of the policy model based on the updated value model and the training data to obtain the updated policy model includes: Extracting the state parameters and action parameters at the current moment in the training data; Based on the state parameters and action parameters at the current moment in the training data, the output of the updated value model is maximized, and based on the maximization of the output of the updated value model, the policy model is updated.

5. A data processing device, characterized in that: include: An acquisition unit is used to acquire various offline data in the industrial control system and construct a sample pool corresponding to each offline data, wherein the offline data includes multiple parameters, and each parameter included in the offline data is a state parameter, an action parameter, a reward parameter, a loss parameter at a current moment, and a state parameter at a next moment; An execution unit, used for initializing a preset value model and a strategy model, and executing a preset model training process on the value model and the strategy model; The model training process includes: extracting any offline data in the sample pool as training data; applying a preset data enhancement mode to perform data enhancement processing on the training data to obtain a plurality of extended data corresponding to the training data; updating the value model based on the strategy model and each of the extended data to obtain an updated value model, and updating the strategy model based on the updated value model and the training data to obtain an updated strategy model; checking whether the updated strategy model and the updated value model have reached a convergence state; if not, continuing to execute the model training process based on the updated value model and the updated strategy model; if so, completing the model training process; The execution unit includes: a third determination subunit, a fourth determination subunit and a processing subunit; The third determination subunit is used to determine each data enhancement method included in the data enhancement mode, and a data enhancement algorithm corresponding to each data enhancement method; The fourth determining subunit is used to determine the data enhancement mode currently supported by the industrial control system; The processing subunit is used to apply the data enhancement algorithm corresponding to the data enhancement method currently supported by the industrial control system to perform data enhancement processing on the training data.

6. The device according to claim 5, characterized in that The execution unit comprises: A first determination subunit, used to determine each first model parameter in the value model and each second model parameter in the strategy model; A second determining subunit, used to determine a preset first parameter value range of each of the first model parameters, and a second parameter value range of each of the second model parameters; A selection subunit, used to randomly select any first parameter value from each of the first parameter value ranges, and randomly select any second parameter value from each of the second parameter value ranges; The initialization subunit is used to apply the selected first parameter values ​​and second parameter values ​​to fill the value model and the strategy model, and complete the initialization of the value model and the strategy model.

7. A storage medium comprising stored instructions, wherein: When the instruction is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 4.

8. An electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the method according to any one of claims 1 to 4.

Images