Control method, device, storage medium and electronic device

Abstract

The present application provides a control method, device, storage medium and electronic device. The control method provided in the embodiment of the present application obtains state detection parameters for characterizing the physical state of the device to be controlled, and inputs the state detection parameters into a preset reinforcement deep learning model to determine the control instructions corresponding to the state detection parameters, wherein the preset reinforcement deep learning model is obtained by training a first data sample set consisting of actual measurement values ​​and a second data sample set consisting of simulated values ​​determined according to a preset simulation model and the first data sample set, so that the distribution of the learning strategy determined by the reinforcement deep learning model obtained based on the combination of real data and simulated data is closer to the real strategy distribution, thereby improving the matching degree between the control instructions determined based on the deep reinforcement learning model and the actual situation.

Description

Technical Field

[0001] The present application relates to the field of device control technology, and in particular to a control method, device, storage medium and electronic device. Background Art

[0002] With the development of the field of intelligent control, it is necessary to improve efficiency as much as possible while ensuring the safe and stable operation of the electronic equipment to be controlled.

[0003] Among them, deep reinforcement learning algorithms have demonstrated their power in different fields. The emergence of deep reinforcement learning has made reinforcement learning technology truly practical and able to solve complex control problems of electronic equipment in real scenarios.

[0004] However, the existing training of the deep reinforcement learning model based on the control algorithm mainly uses offline data to establish an imperfect simulation environment, and treats this simulation environment as a real environment for training. However, since the imperfect simulation environment established based on offline data cannot fully reflect the scene of the real environment, it is easy to cause the control instructions determined based on the deep reinforcement learning model to deviate greatly from the actual situation. Summary of the invention

[0005] The embodiments of the present application provide a control method, device, storage medium and electronic device to solve the technical problem that the control instructions determined according to the deep reinforcement learning model currently deviate greatly from the actual situation.

[0006] In a first aspect, an embodiment of the present application provides a control method, including:

[0007] Acquire a state detection parameter, where the state detection parameter is used to characterize the physical state of the device to be controlled;

[0008] A control instruction is determined according to the state detection parameters and a preset reinforcement deep learning model, and the control instruction is used to control the device to be controlled, wherein the preset reinforcement deep learning model is trained based on a first data sample set and a second data sample set, the first data sample set is a sample set composed of actual measurement values ​​corresponding to the state detection parameters, and the second data sample set is a sample set composed of simulated values ​​determined according to a preset simulation model.

[0009] In a possible design, the control method further includes:

[0010] Acquire the first data sample set;

[0011] Determine the second data sample set according to the first data sample set and the preset simulation model;

[0012] A mixed sample pool is determined according to the first data sample set and the second data sample set, and the mixed sample pool is used to train the preset reinforcement deep learning model.

[0013] In a possible design, determining the second data sample set according to the first data sample set and the preset simulation model includes:

[0014] Determine a data sample set to be selected according to the first data sample set and the preset simulation model;

[0015] Samples that meet a preset joint distribution restriction condition are screened from the candidate data sample set to determine the second data sample set.

[0016] In a possible design, determining a candidate data sample set according to the first data sample set and the preset simulation model includes:

[0017] Determining the to-be-selected data sample set according to the first data sample set and a preset joint conditional probability distribution model, wherein the joint conditional probability distribution model is a deep neural network model;

[0018] Correspondingly, the step of screening samples that meet the preset joint distribution restriction condition from the candidate data sample set includes:

[0019] Samples whose joint conditional probability is greater than a preset probability threshold are screened out from the candidate data sample set.

[0020] In a possible design, determining a candidate data sample set according to the first data sample set and the preset simulation model includes:

[0021] Determine the candidate data sample set according to the first data sample set and a preset adversarial network;

[0022] Correspondingly, the step of screening samples that meet the preset joint distribution restriction condition from the candidate data sample set includes:

[0023] Samples having a discriminant value greater than a preset discriminant threshold are screened out from the candidate data sample set, and the discriminant value is determined according to a discriminant network.

[0024] In a possible design, after determining a mixed sample pool according to the first data sample set and the second data sample set, the method further includes:

[0025] Using the first data sample set to train a preset benchmark strategy model;

[0026] Determine the difference in policy distribution between the trained baseline policy model and the preset learning policy model;

[0027] The preset learning strategy model, the preset reward value model and the preset safety value model are trained by using the mixed sample pool, wherein the strategy distribution difference is used as a regularization term of the preset learning strategy model, the preset reward value model and the preset safety value model;

[0028] When the number of training steps meets the preset conditions, the trained learning strategy model is used as the preset reinforcement deep learning model.

[0029] In a possible design, determining the difference in strategy distribution between the trained baseline strategy model and the preset learning strategy model includes:

[0030] Determine the strategy distribution difference between the trained benchmark strategy model and the preset learning strategy model according to a preset maximum mean difference algorithm; or,

[0031] The strategy distribution difference between the trained baseline strategy model and the preset learning strategy model is determined according to a preset divergence algorithm.

[0032] In one possible design, the preset benchmark strategy model uses a variational encoder.

[0033] In a possible design, during the training process of the preset learning strategy model, the preset reward value model, and the preset safety value model, a preset quadratic term constraint form is used to determine the training target.

[0034] In one possible design, the preset simulation model is any one or a combination of multiple of a deep neural network model, a recurrent neural network model, and a convolutional neural network model.

[0035] In a second aspect, an embodiment of the present application further provides a control device, including:

[0036] A parameter acquisition module, used to acquire state detection parameters, where the state detection parameters are used to characterize the physical state of the device to be controlled;

[0037] A control processing module, used to determine control instructions based on the state detection parameters and a preset reinforcement deep learning model, wherein the control instructions are used to control the device to be controlled, wherein the preset reinforcement deep learning model is trained based on a first data sample set and a second data sample set, the first data sample set is a sample set composed of actual measurement values ​​corresponding to the state detection parameters, and the second data sample set is a sample set composed of simulated values ​​determined according to a preset simulation model.

[0038] In a possible design, the control device further includes:

[0039] A sample acquisition module, which acquires the first data sample set;

[0040] A sample simulation module, used to determine the second data sample set according to the first data sample set and the preset simulation model;

[0041] A sample generation module is used to determine a mixed sample pool based on the first data sample set and the second data sample set, and the mixed sample pool is used to train the preset reinforcement deep learning model.

[0042] In a possible design, the sample simulation module is specifically used to:

[0043] Determine a data sample set to be selected according to the first data sample set and the preset simulation model;

[0044] Samples that meet a preset joint distribution restriction condition are screened from the candidate data sample set to determine the second data sample set.

[0045] In a possible design, the sample simulation module is specifically used to:

[0046] Determining the to-be-selected data sample set according to the first data sample set and a preset joint conditional probability distribution model, wherein the joint conditional probability distribution model is a deep neural network model;

[0047] Samples whose joint conditional probability is greater than a preset probability threshold are screened out from the candidate data sample set.

[0048] In a possible design, the sample simulation module is specifically used to:

[0049] Determine the candidate data sample set according to the first data sample set and a preset adversarial network;

[0050] Samples having a discriminant value greater than a preset discriminant threshold are screened out from the candidate data sample set, and the discriminant value is determined according to a discriminant network.

[0051] In a possible design, the control device further includes:

[0052] A model training module, used to train a preset benchmark strategy model using the first data sample set;

[0053] A difference determination module, used to determine the difference in strategy distribution between the trained baseline strategy model and the preset learning strategy model;

[0054] The model training module is also used to use the mixed sample pool to train the preset learning strategy model, the preset reward value model and the preset safety value model, wherein the strategy distribution difference is used as a regularization term for the preset learning strategy model, the preset reward value model and the preset safety value model, and when the number of training steps meets the preset conditions, the trained learning strategy model is used as the preset reinforcement deep learning model.

[0055] In a possible design, the difference determination module is specifically used to:

[0056] Determine the strategy distribution difference between the trained benchmark strategy model and the preset learning strategy model according to a preset maximum mean difference algorithm; or,

[0057] The strategy distribution difference between the trained baseline strategy model and the preset learning strategy model is determined according to a preset divergence algorithm.

[0058] In one possible design, the preset benchmark strategy model uses a variational encoder.

[0059] In a possible design, during the training process of the preset learning strategy model, the preset reward value model, and the preset safety value model, a preset quadratic term constraint form is used to determine the training target.

[0060] In one possible design, the preset simulation model is any one or a combination of multiple of a deep neural network model, a recurrent neural network model, and a convolutional neural network model.

[0061] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0062] processor; and,

[0063] A memory, configured to store executable instructions of the processor;

[0064] Wherein, the processor is configured to execute any one of the control methods in the first aspect by executing the executable instructions.

[0065] In a fourth aspect, an embodiment of the present application further provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the control methods in the first aspect.

[0066] A control method, device, storage medium and electronic device provided in the embodiments of the present application obtain state detection parameters for characterizing the physical state of a device to be controlled, and input the state detection parameters into a preset reinforcement deep learning model to determine control instructions corresponding to the state detection parameters, wherein the preset reinforcement deep learning model is trained by a first data sample set consisting of actual measured values ​​and a second data sample set consisting of simulated values ​​determined according to a preset simulation model and the first data sample set, so that the distribution of learning strategies determined by the reinforcement deep learning model obtained based on the combination of real data and simulated data is closer to the real strategy distribution, thereby improving the matching degree between the control instructions determined based on the deep reinforcement learning model and the actual situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0068] Figure 1 is an application scenario diagram of a control method according to an exemplary embodiment of the present application;

[0069] Figure 2 is a flowchart of a control method according to an exemplary embodiment of the present application;

[0070] Figure 3 is a schematic diagram of a construction process of a mixed sample pool according to an exemplary embodiment of the present application;

[0071] Figure 4 is a flow chart of a hybrid training method according to an exemplary embodiment of the present application;

[0072] Figure 5 It is a flowchart of a training method for a reinforced deep learning model according to an example embodiment of the present application;

[0073] Figure 6 This is a flowchart of an off-policy reinforcement learning method with safety constraints and policy value regularization according to an example embodiment of the present application;

[0074] Figure 7 is a schematic structural diagram of a control device according to an exemplary embodiment of the present application;

[0075] Figure 8 is a schematic structural diagram of a control device according to another exemplary embodiment of the present application;

[0076] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0079] With the development of the field of intelligent control, it is necessary to improve efficiency as much as possible while ensuring the safe and stable operation of the electronic equipment to be controlled. Among them, deep reinforcement learning algorithms have shown their magical powers in different fields. The emergence of deep reinforcement learning has made reinforcement learning technology truly practical and able to solve complex control problems of electronic equipment in real scenarios. However, the existing training of the deep reinforcement learning model based on the control algorithm is mainly to establish an imperfect simulation environment through offline data, and treat this simulation environment as a real environment for training. However, since the imperfect simulation environment established based on offline data cannot fully reflect the scenario of the real environment, it is easy to cause the control instructions determined based on the deep reinforcement learning model to deviate greatly from the actual situation.

[0080] In addition, it is worth mentioning that deep reinforcement learning is the product of the combination of deep learning and reinforcement learning. It integrates the powerful understanding ability of deep learning in perceptual problems such as vision and the decision-making ability of reinforcement learning, and realizes end-to-end learning. Moreover, the emergence of deep reinforcement learning has made reinforcement learning technology truly practical and able to solve complex problems in real-life scenarios. In the past few years, deep reinforcement learning algorithms have demonstrated their magical powers in different fields, for example, defeating top human players in video games and chess games; controlling complex machinery for operation; allocating network resources; significantly saving energy for data centers; and even automatically adjusting parameters for machine learning algorithms.

[0081] The control field is one of the birthplaces of reinforcement learning ideas and is also the most mature field for the application of reinforcement learning technology. A common example is the use of artificial intelligence to tune machines and equipment, which previously required expert operators to complete. For example, the reinforcement learning technology of DeepMind in the UK has helped Google significantly reduce the energy consumption of its data centers. In the field of autonomous driving, driving is a sequential decision-making process, so it is naturally suitable for reinforcement learning. From ALVINN and TORCS in the 1980s to CARLA today, the industry has been trying to use reinforcement learning to solve the problem of single-vehicle autonomous driving and multi-vehicle traffic scheduling. In addition, similar ideas are also widely used in various aircraft and underwater drone fields. Compared with tasks in the field of computer vision, many tasks in the field of natural language processing are multi-round, that is, they need to seek the optimal solution through multiple iterative interactions, such as: dialogue system; and the feedback signal of the task often needs to be obtained after a series of decisions, such as: machine writing. The characteristics of such problems are naturally suitable for reinforcement. Therefore, in recent years, reinforcement has been applied to many tasks in the field of natural language processing, such as text generation, text summarization, sequence labeling, dialogue robots (text / voice), machine translation, relationship extraction and knowledge graph reasoning, etc. In addition, reinforcement learning is also widely used in product recommendation, news recommendation, online advertising, finance, communications, production scheduling, planning and resource access control and other operational fields, and even in education and training, health care and medicine.

[0082] In addition, the basic idea of ​​reinforcement learning is that the agent learns through interaction with the environment, and the reward obtained guides the behavior. The goal is to find the optimal strategy for the agent to obtain the maximum cumulative reward. The Markov decision process MDP (Markov Decision Processes) in reinforcement learning is defined as a four-tuple M = (S, A, R, T), where S represents the state space of the environment, A represents the action space of the agent, R (s, a) represents the reward function, and the value it returns represents the reward obtained by performing action a in state s, and T (s′|s, a) is the state transition probability function, which represents the probability that the environment transfers to state s′ after performing action a in state s. The goal of reinforcement learning is to find a mapping from environmental state to action, that is, strategy π (a|s), which can maximize the expectation of future rewards. R t is the reward at time t, and γ is the discount factor. The agent selects the optimal action a under the current state s according to the strategy π, executes the action and observes the reward r and the next state s′ fed back by the environment, adjusts and updates the strategy π based on the reward fed back, and iterates continuously until an optimal strategy is found to maximize positive feedback.

[0083] The deep reinforcement learning algorithm training includes two strategies: behavioral strategy and target strategy. The behavioral strategy is a strategy used to interact with the environment to generate data, that is, to make decisions during the exploration process; and the target strategy is to continuously learn and optimize from the data generated by the behavioral strategy, that is, the strategy to be actually applied after training. Based on this, reinforcement learning can be divided into two categories of algorithms: on-policy (same strategy) and off-policy (different strategy). The behavioral strategy and target strategy of the on-policy algorithm are the same strategy. Its advantage is that it is simple and direct, and its strategy can be optimized directly using data. However, since it cannot maintain exploration and utilization well, it is easy to fall into the local optimum; the different-policy algorithm separates the target strategy from the behavioral strategy, which can better obtain the global optimal solution while maintaining exploration. However, the learning process of the different-policy algorithm is more tortuous and converges slowly.

[0084] In many real-world application scenarios, due to many practical factors such as high experimental costs and safety considerations, it is often impossible to directly access the real environment, but some offline experience data sets are collected. Existing technologies mainly use these offline data to establish an imperfect simulation environment, and use this simulation environment as a real environment for training. However, the collected offline data is often a fixed-scale offline experience data set. The amount of data cannot meet the needs of general reinforcement learning algorithms. The data distribution only covers a part of the real sample space and may not cover the real optimal solution. Therefore, the existing algorithms have the following disadvantages:

[0085] 1. The imperfect simulation environment built based on offline data cannot fully reflect the scenario of the real environment. In the sample space with dense offline data, the feedback from the simulation environment can fit the feedback from the real environment very well, but in the sample space with sparse historical data, the feedback from the simulation environment deviates from reality. When exploring the sample space with sparse offline data distribution, since the simulation environment cannot give real feedback, it will cause the strategy to "overfit" the simulation environment and fail to converge to the real optimal solution.

[0086] 2. Actual application scenarios, such as the field of industrial control, usually require the highest possible economic benefits under the premise of ensuring safe and stable operation. Therefore, the actual production environment often has clear and strict safety restrictions. The collected offline data cannot reflect the specific safety restrictions. At the same time, the state and action space are usually high-dimensional and continuous. When the reinforcement learning algorithm learns strategies on such offline data sets, its performance is very sensitive to the distribution of the empirical data set. The reinforcement algorithm trained based on the simulation environment may deviate greatly from the offline data.

[0087] In response to the above technical problems, the present application provides a control method, device, storage medium and electronic device, which obtains state detection parameters used to characterize the physical state of the device to be controlled, and inputs the state detection parameters into a preset reinforcement deep learning model to determine the control instructions corresponding to the state detection parameters, wherein the preset reinforcement deep learning model is trained by a first data sample set consisting of actual measured values ​​and a second data sample set consisting of simulated values ​​determined according to a preset simulation model and the first data sample set, so that the distribution of learning strategies determined by the reinforcement deep learning model obtained based on the combination of real data and simulated data is closer to the real strategy distribution, thereby improving the matching degree between the control instructions determined based on the deep reinforcement learning model and the actual situation.

[0088] Figure 1 is an application scenario diagram of a control method according to an exemplary embodiment of the present application. Figure 1 As shown, the control method provided in this embodiment can be used to control a device to be controlled, wherein the device to be controlled is a car, a drone, an aircraft, a smart phone, or any other electronic device with a control function. The device to be controlled can include an input component, a control component, and an execution component. Here, the device to be controlled can be a car for example. Figure 1 The input component may be a sensor, for example, a speed sensor 101, a temperature sensor 102, and a collision sensor 103. The control component may be a vehicle controller 200. The execution component may be an actuator 300, for example, a motor, an airbag ignition speed, an automatic steering system, etc.

[0089] The vehicle obtains state parameters through sensors, where the state parameters are used to characterize the physical state of the vehicle, such as the current vehicle speed. After obtaining the vehicle speed, the controller 200 determines a control instruction through a built-in preset reinforcement deep learning model, and the control instruction can be an acceleration instruction or a deceleration instruction to dynamically adjust the vehicle speed.

[0090] Among them, it is worth mentioning that the preset reinforcement deep learning model can be trained according to the first data sample set and the second data sample set. The first data sample set is a sample set composed of actual measurement values ​​corresponding to the state detection parameters, for example, it can be the actually detected vehicle speed and the corresponding action, and the second data sample set is a sample set composed of simulation values ​​determined according to the preset simulation model.

[0091] Figure 2 FIG. 1 is a flow chart of a control method according to an exemplary embodiment of the present application. Figure 2 As shown, the control method provided in this embodiment includes:

[0092] Step 101: Obtain status detection parameters.

[0093] Specifically, the device to be controlled can measure the current state detection parameter through the input device, and the detected state detection parameter can be one or more. Among them, the state detection parameter can be understood as a physical state that characterizes the device to be controlled, such as speed, temperature, height, humidity, level state, etc.

[0094] Step 102: Determine control instructions based on state detection parameters and a preset reinforcement deep learning model.

[0095] After obtaining the state detection parameters of the device to be controlled, the controller of the device to be controlled can process the input state detection parameters to output the corresponding control decision. Specifically, the controller of the device to be controlled can make a control decision based on a single state detection parameter or a joint control decision based on multiple state detection parameters. Finally, after the decision is made, a control instruction is output to control the execution component of the device to be controlled to achieve the corresponding control function.

[0096] In this step, the processing logic performed in the controller can be implemented based on a preset enhanced deep learning model. The preset enhanced deep learning model is trained based on a first data sample set and a second data sample set. The first data sample set is a sample set consisting of actual measurement values ​​corresponding to state detection parameters, and the second data sample set is a sample set consisting of simulation values ​​determined based on a preset simulation model. In this way, the first data sample set collected from the real environment is fully utilized to learn an effective control decision strategy while ensuring safety and stability.

[0097] Step 103: Control the device to be controlled according to the control instruction.

[0098] In this embodiment, state detection parameters used to characterize the physical state of the device to be controlled are obtained, and the state detection parameters are input into a preset reinforcement deep learning model to determine the control instructions corresponding to the state detection parameters, wherein the preset reinforcement deep learning model is trained by a first data sample set consisting of actual measured values ​​and a second data sample set consisting of simulated values ​​determined according to a preset simulation model and the first data sample set, so that the distribution of learning strategies determined by the reinforcement deep learning model obtained based on the combination of real data and simulated data is closer to the real strategy distribution, thereby improving the matching degree between the control instructions determined based on the deep reinforcement learning model and the actual situation.

[0099] Figure 3 FIG. 1 is a schematic diagram of a construction process of a mixed sample pool according to an exemplary embodiment of the present application. Figure 3 As shown, the control method provided in this embodiment includes:

[0100] Step 201: Obtain a first data sample set.

[0101] The first data sample set may be offline real data collected from a real environment.

[0102] Step 202: Determine a second data sample set according to the first data sample set and a preset simulation model.

[0103] However, the offline real data collected in the real environment is often limited in quantity, and the use of offline data alone cannot meet the needs of general reinforcement learning algorithms. If the simulation data generated by the imperfect simulation environment is used alone, the reinforcement algorithm strategy trained may deviate greatly from the historical data, which is not conducive to the safety and stability required for the control system in the real world. Therefore, a hybrid training method combining simulation data and real data can be adopted. Adding simulation data, that is, the second data sample set, can solve the problem of insufficient offline data. In addition, adding real offline data can also alleviate the deviation caused by the simulation environment.

[0104] Among them, the simulation model is trained using real offline data, and its model input is the current state s and the executed action a, and the output is the next state s′, reward feedback r and safety limit c. According to the actual application requirements, there are several options for the simulation model here, including deep neural networks (DNN), recurrent neural networks (RNN), convolutional neural networks (CNN) and composite neural networks composed of any of the above three. For environments with strong temporal correlation, such as industrial control, RNN that reflects temporal correlation can be selected; for chess games with weak temporal correlation, DNN can be selected; for video games, both visual features need to be extracted and strong temporal correlation is required, and the structure of RNN and CNN combination can be selected.

[0105] Step 203: Determine a mixed sample pool according to the first data sample set and the second data sample set.

[0106] In addition, since the state and action spaces are usually high-dimensional and continuous, the data distribution in high-dimensional space is usually very sparse, and the simulated data generated by exploration in an imperfect simulation environment will deviate from the real data distribution, creating safety hazards in practical applications. Figure 4 is a flow chart of a hybrid training method according to an exemplary embodiment of the present application, such as Figure 4 As shown, in order to solve the problem of simulation data deviation, reliable simulation data can also be obtained by screening based on the joint distribution restriction strategy.

[0107] Continue to refer to Figure 4 , using the joint distribution restriction strategy to generate reliable simulation data. First, perform conventional exploration, even if a set of data is randomly selected from the offline data as a trajectory (time series state action sequence s1, a1, s2, a2…s t , a t ), use the strategy model in the reinforcement learning algorithm to get the action a at the next moment, and then input the simulation model to get the reward, safety score and the state at the next moment, so as to obtain a set of single-step transfer data (s, a, r, c, s′). Then use the joint distribution restriction strategy to screen whether this set of data is reliable. If it is unreliable, discard this set of single-step transfer data and terminate the exploration; if it is reliable, add this set of single-step transfer data to the mixed sample pool and perform the next step of exploration until the maximum number of exploration steps is reached and the exploration is terminated. As for the specific joint distribution restriction strategy, it can be implemented in the following ways:

[0108] In a possible design, the joint conditional probability distribution model may be trained using real offline data. A candidate data sample set may be determined based on the first data sample set and a preset simulation model, and samples that meet the preset joint distribution restriction conditions may be screened from the candidate data sample set to determine the second data sample set. The candidate data sample set may be determined based on the first data sample set and a preset joint conditional probability distribution model, wherein the joint conditional probability distribution model is a deep neural network model, and samples whose joint conditional probability is greater than a preset probability threshold are screened from the candidate data sample set.

[0109] Specifically, it can be based on the real offline data distribution. Here, there are several options for the joint conditional probability distribution, including traditional continuous probability distribution models, such as normal distribution, exponential distribution, and beta distribution. The model uses a deep neural network DNN to directly fit the joint conditional probability distribution based on the real offline data distribution. The model input is the current state s and the executed action a, and the output is the joint conditional probability p(a|s). A reasonable probability threshold can be selected, and data exceeding the threshold is reliable data.

[0110] In another possible design, a candidate data sample set may be determined based on the first data sample set and a preset adversarial network, and then samples with a discriminant value greater than a preset discriminant threshold may be screened out from the candidate data sample set, and the discriminant value may be determined based on the discriminant network. Specifically, a generative adversarial network (GAN) may be trained using real offline data. A reasonable threshold is selected, and (s, a) is evaluated using a discriminant network. Data exceeding the threshold is considered reliable data.

[0111] Finally, real data is extracted in a certain proportion and added to the mixed sample pool, and a certain number of samples are sampled from the sample pool for training the reinforcement learning algorithm. When the number of training steps reaches the preset maximum number of training steps, the training is stopped.

[0112] In this embodiment, a hybrid training method based on a joint distribution restriction strategy is used to effectively alleviate the problem of learning strategy deviation by screening simulation data generated by an imperfect simulation environment.

[0113] Figure 5 FIG. 1 is a flow chart of a training method for a reinforced deep learning model according to an exemplary embodiment of the present application. Figure 5 As shown, the control method provided in this embodiment includes:

[0114] Step 301: Obtain a first data sample set.

[0115] Step 302: Determine a second data sample set according to the first data sample set and a preset simulation model.

[0116] Step 303: Determine a mixed sample pool according to the first data sample set and the second data sample set.

[0117] It is worth noting that the specific description of steps 301 to 303 in this embodiment can be found in Figure 3 The description of step 201 to step 203 in the illustrated embodiment will not be repeated here.

[0118] Step 304: Use the first data sample set to train the preset benchmark strategy model.

[0119] In actual application environments, certain security constraints often need to be met, and simulation models cannot accurately evaluate the security risks of strategies. Figure 6 This is a flowchart of an off-policy reinforcement learning method with safety constraints and policy value regularization according to an example embodiment of the present application, such as Figure 6 As shown, in this embodiment, a security value model can be introduced to evaluate the security risk of the current strategy, and the security requirements can be met while optimizing the strategy. At the same time, strategy value regularization is introduced to further correct the strategy distribution deviation and value assessment deviation caused by the simulation data distribution deviation.

[0120] Specifically, a certain number of samples can be sampled from real data to train the benchmark policy model. The benchmark policy model is only trained with real data, not the optimal policy, but it can reflect the distribution of real data. The distribution constraint is calculated using the distribution difference between the benchmark policy and the learning policy. Regularizing the learning policy model and the value model can correct the deviation caused by the simulation data.

[0121] For the baseline policy model, a variational auto-encoder (VAE) can be used. VAE consists of two parts of the network, one part is called the encoder, which maps from a high-dimensional input to a low-dimensional latent variable, and the other part is called the decoder, which maps from the low-dimensional latent variable back to the high-dimensional input. It is divided into a model training phase and a sampling phase. In the training phase, the model input is the current state s and the executed action a, and the output is action a. The encoder and decoder are trained together. In the sampling phase, only the decoder part is used to batch sample a certain number of samples.

[0122] Step 305: Determine the difference in strategy distribution between the trained baseline strategy model and the preset learning strategy model.

[0123] The calculation of the above-mentioned strategy distribution difference may be to determine the strategy distribution difference between the trained benchmark strategy model and the preset learning strategy model according to a preset maximum mean discrepancy (MMD) algorithm. Alternatively, the strategy distribution difference between the trained benchmark strategy model and the preset learning strategy model may be determined according to a preset divergence algorithm.

[0124] Specifically, for the preset maximum mean difference algorithm, it can be achieved by θ and π b Sampling is performed to calculate the strategy π θ and π b The estimated value of the MMD distance.

[0125]

[0126] K is the kernel function, which can be a Gaussian kernel function or a Laplace kernel function; π θ is the learning strategy model, π b is the baseline policy model, E is the mathematical expectation, x is the sampling value of the learning policy model in state s, x′ is the sampling value of the learning policy model in state s that is different from x, y is the sampling value of the baseline policy model in state s, and y′ is the sampling value of the baseline policy model in state s that is different from y.

[0127] For the preset divergence algorithm, KL divergence can be used. Directly estimate π θ and π b The KL divergence of θ and π b The probability density function of . The dual form of KL divergence can be used here.

[0128] Step 306: Use the mixed sample pool to train a preset learning strategy model, a preset reward value model, and a preset security value model.

[0129] In this step, the mixed sample pool can be used to train the preset learning strategy model, the preset reward value model, and the preset security value model, wherein the strategy distribution difference is used as a regularization term of the preset learning strategy model, the preset reward value model, and the preset security value model. In addition, during the training process of the preset learning strategy model, the preset reward value model, and the preset security value model, the training target is determined using the preset quadratic term constraint form.

[0130] A certain number of samples are sampled from the mixed sample pool to train the learning strategy model, reward value model and safety value model. The training method adopts the classic "actor-critic" method to learn an action value function by minimizing the Bellman errors on the single-step transfer data (s, a, r, s′), and then the strategy is updated by maximizing the action value function. The quadratic constraint form is used to increase the stability of the model training, and the learning objectives of the strategy are as follows:

[0131]

[0132] Among them, a′ is the action value at the current moment, a″ is the action value at the next moment, and s′ is the state value at the next moment.

[0133] The objective of the reward value function is:

[0134]

[0135] Similarly, the target of the safety value function can be obtained.

[0136] Among them, π b is the benchmark strategy, π θ To learn strategies, is the experience pool, Q ψ is the reward value function or security value function to be learned, represents the target value function, Indicates the distribution difference between the baseline strategy and the learning strategy. Both λ and ρ are adjustable parameters. Alternate iterative training is performed according to the objective function.

[0137] Step 307: When the number of training steps meets the preset conditions, the trained learning strategy model is used as the preset reinforcement deep learning model.

[0138] When the number of training steps reaches the maximum number of training steps, the training is stopped, and the trained learning strategy model is used as the preset reinforcement deep learning model.

[0139] In this embodiment, a safe and effective strategy is learned through joint distribution restricted exploration, safety constraints and policy value regularization. First, the simulation model is trained based on real data, and joint distribution restricted exploration is performed in the simulation environment to generate reliable simulation data. Then, the policy distribution network is trained based on real data to obtain the real policy distribution as the policy value regularization of the reinforcement learning algorithm. Finally, the simulated data and real data are mixed to generate a mixed sample pool for policy learning of the reinforcement learning algorithm, and the policy regularization generated in the second step is introduced to constrain the distribution of the learning strategy while correcting the error of the value evaluation, so that the distribution of the learning strategy is closer to the real policy distribution.

[0140] Specifically, the safety restriction method based on the safety value model solves the problem of inaccurate safety hazard assessment in the simulation environment by introducing the safety value model, thereby ensuring the safety and stability requirements in the actual application environment. In addition, the reinforcement learning method based on policy value regularization of policy distribution difference introduces the distribution difference between the learning policy and the benchmark policy as the regularization term of the value model and the learning policy model, which can remove the policy distribution bias and value estimation bias caused by the simulation data. At the same time, the quadratic constraint form can further improve the stability of training.

[0141] Figure 7 is a schematic diagram of the structure of a control device according to an exemplary embodiment of the present application. Figure 7 As shown, the control device 400 provided in this embodiment includes:

[0142] The parameter acquisition module 401 is used to acquire state detection parameters, where the state detection parameters are used to characterize the physical state of the device to be controlled;

[0143] The control processing module 402 is used to determine the control instructions according to the state detection parameters and the preset reinforcement deep learning model, and the control instructions are used to control the device to be controlled, wherein the preset reinforcement deep learning model is trained based on the first data sample set and the second data sample set, the first data sample set is a sample set composed of the actual measurement values ​​corresponding to the state detection parameters, and the second data sample set is a sample set composed of simulation values ​​determined according to the preset simulation model.

[0144] exist Figure 7 Based on the embodiment shown, Figure 8 FIG. 1 is a schematic diagram of a control device according to another exemplary embodiment of the present application. Figure 8 As shown, the control device 400 further includes:

[0145] The sample acquisition module 403 acquires the first data sample set;

[0146] A sample simulation module 404, configured to determine the second data sample set according to the first data sample set and the preset simulation model;

[0147] The sample generation module 405 is used to determine a mixed sample pool according to the first data sample set and the second data sample set, and the mixed sample pool is used to train the preset reinforcement deep learning model.

[0148] In a possible design, the sample simulation module 404 is specifically used to:

[0149] Determine a data sample set to be selected according to the first data sample set and the preset simulation model;

[0150] Samples that meet a preset joint distribution restriction condition are screened from the candidate data sample set to determine the second data sample set.

[0151] In a possible design, the sample simulation module 404 is specifically used to:

[0152] Determining the to-be-selected data sample set according to the first data sample set and a preset joint conditional probability distribution model, wherein the joint conditional probability distribution model is a deep neural network model;

[0153] Samples whose joint conditional probability is greater than a preset probability threshold are screened out from the candidate data sample set.

[0154] In a possible design, the sample simulation module 404 is specifically used to:

[0155] Determine the candidate data sample set according to the first data sample set and a preset adversarial network;

[0156] Samples having a discriminant value greater than a preset discriminant threshold are screened out from the candidate data sample set, and the discriminant value is determined according to a discriminant network.

[0157] In a possible design, the control device 400 further includes:

[0158] A model training module 406, configured to train a preset benchmark strategy model using the first data sample set;

[0159] A difference determination module 407, used to determine the difference in strategy distribution between the trained baseline strategy model and the preset learning strategy model;

[0160] The model training module 406 is also used to use the mixed sample pool to train the preset learning strategy model, the preset reward value model and the preset safety value model, wherein the strategy distribution difference is used as a regularization term for the preset learning strategy model, the preset reward value model and the preset safety value model, and when the number of training steps meets the preset conditions, the trained learning strategy model is used as the preset reinforcement deep learning model.

[0161] In a possible design, the difference determination module 407 is specifically used to:

[0162] Determine the strategy distribution difference between the trained benchmark strategy model and the preset learning strategy model according to a preset maximum mean difference algorithm; or,

[0163] The strategy distribution difference between the trained baseline strategy model and the preset learning strategy model is determined according to a preset divergence algorithm.

[0164] In one possible design, the preset benchmark strategy model uses a variational encoder.

[0165] In a possible design, during the training process of the preset learning strategy model, the preset reward value model, and the preset safety value model, a preset quadratic term constraint form is used to determine the training target.

[0166] In one possible design, the preset simulation model is any one or a combination of multiple of a deep neural network model, a recurrent neural network model, and a convolutional neural network model.

[0167] This embodiment provides a control device that can be used to execute the above method embodiment. For details not disclosed in the device embodiment of this application, please refer to the method embodiment of this application.

[0168] Fig. 9 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Fig. 9 As shown, this embodiment provides an electronic device 500, including:

[0169] processor 501; and,

[0170] A memory 502, used to store executable instructions of the processor, and the memory may also be a flash memory;

[0171] The processor 501 is configured to execute the steps in the above method by executing the executable instructions. For details, please refer to the above description of the functions of each component of the streaming data processing system.

[0172] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0173] When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include:

[0174] The bus 503 is used to connect the processor 501 and the memory 502 .

[0175] This embodiment also provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes the functions of the various components in the streaming data processing system provided by the various embodiments described above.

[0176] This embodiment also provides a program product, which includes a computer program, which is stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements the functions of each component in the streaming data processing system provided by the various embodiments described above.

[0177] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A control method, characterized in that: include: Acquire state detection parameters through sensors, where the state detection parameters are used to characterize the physical state of the device to be controlled; Determine a control instruction according to the state detection parameter and a preset reinforcement deep learning model, wherein the control instruction is used to control the device to be controlled, wherein the preset reinforcement deep learning model is trained according to a first data sample set and a second data sample set, and the first data sample set is a sample set composed of actual measurement values ​​corresponding to the state detection parameter; The method further includes: acquiring the first data sample set; Determine the second data sample set according to the first data sample set and the preset simulation model; Determine a mixed sample pool according to the first data sample set and the second data sample set; the mixed sample pool is used to train the preset reinforcement deep learning model; Using the first data sample set to train a preset benchmark strategy model; Determine the difference in policy distribution between the trained baseline policy model and the preset learning policy model; The preset learning strategy model, the preset reward value model and the preset safety value model are trained by using the mixed sample pool, wherein the strategy distribution difference is used as a regularization term of the preset learning strategy model, the preset reward value model and the preset safety value model; When the number of training steps meets the preset conditions, the trained learning strategy model is used as the preset reinforcement deep learning model.

2. The control method according to claim 1, characterized in that: The determining the second data sample set according to the first data sample set and the preset simulation model includes: Determine a data sample set to be selected according to the first data sample set and the preset simulation model; Samples that meet a preset joint distribution restriction condition are screened from the candidate data sample set to determine the second data sample set.

3. The control method according to claim 2, characterized in that: The step of determining a candidate data sample set according to the first data sample set and the preset simulation model includes: Determining the to-be-selected data sample set according to the first data sample set and a preset joint conditional probability distribution model, wherein the joint conditional probability distribution model is a deep neural network model; Correspondingly, the step of screening samples that meet the preset joint distribution restriction condition from the candidate data sample set includes: Samples whose joint conditional probability is greater than a preset probability threshold are screened out from the candidate data sample set.

4. The control method according to claim 2, characterized in that: The step of determining a candidate data sample set according to the first data sample set and the preset simulation model includes: Determine the candidate data sample set according to the first data sample set and a preset adversarial network; Correspondingly, the step of screening samples that meet the preset joint distribution restriction condition from the candidate data sample set includes: Samples having a discriminant value greater than a preset discriminant threshold are screened out from the candidate data sample set, and the discriminant value is determined according to a discriminant network.

5. The control method according to claim 1, characterized in that: The determining of the difference in strategy distribution between the trained baseline strategy model and the preset learning strategy model includes: Determine the strategy distribution difference between the trained benchmark strategy model and the preset learning strategy model according to a preset maximum mean difference algorithm; or, The strategy distribution difference between the trained baseline strategy model and the preset learning strategy model is determined according to a preset divergence algorithm.

6. The control method according to claim 1, characterized in that: The preset benchmark strategy model uses a variational encoder.

7. The control method according to claim 1, characterized in that: During the training process of the preset learning strategy model, the preset reward value model and the preset safety value model, the training target is determined using the preset quadratic term constraint form.

8. The control method according to any one of claims 1 to 4, characterized in that: The preset simulation model is any one or a combination of multiple of a deep neural network model, a recurrent neural network model and a convolutional neural network model.

9. A control device, characterized in that: include: A parameter acquisition module, used to acquire state detection parameters through sensors, wherein the state detection parameters are used to characterize the physical state of the device to be controlled; A control processing module, used to determine a control instruction according to the state detection parameter and a preset enhanced deep learning model, wherein the control instruction is used to control the device to be controlled, wherein the preset enhanced deep learning model is trained according to a first data sample set and a second data sample set, and the first data sample set is a sample set composed of actual measurement values ​​corresponding to the state detection parameter; A sample acquisition module, used to acquire the first data sample set; A sample simulation module, used to determine the second data sample set according to the first data sample set and the preset simulation model; A sample generation module, used to determine a mixed sample pool according to the first data sample set and the second data sample set; the mixed sample pool is used to train the preset reinforcement deep learning model; A model training module, used to train a preset benchmark strategy model using the first data sample set; A difference determination module, used to determine the difference in strategy distribution between the trained baseline strategy model and the preset learning strategy model; The model training module is further used to train the preset learning strategy model, the preset reward value model and the preset security value model using the mixed sample pool, wherein the strategy distribution difference is used as a regularization term of the preset learning strategy model, the preset reward value model and the preset security value model; When the number of training steps meets the preset conditions, the trained learning strategy model is used as the preset reinforcement deep learning model.

10. An electronic device, characterized in that: include: processor; as well as a memory for storing a computer program for the processor; Wherein, the processor is configured to implement the control method according to any one of claims 1 to 8 by executing the computer program.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the control method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the control method according to any one of claims 1 to 8 is implemented.

Images