A method for accelerating training of reinforcement learning in a large-scale simulation environment

Abstract

The invention discloses an accelerated training method for reinforcement learning in a large-scale simulation environment, including: building an environment simulation model, an initial reinforcement learning model and a self-supervision model, generating image data through continuous interaction between the environment simulation model and the initial reinforcement learning model; using a distributed architecture to collect image data; extracting low-dimensional features in the image data through a self-supervision model to obtain feature coding; saving multiple historical state information of the feature coding based on the reinforcement learning model, and assigning different weight coefficients to different historical state information, finally obtaining a state coding that satisfies the Markov property, outputting a set of sequence data as training data; performing distributed training on the initial reinforcement learning model based on the training data until the optimization target is met, and obtaining the final reinforcement learning model. The invention has fast training speed and can maximize the utilization of sampled data.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and more specifically to an accelerated training method for reinforcement learning in a large-scale simulation environment. Background Art

[0002] The current mainstream reinforcement learning technology is often very simple in feature extraction. If it is image data, a simple CNN network is used to convert high-dimensional image data into low-dimensional data features, and finally sent to the model for training and reasoning. If it is discrete data, a simple fully connected neural network is used to extract features of discrete data, and finally sent to the model for training and reasoning. Such simple feature extraction makes the state less likely to satisfy the Markov property, which easily leads to unsatisfactory convergence of the reinforcement learning model and also limits the training speed of reinforcement learning.

[0003] At present, mainstream reinforcement learning algorithms often use distributed means to collect data. However, reinforcement learning currently solves distributed problems on smaller scales, such as simple game scenarios. Therefore, most distributed frameworks are single-node multi-process distributed, and simulation environments for high-performance scientific and engineering applications often require a dedicated server. There are relatively few general frameworks for multi-node distributed reinforcement learning in large-scale simulation engineering environments. When reinforcement learning interacts with game simulators, the game simulator can give reinforcement learning corresponding feedback at a response speed of seconds. However, when reinforcement learning interacts with industrial-specific simulation environments, such as Fluent and other fluid mechanics simulation software, these industrial-level simulation software takes a lot of time to simulate once, and the reinforcement learning agent can only get feedback once, resulting in a complete interaction between the reinforcement learning agent and the large-scale simulation environment software. It takes more than ten or even twenty minutes. If reinforcement learning cannot make full use of the obtained data, it will seriously hinder the training process of reinforcement learning.

[0004] Therefore, how to provide a fast reinforcement learning training method in a large-scale simulation environment that can maximize the utilization of the obtained images is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides an accelerated training method for reinforcement learning in a large-scale simulation environment, which has the advantages of fast training speed and can maximize the utilization of sampled data.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for accelerating training of reinforcement learning in a large-scale simulation environment comprises the following steps:

[0008] Building an environment simulation model, an initial reinforcement learning model and a self-supervision model, and generating image data for representing the distribution states of various physical quantities in the environment simulation model through continuous interaction between the environment simulation model and the initial reinforcement learning model;

[0009] Acquiring the image data using a distributed architecture;

[0010] Extracting low-dimensional features from the image data through the self-supervised model to obtain feature encoding;

[0011] Based on the reinforcement learning model, multiple historical state information of the feature encoding is saved, different weight coefficients are assigned to different historical state information, and finally a state encoding that satisfies the Markov property is obtained, and a set of sequence data is output as training data;

[0012] The initial reinforcement learning model is subjected to distributed training based on the training data until an optimization objective is met, thereby obtaining a final reinforcement learning model.

[0013] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the training process of the self-supervisory model is:

[0014] Extracting a set of pictures from the image data generated by the interaction between the initial reinforcement learning model and the environment simulation model;

[0015] Perform two different data enhancement processes on the image data in a group of pictures to obtain a pair of enhanced image pairs i, j;

[0016] The image pair i, j is encoded by the encoder in the self-supervised model to obtain the feature encoding h of the corresponding image data i ,h j ;

[0017] A fully connected layer is used to encode the obtained feature h i ,h j Perform projection mapping to obtain the corresponding eigenvector z i ,z j ;

[0018] For the eigenvector z i ,z j The similarity measure between them is used to measure;

[0019] The encoder is iteratively trained until the encoder can maximize the feature vector z i ,z j The similarity measure between the two representations results in the final self-supervised model.

[0020] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the feature vector z i ,z j The calculation formula for measuring the similarity between is:

[0021]

[0022] Where T represents the transpose of the vector.

[0023] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the initial reinforcement learning model consists of an Actor network and a Critic network;

[0024] The Actor network outputs an action value and a corresponding probability value according to the feature encoding output by the self-supervisory model; the output action value serves as an execution action of the corresponding controller in the environment simulation model;

[0025] The Critic network evaluates the action value output by the Actor network to obtain an evaluation value of the corresponding action value;

[0026] The last layers of the Actor network and the Critic network are different, and the rest are the same; the last layer of the Actor network determines the number of neurons according to the dimension of the action, and outputs the mean of the corresponding Gaussian distribution statistic action and the standard deviation parameter of the same dimension as the action; the last layer of the Critic network contains a neuron, and the output is responsible for evaluating the evaluation value of the current action output by the Actor network.

[0027] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the Actor network and the Critic network both include an LSTM module and a Transformer module; the input of the LSTM module is the feature code output by the self-supervisory model; the LSTM module is used to save multiple historical state information of the feature code, and encode all historical state information into the current state; the Transformer module uses an attention mechanism to assign different weight coefficients to the historical state information according to their importance, and uses the feature code corresponding to the historical state information with the highest weight coefficient as the final state code that satisfies the Markov property.

[0028] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the training data is represented in the form of:

[0029] sequence=[transition_time_step_1, transition_time_step_2, transition_time_step_3,...transition_time_step_20];

[0030]

[0031] Among them, sequence represents training data, and its time series length is 20 steps; transition represents data of one time step; hidden_state represents the output of the LSTM module; state represents the distribution image of various physical quantities returned by the environmental simulation model, and the feature encoding after the self-supervision model; π(a|s) represents the probability value of the reinforcement learning model taking a certain action; action represents the action taken by the reinforcement learning model; reward represents the reward given to the reinforcement learning model by the environmental simulation model after the reinforcement learning model takes the action; done represents the flag indicating whether the interaction between the reinforcement learning model and the environmental simulation model is over.

[0032] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the optimization objective function L of the Actor network is CLIP The expression of (θ) is:

[0033]

[0034] in, represents the advantage function, which means the value of the current action action compared to the average value in the state state. The calculation method uses TruncatedGAE to perform a general estimate of the advantage function; δ t represents the timing difference error of one step; r t (θ) represents the ratio of the probability values ​​of the new strategy and the old strategy reinforcement learning model taking a certain action action under the state state, where the new strategy represents the target strategy; represents the value of the advantage function of state s at time t; Indicates the expectation; clip indicates the restriction function to ensure r t The value of (θ) will not be less than 1-∈, nor greater than 1+∈; ∈ represents a very small value, generally 0.02; γ represents the discount factor; λ represents the weight exponential decay factor; δ t+1 represents the time series differential error at the t+1th moment; T represents the moment T; t represents the moment t; the relationship between T and t: there are T moments in total, T is the last moment, and t is the middle moment; δT-1 represents the time difference error of the corresponding state s at the T-1th moment; r t It represents the ratio of the probability of taking an action action under the state state between the new strategy and the old strategy; V(s t+1 ) represents the state value of state s at time t+1; s t+1 represents the state value at time t+1; s represents the state;

[0035] The expression of the optimization objective function of the Critic network is:

[0036]

[0037] in, V means to seek expectation; φ (s t ) represents the state value of state s at time t; represents an estimate of the state value of state s at time t calculated from simulation data; represents the advantage function of the corresponding state at time t; V(s t ) represents the state value of state s at time t.

[0038] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the distributed architecture includes multiple Actor servers and a Learner server; each of the Actor servers pushes the image data generated by the interaction between itself and the environmental simulation model to the Learner server; the Learner server waits for all the Actor servers to transmit the image data back, and then trains the initial reinforcement learning model and the self-supervised model respectively, and synchronizes the latest model parameters to all the Actor servers.

[0039] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the Actor server includes: a first main thread, a blocking thread and a first network service thread;

[0040] The first main thread interacts with the environment simulation model, samples and caches the interaction data, and saves the sampled data in the cache to a local disk after the capacity in the cache reaches a preset threshold;

[0041] After the first main thread saves the sampled data in the cache to the local disk, the first network service thread pushes all the saved sampled data to the Learner server and receives the latest model parameters sent back by the Learner server;

[0042] The blocking thread blocks the first main thread when the first network service thread pushes sample data to the Learner server, and wakes up the first main thread when the first main thread receives the latest model parameters sent back by the Learner server.

[0043] Preferably, in the above-mentioned accelerated training method for reinforcement learning in a large-scale simulation environment, the Learner server includes: a second main thread, a Redis thread, and a second network service thread;

[0044] The second network service thread receives the sampled data transmitted back by the Actor server, and saves the sampled data into a Redis cache database;

[0045] The Redis thread checks whether all the Actor servers have completely transmitted the sampled data, and when the transmission is completed, pushes the sampled data to a memory pool for use by the second main thread;

[0046] After the second main thread waits for all sampled data to be transmitted, it reads the sampled data from the memory pool, trains the initial reinforcement learning model and the self-supervised model in sequence, and saves the updated model file of the initial reinforcement learning model and the model file of the self-supervised model to the disk;

[0047] The second network service thread also reads the model parameters on the disk and transmits them to all the Actor servers synchronously.

[0048] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for accelerating training of reinforcement learning in a large-scale simulation environment, and the beneficial effects are:

[0049] 1. The present invention uses the encoder in the self-supervised model to fully extract features from the obtained image, includes historical information in the state in the LSTM stage, and finally assigns different weight coefficients to different states that have occurred in history through the self-attention mechanism in the Transformer. Finally, a state encoding that greatly satisfies the Markov property is obtained, so that the image data returned to the reinforcement learning model after a long period of simulation in the large-scale simulation environment is maximized, and the training speed of the reinforcement learning model in the interaction process with the large-scale simulation environment software is accelerated. The trained reinforcement learning model is applied to the real world, and the effect of automatic control is finally achieved.

[0050] 2. The present invention collects data through a distributed architecture, accelerates the interaction between the reinforcement learning model and the large-scale environmental simulation model, and has a scalable, multi-node, and universal distributed framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A flow chart of the accelerated training method for reinforcement learning in a large-scale simulation environment provided by the present invention;

[0053] Figure 2 A schematic diagram of the training process of the self-supervisory model provided by the present invention;

[0054] Figure 3 A schematic diagram of the structure of the Actor network provided by the present invention;

[0055] Figure 4 A schematic diagram of the structure of the Critic network provided by the present invention;

[0056] Figure 5 A schematic diagram of the structure of the distributed architecture provided by the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] like Figure 1 As shown, the embodiment of the present invention discloses a method for accelerating training of reinforcement learning in a large-scale simulation environment, comprising the following steps:

[0059] S1. Building an environment simulation model, an initial reinforcement learning model and a self-supervision model, and generating image data for representing the distribution states of various physical quantities in the environment simulation model through continuous interaction between the environment simulation model and the initial reinforcement learning model;

[0060] S2, using distributed architecture to collect image data;

[0061] S3, extracting low-dimensional features from image data through a self-supervised model to obtain feature encoding;

[0062] S4. Based on the reinforcement learning model, multiple historical state information of feature encoding is saved, different historical state information is assigned different weight coefficients, and finally a state encoding that satisfies the Markov property is obtained, and a set of sequence data is output as training data;

[0063] S5. Perform distributed training on the initial reinforcement learning model based on the training data until the optimization goal is met to obtain the final reinforcement learning model.

[0064] The present invention accelerates the training process of reinforcement learning and environmental simulation models from two aspects: one is to accelerate the interaction between the reinforcement learning model and the environmental simulation model from the state encoding level; the other is to accelerate the training process of the model from the distributed data collection and training method.

[0065] The standard modeling process of reinforcement learning is: first, three-dimensional modeling of the real world is performed, and an environmental simulation model similar to the real environment is built. Then, a reinforcement learning model is built, and a large amount of data generated by the continuous interaction between the reinforcement learning model and the environmental simulation model is used to train the reinforcement learning model, and finally the trained reinforcement learning model is transferred to the real world. Since the environmental simulation model of the embodiment of the present invention can be any large-scale simulation environment software (it takes a long time to simulate once), it is assumed here that a simulation environment that interacts with reinforcement learning has been established using any large-scale simulation environment software.

[0066] Below, the training process will be described in detail from the above two levels.

[0067] 1. Accelerate the interaction between the reinforcement learning model and the environment simulation model at the state encoding level:

[0068] Many control problems in the real world often rely on high-dimensional data obtained from different sensors, such as image data. Compressing the high-dimensional space data transmitted by the simulation environment software into a low-dimensional and relatively good dimensional space (the standard for a relatively good standard is that the low-dimensional feature vector can represent the original high-dimensional features to the greatest extent) can solve the problem that reinforcement learning cannot learn fast and effective feature extraction during the interaction with the environmental simulation model, and accelerate the convergence of the reinforcement learning algorithm.

[0069] During the interaction between the reinforcement learning model and the environmental simulation model, a large amount of image data will be obtained from the environmental simulation model to represent the state of the current environment. These image data are all unlabeled data, so a self-supervised learning method is used to automatically learn useful low-dimensional features from these image data.

[0070] After the self-supervised model is fully trained, it only needs to use the Encoder in the self-supervised model when interacting with the simulation environment software: after the simulation environment returns the image data, the Encoder is used to encode the image data into a low-dimensional feature vector, which is finally sent to the reinforcement learning model. The reinforcement learning model takes the feature vector as input and outputs the corresponding action and the corresponding action value.

[0071] Specifically include:

[0072] 1. Construction and training of self-supervised models

[0073] Since the reinforcement learning model generates a large amount of unlabeled image data during the interaction with the environment simulation model, the self-supervised model should make full use of the image data returned by the environment simulation model to obtain a better image feature encoder. This encoder is used to encode the image data returned by the environment simulation model, compress the high-dimensional image data into a low-dimensional feature vector, and then pass it to the reinforcement learning model.

[0074] like Figure 2 As shown in Figure 2, the training process of the self-supervised model is:

[0075] ① Extract a set of images from the image data generated by the interaction between the initial reinforcement learning model and the environment simulation model;

[0076] ② Perform two different data enhancement processes on the image data in a group of pictures to obtain a pair of enhanced image pairs i, j; the enhancement process can be any of the three methods: random cropping, random color distortion, and random Gaussian blur;

[0077] ③ Encode the image pair i, j through the encoder in the self-supervised model to obtain the feature encoding h of the corresponding image data i ,h j ;

[0078] ④ Use a fully connected layer to encode the obtained feature h i ,h j Perform projection mapping to obtain the corresponding eigenvector z i ,z j ;

[0079] ⑤ For the eigenvector z i ,z j The similarity measure between them is used to measure;

[0080] ⑥ Iteratively train the encoder until the encoder can maximize the feature vector z j ,z jThe similarity measure between the two representations results in the final self-supervised model.

[0081] Among them, the loss function of the self-supervised model to be optimized is as follows:

[0082] For the eigenvector z i ,z j The similarity between them is measured by cosine similarity, and the calculation formula is:

[0083]

[0084] Where T represents the transpose of the vector.

[0085] Assuming that the number of a batch is N, the loss calculation formula between image data i and the 2N image data after other image data augmentation is as follows:

[0086]

[0087] Among them, loss i,j Represents the loss of an image, and the parameter τ is an adjustable hyperparameter; similarity i,j Represents the similarity between image i and image j after image i is enhanced; similarity i,k Represents the similarity between image i and image k after other image enhancement; k represents the image index after other image data enhancement.

[0088] Therefore, the total loss in a batch is:

[0089]

[0090] The above formula L represents the loss function of the self-supervised model, which is consistent with the above loss i,j The relationship is: there are N pictures in total, each picture is randomly transformed twice, and 2N pictures are obtained, loss i,j It represents the loss of comparing one picture i with other pictures among the 2N pictures, and L represents the total loss of the 2N pictures.

[0091] 2. Construction of reinforcement learning model

[0092] 1) Modeling of Markov decision process in reinforcement learning

[0093] In reinforcement learning, there are several very important concepts that need to be carefully designed in the modeling of Markov decision processes, namely the design of Agent, Environment, State, Action, and Reward. In various control problems, Agent is a reinforcement learning model responsible for the output of strategy π(a|s); Environment is an environmental simulation model; Agent and Environment interact continuously to generate relevant data such as State, Action, and Reward; state is defined as image data of the distribution of various physical quantities returned by the environmental simulation model, and feature encoding after the self-supervision model; action is defined as a vector related to the control task, and the Reward function is also designed to be related to different control tasks.

[0094] 2) Construction of reinforcement learning model agent

[0095] The initial reinforcement learning model consists of two parts: the Actor network and the Critic network. Figure 3 and Figure 4 shown.

[0096] The Actor network outputs action values ​​and corresponding probability values ​​according to the feature encoding output by the self-supervisory model; the output action values ​​serve as the execution actions of the corresponding controllers in the environment simulation model;

[0097] The Critic network evaluates the action value output by the Actor network and obtains the evaluation value of the corresponding action value;

[0098] The last layer of the Actor network and the Critic network is different, and the rest are the same; the last layer of the Actor network determines the number of neurons according to the dimension of the action, and outputs the corresponding Gaussian distribution statistic action mean and the standard deviation parameter of the same dimension as the action; the last layer of the Critic network contains one neuron, and the output is responsible for evaluating the evaluation value of the current action output by the Actor network.

[0099] That is to say, the Actor network is responsible for the output of the strategy π(a|s), and models the probability of selecting the action corresponding to each state into a Gaussian distribution. The last layer of the Actor network determines the number of neurons according to the dimension of the action, and outputs the mean of the corresponding Gaussian distribution statistic action, and also has an independent learnable standard deviation parameter with the same dimension as the action. The last layer of the Critic network has only one neuron, and the output of the neuron is responsible for evaluating the current state action value Q(s,a). The Actor network and the Critic network are the same except for the last layer.

[0100] The Actor network outputs the action that the reinforcement learning model should take given a state. The output value of the Critic network is responsible for evaluating the quality of the action output by the Actor network and serves as a baseline to guide the Actor network's next action selection.

[0101] The inputs of the Acot network and the Critic network are the same. In this embodiment, a time series is used as the input. Each time series has a step length of 20 time steps. The specific format is as follows:

[0102] sequence=[transition_time_step_1, transition_time_step_2, transition_time_step_3,...transition_time_step_20];

[0103] There are 20 transitions in total. Transition is the minimum measure of data collected in a time step. The format of the data in transition is as follows:

[0104]

[0105] The meaning of each item is explained as follows:

[0106] sequence represents training data, and the length of its time series is 20 steps; transition represents data of one time step; hidden_state represents the output of the LSTM module; state represents the distribution images of various physical quantities returned by the environmental simulation model, and the feature encoding after the self-supervision model; π(a|s) represents the probability value of the reinforcement learning model taking a certain action; action represents the action taken by the reinforcement learning model; reward represents the reward given to the reinforcement learning model by the environmental simulation model after the reinforcement learning model takes the action; done represents the flag indicating whether the interaction between the reinforcement learning model and the environmental simulation model is over, and it is stipulated that the interaction between the environmental simulation model and the reinforcement learning model for 120 times marks the end of one scene.

[0107] Both the Actor network and the Critic network include LSTM modules and Transformer modules; the input of the LSTM module is the feature code output by the self-supervised model; the LSTM module is used to save multiple historical state information of the feature code and encode all the historical state information into the current state; the Transformer module uses the attention mechanism to assign different weight coefficients to the historical state information according to its importance, and takes the feature code corresponding to the historical state information with the highest weight coefficient as the final state code that satisfies the Markov property.

[0108] That is to say, the distribution images of various physical quantities returned by the environmental simulation model are used as the input of the self-supervised model for feature extraction, and the feature output extracted by the Encoder encoder of the self-supervised model is Encoder It can retain the most distinctive features of the original image data to the greatest extent possible, and output Encoder As the input of the LSTM module, because the LSTM module can well save historical information, the historical information is encoded in the current state, alleviating the problem of slow model convergence caused by the locally observable Markov decision process; finally, when reasoning, there will be a sequence of 20 historical states. Since the current state is definitely affected differently by different states that have occurred in history, and the current state is transferred from the 20 historical states that have been saved in history, the Transformer's self-attention mechanism is introduced here, so that the more important state information in history is given a higher weight under the attention mechanism, so that it can be better encoded in the current state. Finally, a layer of neural network outputs the statistics of the policy distribution or the state action value.

[0109] 3) Optimization objectives of reinforcement learning models

[0110] During initialization, the environment simulation model will first return a distribution map of various physical quantities, which is then encoded by the self-supervised model. Encoder As the input of the reinforcement learning model, at this time, the reinforcement learning model will output an action value as the action in the actual controller in the environment simulation model. After the controller executes the action, the environment simulation model will transfer to the next state. The distribution map image of various physical quantities representing the state of the environment simulation model is first input into the self-supervision model, and the feature encoding of the image is output as the input of the reinforcement learning model. The reinforcement learning model outputs an action value at this time...continuous interaction.

[0111] During the interaction process, the continuous interaction between the Agent and the simulation environment generates a large amount of data, all in the form of sequences. The specific format is as follows:

[0112] [<hidden_state,state,π(a|s),action,reward,done> ,<hidden_state,state,π(a|s),action,reward,done> …], 20 steps are saved in one sequence<state,π(a|s),action,reward,done> .

[0113] Use these time series data to optimize the objective function, where the optimization objective function L of the Actor network CLIP The expression of (θ) is:

[0114]

[0115] in, represents the advantage function, which means the value of the current action action compared to the average value in the state state. The calculation method uses Truncated GAE to perform a general estimate of the advantage function; δ t represents the timing difference error of one step; r t (θ) represents the ratio of the probability values ​​of the new strategy and the old strategy reinforcement learning model taking a certain action action under the state state, where the new strategy represents the target strategy; represents the value of the advantage function of state s at time t; Indicates the expectation; clip indicates the restriction function to ensure r t The value of (θ) will not be less than 1-∈, nor greater than 1+∈; ∈ represents a very small value, generally 0.02; γ represents the discount factor; λ represents the weight exponential decay factor; δ t+1represents the time series differential error at the t+1th moment; T represents the moment T; t represents the moment t; the relationship between T and t: there are T moments in total, T is the last moment, and t is the middle moment; δ T-1 represents the time difference error of the corresponding state s at the T-1th moment; r t It represents the ratio of the probability of taking an action action under the state state between the new strategy and the old strategy; V(s t+1 ) represents the state value of state s at time t+1; s t+1 It represents the state value at time t+1; s represents the state.

[0116] The expression of the optimization objective function of the Critic network is:

[0117]

[0118] Where L(φ) represents the objective function of the Critic network; V means to seek expectation; φ (s t ) represents the state value of state s at time t; represents an estimate of the state value of state s at time t calculated from simulation data; represents the advantage function of the corresponding state at time t; V(s t ) represents the state value of state s at time t.

[0119] 3) Combination of self-supervised model and reinforcement learning model

[0120] After the self-supervised model in 1) above is fully trained, the reinforcement learning model only needs to use the Encoder in the self-supervised model for encoding during the interaction with the simulation environment software: the simulation environment returns image data, and the self-supervised model uses the Encoder to encode the image data into a low-dimensional feature vector, which is then sent to the reinforcement learning model. The reinforcement learning model takes the feature vector as input and outputs the corresponding action and the corresponding action value.

[0121] 2. Accelerate the training process from the distributed architecture

[0122] In order to speed up the interaction between the model and the large-scale simulation environment software, the present invention adopts a distributed training method to accelerate data collection and model training. The specific distributed architecture is as follows: Figure 5 shown.

[0123] The distributed architecture includes multiple Actor servers and one Learner server; each Actor server pushes the image data generated by the interaction between itself and the environmental simulation model to the Learner server; the Learner server waits for all Actor servers to transmit the image data back, and then trains the initial reinforcement learning model and the self-supervised model respectively, and synchronizes the latest model parameters to all Actor servers.

[0124] Specifically, 1) the Actor server includes: a first main thread, a blocking thread and a first network service thread;

[0125] The first main thread interacts with the environment simulation model, samples and caches the interaction data until the capacity in the cache reaches a preset threshold, and then saves the sampled data in the cache to the local disk;

[0126] After the first main thread saves the sampled data in the cache to the local disk, the first network service thread pushes all the saved sampled data to the Learner server and receives the latest model parameters sent back by the Learner server;

[0127] The blocking thread blocks the first main thread when the first network service thread pushes sample data to the Learner server, and wakes up the first main thread when the first main thread receives the latest model parameters sent back by the Learner server.

[0128] When the Actor server collects data, it stipulates that one scene has only 120 steps. That is to say, the interaction between the reinforcement learning model and the environment simulation model only has a 120-step process. After 120 steps are executed, one scene ends, and then the simulation environment software is restarted to start a new scene. This cycle continues to generate new data.

[0129] 2) The Learner server includes: a second main thread, a Redis thread, and a second network service thread;

[0130] The second network service thread receives the sampled data transmitted back by the Actor server and saves the sampled data into the Redis cache database;

[0131] The Redis thread checks whether all Actor servers have completely transmitted the sampled data, and when the transmission is complete, pushes the sampled data to a memory pool for use by the second main thread;

[0132] After the second main thread waits for all sampled data to be transmitted, it reads the sampled data from the memory pool, trains the initial reinforcement learning model and the self-supervised model in turn, and saves the updated model files of the initial reinforcement learning model and the self-supervised model to the disk;

[0133] The second network service thread also reads the model parameters from disk and transmits them synchronously to all Actor servers.

[0134] When the Learner server is training, the model is updated and iterated once after receiving the data collected by the old strategy. At this time, the model parameters are more "new" than the strategy parameters when collecting data. The hidden_state at this time will appear outdated compared to the parameters of the updated model, affecting the convergence of the model. In order to ensure that the model can resume training from a better hidden_state, a burn_in technique is used in the algorithm: assuming that the length of the collected sequence is 20, the sequence data of the first 5 steps will be fed into the model to generate a new hidden_state. This new hidden_state and the remaining sequence of length 15 are used to train the model to accelerate the convergence of the model.

[0135] 3. Migrate the model from the virtual environment simulation environment to the real world

[0136] In the first two steps, the reinforcement learning model and the self-supervised model (hereinafter referred to as the model) are trained, and the models are transferred to the real world for reasoning verification. There are enough sensors in the real world, and the data sent back by the sensors can be used to obtain the distribution map images of various physical quantities in the real environment. The distribution map image of this physical quantity is used as the input of the model, and the image is used as the input of the encoder in the self-supervised model. The feature encoding of the image is used as the state as the input of the LSTM module in the reinforcement learning model. At this moment, the reinforcement learning model will output the probability distribution of the corresponding action, and use the mean of this probability distribution [action 1 ,action 2 ,...,action n ] can be used as the action taken by the control device in the real world, and finally the reinforcement learning model can achieve the effect of automatic control.

[0137] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0138] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for accelerating training of reinforcement learning in a large-scale simulation environment, characterized in that: The following steps are involved: Building an environment simulation model, an initial reinforcement learning model and a self-supervision model, and generating image data for representing the distribution states of various physical quantities in the environment simulation model through continuous interaction between the environment simulation model and the initial reinforcement learning model; Acquiring the image data using a distributed architecture; Extracting low-dimensional features from the image data through the self-supervised model to obtain feature encoding; Based on the reinforcement learning model, multiple historical state information of the feature encoding is saved, different weight coefficients are assigned to different historical state information, and finally a state encoding that satisfies the Markov property is obtained, and a set of sequence data is output as training data; Performing distributed training on the initial reinforcement learning model based on the training data until the optimization goal is met to obtain a final reinforcement learning model; The training process of the self-supervised model is: Extracting a set of pictures from the image data generated by the interaction between the initial reinforcement learning model and the environment simulation model; Perform two different data enhancement processes on the image data in a group of pictures to obtain a pair of enhanced image pairs i, j; The image pair i, j is encoded by the encoder in the self-supervised model to obtain the feature encoding h of the corresponding image data i ,h j ; A fully connected layer is used to encode the obtained feature h i ,h j Perform projection mapping to obtain the corresponding eigenvector z i ,z j ; For the eigenvector z i ,z j The calculation formula for measuring the similarity between is: Where T represents the transpose of a vector; For the eigenvector z i ,z j The similarity measure between them is used to measure; The encoder is iteratively trained until the encoder can maximize the feature vector z i ,z j The similarity measure between the two representations yields the final self-supervised model; The initial reinforcement learning model consists of an Actor network and a Critic network; The Actor network outputs an action value and a corresponding probability value according to the feature encoding output by the self-supervisory model; the output action value serves as an execution action of the corresponding controller in the environment simulation model; The Critic network evaluates the action value output by the Actor network to obtain an evaluation value of the corresponding action value; The last layer of the Actor network and the Critic network are different, and the rest are the same; the last layer of the Actor network determines the number of neurons according to the dimension of the action, and outputs the mean of the corresponding Gaussian distribution statistic action and the standard deviation parameter of the same dimension as the action; the last layer of the Critic network contains one neuron, and the output is responsible for evaluating the evaluation value of the current action output by the Actor network; The Actor network and the Critic network both include an LSTM module and a Transformer module; the input of the LSTM module is the feature code output by the self-supervisory model; the LSTM module is used to save multiple historical state information of the feature code and encode all the historical state information into the current state; the Transformer module uses an attention mechanism to assign different weight coefficients to the historical state information according to the importance, and uses the feature code corresponding to the historical state information with the highest weight coefficient as the final state code that satisfies the Markov property; The distributed architecture includes multiple Actor servers and a Learner server; each Actor server pushes the image data generated by the interaction between itself and the environment simulation model to the Learner server; the Learner server waits for all the Actor servers to transmit the image data back, then trains the initial reinforcement learning model and the self-supervised model respectively, and synchronizes the latest model parameters to all the Actor servers; The Actor server includes: a first main thread, a blocking thread and a first network service thread; The first main thread interacts with the environment simulation model, samples and caches the interaction data, and saves the sampled data in the cache to a local disk after the capacity in the cache reaches a preset threshold; After the first main thread saves the sampled data in the cache to the local disk, the first network service thread pushes all the saved sampled data to the Learner server and receives the latest model parameters sent back by the Learner server; The blocking thread blocks the first main thread when the first network service thread pushes sample data to the Learner server, and wakes up the first main thread when the first main thread receives the latest model parameters sent back by the Learner server; The Learner server includes: a second main thread, a Redis thread, and a second network service thread; The second network service thread receives the sampled data transmitted back by the Actor server, and saves the sampled data into a Redis cache database; The Redis thread checks whether all the Actor servers have completely transmitted the sampled data, and when the transmission is completed, pushes the sampled data to a memory pool for use by the second main thread; After the second main thread waits for all sampled data to be transmitted, it reads the sampled data from the memory pool, trains the initial reinforcement learning model and the self-supervised model in sequence, and saves the updated model file of the initial reinforcement learning model and the model file of the self-supervised model to the disk; The second network service thread also reads the model parameters on the disk and transmits them to all the Actor servers synchronously.

2. The accelerated training method for reinforcement learning in a large-scale simulation environment according to claim 1, characterized in that: The training data is represented as: sequence=[transition_time_step_1,transition_time_step_2,transition_time_step_3,…transition_time_step_20]; Among them, sequence represents training data, and its time series length is 20 steps; transition represents data of one time step; hidden_state represents the output of the LSTM module; state represents the distribution image of various physical quantities returned by the environmental simulation model, and the feature encoding after the self-supervision model; π(a|s) represents the probability value of the reinforcement learning model taking a certain action; action represents the action taken by the reinforcement learning model; reward represents the reward given to the reinforcement learning model by the environmental simulation model after the reinforcement learning model takes the action; done represents the flag indicating whether the interaction between the reinforcement learning model and the environmental simulation model is over.

3. The accelerated training method for reinforcement learning in a large-scale simulation environment according to claim 1, characterized in that: The optimization objective function L of the Actor network CLIP The expression of (θ) is: in, represents the advantage function, which means the value of the current action action compared to the average value in the state state. The calculation method uses Truncated GAE to perform a general estimate of the advantage function; δ t represents the timing difference error of one step; r t (θ) represents the ratio of the probability values ​​of the new strategy and the old strategy reinforcement learning model taking a certain action action under the state state, where the new strategy represents the target strategy; represents the value of the advantage function of state s at time t; Indicates the expectation; clip indicates the restriction function to ensure r t The value of (θ) will not be less than 1-∈, nor greater than 1+∈; ∈ represents a very small value, generally 0.02; γ represents the discount factor; λ represents the weight exponential decay factor; δ t+1 represents the time series differential error at the t+1th moment; T represents the moment T; t represents the moment t; the relationship between T and t: there are T moments in total, T is the last moment, and t is the middle moment; δ T-1 represents the time difference error of the corresponding state s at the T-1th moment; r t It represents the ratio of the probability of taking an action action under the state state between the new strategy and the old strategy; V(s t+1 ) represents the state value of state s at time t+1; s t+1 represents the state value at time t+1; s represents the state; The expression of the optimization objective function of the Critic network is: in, V means to seek expectation; φ (s t ) represents the state value of state s at time t; represents an estimate of the state value of state s at time t calculated from the simulation data; represents the advantage function of the corresponding state at time t; V(s t ) represents the state value of state s at time t.

Images