A Security Policy Learning Method Based on Offline Reinforcement Policy Learning
By constructing a style transfer model and a conditional variational autoencoder, and combining virtual and real data, a security policy is generated. This solves the problem of insecure policies caused by directly migrating virtual data to the real environment, and realizes the generation and execution of security policies in the real environment.
Patent Information
- Application Number
- CN202410612181.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-16
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-05-16
AI Technical Summary
In existing technologies, security policies trained using virtual data in virtual simulation scenarios are directly transferred to real-world scenarios, which may result in insecure policies and lead to security incidents.
By acquiring datasets from both virtual and real-world environments, a style transfer model is constructed to generate a target virtual dataset, which is then fused with real data. Using a conditional variational autoencoder and a critic network, the policy is updated to generate a safe policy, ensuring the safety of actions performed in the real-world environment.
It improves the accuracy of virtual data to real data, reduces the security risks caused by direct migration strategies, and ensures that the generated strategies are safe and reliable in real environments.
Smart Images

Figure CN118551799B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of reinforcement learning and deep learning technologies, and in particular to a security policy learning method based on offline reinforcement policy learning. Background Technology
[0002] With the rapid development of science and technology, especially information science and technology, unmanned systems are playing an increasingly important role in real life. From robotic vacuum cleaners and logistics robots to unmanned vehicles, unmanned boats, and drones, the application areas of unmanned systems are constantly expanding.
[0003] As the application scenarios of unmanned systems expand, their working environments become increasingly complex. However, current decision-making in unmanned systems primarily relies on a pre-defined static strategy, with control strategies determined by the output of upper-level modules. This approach has significant drawbacks, over-reliance on upper-level module outputs leading to poor system robustness and generalization. Furthermore, since these decision rules are largely formulated by humans based on prior knowledge, each decision module may only be suitable for specific scenarios, making it difficult to adapt to complex and changing environments. Therefore, unmanned systems must possess strong autonomous learning capabilities to achieve environmental adaptability.
[0004] Offline reinforcement learning (Offline RL) is a reinforcement learning method that does not require real-time data collection during interaction with the environment. It uses a previously collected dataset to train the agent. The underlying technology of offline reinforcement learning is built upon the fundamental principles of reinforcement learning, including concepts such as state, action, reward, and value function. It primarily focuses on how to learn a policy through interaction with the environment to maximize the expected cumulative reward.
[0005] Offline reinforcement learning typically utilizes virtual data collected in virtual simulation environments for policy learning. However, in real-world scenarios, due to the differences between real and virtual data—namely, differences in perception, dynamics, and uncertainty—when a security policy obtained using virtual data is deployed in a real environment, the agent's next action, executed according to the security policy, may still be unsafe.
[0006] Current offline reinforcement learning strategies, due to the differences between virtual simulation scenarios and real scenarios, inevitably lead to unsafe policies when models trained using virtual data to generate security policies are directly transferred to real scenarios, thus triggering a series of security incidents. Summary of the Invention
[0007] The purpose of this invention is to provide an offline reinforcement policy learning method for security policy execution, in order to solve the problem that in the prior art, the model for generating security policies is trained using only virtual data in a virtual simulation scenario, and then the model for generating security policies is directly transferred to the real scenario for use, which may lead to the generation of insecure policies.
[0008] To achieve the above objectives, the present invention provides the following solution:
[0009] A security policy learning method based on offline reinforcement policy learning, the method specifically includes:
[0010] A virtual-real dataset is obtained, wherein the virtual-real dataset includes virtual data in a virtual simulation environment and real data in a real environment; the virtual data corresponds to the real data; the virtual data includes the agent's current virtual observation state and current virtual action in the virtual simulation environment, the current virtual reward from environmental feedback, the cost of virtual environmental feedback, the virtual observation state at the next moment, and a flag indicating whether the current virtual trajectory has ended; the real data includes the agent's current real observation state and current real action in the real environment, the real reward from environmental feedback, the real cost of environmental feedback, the real observation state at the next moment, and a flag indicating whether the current real trajectory has ended.
[0011] Based on the aforementioned virtual and real datasets, a style transfer model is constructed;
[0012] The virtual data is input into the style transfer model to generate the target virtual dataset;
[0013] The target virtual data and the real data are fused to obtain a fused dataset; the fused dataset includes: the current fused observation state, the current fused action, the fused reward from environmental feedback, the fused cost from environmental feedback, the fused observation state at the next moment, the fused action at the next moment, and a flag indicating whether the current target fused trajectory has ended;
[0014] The next moment's fused observation state is input into a conditional variational autoencoder, which outputs the action at the next moment.
[0015] Determine whether the action safety value of the action at the next moment is less than a preset safety threshold;
[0016] If the action safety value of the action at the next moment is less than the preset safety threshold, the temporal difference error is determined based on the action at the next moment and the fused action at the current moment. The critic network is updated based on the temporal difference error, and the conditional variational autoencoder is updated using the updated critic network to obtain the action at the next moment. The action at the next moment is used as the safety policy to drive the agent's movement.
[0017] If the action safety value of the action at the next moment is not less than the preset safety threshold, the temporal difference error is determined based on the current fused action and the action at the previous moment. The critic network is updated based on the temporal difference error. The updated critic network is used to update the conditional variational autoencoder to obtain the updated target action at the next moment. The updated target action at the next moment is used as the safety policy to drive the agent's movement.
[0018] Furthermore, the training process of the critic network specifically includes:
[0019] If the action safety value of the action at the next moment is less than the preset safety threshold, the commentator network is trained based on the temporal difference error determined by the action at the next moment and the fused action at the current moment, until the preset iteration threshold is reached.
[0020] If the action safety value of the next action is not less than a preset safety threshold, the temporal difference error is determined based on the current fused action and the action of the previous action, and the commentator network is trained until the preset iteration threshold is reached.
[0021] Furthermore, the action safety value of the target action at the next moment is equal to:
[0022]
[0023] Among them, a t+1 This represents the fusion action at time t+1 in the fused dataset; The conditional variational autoencoder outputs the action at time t+1 based on the fused observation state at time t+1.
[0024] Furthermore, the construction of the style transfer model based on the virtual and real datasets specifically includes:
[0025] The style transfer model is a recurrent generative adversarial network (RGAN); the RGAN includes a generator G, a generator F, and a discriminator D. X Discriminator D Y ;
[0026] The recurrent generative adversarial network is trained based on several virtual data X and several real data Y, and a first generative adversarial loss and a second generative adversarial loss are determined.
[0027] The target loss of the cyclic generative adversarial network is determined based on the first generative adversarial loss, the second generative adversarial loss, and the cyclic consistency loss.
[0028] The recurrent generative adversarial network is updated based on the target loss, and the updated recurrent generative adversarial network is used as the style transfer model.
[0029] Furthermore, the recurrent generative adversarial network is trained based on several virtual data X and several real data Y, and a first generative adversarial loss and a second generative adversarial loss are determined. Specifically, the training process of the recurrent generative adversarial network includes a forward training process and a backward training process.
[0030] The recurrent generative adversarial network is trained in a forward manner using several virtual data X and several real data Y to generate a first forward generative adversarial loss and a second forward generative adversarial loss.
[0031] The recurrent generative adversarial network is trained in reverse using several virtual data X and several real data Y to generate a first inverse generative adversarial loss and a second inverse generative adversarial loss.
[0032] The first generation adversarial loss is determined based on the first forward generation adversarial loss and the first reverse generation adversarial loss;
[0033] The second generation adversarial loss is determined based on the second forward generation adversarial loss and the second reverse generation adversarial loss.
[0034] Furthermore, the forward training process specifically includes:
[0035] The virtual data is input into the generator G to generate the first real data;
[0036] Determine the first positive generative adversarial loss between the first real data and the real data, and update the parameters of the generator G and the discriminator D. Y The parameters, until the discriminator D Y Once the loss of the first positive generative adversarial interaction is determined to be zero, updates are stopped.
[0037] The first real data is input into generator F to generate the first virtual data;
[0038] Determine the second positive generative adversarial loss of the first virtual data and the virtual data, and update the parameters of the generator F and the discriminator D. X The parameters, until the discriminator D X Once the second positive generative adversarial loss is determined to be 0, updates are stopped.
[0039] Furthermore, the reverse training process specifically includes:
[0040] The real data Y is input into the generator F to generate the second virtual data;
[0041] Determine the second reverse generative adversarial loss between the second virtual data and the virtual data; and update the parameters of the generator F and the discriminator D. XThe parameters, until the discriminator D X Once the second counter-attack loss is determined to be zero, updates are stopped.
[0042] The second virtual data is input into generator G to generate the second real data;
[0043] Determine the first reverse generative adversarial loss between the second real data and the real data; and update the parameters of the generator G and the discriminator D. Y The parameters, until the discriminator D Y Once the first reverse generative adversarial loss is determined to be equal to 0, updates are stopped.
[0044] Furthermore, the formula for calculating the first generative adversarial loss is as follows:
[0045]
[0046] In the formula, P X X represents the set of data consisting of all current virtual observation states and the next virtual observation state in the virtual data; X represents the data from P. X The sampled virtual data, denoted by X∈P X Indicates the sampling process; P Y Y represents the set of data consisting of all current and next real observation states in the real data; Y represents the data from P. Y The actual data sampled; D x (X) indicates that the discriminator D is returned. X The probability of a decision given virtual data; F(Y) represents the current virtual observation state generated by generator F based on the current real observation state in the real data, or the next virtual observation state generated based on the next real observation state in the real data; D X (F(Y)) represents the discriminator D X The probability that the virtual observation state generated by generator F using the current real observation state in the real data is virtual data; Representing variables (D) X (X)-1) is the expectation relative to the virtual data X. Represents variable D X (F(Y)) is the expectation relative to the true data Y; L GAN (F, D) X P Y P X ) represents the generator F and the discriminator D. X The first generative adversarial loss generated;
[0047] The formula for calculating the second generative adversarial loss is as follows:
[0048]
[0049] In the formula, D Y (Y) indicates a return to the discriminator D. Y The probability of determining the given real data; G(X) represents the current real observation state generated by generator G based on the current virtual observation state in the original virtual data, or the next real observation state generated based on the next virtual observation state in the real data; D Y (G(X)) represents the discriminator D Y The determination of the real observation state generated by generator G using the current virtual observation state in virtual data X is the real data probability; Represents variable D Y (G(X)) is the expectation of the virtual data X. Representing variables (D) Y (Y)-1) Expectation relative to the true data Y; L GAN (G, D) Y P X P Y ) represents the generator G and the discriminator D. Y The generated second generative adversarial loss.
[0050] Furthermore, the formula for calculating the cycle consistency loss is as follows:
[0051]
[0052] In the formula, This represents the expectation of the variable ||F(G(X))-X||2 relative to the dummy data X. Let ||G(F(Y))-Y||2 represent the expectation of the variable ||G(F(Y))-Y||2 relative to the actual data Y.
[0053] Furthermore, the formula for calculating the target loss is as follows:
[0054] Loss = L GAN (G,D Y ,P X ,P Y )+L GAN (F,D X ,P X ,P Y )+λL(G,F)...(5);
[0055] In the formula, L GAN (G,D Y ,P X ,P Y ) represents the first generation adversarial loss, L GAN (F,D X ,PX ,P Y ) represents the second generative adversarial loss, L(G,F) represents the cycle consistency loss, and λ represents the weighting factor that balances the losses.
[0056] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0057] This application utilizes a style transfer model trained on a virtual dataset to transform virtual data collected in a simulation environment into target virtual data. This target virtual data is more realistic than the virtual data, thus increasing the accuracy of the virtual data relative to real data. Further, the target virtual data and real data are fused to obtain a fused dataset. A conditional variational autoencoder (CVA) is trained using the current fused observation state and current fused action from the fused dataset. The next-time fused observation state is input into the CVA, which then outputs the next-time action (safety policy). In other words, instead of using only virtual data to train the existing model that generates safe policies, this application uses a CVA trained on the fused dataset. This reduces the risk of generating unsafe policies when the existing model that generates safe policies is directly transferred from a virtual simulation environment to a real environment.
[0058] Furthermore, this invention compares the action generated by the conditional variational autoencoder at the next time step with a preset safety threshold to determine whether to calculate the temporal difference error by using the action at the next time step or to calculate the temporal difference error by using the fused action at the next time step. The commentator network is then updated using the temporal difference error. This ensures that the actions used to update the commentator network are safe, and that the policies output by the conditional variational autoencoder after training are safe, thus avoiding potential safety hazards in the prior art. Attached Figure Description
[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0060] Figure 1 This is a schematic diagram of a security policy execution method for offline reinforcement policy learning provided in Embodiment 1 of the present invention;
[0061] Figure 2 This is a schematic diagram of the training process of the style transfer model provided in Embodiment 1 of the present invention;
[0062] Figure 3This is a structural block diagram of a security policy execution method for offline reinforcement policy learning provided in Embodiment 1 of the present invention;
[0063] Figure 4 This is a processing block diagram of a security policy execution method for offline reinforcement policy learning provided in Embodiment 1 of the present invention;
[0064] Figure 5 This is a schematic diagram of the overall experimental environment of UGV provided in Embodiment 1 of the present invention;
[0065] Figure 6 This is a schematic diagram of the soldier's detection range provided in Embodiment 1 of the present invention;
[0066] Figure 7 This is a schematic diagram of obstacles in a map provided in Embodiment 1 of the present invention;
[0067] Figure 8(a) shows the experimental results of intelligent vehicle B intercepting the enemy forward;
[0068] Figure 8(b) shows the experimental results of intelligent vehicle B intercepting the enemy forward;
[0069] Figure 8(c) shows the experimental results of intelligent vehicle B intercepting the enemy forward;
[0070] Figure 8(d) shows the experimental results of intelligent vehicle B intercepting the enemy forward;
[0071] Figure 9 This is an experimental result diagram showing the successful avoidance of scene obstacles and execution of safety strategies provided in Embodiment 1 of the present invention. Detailed Implementation
[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0073] The purpose of this invention is to provide an offline reinforcement policy learning method for security policy execution, which aims to solve the problem that in the prior art, models for generating security policies are trained using only virtual data in virtual simulation scenarios, and then the models for generating security policies are directly transferred to real scenarios for use, which may lead to the generation of insecure policies.
[0074] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0075] like Figure 1As shown in this embodiment, a security policy learning method based on offline reinforcement policy learning specifically includes:
[0076] Step 11: Obtain the virtual and real datasets, wherein the virtual and real datasets include virtual data in the virtual simulation environment and real data in the real environment; the virtual data corresponds to the real data; the virtual data includes the agent's current virtual observation state, current virtual action, current virtual reward from environmental feedback, cost of virtual environmental feedback, virtual observation state at the next moment, and whether the current virtual trajectory has ended; the real data includes the agent's current real observation state, current real action, real reward from environmental feedback, real cost of environmental feedback, real observation state at the next moment, and whether the current real trajectory has ended.
[0077] In the virtual simulation environment, a pre-prepared script is used to execute the agent's data collection task, which is then stored in virtual data D. V ={τ1,τ2,…,τ i Within ,…}, each demonstration transitionτ t A 7-tuple τ is formed by including the agent's current virtual observation state, current virtual action, virtual reward from environmental feedback, virtual cost from environmental feedback, next virtual observation state, next virtual action, and whether the current virtual trajectory has ended. t ={s t ,a t ,r t ,c t ,s t+1 ,a t+1 In real-world scenarios, real data collected through methods such as manual operation by experts or knowledge bodies obtained through training with prior expert knowledge is stored in real data D. R ={τ1,τ2,…,τ t ,…}, each demonstration transitionτ t A 7-tuple τ is formed by the agent's current real observation state, current real action, real reward from environmental feedback, real cost from environmental feedback, next real observation state, next real action, and whether the current real trajectory has ended. t ={s t ,a t ,r t ,c t ,s t+1 ,a t+1 ,done}.
[0078] Step 12: Construct a style transfer model based on the virtual and real datasets.
[0079] In some embodiments, constructing a style transfer model based on the virtual and real datasets specifically includes: the style transfer model being a recurrent generative adversarial network (RGAN); the RGAN includes a generator G, a generator F, and a discriminator D. X Discriminator D Y The recurrent generative adversarial network (RGAN) is trained based on several virtual data X and several real data Y, and a first generative adversarial loss and a second generative adversarial loss are determined. Based on the first generative adversarial loss, the second generative adversarial loss and the recurrent consistency loss, a target loss of the RGAN is determined. The RGAN is updated based on the target loss, and the updated RGAN is used as the style transfer model.
[0080] In this invention, the recurrent generative adversarial network (RGAN) can be a neural network. The pre-trained RGAN is used as a style transfer model. The generator G generates the first real data in the real scene from the virtual data in the simulation environment; the generator F generates the first virtual data in the virtual simulation environment from the real data in the real scene; the discriminator DY distinguishes whether the generated first real data is virtual or real data, and the discriminator Dx distinguishes whether the generated first real data is virtual or real data.
[0081] In some embodiments, training the recurrent generative adversarial network (RGAN) using a plurality of virtual data X and a plurality of real data Y, and determining a first generative adversarial loss and a second generative adversarial loss, specifically includes: the training process of the RGAN includes a forward training process and a backward training process; forward training the RGAN using a plurality of virtual data X and a plurality of real data Y to generate a first forward generative adversarial loss and a second forward generative adversarial loss; backward training the RGAN using a plurality of virtual data X and a plurality of real data Y to generate a first backward generative adversarial loss and a second backward generative adversarial loss; determining the first generative adversarial loss based on the first forward generative adversarial loss and the first backward generative adversarial loss; and determining the second generative adversarial loss based on the second forward generative adversarial loss and the second backward generative adversarial loss.
[0082] Reference Figure 2 The training process of the style transfer model specifically includes:
[0083] In some embodiments, the forward training process specifically includes: inputting the virtual data into the generator G to generate first real data; determining a first forward generative adversarial loss between the first real data and the real data; and updating the parameters of the generator G and the discriminator D. Y The parameters, until the discriminator DY The first positive generative adversarial loss is determined to be equal to 0, and updates are stopped; the first real data is input into generator F to generate first virtual data; the second positive generative adversarial loss between the first virtual data and the virtual data is determined, and the parameters of generator F and discriminator D are updated. X The parameters, until the discriminator D X Once the second positive generative adversarial loss is determined to be 0, updates are stopped.
[0084] Here, updating parameters refers to updating the parameters of the generator G in this neural network, that is, updating the weights and biases of the neural network.
[0085] In some embodiments, the reverse training process specifically includes: inputting the real data Y into the generator F to generate second virtual data; determining the second reverse generative adversarial loss between the second virtual data and the virtual data; and updating the parameters of the generator F and the discriminator D. X The parameters, until the discriminator D X Determine that the second reverse adversarial loss is equal to 0, and stop updating; input the second virtual data into generator G to generate second real data; determine the first reverse generation adversarial loss between the second real data and the real data; and update the parameters of generator G and discriminator D. Y The parameters, until the discriminator D Y Once the first reverse generative adversarial loss is determined to be equal to 0, updates are stopped.
[0086] In some embodiments, the formula for calculating the first generative adversarial loss is:
[0087]
[0088] In the formula, P X X represents the set of data consisting of all current virtual observation states and the next virtual observation state in the virtual data; X represents the data from P. X The sampled virtual data, denoted by X∈P X Indicates the sampling process; P Y Y represents the set of data consisting of all current and next real observation states in the real data; Y represents the data from P. Y The actual data sampled; D X (X) indicates that the discriminator D is returned. X The probability of a decision given virtual data; F(Y) represents the current virtual observation state generated by generator F based on the current real observation state in the real data, or the next virtual observation state generated based on the next real observation state in the real data; D X (F(Y)) represents the discriminator DX The probability that the virtual observation state generated by generator F using the current real observation state in the real data is virtual data; Representing variables (D) X (X)-1) is the expectation relative to the virtual data X. Represents variable D X (F(Y)) is the expectation relative to the true data Y; L GAN (F, D) X P Y P X ) represents the generator F and the discriminator D. X The first generative adversarial loss generated;
[0089] The formula for calculating the second generative adversarial loss is as follows:
[0090]
[0091] In the formula, D Y (Y) indicates a return to the discriminator D. Y The probability of determining the given real data; G(X) represents the current real observation state generated by generator G based on the current virtual observation state in the original virtual data, or the next real observation state generated based on the next virtual observation state in the real data; D Y (G(X)) represents the discriminator D Y The determination of the real observation state generated by generator G using the current virtual observation state in virtual data X is the real data probability; Represents variable D Y (G(X)) is the expectation of the virtual data X. Representing variables (D) Y (Y)-1) Expectation relative to the true data Y; L GAN (G, D) Y P X P Y ) represents the generator G and the discriminator D. Y The generated second generative adversarial loss.
[0092] In some embodiments, the formula for calculating the cycle consistency loss is:
[0093]
[0094] In the formula, This represents the expectation of the variable ||F(G(X))-X||2 relative to the dummy data X. Let ||G(F(Y))-Y||2 represent the expectation of the variable ||G(F(Y))-Y||2 relative to the actual data Y.
[0095] In some embodiments, the formula for calculating the target loss is:
[0096] Loss = L GAN (G,D Y ,P X ,P Y )+L GAN (F,D X ,P X ,P Y )+λL(G,F)...(5);
[0097] In the formula, L GAN (G,D Y ,P X ,P Y ) represents the first generation adversarial loss, L GAN (F,D X ,P X ,P Y ) represents the second generative adversarial loss, L(G,F) represents the cycle consistency loss, and λ represents the weighting factor that balances the losses.
[0098] Step 13: Input the virtual data into the style transfer model to generate the target virtual dataset.
[0099] Among them, the style transfer model is trained based on real data and virtual data, which makes the target virtual data generated by the style transfer model closer to real data than the virtual data in the virtual simulation environment, thus reducing the error between virtual data and real data.
[0100] Step 14: Merge the target virtual data with the real data to obtain a fused dataset; the fused dataset includes: the current fused observation state, the current fused action, the fused reward from environmental feedback, the fused cost from environmental feedback, the fused observation state at the next moment, the fused action at the next moment, and a flag indicating whether the current target fused trajectory has ended.
[0101] The fused dataset contains relatively realistic target virtual data and real data, which reduces the risk of unsafe strategies that may occur in the subsequent training of related models using only virtual data from virtual simulation environments in existing technologies.
[0102] Step 15: Input the current fused observation state and the current fused action into the conditional variational autoencoder to train the conditional variational autoencoder. Input the next time-step fused observation state into the conditional variational autoencoder to output the next time-step action.
[0103] In practical applications, such as autonomous driving scenarios, pre-trained conditional variational autoencoders are utilized. When used in autonomous vehicles, these encoders acquire the actual observations of the vehicle's cameras in a real-world scene at the current moment and output the actual action for the next moment. Because the pre-trained encoders output highly accurate safety policies, the actual actions for the next moment represent very safe safety strategies.
[0104] For example, the input to the conditional variational autoencoder is the current fused observation state s. t and the current fusion action a t The output is the action at the next moment. The corresponding mean and variance (normal distribution) are used to derive the action at the next moment based on the normal distribution.
[0105] Among them, the loss function L of the conditional variational autoencoder CVAE From formula (6): L CVAE =L recon +β·D KL +α·(r+γQ(s′,a′)-Q(s,a)) 2 Calculated.
[0106] Where L recon It is the reconstruction loss, L KL is the KL divergence loss, β is a hyperparameter, preferably 0.5, α is a hyperparameter, Q is the critic network, r is the sum of the fusion reward and fusion cost of the environment feedback obtained from the sampling of the fusion dataset, s is the current fusion state obtained from the sampling of the fusion dataset, a is the current fusion action obtained from the sampling of the fusion dataset, s′ is the fusion observation state at the next time step obtained from the sampling of the fusion dataset, γ is the discount factor, and a′ is the action output by the conditional variational autoencoder based on s′.
[0107] In practical applications of the training process of a conditional variational autoencoder, the reconstruction loss is obtained through formula (7):
[0108]
[0109] Among them, a t This represents the current fusion action at time t in the fused dataset. This represents the action generated by the autoencoder at any given moment.
[0110] Among them, D KL This is the KL divergence loss, which is obtained through formula (8). The specific calculation formula is as follows:
[0111]
[0112] in, This represents a Gaussian distribution with mean μ and standard deviation σ. This represents a Gaussian distribution with a mean of 0 and a standard deviation of 1. Indicates to Take the logarithm. Representing variables Compared to The expectation.
[0113] Step 16: Determine whether the action safety value of the next action is less than the preset safety threshold. If yes, proceed to step 17; otherwise, proceed to step 18.
[0114] In the training process of the Conditional Variational Autoencoder (CVA), in order to ensure that the accuracy of the trained CVA is high, the action generated by the CVA for the next time step is verified before each CVA update. This ensures that the actions generated by the trained CVA for the next time step are safe actions, that is, safe policies are generated.
[0115] Step 17: Determine the temporal difference error based on the action at the next moment and the fused action at the current moment, update the critic network based on the temporal difference error, update the conditional variational autoencoder using the updated critic network, obtain the updated action at the next moment, and use the updated action at the next moment as a safety policy to drive the agent's movement.
[0116] Step 18: Determine the temporal difference error based on the current fused action and the action at the previous moment, update the critic network based on the temporal difference error, use the updated critic network to update the conditional variational autoencoder, obtain the updated target action at the next moment, and use the updated target action at the next moment as a safety policy to drive the agent's movement.
[0117] The critic network (also known as the Q-network) is used to output a score for the action at the next moment, and the score is used to determine whether the action at the next moment is safe or unsafe.
[0118] The critic network is updated using formulas (9) and (10):
[0119] δ=r+γQ(s′,a′)-Q(s,a) (9)
[0120]
[0121] Where Q is the critic network, w is the critic network parameter, α is the learning rate, r is the sum of the fusion reward and fusion cost of the environment feedback obtained from the sampling of the fusion dataset, s is the current fusion state obtained from the sampling of the fusion dataset, a is the current fusion action obtained from the sampling of the fusion dataset, s′ is the fusion observation state at the next time step obtained from the sampling of the fusion dataset, γ is the discount factor, a′ is the safe action determined in step 16, and δ is the temporal difference error.
[0122] Understandably, a well-trained recurrent generative adversarial network (style transfer model) is responsible for generating the observed state s in the current real-world environment. A conditional variational autoencoder (VAE) is responsible for generating the action s in the observed state, i.e., the next action s, a at the next time step. Q(s, a) is used to train the critic network, where Q(s, a) is the critic network's output, which scores the next action s, a. Initially, the scores are poor, but through training, a good and accurate neural network is obtained. This allows the network to distinguish between good and bad actions, and then enables the VAE to generate actions with high scores, i.e., the optimal safety policy.
[0123] In some embodiments, the training process of the commentator network specifically includes: if the action safety value of the action at the next time step is less than a preset safety threshold, training the commentator network based on the temporal difference error determined by the action at the next time step and the fused action at the current time step, until a preset iteration threshold is reached. If the action safety value of the action at the next time step is not less than the preset safety threshold, determining the temporal difference error based on the current fused action and the action at the previous time step, training the commentator network until a preset iteration threshold is reached.
[0124] The maximum number of training iterations is set to N (i.e., the preset iteration threshold). If the current iteration count has reached N, the training ends. If the number of training iterations has not reached N, the process jumps to step 2 to continue iterative training.
[0125] In some embodiments, the action safety value of the target action at the next moment is equal to:
[0126]
[0127] Among them, a t+1 This represents the fusion action at time t+1 in the fused dataset; This represents the action at time t+1 output by the conditional variational autoencoder based on the state at time t+1.
[0128] The method described in this invention fuses data collected from a virtual simulation environment and a real environment, so that the fused dataset has both the characteristics of real data and a large quantity. Moreover, by performing safety verification on the actions generated by the conditional variational autoencoder in the next moment, it ensures that the decisions made by the conditional variational autoencoder in the real environment are safe policies.
[0129] Reference Figures 3-9 According to Table 1, the UGV's mission scenario is: Intelligent Vehicle A surrounds and intercepts Intelligent Vehicle B; at the same time, it needs to avoid enemy soldiers' reconnaissance and obstacles on the map. Intelligent Vehicle A receives a reward for passing through a checkpoint; it also receives rewards for reaching the front, rear, left, and right of Intelligent Vehicle B. Intelligent Vehicle A incurs a penalty for hitting an obstacle or being detected by enemy soldiers.
[0130] In UGV scenarios, this invention reduces the cost by 19% compared to the constraint-penalty Q-learning algorithm and increases the cumulative reward by 3%. Intelligent vehicle A successfully avoids obstacles in the scene, reaching intelligent vehicle B to intercept it from all sides. In most public experimental scenarios, this invention achieves the maximum cumulative reward with the minimum cost.
[0131] Table 1 Test Results
[0132]
[0133] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0134] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0136] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A security policy learning method based on offline reinforcement policy learning, characterized in that, The method specifically includes: A virtual-real dataset is acquired, comprising virtual data in a virtual simulation environment and real data in a real environment; the virtual data corresponds to the real data; the virtual data includes the agent's current virtual observation state, current virtual action, current virtual reward from environmental feedback, cost of virtual environmental feedback, virtual observation state at the next moment, virtual action at the next moment, and a flag indicating whether the current virtual trajectory has ended; the real data includes the agent's current real observation state, current real action, real reward from environmental feedback, real cost of environmental feedback, real observation state at the next moment, real action at the next moment, and a flag indicating whether the current real trajectory has ended; the agent is an unmanned system, which includes at least a robotic vacuum cleaner, a logistics robot, an unmanned vehicle, an unmanned boat, and a drone; Based on the aforementioned virtual and real datasets, a style transfer model is constructed; The virtual data is input into the style transfer model to generate the target virtual dataset; The target virtual data and the real data are fused to obtain a fused dataset; the fused dataset includes: the current fused observation state, the current fused action, the fused reward from environmental feedback, the fused cost from environmental feedback, the fused observation state at the next moment, the fused action at the next moment, and a flag indicating whether the current target fused trajectory has ended; The fused observation state at the next moment is input into the conditional variational autoencoder, which outputs the action at the next moment. Determine whether the action safety value of the action at the next moment is less than a preset safety threshold; If the action safety value of the action at the next moment is less than the preset safety threshold, the temporal difference error is determined based on the action at the next moment and the fused action at the current moment. The critic network is updated based on the temporal difference error, and the conditional variational autoencoder is updated using the updated critic network to obtain the action at the next moment. The action at the next moment is used as the safety policy to drive the agent's movement. If the action safety value of the action at the next moment is not less than the preset safety threshold, the temporal difference error is determined based on the fused action at the current moment and the action at the previous moment. The critic network is updated based on the temporal difference error. The conditional variational autoencoder is updated using the updated critic network to obtain the updated target action at the next moment. The updated target action at the next moment is used as the safety policy to drive the agent's movement.
2. The security policy learning method based on offline reinforcement policy learning according to claim 1, characterized in that, The training process of the critic network specifically includes: If the action safety value of the action at the next moment is less than the preset safety threshold, the commentator network is trained based on the temporal difference error determined by the action at the next moment and the fused action at the current moment, until the preset iteration threshold is reached. If the action safety value of the next action is not less than a preset safety threshold, the temporal difference error is determined based on the current fused action and the action of the previous action, and the commentator network is trained until the preset iteration threshold is reached.
3. The security policy learning method based on offline reinforcement policy learning according to claim 2, characterized in that, The action safety value for the action at the next moment is equal to: ...(1) in, Representative fusion dataset t The fusion action at +1 moment; The representative conditional variational autoencoder is based on t The fused observation state output at time +1 t Action at time +1.
4. The security policy learning method based on offline reinforcement policy learning according to claim 1, characterized in that, The construction of the style transfer model based on the virtual and real datasets specifically includes: The style transfer model is a recurrent generative adversarial network (RGAN); the RGAN includes a generator G, a generator F, and a discriminator D. X and discriminator D Y ; The recurrent generative adversarial network is trained based on several virtual data X and several real data Y, and a first generative adversarial loss and a second generative adversarial loss are determined. The target loss of the cyclic generative adversarial network is determined based on the first generative adversarial loss, the second generative adversarial loss, and the cyclic consistency loss. The recurrent generative adversarial network is updated based on the target loss, and the updated recurrent generative adversarial network is used as the style transfer model.
5. The security policy learning method based on offline reinforcement policy learning according to claim 4, characterized in that, The recurrent generative adversarial network is trained based on several virtual data X and several real data Y, and a first generative adversarial loss and a second generative adversarial loss are determined. Specifically, the training process of the recurrent generative adversarial network includes a forward training process and a backward training process. The recurrent generative adversarial network is trained in a forward manner using several virtual data X and several real data Y to generate a first forward generative adversarial loss and a second forward generative adversarial loss. The recurrent generative adversarial network is trained in reverse using several virtual data X and several real data Y to generate a first inverse generative adversarial loss and a second inverse generative adversarial loss. The first generation adversarial loss is determined based on the first forward generation adversarial loss and the first reverse generation adversarial loss; The second generation adversarial loss is determined based on the second forward generation adversarial loss and the second reverse generation adversarial loss.
6. The security policy learning method based on offline reinforcement policy learning according to claim 5, characterized in that, The positive training process specifically includes: The virtual data is input into the generator G to generate the first real data; Determine the first positive generative adversarial loss between the first real data and the real data, and update the parameters of the generator G and the discriminator D. Y The parameters, until the discriminator D Y Once the loss of the first positive generative adversarial interaction is determined to be 0, updates are stopped. The first real data is input into generator F to generate the first virtual data; Determine the second positive generative adversarial loss of the first virtual data and the virtual data, and update the parameters of the generator F and the discriminator D. X The parameters, until the discriminator D X Once the second positive generative adversarial loss is determined to be 0, updates are stopped.
7. The security policy learning method based on offline reinforcement policy learning according to claim 5, characterized in that, The reverse training process specifically includes: The real data is input into the generator F to generate the second virtual data; Determine the second reverse generative adversarial loss between the second virtual data and the virtual data; and update the parameters of the generator F and the discriminator D. X The parameters, until the discriminator D X Once the second counter-attack loss is determined to be zero, updates are stopped. The second virtual data is input into generator G to generate the second real data; Determine the first reverse generative adversarial loss between the second real data and the real data; and update the parameters of the generator G and the discriminator D. Y The parameters, until the discriminator D Y Once the first reverse generative adversarial loss is determined to be equal to 0, updates are stopped.
8. The security policy learning method based on offline reinforcement policy learning according to claim 7, characterized in that, The formula for calculating the first generative adversarial loss is: ...(2); In the formula, X represents the set of data consisting of all current virtual observation states and the next virtual observation state in the virtual data; X represents the data from... Sampling virtual data, using Indicates the sampling process; Y represents the set of data consisting of all current and next real observation states in the real data; The actual data sampled; Indicates the return discriminator The probability of determining a given set of virtual data; Represents generator The current virtual observation state generated based on the current real observation state in the real data, or the next virtual observation state generated based on the next real observation state in the real data; Discriminator For generator The probability of determining a virtual observation state generated using the current real observation state in real data as virtual data; Representing variables Relative to the expectation of virtual data X, Representing variables The expectation relative to the actual data Y; Represents the generator F and the discriminator The first generative adversarial loss generated; The formula for calculating the second generative adversarial loss is as follows: ...(3); In the formula, Indicates the return discriminator The probability of determining the given real data; Represents generator The current real observation state generated from the current virtual observation state in the original virtual data, or the next real observation state generated from the next virtual observation state in the real data; Discriminator For generator The determination of the real observation state generated using the current virtual observation state in virtual data X is the real data probability; Representing variables Relative to the expectation of virtual data X, Representing variables The expectation relative to the actual data Y; Represents the generator G and the discriminator The generated second generative adversarial loss.
9. The security policy learning method based on offline reinforcement policy learning according to claim 7, characterized in that, The formula for calculating the cycle consistency loss is as follows: ....(4); In the formula, Representing variables Relative to the expectation of virtual data X, Representing variables The expectation relative to the actual data Y.
10. The security policy learning method based on offline reinforcement policy learning according to claim 9, characterized in that, The formula for calculating the target loss is: ...(5); In the formula, This represents the first generation adversarial loss. This represents the second generation adversarial loss. Indicates the loss of cycle consistency. This represents the weighting factor that balances the various losses.
