A virtual-real seamless migration method and system, electronic device and medium

Abstract

The application discloses a virtual-real seamless migration method and system, electronic equipment and medium, and relates to the fields of representation learning and reinforcement learning. The method comprises the following steps: acquiring interaction data of an agent and an environment in a virtual environment; performing data enhancement on observation of a current state; inputting the enhanced observation into a twin representation network for dimension reduction to obtain vectorized state data; inputting the vectorized state data and corresponding actions into a twin value network to obtain corresponding values; constructing a representation loss function according to the vectorized state data, the corresponding values and observation of a next state; training the twin representation network, the twin value network and a decision network by using a gradient descent method according to the representation loss function; and obtaining trained representation networks and decision networks when the total number of training times is greater than a preset threshold, and obtaining actions responding to a real environment according to observation of a current state in the real environment. The application can improve the generalization performance of a strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of representation learning and reinforcement learning, and in particular to a virtual-real seamless transfer method, system, electronic device and medium. BACKGROUND

[0002] The application landing of reinforcement learning is an important research topic in the field of reinforcement learning. Due to the safety and efficiency considerations of reinforcement learning, it is generally necessary to learn the strategy in a virtual simulation scene, so the efficient utilization of high-dimensional observation samples and the generalization of the strategy are difficult problems in the application landing of reinforcement learning. At present, representation learning methods are combined with reinforcement learning methods to improve the sample utilization rate and the strategy generalization to realize virtual-real seamless transfer.

[0003] Representation learning with a twin network is a relatively common self-supervised representation architecture at present, which usually takes two or more enhanced data with the same semantics as input, passes through a twin encoder with the same architecture, and outputs low-dimensional representation information. The representation method based on comparison will pull the same semantic representations closer and the different semantic representations farther apart according to the similarity of the low-dimensional representations; the representation method based on promotion will also pull the same semantic representations closer according to the similarity of the low-dimensional representations. Representation learning with a twin network can make the encoder ignore the surface factors of high-dimensional original data and learn the internal consistent information of the data, such as occlusion invariance, rotation invariance and color invariance.

[0004] However, training representation learning and reinforcement learning as two independent sub-modules separately will cause the following problems:

[0005] 1. The loss of representation learning attempts to map high-dimensional observations with the same semantics to the same low-dimensional space, but the reinforcement learning loss based on reward feedback will pull the low-dimensional representations apart, causing the value network to learn some overfitting features to distinguish these different low-dimensional representations.

[0006] 2. The representation network trained separately from the reinforcement learning will lose features related to task decision-making, resulting in poor strategy generalization. SUMMARY

[0007] The purpose of the present application is to provide a virtual-real seamless transfer method, system, electronic device and medium, which uses a twin value network to pull the distance of high-dimensional input with the same semantics in the latent space at the task value level, so that the low-dimensional representation contains task features and improves the generalization performance of the strategy.

[0008] To achieve the above purpose, the present application provides the following solutions:

[0009] A virtual-real seamless transfer method, the method comprising:

[0010] interaction data of an agent and an environment in a virtual environment; the interaction data comprises an observation of a current state, an action, an observation of a next state, and a reward value; wherein the action is obtained by selection of a representation network and a decision network according to the observation of the current state; the observation of the next state and the reward value are determined according to the action; the observation comprises image data and radar data;

[0011] performing data enhancement on the observation of the current state to obtain an enhanced observation;

[0012] inputting the enhanced observation into a twin representation network to obtain vectorized state data; the twin representation network comprises the representation network and a target representation network with same architecture and different parameters as the representation network;

[0013] inputting the vectorized state data and the corresponding action into a twin value network to obtain a corresponding value; the twin value network comprises a value network and a target value network with same architecture and different parameters as the value network;

[0014] constructing a representation loss function according to the vectorized state data, the corresponding value, and the observation of the next state;

[0015] applying gradient descent method to train the twin representation network, the twin value network, and the decision network according to the representation loss function, and recording a total training number;

[0016] judging whether the total training number is greater than a preset threshold;

[0017] when the total training number is less than or equal to the preset threshold, reacquiring the interaction data of the agent and the environment in the virtual environment, and continuing the training; when the total training number is greater than the preset threshold, obtaining a trained representation network and a trained decision network;

[0018] inputting an observation of a current state in a real environment into the trained representation network and the trained decision network to obtain an action in response to the observation of the current state in the real environment.

[0019] Optionally, the interaction data of the agent and the environment comprises:

[0020] acquiring an observation of a current state of the agent in the environment;

[0021] inputting the observation of the current state into a representation network of the agent, the representation network being used for dimension reduction of the observation of the current state;

[0022] inputting the observation of the reduced dimension current state into a decision network of the agent, the decision network being configured to generate an action in response to the observation of the current state;

[0023] outputting the action by the agent to the environment;

[0024] obtaining, by the agent, an observation of a next state and a reward value in response to the action from the environment, and taking the observation of the current state, the action, the observation of the next state and the reward value as interaction data.

[0025] Optionally, the observation of the current state is data enhanced to obtain an enhanced observation, specifically including:

[0026] applying a first enhancement algorithm to data enhance the observation of the current state to obtain a first enhanced observation;

[0027] applying a second enhancement algorithm to data enhance the observation of the current state to obtain a second enhanced observation, and taking the first enhanced observation and the second enhanced observation as the enhanced observation; the data enhancement algorithm includes an image data enhancement algorithm and a radar data enhancement algorithm; the image data enhancement algorithm includes image salt and pepper noise enhancement, image Gaussian enhancement, image Gaussian blur and image random cropping; the radar data enhancement algorithm is radar ray random failure; the first enhancement algorithm is one of the image data enhancement algorithm or the radar data enhancement algorithm; the second enhancement algorithm is one of the image data enhancement algorithm or the radar data enhancement algorithm, and the first enhancement algorithm and the second enhancement algorithm use different algorithms.

[0028] Optionally, the enhanced observation is input into a twin representation network for dimension reduction to obtain vectorized state data, specifically including:

[0029] inputting the first enhanced image observation into the representation network for dimension reduction to obtain first vectorized state data;

[0030] inputting the first enhanced radar observation into the representation network for dimension reduction to obtain second vectorized state data;

[0031] splicing the first vectorized state data and the second vectorized state data to obtain third vectorized state data;

[0032] inputting the second enhanced image observation into the target representation network for dimension reduction to obtain fourth vectorized state data;

[0033] inputting the second enhanced radar observation into the target representation network for dimension reduction to obtain fifth vectorized state data;

[0034] concatenate the fourth vectorized state data and the fifth vectorized state data to obtain sixth vectorized state data, and take the third vectorized state data and the sixth vectorized state data as vectorized state data.

[0035] Optionally, the feature network and the target feature network both employ a two-dimensional convolutional neural network to reduce the dimensionality of image data in the first enhanced observation and the second enhanced observation; and the feature network and the target feature network both employ a one-dimensional convolutional neural network to reduce the dimensionality of radar data in the first enhanced observation and the second enhanced observation.

[0036] Optionally, a feature loss function is constructed according to the vectorized state data, the corresponding value and the observation of the next state, and specifically includes:

[0037] A contrast learning method is applied to construct a feature loss:

[0038]

[0039] sim(c,b)=c T b;

[0040] Wherein, sim() is a similarity calculation function using vector dot product, x i is the third vectorized state data of the i th set of interaction data; x' i is the sixth vectorized state data of the i th set of interaction data; x' j is the sixth vectorized state data of the j th set of interaction data.

[0041] Loss of twin value network:

[0042]

[0043] Wherein, Q θ (x,a) is the value corresponding to the third vectorized state data x and the action a; Q θ' (x',a) is the value corresponding to the sixth vectorized state data x' and the action a.

[0044] Value loss in reinforcement learning:

[0045]

[0046]

[0047] Wherein, r represents the reward value, γ represents the reward decay coefficient, a represents the action, represents the observation of the next state.

[0048] Policy loss in reinforcement learning:

[0049]

[0050] According to the representation loss, the twin value network loss and the value loss in reinforcement learning, an initial representation loss function is constructed, and the initial representation loss function and the policy loss in reinforcement learning are taken as a representation loss function; wherein the initial representation loss function is:

[0051]

[0052] Wherein λ Siamese-Q And λ Q Is an adjustable loss weight.

[0053] Optionally, according to the representation loss function, gradient descent method is applied to train the twin representation network, the twin value network and the decision network, specifically including:

[0054] According to the initial representation loss function, gradient descent method is applied to train the representation network, and the training times are recorded to obtain a first training time;

[0055] According to the value loss in reinforcement learning, gradient descent method is applied to train the value network, and the training times are recorded to obtain a second training time;

[0056] According to the policy loss in reinforcement learning, gradient descent method is applied to train the decision network, and the training times are recorded to obtain a third training time;

[0057] The application of And The weights of the twin value network loss and the value loss in reinforcement learning are adaptively adjusted to obtain the trained initial representation loss function;

[0058] The target representation network is trained by applying ξ'←(1-τ)ξ'+τξ, and the training times are recorded to obtain a fourth training time;

[0059] And the target value network is trained by applying θ'←(1-τ)θ'+τθ, and the training times are recorded to obtain a fifth training time;

[0060] According to the first training time, the second training time, the third training time, the fourth training time and the fifth training time, the training times are determined.

[0061] A virtual-real seamless migration system is applied to the virtual-real seamless migration method described above, and the system comprises:

[0062] an acquisition module, configured to acquire interaction data of an agent and an environment in a virtual environment, the interaction data comprising observation of a current state, an action, observation of a next state, and a reward value, wherein the action is obtained according to the observation of the current state through selection of a representation network and a decision network, the observation of the next state and the reward value are determined according to the action, and the observation comprises image data and radar data;

[0063] an enhancement module, configured to perform data enhancement on the observation of the current state to obtain enhanced observation;

[0064] a dimension reduction module, configured to input the enhanced observation into a twin representation network to obtain vectorized state data, wherein the twin representation network comprises the representation network and a target representation network with same architecture and different parameters as the representation network;

[0065] a value determination module, configured to input the vectorized state data and the corresponding action into a twin value network to obtain a corresponding value, wherein the twin value network comprises a value network and a target value network with same architecture and different parameters as the value network;

[0066] a loss determination module, configured to construct a representation loss function according to the vectorized state data, the corresponding value, and the observation of the next state;

[0067] a training module, configured to train the twin representation network, the twin value network, and the decision network by using gradient descent method according to the representation loss function, and record a total training number;

[0068] a judgment module, configured to judge whether the total training number is greater than a preset threshold;

[0069] a judgment result determination module, configured to reacquire the interaction data of the agent and the environment in the virtual environment and continue the training when the total training number is less than or equal to the preset threshold, and obtain a trained representation network and a trained decision network when the total training number is greater than the preset threshold;

[0070] a prediction module, configured to acquire observation of a current state in a real environment, and input the observation of the current state in the real environment into the trained representation network and the trained decision network to obtain an action in response to the observation of the current state in the real environment.

[0071] An electronic device, comprising a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the above-mentioned virtual-real seamless migration method.

[0072] A computer readable storage medium stores a computer program, the computer program is executed by a processor to implement the virtual-real seamless migration method.

[0073] According to the specific embodiments of the present application, the following technical effects are disclosed:

[0074] The present application narrows the distance between the same semantic high-dimensional observation input in the low-dimensional hidden space at the task value level through the representation of the twin value network, so that the representation network contains task features, improves the generalization performance of the strategy, and then seamlessly migrates the representation strategy trained in the virtual simulation environment to the real world for inference decision without any real world data, interaction and tuning, and maintains the performance. BRIEF DESCRIPTION OF DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0076] Figure 1 The flowchart of the virtual-real seamless migration method of the present application is shown in the figure.

[0077] Figure 2 The flowchart of the virtual-real seamless migration network training method based on the twin value network representation of the present application is shown in the figure.

[0078] Figure 3 The twin value network representation architecture of the present application is shown in the figure.

[0079] Figure 4 The simulation environment schematic diagram of the present application is shown in the figure.

[0080] Figure 5 The real environment schematic diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0081] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0082] The application aims to provide a virtual-real seamless migration method, system, electronic device and medium, which narrows the distance of high-dimensional inputs with the same semantics in the hidden space at the task value level by using a twin value network, so that the low-dimensional representation contains task features, and the generalization performance of the strategy is improved.

[0083] The method of the application is a reinforcement learning representation method based on a twin value network in the case of no real data and no real interaction, which solves the problem that the reinforcement learning strategy trained in a virtual simulation scene has low generalization and large virtual-real difference, which makes it difficult to migrate the strategy to a real scene and land it, and can be applied to decision learning and application landing of unmanned systems (unmanned vehicles, unmanned aerial vehicles, etc.). The application narrows the distance of high-dimensional inputs with the same semantics in the hidden space at the task value level by using a twin value network, so that the low-dimensional representation contains task features, and the generalization performance of the strategy is improved, so that the representation network and the decision network trained in the virtual environment can be seamlessly migrated to the real environment for inference and decision making, and maintain considerable performance.

[0084] The basic features of the application mainly include the following aspects: first, interacting with a virtual simulation environment to obtain interaction data; second, constructing representation loss, reinforcement learning loss and twin value network representation loss, and training the representation and strategy together; third, the trained representation network and decision network can be directly migrated to the real environment without real data and real interaction, and can maintain considerable performance.

[0085] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0086] Embodiment one

[0087] As shown in Figure 1 The application provides a virtual-real seamless migration method, which is characterized in that the method comprises:

[0088] Step S101: obtaining interaction data of an agent and an environment in a virtual environment; the interaction data comprises observation of a current state, action, observation of a next state and reward value; wherein the action is obtained after selection of a representation network and a decision network according to the observation of the current state; the observation of the next state and the reward value are determined according to the action; the observation comprises image data and radar data. S101 specifically comprises:

[0089] Step S1011: obtaining observation of a current state of the agent in the environment.

[0090] Step S1012: inputting the observation of the current state to a representation network of the agent, and the representation network is used for dimension reduction of the observation of the current state.

[0091] Step S1013: Input the observation of the current state after dimensionality reduction into the decision network of the agent, the decision network being used to generate an action in response to the observation of the current state.

[0092] Step S1014: Use the intelligent agent to output the action to the environment.

[0093] Step S1015: Use the agent to obtain the observation and reward value of the next state in response to the action from the environment, and use the observation of the current state, the action, the observation of the next state, and the reward value as interaction data.

[0094] In practical applications, interaction with a virtual simulation environment is used to obtain batch interaction data: In the virtual simulation scenario, the simulated agent receives observations from the simulation environment at each time t. t The observations are high-dimensional, such as raw image pixels (usually expressed as length × width × 3) or radar data (usually expressed as 2 × ray number). The agent then uses these observations to determine the appropriate parameters. t After dimensionality reduction of the representation network f and selection of the decision network π, action a is obtained. t This feedback is then sent to the simulation environment, which transitions to the next state and returns the next observation to the simulated agent. t+1 The reward value r generated by this action t Interactive data is stored in the experience pool: (Note: The original text contains several typos and inconsistencies. A more accurate translation would require the full context.) <o t ,a r ,r t ,o t+1 >This is a frame of interactive data, and the frame is stored in an experience pool D with a first-in-first-out queue as the data structure; the structure of the decision network is a general fully connected neural network.

[0095] Specifically, taking the interaction with the virtual simulation unmanned vehicle target search environment (Unity3D) as an example, such as Figure 4 As shown, batch interaction data (observations, actions, and reward values) are obtained and saved to the experience pool. The image pixel dimensions are 84×84×3, and the radar data dimensions are 2×400.

[0096] Step S102: Perform data augmentation on the observations of the current state to obtain augmented observations. S102 specifically includes:

[0097] Step S1021: Apply the first enhancement algorithm to enhance the observations of the current state to obtain the first enhanced observations.

[0098] Step S1022: applying a second enhancement algorithm to the observation of the current state to obtain a second enhanced observation, and taking the first enhanced observation and the first enhanced observation as enhanced observations; the data enhancement algorithm includes an image data enhancement algorithm and a radar data enhancement algorithm; the image data enhancement algorithm includes image salt and pepper noise enhancement, image Gaussian enhancement, image Gaussian blur, and image random cropping; the radar data enhancement algorithm is radar ray random failure; the first enhancement algorithm is one of the image data enhancement algorithms or the radar data enhancement algorithm; the second enhancement algorithm is one of the image data enhancement algorithms or the radar data enhancement algorithm, and the first enhancement algorithm and the second enhancement algorithm use different algorithms.

[0099] In practical applications, a batch of interaction data is randomly taken out from the experience pool, and each high-dimensional observation image, radar is enhanced twice in different ways. Specifically, a batch of interaction data is randomly taken out from the experience pool Each high-dimensional observation o is enhanced twice in different ways The enhancement method can be randomly selected from image salt and pepper noise enhancement, image Gaussian enhancement, image Gaussian blur, image random cropping, and radar ray random failure.

[0100] Specifically, image salt and pepper noise enhancement: randomly increase or decrease the value of each pixel point.

[0101]

[0102] wherein ε is a random value between 0 and 1, v sp = 0.2; v sp is the noise intensity; p sp = 0.5; p sp is the noise occurrence probability.

[0103] Image Gaussian enhancement: for each pixel point, add a value randomly sampled from a unit Gaussian distribution.

[0104]

[0105] wherein μ go = 0; μ go is the Gaussian mean, σ go = 0; σ go is the Gaussian standard deviation.

[0106] Image Gaussian blur: Gaussian filtering is performed on the original image.

[0107] ​​

[0108] where w gb = 9; w gb is the filter window size, s gb = 9; s gb is the Gaussian standard deviation; m and n represent the pixel points to which the filter is applied.

[0109] Image random crop: randomly crop the original image HxWx3 into H'xW'x3 and then enlarge it into HxWx3; specifically, randomly crop the original image 84x84x3 into 50x50x3 and then enlarge it into 84x84x3.

[0110] Radar ray random failure: fail each radar ray with a probability of p, i.e., reset the failed ray to no object detected; specifically, fail each radar ray with a probability of 0.5, i.e., reset the failed ray to no object detected.

[0111] Step S103: input the enhanced observation into a twin representation network for dimension reduction to obtain vectorized state data; the twin representation network includes the representation network and a target representation network with the same architecture but different parameters. The representation network and the target representation network are both general fully connected neural networks. S103 specifically includes:

[0112] Step S1031: input the first enhanced image observation into the representation network for dimension reduction to obtain first vectorized state data.

[0113] Step S1032: input the first enhanced radar observation into the representation network for dimension reduction to obtain second vectorized state data.

[0114] Step S1033: concatenate the first vectorized state data and the second vectorized state data to obtain third vectorized state data.

[0115] Step S1034: input the second enhanced image observation into the target representation network for dimension reduction to obtain fourth vectorized state data.

[0116] Step S1035: input the second enhanced radar observation into the target representation network for dimension reduction to obtain fifth vectorized state data.

[0117] Step S1036: concatenate the fourth vectorized state data and the fifth vectorized state data to obtain sixth vectorized state data, and take the third vectorized state data and the sixth vectorized state data as the vectorized state data.

[0118] In addition, the representation network and the target representation network both employ a two-dimensional convolutional neural network to reduce dimensionality of image data in the first enhanced observation and the second enhanced observation; the representation network and the target representation network both employ a one-dimensional convolutional neural network to reduce dimensionality of radar data in the first enhanced observation and the second enhanced observation.

[0119] In actual application, the two enhanced high-dimensional observations are input into the twin representation network to obtain two low-dimensional representations, and the process is as follows:

[0120] Two twin representation networks with the same architecture but different initial parameters are constructed: f ξ and f ξ , and the enhanced high-dimensional observations are reduced in dimension to vectorized state data. For image high-dimensional observations, a two-dimensional convolutional neural network is used for dimensionality reduction representation; specifically, a 64-dimensional low-dimensional vector is obtained; for radar high-dimensional observations, a one-dimensional convolutional neural network is used for dimensionality reduction representation, and a 64-dimensional low-dimensional vector is obtained, and all low-dimensional vectors are spliced to obtain and two 128-dimensional low-dimensional vectorized representations.

[0121] Step S104: input the vectorized state data and the corresponding action into a twin value network to obtain the corresponding value; the twin value network includes a value network and a target value network with the same architecture but different parameters. The value network and the target value network are both general fully connected neural networks.

[0122] In actual application, as shown in Figure 3 , the two low-dimensional representations and the corresponding actions are input into the twin value network to obtain two values, and the process is as follows:

[0123] The two 128-dimensional low-dimensional vectorized representations and the action data a are respectively input into the value network Q θ and the target value network Q θ' to obtain the representation state-action values Q θ (x, a) and Q θ' (x', a).

[0124] Step S105: construct a representation loss function according to the vectorized state data, the corresponding value, and the next state observation. S105 specifically includes:

[0125] (1) apply a contrastive learning method to construct a representation loss:

[0126]

[0127] sim(c,b) = c T b (6)

[0128] wherein sim() is a similarity calculation function using vector dot product, x i is the third vectorized state data of the i-th set of interaction data; x' i is the sixth vectorized state data of the i-th set of interaction data; x' j is the sixth vectorized state data of the j-th set of interaction data; c and b are two different vectors, sim(c,b) is only to measure the similarity of two different vectors, and c and b can be any vector.

[0129] (2) Twin Value Network Loss:

[0130]

[0131] wherein Q θ (x,a) is the value corresponding to the third vectorized state data x and action a; Q θ' (x',a) is the value corresponding to the sixth vectorized state data x' and action a.

[0132] (3) Value Loss in Reinforcement Learning:

[0133]

[0134]

[0135] wherein r represents the reward value, γ represents the reward decay coefficient, a represents the action, represents the observation of the next state; in the application, o is the original observation (including image and radar); is the first enhanced observation (including the first enhanced image and radar); is the second enhanced observation (including the second enhanced image and radar); x is the third vectorized state data (after the representation network); x' is the sixth vectorized state data (after the target representation network).

[0136] (4) Policy Loss in Reinforcement Learning:

[0137]

[0138] According to the representation loss, the twin value network loss and the value loss in reinforcement learning, an initial representation loss function is constructed, and the initial representation loss function and the policy loss in reinforcement learning are taken as the representation loss function; wherein the initial representation loss function is:

[0139]

[0140] where λ Siamese-Q and λ Q is an adjustable loss weight.

[0141] Step S106: According to the feature loss function, applying gradient descent method to train the twin feature network, the twin value network and the decision network, and record the total training times. S106 specifically includes:

[0142] Step S1061: According to the initial feature loss function, applying gradient descent method to train the feature network, and record the training times, to obtain the first training times.

[0143] Step S1062: According to the value loss in reinforcement learning, applying gradient descent method to train the value network, and record the training times, to obtain the second training times.

[0144] Step S1063: According to the policy loss in reinforcement learning, applying gradient descent method to train the decision network, and record the training times, to obtain the third training times.

[0145] Step S1064: Applying and Adaptive adjustment of the weight of the twin value network loss and the value loss in reinforcement learning, to obtain the trained initial feature loss function. Wherein represents the feature loss gradient, represents the value loss gradient, represents the twin value loss gradient.

[0146] Step S1065: Applying ξ'←(1-τ)ξ'+τξ to train the target feature network, and record the training times, to obtain the fourth training times.

[0147] Step S1066: Applying θ'←(1-τ)θ'+τθ to train the target value network, and record the training times, to obtain the fifth training times. Wherein τ represents the target training momentum coefficient.

[0148] Step S1067: According to the first training times, the second training times, the third training times, the fourth training times and the fifth training times, determine the training times.

[0149] In practical application, using Adam gradient descent method, updating the feature network f ξ according to formula (11), updating the value network Q θ according to formula (8) and formula (9), and updating the decision network π φ according to formula (10).

[0150] with main loss, and auxiliary loss, and the weight of the auxiliary loss is adaptively adjusted:

[0151]

[0152]

[0153] For the twin representation network f ξ' , the twin value network Q θ' is updated in a momentum manner:

[0154] ξ'←(1-τ)ξ'+τξ (14)

[0155] θ'←91-τ)θ'+τθ (15)

[0156] where τ is set to 0.02.

[0157] Step S107: determining whether the total number of training is greater than a preset threshold.

[0158] Step S108: when the total number of training is less than or equal to the preset threshold, reacquiring the interaction data of the agent and the environment in the virtual environment and continuing training; when the total number of training is greater than the preset threshold, obtaining the trained representation network and the trained decision network.

[0159] In actual application, when the total number of training is less than or equal to the preset threshold, returning to step S101, and each time training is completed f ξ , Q θ , π φ , f ξ ' and Q θ' update the number of iterations, if the number of iterations is greater than the predetermined upper limit, the training is completed, otherwise, starting from step 101. Specifically, the preset threshold of the number of training is 100000.

[0160] Step S109: acquiring the observation of the current state in the real environment, and inputting the observation of the current state in the real environment into the trained representation network and the trained decision network to obtain the action in response to the observation of the current state in the real environment.

[0161] In actual application, the value network and all the twin networks f ξ ', Q θ , Q θ' are discarded, and only the representation network f ξ and the decision network π φ are retained.

[0162] Directly migrate the representation network and the decision network to the real environment, when encountering a new real scene high-dimensional observation o, the action generation formula is:

[0163] a = π φ (f ξ (o)) (16)

[0164] As a specific embodiment, the virtual-real seamless migration method provided by the application obtains the training process of the trained representation network and the trained decision network, as shown in Figure 2 , specifically comprising:

[0165] Step S1: interact with the virtual simulation environment, obtain a batch of interaction data (observation, action and reward value) and save to the experience pool.

[0166] Step S2: randomly take out a batch of interaction data from the experience pool, and do two different enhancements for each high-dimensional observation image and radar.

[0167] Step S3: input the enhanced two high-dimensional observations into the twin representation network to obtain two low-dimensional representations.

[0168] Step S4: input the two low-dimensional representations and the corresponding actions into the twin value network to obtain two values.

[0169] Step S5: construct the representation loss to reduce the distance of the two low-dimensional representations from the same observation; construct the twin value network representation loss to reduce the value distance of the two low-dimensional representations and actions from the same observation; construct the reinforcement learning loss to update the decision policy and value through the reward value feedback only taking one low-dimensional representation.

[0170] Step S6: whether the current training reaches the iteration upper limit, if the upper limit is reached, execute step S7, if the upper limit is not reached, execute step S1.

[0171] Step S7: keep the representation and strategy, and do not optimize to migrate to the real environment for inference decision.

[0172] The experimental results of interacting with the virtual simulation unmanned vehicle target search environment (Unity3D) are as follows:

[0173] As Figure 5As shown, the unmanned vehicle agent needs to search and reach the position of the target object in a closed scene through high-dimensional image sensors and high-dimensional radar sensors, while not colliding with the fence and random obstacles. Table 1 shows the target search success rate of the model trained by different training methods when inferring in a virtual simulation environment with added random noise. It can be seen that the success rate of the present application reaches 90.07%, which is 22.86 percentage points higher than other methods. Table 2 shows the target search success rate of the model trained in the virtual simulation environment when seamlessly migrated to the real environment for inference (10 experiments per group). It can be seen that the success rate of the present application reaches 90%, which is 60 percentage points higher than other methods.

[0174] Table 1 Target search success rate statistics table in random noise virtual simulation environment

[0175] No representation Representation learning + reinforcement learning Representation learning + reinforcement learning + twin value network 19.12％ 67.21％ 90.07％

[0176] Table 2 Target search success rate statistics table in real environment inference

[0177] No representation Representation learning + reinforcement learning Representation learning + reinforcement learning + twin value network 0％ 30％ 90％

[0178] Compared with the reinforcement learning method with a representation network and combined with representation learning, the method of the present application can make the low-dimensional representation contain task features by narrowing the distance between semantic high-dimensional inputs in the hidden space at the task value level, improve the generalization performance of the strategy, and thus enable the representation network and the decision network trained in the virtual scene to seamlessly migrate to the real scene for decision-making.

[0179] Embodiment two

[0180] In order to perform the method corresponding to the above-mentioned embodiment one to achieve the corresponding functions and technical effects, the following provides a virtual-real seamless migration system, the system comprises:

[0181] An acquisition module is configured to acquire interaction data of an agent and an environment; the interaction data comprises observation of a current state, action, observation of a next state and reward value; wherein the action is obtained by selecting the observation of the current state through a representation network and a decision network; the observation of the next state and the reward value are determined according to the action; the observation comprises image data and radar data.

[0182] An enhancement module is configured to perform data enhancement on the observation of the current state to obtain enhanced observation.

[0183] A dimension reduction module is configured to input the enhanced observation into a twin representation network to obtain vectorized state data; the twin representation network comprises the representation network and a target representation network with the same architecture and different parameters as the representation network.

[0184] a value determination module configured to input the vectorized state data and the corresponding action into a twin value network to obtain a corresponding value, the twin value network comprising a value network and a target value network with the same architecture as the value network but different parameters.

[0185] a loss determination module configured to construct a representation loss function according to the vectorized state data, the corresponding value and the observation of the next state.

[0186] a training module configured to train the twin representation network, the twin value network and the decision network according to the representation loss function by using a gradient descent method, and record a total number of training times.

[0187] a judgment module configured to judge whether the total number of training times is greater than a preset threshold.

[0188] a judgment result determination module configured to, when the total number of training times is less than or equal to the preset threshold, reacquire the interaction data between the agent and the environment in the virtual environment and continue the training, and when the total number of training times is greater than the preset threshold, obtain the trained representation network and the trained decision network.

[0189] a prediction module configured to acquire an observation of a current state in a real environment, and input the observation of the current state in the real environment into the trained representation network and the trained decision network to obtain an action in response to the observation of the current state in the real environment.

[0190] Embodiment Three

[0191] An electronic device is provided in the embodiments of the present application, which comprises a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to execute the method for privacy protection of security monitoring video in Embodiment One.

[0192] Optionally, the electronic device can be a server.

[0193] In addition, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method for privacy protection of security monitoring video in Embodiment One.

[0194] In the description, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0195] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. A method for seamless virtual-real migration, characterized in that, The method comprises: acquiring interaction data of an agent and an environment in a virtual environment; the interaction data comprises observation of a current state, an action, observation of a next state, and a reward value; wherein the action is obtained by selecting a representation network and a decision network according to the observation of the current state; the observation of the next state and the reward value are determined according to the action; the observation comprises image data and radar data; performing data enhancement on the observation of the current state to obtain enhanced observation; inputting the enhanced observation into a twin representation network to obtain vectorized state data; the twin representation network comprises the representation network and a target representation network with the same architecture and different parameters as the representation network; inputting the vectorized state data and the corresponding action into a twin value network to obtain a corresponding value; the twin value network comprises a value network and a target value network with the same architecture and different parameters as the value network; constructing a representation loss function according to the vectorized state data, the corresponding value, and the observation of the next state; applying gradient descent method to train the twin representation network, the twin value network, and the decision network according to the representation loss function, and recording the total number of training times; determining whether the total number of training times is greater than a preset threshold; when the total number of training times is less than or equal to the preset threshold, reacquiring the interaction data of the agent and the environment in the virtual environment and continuing the training; when the total number of training times is greater than the preset threshold, obtaining the trained representation network and the trained decision network; acquiring observation of a current state in a real environment, and inputting the observation of the current state in the real environment into the trained representation network and the trained decision network to obtain an action in response to the observation of the current state in the real environment.

2. The method of claim 1, wherein, The acquisition of the interaction data of the agent and the environment specifically comprises: acquiring observation of a current state of the agent in the environment; inputting the observation of the current state into a representation network of the agent, the representation network being used for dimension reduction of the observation of the current state; inputting the dimension-reduced observation of the current state into a decision network of the agent, the decision network being used for generating an action in response to the observation of the current state; outputting the action from the agent to the environment; acquiring observation of a next state and a reward value in response to the action from the environment by the agent, and taking the observation of the current state, the action, the observation of the next state, and the reward value as interaction data.

3. The method of claim 1, wherein, The data enhancement on the observation of the current state to obtain enhanced observation specifically comprises: applying a first enhancement algorithm to perform data enhancement on the observation of the current state to obtain first enhanced observation; applying a second enhancement algorithm to the observation of the current state to obtain second enhanced observation, and taking the first enhanced observation and the second enhanced observation as enhanced observation; the data enhancement algorithm includes an image data enhancement algorithm and a radar data enhancement algorithm; the image data enhancement algorithm includes image salt and pepper noise enhancement, image Gaussian enhancement, image Gaussian blur and image random clipping; the radar data enhancement algorithm is radar ray random failure; the first enhancement algorithm is one of the image data enhancement algorithms or the radar data enhancement algorithm; the second enhancement algorithm is one of the image data enhancement algorithms or the radar data enhancement algorithm, and the first enhancement algorithm and the second enhancement algorithm use different algorithms.

4. The method of claim 3, wherein, inputting the enhanced observation into a twin representation network for dimension reduction to obtain vectorized state data, specifically including: inputting the first enhanced image observation into the representation network for dimension reduction to obtain first vectorized state data; inputting the first enhanced radar observation into the representation network for dimension reduction to obtain second vectorized state data; splicing the first vectorized state data and the second vectorized state data to obtain third vectorized state data; inputting the second enhanced image observation into the target representation network for dimension reduction to obtain fourth vectorized state data; inputting the second enhanced radar observation into the target representation network for dimension reduction to obtain fifth vectorized state data; splicing the fourth vectorized state data and the fifth vectorized state data to obtain sixth vectorized state data, and taking the third vectorized state data and the sixth vectorized state data as vectorized state data.

5. The method of claim 4, wherein, The representation network and the target representation network both use a two-dimensional convolutional neural network to reduce the dimension of image data in the first enhanced observation and the second enhanced observation; the representation network and the target representation network both use a one-dimensional convolutional neural network to reduce the dimension of radar data in the first enhanced observation and the second enhanced observation.

6. The method of claim 4, wherein, constructing a representation loss function according to the vectorized state data, the corresponding value and the observation of the next state, specifically including: applying a contrast learning method to construct a representation loss: sim(c,b) = c T b; wherein sim() is a similarity calculation function using vector dot product, x i is the third vectorized state data of the i-th set of interaction data; x' i is the sixth vectorized state data of the i-th set of interaction data; x' j is the sixth vectorized state data of the j-th set of interaction data; c and b are two different vectors, and sim(c, b) is only a measure of the similarity of two different vectors, and c and b can be any vector; twin value network loss: wherein Q θ (x, a) is the value corresponding to the third vectorized state data x and the action a; Q θ' (x', a) is the value corresponding to the sixth vectorized state data x' and the action a; value loss in reinforcement learning: where r represents the reward value, γ represents a reward decay coefficient, a represents the action, an observation representing the next state; policy loss in reinforcement learning: constructing an initial representation loss function according to the representation loss, the twin value network loss and the value loss in reinforcement learning, and taking the initial representation loss function and the policy loss in reinforcement learning as a representation loss function; wherein the initial representation loss function: Where, λ Siamese-Q With λ Q The loss weight is adjustable.

7. The method of claim 6, wherein, applying gradient descent method to train the twin representation network, the twin value network and the decision network according to the representation loss function, specifically including: applying gradient descent method to train the representation network according to the initial representation loss function, and recording the number of training times to obtain a first number of training times; According to the value loss in the reinforcement learning, a gradient descent method is applied to train the value network, and a training number is recorded to obtain a second training number; According to the policy loss in the reinforcement learning, a gradient descent method is applied to train the decision network, and a training number is recorded to obtain a third training number; Applications and adaptively adjusting weights of the twin value network loss and the value loss in the reinforcement learning to obtain a trained initial representation loss function; denotes a representation loss gradient, denotes a value loss gradient, denotes a twin value loss gradient; The target representation network is trained by applying ξ'←(1-τ)ξ'+τξ, and a training number is recorded to obtain a fourth training number, where ξ is a representation network parameter, ξ' is a target representation network parameter, and τ represents a target training momentum coefficient; The target value network is trained by applying θ'←(1-τ)θ'+τθ, and a training number is recorded to obtain a fifth training number; where θ is a value network parameter, θ' is a target value network parameter, and τ represents a target training momentum coefficient; The training number is determined according to the first training number, the second training number, the third training number, the fourth training number, and the fifth training number.

8. A virtual-real seamless migration system, characterized by, The system comprises: An acquisition module configured to acquire interaction data of an agent and an environment; the interaction data comprises an observation of a current state, an action, an observation of a next state, and a reward value; the action is obtained by selecting a representation network and a decision network according to the observation of the current state; the observation of the next state and the reward value are determined according to the action; the observation comprises image data and radar data; An enhancement module configured to perform data enhancement on the observation of the current state to obtain an enhanced observation; A dimension reduction module configured to input the enhanced observation into a twin representation network to obtain vectorized state data; the twin representation network comprises the representation network and a target representation network with the same architecture but different parameters; A value determination module configured to input the vectorized state data and a corresponding action into a twin value network to obtain a corresponding value; the twin value network comprises a value network and a target value network with the same architecture but different parameters; A loss determination module configured to construct a representation loss function according to the vectorized state data, the corresponding value, and the observation of the next state; A training module configured to train the twin representation network, the twin value network, and the decision network by applying a gradient descent method according to the representation loss function, and record a total training number; A judgment module configured to judge whether the total training number is greater than a preset threshold; A judgment result determination module configured to, when the total training number is less than or equal to the preset threshold, reacquire interaction data of an agent and an environment in a virtual environment and continue training; and when the total training number is greater than the preset threshold, obtain a trained representation network and a trained decision network; A prediction module configured to acquire an observation of a current state in a real environment, and input the observation of the current state in the real environment into the trained representation network and the trained decision network to obtain an action in response to the observation of the current state in the real environment.

9. An electronic device, comprising: An electronic device comprising a memory for storing a computer program and a processor for running the computer program to cause the electronic device to perform the seamless migration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program product, which stores a computer program, and the computer program is executed by a processor to implement the seamless migration method according to any one of claims 1 to 7.

Images