Reinforcement learning generalization method and device for visually changing environment

Abstract

The present invention provides a reinforcement learning generalization method and device for a visually changing environment, comprising: associating and storing the original state of an intelligent agent at a current moment, the action of the intelligent agent at a current moment, the environmental reward of the intelligent agent at a current moment, and the original state of the intelligent agent at a next moment in a replay pool; enhancing the original state using an enhancement function to obtain an enhanced state corresponding to the original state; adding saliency guidance information to the original state and the enhanced state respectively, obtaining the value consistency loss of the intelligent agent in combination with the data in the replay pool and the enhanced state, and updating the value network and representation network of the intelligent agent based on the value consistency loss; obtaining the strategy consistency loss of the intelligent agent corresponding to the original state and the enhanced state, and updating the strategy network of the intelligent agent based on the strategy consistency loss; obtaining the dynamics loss of the intelligent agent corresponding to the original state and the enhanced state through a dynamics model, and updating the representation network and dynamics model of the intelligent agent based on the dynamics loss.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning generalization method and device in a visual changing environment. Background Art

[0002] In recent years, reinforcement learning based on visual input has achieved great success in various applications, such as video games, robotic manipulation, and autonomous navigation. However, it is still challenging to generalize the strategies learned in the current environment to new and changing environments. For example, changes in visual input that are irrelevant to the task can easily interfere with the agent's attention and decision-making, thereby affecting the performance of the reinforcement learning algorithm. Summary of the invention

[0003] The present invention provides a reinforcement learning generalization method and device in a visual changing environment, which are used to improve the generalization of reinforcement learning based on visual input.

[0004] The present invention provides a reinforcement learning generalization method for a visually changing environment, which is applied to an intelligent agent that makes decisions based on visual input, wherein the state of the intelligent agent is image data of the visual input, and the action of the intelligent agent is a control variable of a control target, including:

[0005] The original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment are associated and stored in the replay pool;

[0006] Using an enhancement function to enhance the original state in the playback pool, to obtain an enhanced state corresponding to the original state in the playback pool;

[0007] Adding saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the replay pool and the enhanced state, and updating the value network and representation network of the agent based on the value consistency loss;

[0008] Obtaining the original state in the replay pool and the policy consistency loss of the agent corresponding to the enhanced state, and updating the policy network of the agent based on the policy consistency loss;

[0009] The original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state are obtained through the dynamics model, and the representation network and the dynamics model of the agent are updated based on the dynamics loss.

[0010] According to a reinforcement learning generalization method for a visually changing environment provided by the present invention, the original state in the replay pool and the enhanced state are respectively added with saliency guidance information, and the value consistency loss of the intelligent agent is obtained by combining the data in the replay pool and the enhanced state, including:

[0011] adding saliency guidance information to the original state and the enhanced state in the playback pool respectively, to obtain a first saliency map corresponding to the original state in the playback pool and a second saliency map corresponding to the enhanced state;

[0012] Obtaining a first predicted value of the agent according to the original state of the agent at the current moment in the playback pool and the action of the agent at the current moment;

[0013] Obtaining a target value of the agent according to the environment reward of the agent at the current moment, the original state of the agent at the next moment, and the action of the agent at the next moment in the replay pool;

[0014] Obtaining the original state value loss of the agent according to the first predicted value and the target value of the agent;

[0015] Obtaining a second predicted value of the agent according to the enhanced state of the agent at the current moment and the action of the agent at the current moment in the playback pool;

[0016] Obtaining an enhanced state value loss of the agent according to the first predicted value and the target value of the agent;

[0017] Obtaining a third predicted value of the agent according to the first saliency map of the agent at the current moment and the action of the agent at the current moment in the playback pool;

[0018] Obtaining an original state saliency-guided consistency loss of the agent according to the first prediction value and the third prediction value of the agent;

[0019] Obtaining a fourth predicted value of the agent according to the second saliency map of the agent at the current moment and the action of the agent at the current moment in the playback pool;

[0020] Obtaining an enhanced state saliency guided consistency loss of the agent according to the second prediction value and the fourth prediction value of the agent;

[0021] The value consistency loss of the agent is obtained according to the original state value loss, the enhanced state value loss, the original state saliency-guided consistency loss, the enhanced state saliency-guided consistency loss and the value consistency coefficient of the agent.

[0022] According to a reinforcement learning generalization method for a visually changing environment provided by the present invention, the original state and the enhanced state in the playback pool are respectively added with saliency guidance information to obtain a first saliency map corresponding to the original state in the playback pool and a second saliency map corresponding to the enhanced state, including:

[0023] Obtaining a first saliency gradient map corresponding to an original state in the playback pool;

[0024] The pixel values ​​in the first saliency gradient map that are equal to or higher than a preset threshold are set to 1, and the pixel values ​​that are lower than the preset threshold are set to 0, to obtain a first saliency attribute map of the significant area, and the first saliency attribute map is used as saliency guidance information of the original state in the playback pool to obtain a first saliency map corresponding to the original state in the playback pool;

[0025] Acquire a second saliency gradient map corresponding to the enhanced state;

[0026] The pixel values ​​in the second saliency gradient map that are equal to or higher than a preset threshold are set to 1, and the pixel values ​​that are less than the preset threshold are set to 0, so as to obtain a second saliency attribute map of the significant area, and the second saliency attribute map is used as the saliency guidance information of the enhanced state to obtain a second saliency map corresponding to the enhanced state.

[0027] According to a reinforcement learning generalization method for a visually changing environment provided by the present invention, the obtaining of the original state in the playback pool and the strategy consistency loss of the intelligent agent corresponding to the enhanced state includes:

[0028] Obtaining the strategy loss corresponding to the original state in the replay pool;

[0029] Obtaining a policy constraint loss between an original state in the replay pool and the enhanced state;

[0030] The strategy consistency loss of the agent is obtained according to the strategy loss, the strategy constraint loss and the strategy consistency coefficient.

[0031] According to a reinforcement learning generalization method for a visually changing environment provided by the present invention, the obtaining of the original state in the playback pool and the dynamics loss of the intelligent agent corresponding to the enhanced state includes:

[0032] Obtaining a first dynamic loss corresponding to an original state in the playback pool through a dynamic model;

[0033] Obtaining a second dynamic loss corresponding to the enhanced state through a dynamic model;

[0034] The dynamics loss of the agent is obtained according to the first dynamics loss and the second dynamics loss.

[0035] A reinforcement learning generalization method for a visually changing environment provided by the present invention further includes:

[0036] Initialize the agent's network parameters, enhancement function, value consistency coefficient and strategy consistency coefficient, wherein the network parameters include value network parameters, strategy network parameters, representation network parameters and dynamic model parameters.

[0037] The present invention also provides a reinforcement learning generalization device for a visually changing environment, which is applied to an intelligent agent that makes decisions based on visual input, wherein the state of the intelligent agent is the image data of the visual input, and the action of the intelligent agent is the control variable of the control target, including:

[0038] The data collection module is used to associate and store the original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment in the replay pool;

[0039] A state enhancement module, used to enhance the original state in the playback pool using an enhancement function to obtain an enhanced state corresponding to the original state in the playback pool;

[0040] A value network updating module, used to add saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtain value consistency loss in combination with the data in the replay pool and the enhanced state, and update the value network and representation network of the agent based on the value consistency loss;

[0041] A policy network updating module, used to obtain the original state in the replay pool and the policy consistency loss of the agent corresponding to the enhanced state, and update the policy network of the agent based on the policy consistency loss;

[0042] The representation network update module is used to obtain the original state in the playback pool and the dynamics loss of the intelligent agent corresponding to the enhanced state through a dynamics model, and update the representation network and dynamics model of the intelligent agent based on the dynamics loss.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the reinforcement learning generalization method for a visually changing environment as described in any one of the above are implemented.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described reinforcement learning generalization methods for visually changing environments.

[0045] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned reinforcement learning generalization methods for visual changing environments.

[0046] The reinforcement learning generalization method and device for visual change environment provided by the present invention updates the value function by using value consistency loss, adds saliency guidance information training while performing value training for the original state and the enhanced state, uses the original state and the enhanced state to train the value network and the representation network at the same time, and uses the saliency map to perform saliency guidance for the two states respectively. In this way, the intelligent agent pays attention to accurate and consistent task-related areas; the policy network is trained by using the update method of policy consistency loss, and the KL divergence between the policy distributions made by the intelligent agent for the original state and the enhanced state is limited while the policy is updated. In this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information; for the original state and the enhanced state, the dynamic loss is used as an auxiliary task to train the representation network and the dynamic model, so that the representation generated by the intelligent agent for the original state and the enhanced state conforms to the state transition of the environment, and in this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information, thereby improving the visual generalization ability of the intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flow chart of the reinforcement learning generalization method for visual change environment provided by the present invention;

[0049] Figure 2 This is one of the schematic diagrams of application scenarios of the reinforcement learning generalization method for visually changing environments provided by the present invention;

[0050] Figure 3 This is the second schematic diagram of the application scenario of the reinforcement learning generalization method for visual change environment provided by the present invention;

[0051] Figure 4This is the third schematic diagram of the application scenario of the reinforcement learning generalization method for visual change environment provided by the present invention;

[0052] Figure 5 It is a flow chart of the value consistency loss calculation method provided by the present invention;

[0053] Figure 6 It is a flow chart of the strategy consistency loss calculation method provided by the present invention;

[0054] Figure 7 It is a schematic flow chart of the dynamic loss calculation method provided by the present invention;

[0055] Figure 8 It is a structural schematic diagram of the network model of the intelligent agent provided by the present invention;

[0056] Fig. 9 It is a schematic diagram of the structure of the reinforcement learning generalization device for visual change environment provided by the present invention;

[0057] Fig.10 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In recent years, reinforcement learning based on visual input refers to a method that uses computer vision technology as input and combines it with reinforcement learning algorithms for decision-making and learning. In traditional reinforcement learning, agents usually use abstract state representations to make decisions, while reinforcement learning based on visual input directly uses visual input (such as images or videos) as the state of the agent.

[0060] In this approach, the agent perceives visual input and extracts information about the environment from it. It then uses this information to decide on its next action and obtains reward signals through interaction with the environment to further optimize its behavior strategy.

[0061] Reinforcement learning based on visual input has wide applications in many fields, such as video games, robotic manipulation, and autonomous navigation. However, it is still challenging to generalize the strategy learned in the current environment to a new and changing environment.

[0062] In the decision-making tasks of deep reinforcement learning, the visual input of the agent will change with the changes of the scene. Studies have found that changes in visual input that are not related to the task can easily interfere with the agent's attention and decision-making actions, thereby affecting the performance of the reinforcement learning algorithm. This is mainly due to three reasons: First, in new scenes, the features learned by the agent's value network from the visual input can easily be concentrated in the changing task-irrelevant areas, causing the agent to make wrong decisions based on task-irrelevant features; second, the agent's policy network is difficult to make stable decisions for visual inputs with the same control target in different changing environments, resulting in the policy performance in new scenes being inferior to the decision-making effect during training; finally, the agent's representation network is difficult to learn efficient and robust representations of unknown changing environments, which hinders the agent's generalization ability.

[0063] Based on the defects in the above-mentioned technology, a reinforcement learning generalization method for visual change environment is proposed in this embodiment. The value function is updated by using value consistency loss. The original state and the enhanced state are trained with saliency guidance information training at the same time. The value network is trained with the original state and the enhanced state, and the saliency map is used to guide the saliency of the two states respectively. In this way, the intelligent agent pays attention to the accurate and consistent task-related areas; the policy network is trained by using the update method of policy consistency loss, and the KL divergence between the policy distributions made by the intelligent agent for the original state and the enhanced state is limited while updating the policy. In this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information; the dynamic loss is used as an auxiliary task to train the representation network for the original state and the enhanced state, so that the representation generated by the intelligent agent for the original state and the enhanced state conforms to the state transition of the environment. In this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information, thereby improving the visual generalization ability of the intelligent agent.

[0064] The following describes the reinforcement learning generalization method and device for visual change environment of the present invention in conjunction with the accompanying drawings.

[0065] Figure 1 It is a flow chart of the reinforcement learning generalization method for visual changing environment provided by the present invention, wherein the method is applied to an intelligent agent that makes decision-making control based on visual input, wherein the state of the intelligent agent is the control target image data in the visual input, and the action of the intelligent agent is the control variable of the control target.

[0066] In this embodiment, the agent uses visual input to make decisions and control. In this scenario, the state of the agent refers to the image data about the control target obtained from the visual input. This data describes the target or task that the agent is concerned about. The agent obtains information about the target position, posture or other relevant information by observing the current visual input in order to make decisions and control.

[0067] The action of the agent refers to the control variables of the control target, which are determined according to the preset control strategy. The preset control strategy is a set of rules or algorithms that calculate the corresponding actions based on the image data of the visual input. These rules or algorithms can select appropriate control variables based on the image data of the visual input so that the agent can reach an agreement with the target or perform a specific task. For example, in the agent walking control task, the action is the joint control variable of the agent, and the environmental reward includes two parts: health reward and forward walking reward. In the agent inverted pendulum control task, the action is the joint control variable of the inverted pendulum, and the environmental reward is the number of time steps in the inverted state. In the agent rotation control task in the rectangular environment, the action is the joint control variable of the agent's fingers, and the environmental reward is the distance between the fingers and the target point.

[0068] The goal of the reinforcement learning generalization method is to enable the agent to make adaptive decisions and control under different visual inputs. It enhances the state during the training phase so that it can learn a decision-making strategy with generalization ability. In this way, when faced with new visual inputs, the agent can choose appropriate actions to deal with different situations based on previous experience and learning.

[0069] Specifically, the method comprises:

[0070] Step 100, the original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment are associated and stored in the replay pool;

[0071] It should be understood that the original state of the agent refers to the original image data in the visual input that has not been processed, and will not be elaborated here.

[0072] In this embodiment, the state, action and reward information of the agent at the current moment are collected (s t ,a t ,r t ,s t+1 ), where s t is the original state of the agent at the current moment, a t is the action of the agent at the current moment, r t is the current agent’s environmental reward, s t+1 is the original state of the agent at the next moment.

[0073] Specifically, the original state s of the agent at the current moment t Input to the policy network π(α|s t ) to obtain the action decision a t , and then make action decision a t Input into the environment to get reward r t and the original state s of the agent at the next moment t+1 .

[0074] Step 200, using an enhancement function to enhance the original state in the playback pool to obtain an enhanced state corresponding to the original state in the playback pool;

[0075] Specifically, the enhancement function s is used on the state α =τ(s t ) to gain enhanced status.

[0076] In this embodiment, the enhancement function method includes two types: random convolution and image superposition. The random convolution is to use the same convolution kernel (such as a 5*5 random convolution kernel) to perform a convolution operation on the original state to obtain an enhanced state corresponding to the original state. The image superposition is to mix the original state with the public data set image in proportion to obtain the enhanced state corresponding to the original state.

[0077] Step 300, adding saliency guidance information to the original state and the enhanced state in the playback pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the playback pool and the enhanced state, and updating the value network and representation network of the agent based on the value consistency loss;

[0078] In this embodiment, by simultaneously utilizing the original state and the enhanced state, the value network and the representation network are updated using value consistency loss, and saliency guided training is added while training the value functions of the original state and the enhanced state to ensure that the agent can focus on the same task-related areas in different visual environments.

[0079] By adding saliency guidance information to the original state and the enhanced state, we can ensure that the agent focuses on the areas directly related to the task. These saliency guidance information can be specific goals of the task, important areas, or other task-related information. By integrating these guidance information into the state representation, the agent can focus on the task-related areas more accurately, thereby improving its decision-making and control capabilities.

[0080] Step 400, obtaining the original state in the playback pool and the policy consistency loss of the agent corresponding to the enhanced state, and updating the policy network of the agent based on the policy consistency loss;

[0081] In this embodiment, the policy consistency of the agent is measured by comparing the difference between the original policy and the enhanced policy in the same state. This can be achieved by calculating the loss function between the two policy distributions, such as using KL divergence or other measurement methods.

[0082] Using this loss function as the training objective, the agent's policy network is updated by minimizing the policy consistency loss. In this way, the agent's policy network will be adjusted so that the policy distribution in the original state and the augmented state is more consistent.

[0083] That is, in this embodiment, the difference between the policy distribution of the original state and the enhanced state is used as the training target, and the policy network of the agent is adjusted by minimizing the loss to achieve consistent decision-making and actions in different states.

[0084] Step 500, obtaining the original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state through the dynamics model, and updating the representation network and dynamics model of the agent based on the dynamics loss.

[0085] Specifically, in this embodiment, the sum of the dynamic losses calculated by the original state and the enhanced state is used by the dynamic model to update the parameters of the representation network and the dynamic model to minimize the sum of the dynamic losses. When minimizing the sum of the losses, an optimization method such as gradient descent can be used to update the parameters. The updated representation network is applied to the decision-making process of the intelligent agent to improve the decision-making performance of the intelligent agent.

[0086] In this embodiment, the value function is updated by using the value consistency loss, and the original state and the enhanced state are trained with the saliency guidance information training at the same time. The original state and the enhanced state are used to train the value network and the representation network at the same time, and the saliency map is used to guide the saliency of the two states respectively. In this way, the intelligent agent pays attention to the accurate and consistent task-related areas; the policy network is trained by using the update method of the policy consistency loss, and the KL divergence between the policy distributions made by the intelligent agent for the original state and the enhanced state is limited while the policy is updated. In this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information; for the original state and the enhanced state, the dynamic loss is used as an auxiliary task to train the representation network and the dynamic model, so that the representations generated by the intelligent agent for the original state and the enhanced state conform to the state transition of the environment. In this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information, thereby improving the visual generalization ability of the intelligent agent.

[0087] Furthermore, during the iterative training of the agent, the above steps 100 to 500 are repeatedly executed until the preset number of iterations is reached, the training is terminated, and the last updated agent network parameters are obtained as the training results. For example, the preset training length unit is time step, and the training stops after sampling 500,000 time steps. After the training is completed, the agent network parameters are retained, and the agent's policy network is used to select actions for the agent in subsequent executions.

[0088] In addition, in the testing and training of the intelligent agent, the control target image data with a pure background can be used for intelligent agent training and learning, and the control target image data with a video background can be used for intelligent agent testing, which improves the visual generalization ability of the intelligent agent.

[0089] For example, reference Figure 2 As shown in Figure 2, the agent learns walking movements in a pure simulation space and tests walking control in a visually changing environment with a video background. Figure 3 As shown in the figure, the agent performs inverted pendulum balancing in a pure simulation space and performs inverted pendulum balancing control tests in a visually changing environment with a video background. Figure 4 As shown, the agent learns to rotate a cuboid in a pure simulation space and performs a rotating cuboid control test in a visually changing environment with a video background.

[0090] In some embodiments, adding saliency guidance information to the original state and the enhanced state in the playback pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the playback pool and the enhanced state, comprises:

[0091] adding saliency guidance information to the original state and the enhanced state in the playback pool respectively, to obtain a first saliency map corresponding to the original state in the playback pool and a second saliency map corresponding to the enhanced state;

[0092] Obtaining a first predicted value of the agent according to the original state of the agent at the current moment in the playback pool and the action of the agent at the current moment;

[0093] Obtaining a target value of the agent according to the environment reward of the agent at the current moment, the original state of the agent at the next moment, and the action of the agent at the next moment in the replay pool;

[0094] Obtaining the original state value loss of the agent according to the first predicted value and the target value of the agent;

[0095] Obtaining a second predicted value of the agent according to the enhanced state of the agent at the current moment and the action of the agent at the current moment in the playback pool;

[0096] Obtaining an enhanced state value loss of the agent according to the first predicted value and the target value of the agent;

[0097] Obtaining a third predicted value of the agent according to the first saliency map of the agent at the current moment and the action of the agent at the current moment in the playback pool;

[0098] Obtaining an original state saliency-guided consistency loss of the agent according to the first prediction value and the third prediction value of the agent;

[0099] Obtaining a fourth predicted value of the agent according to the second saliency map of the agent at the current moment and the action of the agent at the current moment in the playback pool;

[0100] Obtaining an enhanced state saliency guided consistency loss of the agent according to the second prediction value and the fourth prediction value of the agent;

[0101] The value consistency loss of the agent is obtained according to the original state value loss, the enhanced state value loss, the original state saliency-guided consistency loss, the enhanced state saliency-guided consistency loss and the value consistency coefficient of the agent.

[0102] refer to Figure 5 In this embodiment, the value consistency loss of the intelligent agent consists of four parts: the original state value loss L Q1 , Enhanced state value loss L Q2 , the original state saliency guides the consistency loss L QC1 , Enhanced state saliency guides consistency loss L QC2 .

[0103] Specifically, the agent’s value consistency loss L Q =L Q1 +L Q2 +λ(L QC1 +L QC2 ), where λ is the value consistency coefficient.

[0104] In this embodiment, the original state s and enhanced state s of the agent at the current moment in the playback pool are a Add the saliency guidance information respectively to obtain the first saliency map s corresponding to the original state s + (i.e. Figure 5 In ) and enhanced state s a The corresponding second saliency map s a + (i.e. Figure 5 In ).

[0105] The original state s of the agent at the current moment is input into the representation network f to obtain the representation f(s) of the state. Then the representation f(s) of the state and the action a of the agent at the current moment are input into the value network Q to obtain the first predicted value Q(f(s),a) of the agent (i.e. Figure 5 Q s ).

[0106] According to the current agent's environmental reward r in the replay pool, the original state s of the agent at the next moment t+1 and the next action a of the agent t+1 , get the target value y of the agent t .

[0107] The enhanced state s of the agent at the current moment a Input into the representation network f to obtain the representation f(s a ), and then the state representation f(s a ) is input into the value network Q together with the action a of the agent at the current moment, and the second predicted value Q(f(s a ),a)(that is Figure 5 Q sa ).

[0108] Then combine the first predicted value Q(f(s),a) with the target value y of the agent t Calculate the original state value loss L Q1 :

[0109] Among them, E s,a is the expected value corresponding to the original state s.

[0110] Combined with the second prediction value Q(f(s a ),a) and the target value y of the agent t Calculate the original state value loss L Q2 :

[0111] in, For enhanced status a The corresponding expected value.

[0112] Furthermore, in this embodiment, the target value y t It is calculated in the following way:

[0113] y t = r + γQ (s t+1 ,a t+1 );

[0114] Among them, r is the environmental reward, Q(s t+1 ,a t+1 ) is the value calculated based on the original state of the agent at the next moment and the action of the agent at the next moment, and γ is the coefficient.

[0115] The first saliency map s of the agent at the current moment + (i.e. Figure 5 In ) is input into the representation network f to obtain the representation f(s + ), and then the state representation f(s + ) is input into the value network Q together with the action a of the agent at the current moment, and the output agent's third predicted value Q(f(s + ),a)(that is Figure 5 In ).

[0116] The second saliency map s of the agent at the current moment a + (i.e. Figure 5 In ) is input into the representation network f to obtain the representation f(s a + ), and then the state representation f(s a + ) is input into the value network Q together with the action a of the agent at the current moment, and the fourth predicted value Q(f(s a + ),a)(that is Figure 5 In ).

[0117] Combining the first prediction value Q(f(s),a) with the third prediction value Q(f(s + ),a) Calculate the original state saliency guided consistency loss of the agent:

[0118] in, is the first saliency map s + The corresponding expected value.

[0119] Combined with the second prediction value Q(f(s a ),a) and the fourth prediction value Q(f(s a + ),a) Calculate the original state saliency guided consistency loss of the agent:

[0120] in, is the first saliency map s a+ The corresponding expected value.

[0121] Further, in some embodiments, adding saliency guidance information to the original state and the enhanced state in the playback pool respectively to obtain a first saliency map corresponding to the original state in the playback pool and a second saliency map corresponding to the enhanced state includes:

[0122] Obtaining a first saliency gradient map corresponding to an original state in the playback pool;

[0123] The pixel values ​​in the first saliency gradient map that are equal to or higher than a preset threshold are set to 1, and the pixel values ​​that are lower than the preset threshold are set to 0, to obtain a first saliency attribute map of the significant area, and the first saliency attribute map is used as saliency guidance information of the original state in the playback pool to obtain a first saliency map corresponding to the original state in the playback pool;

[0124] Acquire a second saliency gradient map corresponding to the enhanced state;

[0125] The pixel values ​​in the second saliency gradient map that are equal to or higher than a preset threshold are set to 1, and the pixel values ​​that are less than the preset threshold are set to 0, so as to obtain a second saliency attribute map of the significant area, and the second saliency attribute map is used as the saliency guidance information of the enhanced state to obtain a second saliency map corresponding to the enhanced state.

[0126] In this embodiment, the first saliency gradient map M (f) corresponding to the original state s is first obtained by the following method: 0 Q,s,a):

[0127]

[0128] Set the retention ratio of the saliency attribute map to ρ, and then convert the first saliency gradient map M(f 0 The 1-ρ pixels with higher values ​​in Q, s, a) are set to 1, and the others are set to 0, and the first saliency attribute map M of the saliency region is obtained. ρ (f 0 Q,s,a).

[0129] Finally, by transforming the first saliency attribute map M ρ (f 0 Q,s,a) is dot-multiplied with the original state s to get the first saliency map s + :

[0130] s + =s·M ρ (f 0 Q,s,a);

[0131] Similarly, the enhanced state s is obtained by the following method a The corresponding second saliency gradient map M(f 0 Q,s a ,a):

[0132]

[0133] Set the retention ratio of the saliency attribute map to ρ, and then convert the second saliency gradient map M(f 0 Q,s a ,a) The 1-ρ pixels with higher median values ​​are set to 1, and the others are set to 0, to obtain the second saliency attribute map M of the saliency region ρ (f 0 Q,s a ,a).

[0134] Finally, by transforming the second saliency attribute map M ρ (f 0 Q,s a ,a) with the original state s a Perform point multiplication to obtain the second saliency map s a + :

[0135] s a + =s a ·M ρ (f 0 Q,s a ,a).

[0136] In this embodiment, by performing value training on the original state and the enhanced state, saliency guidance information training is added, and saliency maps are used to perform saliency guidance on the two states respectively. In this way, the intelligent agent can focus on accurate and consistent task-related areas.

[0137] In some embodiments, obtaining the original state in the playback pool and the strategy consistency loss of the agent corresponding to the enhanced state includes:

[0138] Obtaining the strategy loss corresponding to the original state in the replay pool;

[0139] Obtaining a policy constraint loss between an original state in the replay pool and the enhanced state;

[0140] The strategy consistency loss of the agent is obtained according to the strategy loss, the strategy constraint loss and the strategy consistency coefficient.

[0141] refer to Figure 6 In this embodiment, the strategy consistency loss of the agent consists of two parts: the strategy loss L corresponding to the original stateπo , strategy constraint loss L πc .

[0142] Specifically, the agent’s policy consistency loss L π =L πo +βL πc ; Where β is the strategy consistency coefficient.

[0143] Furthermore, the strategy loss L corresponding to the original state πo Obtained by:

[0144] L πo =E s,a [αlogπ(a|f(s)-Q(f(s),a))]; where π is the policy network.

[0145] Furthermore, the policy constraint loss L πc Obtained by:

[0146] Among them, D KL for KL Divergence.

[0147] In this embodiment, the policy network updates parameters through back propagation, and the parameter optimization can use the Adam optimizer, which is not limited.

[0148] In this embodiment, the policy network is trained by using a policy consistency loss update method, and the KL divergence between the policy distributions made by the agent for the original state and the enhanced state is limited while the policy is updated. In this way, the agent can make stable and consistent decisions for states with different visual information but the same task-related regional information.

[0149] In some embodiments, obtaining the original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state includes:

[0150] Obtaining a first dynamic loss corresponding to an original state in the playback pool through a dynamic model;

[0151] Obtaining a second dynamic loss corresponding to the enhanced state through a dynamic model;

[0152] Obtaining a dynamics loss of the agent according to the first dynamics loss and the second dynamics loss;

[0153] The dynamic model includes a state transition prediction network, a reward prediction network and an agent representation network. The state transition prediction network is used to predict the next state based on the current state and action, and the reward prediction network is used to predict the reward based on the current state and action.

[0154] Specifically, refer to Figure 7 In this embodiment, the dynamics loss of the agent consists of two parts: the dynamics loss corresponding to the original state Dynamical loss corresponding to the enhanced state

[0155] Specifically, the agent's dynamics loss

[0156] Furthermore, the dynamic loss corresponding to the original state is obtained by:

[0157] Among them, P(f(s),a) is the state transition prediction network, which predicts the next state based on the current state and action, and R(f(s),a) is the reward prediction network, which predicts the reward based on the current state and action.

[0158] Furthermore, the dynamic loss corresponding to the original state is obtained by:

[0159] Among them, P(f(s a ), a) is the state transition prediction network, which predicts the next enhancement state according to the current enhancement state and action, R(f(s a ),a) is the reward prediction network, which predicts the reward based on the current state and action.

[0160] In this embodiment, dynamics is used as an auxiliary task to train the representation network and dynamics model for the original state and the enhanced state, and the parameters of the representation network and the dynamics model are updated to make the representation of the original state and the enhanced state generated by the intelligent agent conform to the state transition of the environment.

[0161] In some embodiments, it also includes:

[0162] Initialize the agent's network parameters, enhancement function, value consistency coefficient and strategy consistency coefficient, wherein the network parameters include value network parameters, strategy network parameters, representation network parameters and dynamic model parameters.

[0163] refer to Figure 8 , Figure 8 The network model of the agent in this embodiment includes a representation network f, a policy network π, a value network Q, a state transition network P and a reward prediction network R. Before the agent is trained and tested, all network parameters are updated.

[0164] The reinforcement learning generalization device for a visually changing environment provided by the present invention is described below. The reinforcement learning generalization device for a visually changing environment described below and the reinforcement learning generalization method for a visually changing environment described above can be referenced to each other.

[0165] The device is applied to an intelligent body that makes decision-making control based on visual input, wherein the state of the intelligent body is the image data of the visual input, and the action of the intelligent body is the control variable of the control target.

[0166] Fig. 9 is a schematic diagram of the structure of the reinforcement learning generalization device for visual change environment provided by the present invention, such as Fig. 9 As shown, the device comprises:

[0167] The data collection module 910 is used to associate the original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment and store them in a replay pool;

[0168] A state enhancement module 920, configured to enhance the original state in the playback pool using an enhancement function to obtain an enhanced state corresponding to the original state in the playback pool;

[0169] A value network updating module 930 is used to add saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtain value consistency loss in combination with the data in the replay pool and the enhanced state, and update the value network and representation network of the agent based on the value consistency loss;

[0170] A policy network updating module 940 is used to obtain the original state in the playback pool and the policy consistency loss of the agent corresponding to the enhanced state, and update the policy network of the agent based on the policy consistency loss;

[0171] The representation network update module 950 is used to obtain the original state in the playback pool and the dynamics loss of the intelligent agent corresponding to the enhanced state through the dynamics model, and update the representation network and dynamics model of the intelligent agent based on the dynamics loss.

[0172] The reinforcement learning generalization device for visual change environment provided by the present invention updates the value function by using value consistency loss, adds saliency guidance information training while performing value training for the original state and the enhanced state, uses the original state and the enhanced state to train the value network and the representation network at the same time, and uses the saliency map to perform saliency guidance for the two states respectively. In this way, the intelligent agent pays attention to accurate and consistent task-related areas; the policy network is trained by using the update method of policy consistency loss, and the KL divergence between the policy distributions made by the intelligent agent for the original state and the enhanced state is limited while the policy is updated. In this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information; for the original state and the enhanced state, the dynamic loss is used as an auxiliary task to train the representation network and the dynamic model, so that the representation generated by the intelligent agent for the original state and the enhanced state conforms to the state transition of the environment, and in this way, the intelligent agent makes stable and consistent decisions for states with different visual information but the same task-related regional information, thereby improving the visual generalization ability of the intelligent agent.

[0173] It should be noted that the reinforcement learning generalization device for a visually changing environment provided by the present invention can implement the various embodiments of the reinforcement learning generalization method for a visually changing environment described above and achieve the same technical effects, which will not be described in detail herein.

[0174] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the reinforcement learning generalization method for the visual change environment, including:

[0175] The original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment are associated and stored in the replay pool;

[0176] Using an enhancement function to enhance the original state in the playback pool, to obtain an enhanced state corresponding to the original state in the playback pool;

[0177] Adding saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the replay pool and the enhanced state, and updating the value network and representation network of the agent based on the value consistency loss;

[0178] Obtaining the original state in the replay pool and the policy consistency loss of the agent corresponding to the enhanced state, and updating the policy network of the agent based on the policy consistency loss;

[0179] The original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state are obtained through the dynamics model, and the representation network and the dynamics model of the agent are updated based on the dynamics loss.

[0180] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0181] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the reinforcement learning generalization method for a visual change environment provided by the above methods, including:

[0182] The original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment are associated and stored in the replay pool;

[0183] Using an enhancement function to enhance the original state in the playback pool, to obtain an enhanced state corresponding to the original state in the playback pool;

[0184] Adding saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the replay pool and the enhanced state, and updating the value network and representation network of the agent based on the value consistency loss;

[0185] Obtaining the original state in the replay pool and the policy consistency loss of the agent corresponding to the enhanced state, and updating the policy network of the agent based on the policy consistency loss;

[0186] The original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state are obtained through the dynamics model, and the representation network and the dynamics model of the agent are updated based on the dynamics loss.

[0187] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the reinforcement learning generalization method for a visually changing environment provided by the above methods, including:

[0188] The original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment are associated and stored in the replay pool;

[0189] Using an enhancement function to enhance the original state in the playback pool, to obtain an enhanced state corresponding to the original state in the playback pool;

[0190] Adding saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the replay pool and the enhanced state, and updating the value network and representation network of the agent based on the value consistency loss;

[0191] Obtaining the original state in the replay pool and the policy consistency loss of the agent corresponding to the enhanced state, and updating the policy network of the agent based on the policy consistency loss;

[0192] The original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state are obtained through the dynamics model, and the representation network and the dynamics model of the agent are updated based on the dynamics loss.

[0193] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning generalization method for visually changing environments, applied to intelligent agents that make decisions based on visual input, where: The state of the agent is the image data of the visual input, the image data describes the task that the agent is concerned about, and the action of the agent is the control variable of the control target, which is characterized by including: The original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment are associated and stored in the playback pool; wherein, in the agent walking control task, the action is the joint control variable of walking, and the environmental reward includes a health reward and a forward walking reward; or, in the agent inverted pendulum control task, the action is the joint control variable of the inverted pendulum, and the environmental reward is the number of time steps in the inverted state; or, in the agent rotation control task in a rectangular environment, the action is the joint control variable of the finger, and the environmental reward is the distance between the finger and the target point; Using an enhancement function to enhance the original state in the playback pool, to obtain an enhanced state corresponding to the original state in the playback pool; Adding saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the replay pool and the enhanced state, and updating the value network and representation network of the agent based on the value consistency loss; the saliency guidance information is information related to the task that the agent is concerned about; Obtaining the policy consistency loss of the agent corresponding to the original state and the enhanced state in the replay pool, and updating the policy network of the agent based on the policy consistency loss; the policy consistency loss is obtained based on the KL divergence between the policy distribution in the original state and the policy distribution in the enhanced state; Obtaining the original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state through a dynamics model, and updating the representation network and the dynamics model of the agent based on the dynamics loss; The obtaining the original state in the playback pool and the dynamics loss of the intelligent agent corresponding to the enhanced state through the dynamics model includes: Obtaining a first dynamic loss corresponding to an original state in the playback pool through a dynamic model; Obtaining a second dynamic loss corresponding to the enhanced state through a dynamic model; Obtaining a dynamics loss of the agent according to the first dynamics loss and the second dynamics loss; The dynamic model includes a state transition prediction network, a reward prediction network and an agent representation network. The state transition prediction network is used to predict the next state based on the current state and action, and the reward prediction network is used to predict the reward based on the current state and action.

2. The reinforcement learning generalization method for visually changing environments according to claim 1, characterized in that: The adding saliency guidance information to the original state and the enhanced state in the replay pool, respectively, and obtaining the value consistency loss of the agent in combination with the data in the replay pool and the enhanced state, comprises: adding saliency guidance information to the original state and the enhanced state in the playback pool respectively, to obtain a first saliency map corresponding to the original state in the playback pool and a second saliency map corresponding to the enhanced state; Obtaining a first predicted value of the agent according to the original state of the agent at the current moment in the playback pool and the action of the agent at the current moment; Obtaining a target value of the agent according to the environment reward of the agent at the current moment, the original state of the agent at the next moment, and the action of the agent at the next moment in the replay pool; Obtaining the original state value loss of the agent according to the first predicted value and the target value of the agent; Obtaining a second predicted value of the agent according to the enhanced state of the agent at the current moment and the action of the agent at the current moment in the playback pool; Obtaining an enhanced state value loss of the agent according to the first predicted value and the target value of the agent; Obtaining a third predicted value of the agent according to the first saliency map of the agent at the current moment and the action of the agent at the current moment in the playback pool; Obtaining an original state saliency-guided consistency loss of the agent according to the first prediction value and the third prediction value of the agent; Obtaining a fourth predicted value of the agent according to the second saliency map of the agent at the current moment and the action of the agent at the current moment in the playback pool; Obtaining an enhanced state saliency guided consistency loss of the agent according to the second prediction value and the fourth prediction value of the agent; The value consistency loss of the agent is obtained according to the original state value loss, the enhanced state value loss, the original state saliency-guided consistency loss, the enhanced state saliency-guided consistency loss and the value consistency coefficient of the agent.

3. The reinforcement learning generalization method for visually changing environments according to claim 2, characterized in that: The adding saliency guidance information to the original state and the enhanced state in the playback pool respectively to obtain a first saliency map corresponding to the original state in the playback pool and a second saliency map corresponding to the enhanced state includes: Obtaining a first saliency gradient map corresponding to an original state in the playback pool; The pixel values ​​in the first saliency gradient map that are equal to or higher than a preset threshold are set to 1, and the pixel values ​​that are lower than the preset threshold are set to 0, to obtain a first saliency attribute map of the significant area, and the first saliency attribute map is used as saliency guidance information of the original state in the playback pool to obtain a first saliency map corresponding to the original state in the playback pool; Acquire a second saliency gradient map corresponding to the enhanced state; The pixel values ​​in the second saliency gradient map that are equal to or higher than a preset threshold are set to 1, and the pixel values ​​that are less than the preset threshold are set to 0, so as to obtain a second saliency attribute map of the significant area, and the second saliency attribute map is used as the saliency guidance information of the enhanced state to obtain a second saliency map corresponding to the enhanced state.

4. The reinforcement learning generalization method for visually changing environments according to claim 1, characterized in that: The obtaining of the original state in the playback pool and the strategy consistency loss of the agent corresponding to the enhanced state includes: Obtaining the strategy loss corresponding to the original state in the replay pool; Obtaining a policy constraint loss between an original state in the replay pool and the enhanced state; The strategy consistency loss of the agent is obtained according to the strategy loss, the strategy constraint loss and the strategy consistency coefficient.

5. The reinforcement learning generalization method for visually changing environments according to any one of claims 1 to 4, characterized in that: Also includes: Initialize the agent's network parameters, enhancement function, value consistency coefficient and strategy consistency coefficient, wherein the network parameters include value network parameters, strategy network parameters, representation network parameters and dynamic model parameters.

6. A reinforcement learning generalization device for visually changing environments, applied to an intelligent agent that makes decisions based on visual input, wherein: The state of the agent is the image data of the visual input, the image data describes the task that the agent is concerned about, and the action of the agent is the control variable of the control target, which is characterized by including: A data collection module is used to associate and store the original state of the agent at the current moment, the action of the agent at the current moment, the environmental reward of the agent at the current moment, and the original state of the agent at the next moment in a replay pool; wherein, in the agent walking control task, the action is the joint control variable of walking, and the environmental reward includes a health reward and a forward walking reward; or, in the agent inverted pendulum control task, the action is the joint control variable of the inverted pendulum, and the environmental reward is the number of time steps in the inverted state; or, in the agent rotation control task in a rectangular environment, the action is the joint control variable of the finger, and the environmental reward is the distance between the finger and the target point; A state enhancement module, used to enhance the original state in the playback pool using an enhancement function to obtain an enhanced state corresponding to the original state in the playback pool; a value network updating module, configured to add saliency guidance information to the original state in the replay pool and the enhanced state respectively, and obtain the value consistency loss of the agent in combination with the data in the replay pool and the enhanced state, and update the value network and representation network of the agent based on the value consistency loss; the saliency guidance information is information related to the task that the agent is concerned about; A policy network update module, used to obtain the policy consistency loss of the agent corresponding to the original state and the enhanced state in the replay pool, and update the policy network of the agent based on the policy consistency loss; the policy consistency loss is obtained based on the KL divergence between the policy distribution in the original state and the policy distribution in the enhanced state; A representation network update module, used to obtain the original state in the playback pool and the dynamics loss of the agent corresponding to the enhanced state through a dynamics model, and update the representation network and the dynamics model of the agent based on the dynamics loss; The obtaining the original state in the playback pool and the dynamics loss of the intelligent agent corresponding to the enhanced state through the dynamics model includes: Obtaining a first dynamic loss corresponding to an original state in the playback pool through a dynamic model; Obtaining a second dynamic loss corresponding to the enhanced state through a dynamic model; Obtaining a dynamics loss of the agent according to the first dynamics loss and the second dynamics loss; The dynamic model includes a state transition prediction network, a reward prediction network and an agent representation network. The state transition prediction network is used to predict the next state based on the current state and action, and the reward prediction network is used to predict the reward based on the current state and action.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the reinforcement learning generalization method for a visually changing environment as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the reinforcement learning generalization method for a visually changing environment as claimed in any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the reinforcement learning generalization method for a visually changing environment as claimed in any one of claims 1 to 5 are implemented.

Images