A reinforcement learning method for learning intrinsic rewards based on state semantic representation
By constructing and fusion of intrinsic reward values for visual state image description and semantic information, the sample efficiency problem of reinforcement learning in the case of sparse rewards is solved, and more effective strategy optimization and performance improvement are achieved.
Patent Information
- Application Number
- CN202411366573.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-29
AI Technical Summary
In the case of sparse rewards, it is difficult for reinforcement learning algorithms to obtain effective samples for strategy optimization, and it is easy to ignore the semantic information representation of natural language descriptions.
By extracting the visual state image description of the reinforcement learning algorithm interacting with the environment, an intrinsic reward value based on image information and semantic information is constructed and fused to obtain the final intrinsic reward value in the current step, and finally, a reinforcement learning strategy training is carried out based on the environmental feedback value. .
It realizes more efficient estimation of the current state value in a sparse reward environment, improves the performance and efficiency of reinforcement learning, and ensures the effectiveness of strategy optimization.
Smart Images

Figure CN118886476B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of deep learning and machine learning, and in particular to a reinforcement learning method based on state semantic representation learning intrinsic rewards. Background Art
[0002] Since the advent of AlphaGo, reinforcement learning has become an effective means of training intelligent agents and has achieved great development and application in the fields of games, e-commerce recommendations, resource scheduling, etc. However, the trial-and-error mechanism of reinforcement learning leads to the huge problem of sample efficiency. Especially when the rewards are sparse, it is difficult for reinforcement learning algorithms to obtain effective samples for strategy optimization. For the high-dimensional data representation of visual state, existing methods often rely on neural network to extract high-level information of visual descriptions and calculate intrinsic rewards based on this information, ignoring the semantic information representation of natural language descriptions, which can often extract state information from the semantic level and often reflect the attributes of the task. This method combines visual and language representations to calculate and integrate intrinsic rewards, which can overcome the above problems and provide more effective intrinsic rewards for reinforcement learning, especially reinforcement learning under sparse rewards. Summary of the invention
[0003] The purpose of the present invention is to provide a reinforcement learning method based on state semantic representation to learn intrinsic rewards, so as to solve the problems in the prior art that when the rewards are sparse, the reinforcement learning algorithm is difficult to obtain effective samples for strategy optimization and is prone to ignore the semantic information representation of natural language descriptions.
[0004] To achieve the above object, the present invention provides a reinforcement learning method based on state semantic representation learning intrinsic reward, and the specific steps are as follows:
[0005] Step S1, extracting the visual state image description of the current step of the interaction between the reinforcement learning algorithm and the environment;
[0006] Step S2, constructing an intrinsic reward value 1 based on image information based on the visual state image description;
[0007] Step S3, converting the visual state image description in step S1 into a visual state image text representation using a pre-training method, and constructing an intrinsic reward value 2 based on semantic information based on the visual state image text representation;
[0008] Step S4, fusing the intrinsic reward value 1 and the intrinsic reward value 2 to obtain the final intrinsic reward value of the current step;
[0009] Step S5: The final intrinsic reward value of the current step is combined with the environmental feedback value to perform reinforcement learning strategy training to obtain the final reinforcement learning strategy.
[0010] Preferably, the specific process of step S2 is as follows:
[0011] Step S2.1, using the ResNet neural network to construct a target ResNet neural network for image description;
[0012] Step S2.2, using the same ResNet neural network as step S2.1 to construct a learnable ResNet neural network for image description;
[0013] Step S2.3, the difference in state representation between the target ResNet neural network and the learnable ResNet neural network is taken as the intrinsic reward value 1, and the calculation expression is as follows:
[0014] ;
[0015] In the formula, represents an intrinsic reward value of 1, and They are the target ResNet neural network and the learnable ResNet neural network, It is the description of the current step visual state. is a hyperparameter with a value range of , the initial value is 1, and it decreases linearly to 0 with reinforcement learning training. Representation vector of 2The square of the norm.
[0016] Preferably, the target ResNet neural network in step S2.1 uses network parameters pre-trained by ImageNet and remains fixed during the reinforcement learning process.
[0017] Preferably, the specific process of step S3 is as follows:
[0018] Step S3.1, using the pre-trained ClipCap model to describe the state in text form;
[0019] Step S3.2, using a pre-trained Bert model to vectorize and encode the above text description;
[0020] Step S3.3, repeating steps S3.1 and S3.2 to vectorize the next state after the reinforcement learning action is performed and the previous state of the state;
[0021] Step S3.4, the difference between the current state vector code and the previous state and the next state vector code is taken as the intrinsic reward value 2, and the calculation expression is as follows:
[0022] ;
[0023] in, represents the intrinsic reward value 2, Describes the current step visual state The vectorized coding expression obtained after ClipCap model and Bert model, Describes the visual state of the previous step The vectorized coding expression obtained after ClipCap model and Bert model, Describe the next visual state The vectorized coding expression obtained after ClipCap model and Bert model, Represents a vector All elements in are processed by absolute value. Represents vector With vector The maximum value in the same position, Representation vector of 1 norm.
[0024] Preferably, the pre-trained ClipCap model and the Bert model in step S3.1 and step S3.2 do not update parameters during the reinforcement learning process.
[0025] Preferably, in step S4, the intrinsic reward value 1 and the intrinsic reward value 2 are merged to obtain the final intrinsic reward value of the current step by specifically adopting a pooling operation method, and the calculation method is as follows:
[0026] ;
[0027] In the formula, represents the final intrinsic reward value of the current step, For a reinforcement learning training batch data The maximum value of For a reinforcement learning training batch data The maximum value of .
[0028] Preferably, in step S5, the value method or the policy gradient method is used to perform reinforcement learning strategy training, and the final intrinsic reward value of the current step in the training process is calculated in combination with the environmental feedback value as follows:
[0029] ;
[0030] in, is the environmental feedback value, is a hyperparameter whose value decreases linearly over time.
[0031] Preferably, during the reinforcement learning strategy training in step S5, the parameters of the learnable ResNet neural network in step S2.2 are updated at fixed time steps to update the objective function for:
[0032] ;
[0033] In the formula, To learn the parameters of the ResNet neural network, is the set of sampled states, when When sampling the same state , The value of is 1, when , sampling different states, The value of is 0. Indicates a positive value or 0. and is a hyperparameter with values of 0.2 and 0.9 respectively.
[0034] Therefore, the present invention adopts the above-mentioned reinforcement learning method based on state semantic representation learning intrinsic reward, considers the intrinsic reward under state image description and state image text representation, realizes effective estimation of current state value or novelty, and improves the performance and efficiency of reinforcement learning to solve sparse rewards in the environment.
[0035] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is an architecture diagram of a reinforcement learning method for learning intrinsic rewards based on state semantic representation of the present invention. DETAILED DESCRIPTION
[0037] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] See also Figure 1 , a reinforcement learning method based on state semantic representation to learn intrinsic rewards, the specific steps are as follows:
[0039] Step S1, extracting the visual state image description of the current step of the interaction between the reinforcement learning algorithm and the environment;
[0040] Step S2: construct an intrinsic reward value 1 based on image information based on the visual state image description; the specific process is as follows:
[0041] Step S2.1, using the ResNet neural network to construct a target ResNet neural network for image description; wherein the target ResNet neural network uses network parameters pre-trained by ImageNet and remains fixed during the reinforcement learning process;
[0042] Step S2.2, using the same ResNet neural network as step S2.1 to construct a learnable ResNet neural network for image description;
[0043] Step S2.3, the difference in state representation between the target ResNet neural network and the learnable ResNet neural network is taken as the intrinsic reward value 1, and the calculation expression is as follows:
[0044] ;
[0045] In the formula, represents an intrinsic reward value of 1, and They are the target ResNet neural network and the learnable ResNet neural network, It is the description of the current step visual state. is a hyperparameter with a value range of , the initial value is 1, and it decreases linearly to 0 with reinforcement learning training. Representation vector of 2The square of the norm.
[0046] Step S3, converting the visual state image description in step S1 into a visual state image text representation using a pre-training method, and constructing an intrinsic reward value 2 based on semantic information based on the visual state image text representation; the specific process is as follows:
[0047] Step S3.1, using the pre-trained ClipCap model to describe the state in text form;
[0048] Step S3.2, using a pre-trained Bert model to vectorize and encode the above text description;
[0049] Step S3.3, repeating steps S3.1 and S3.2 to vectorize the next state after the reinforcement learning action is performed and the previous state of the state;
[0050] Step S3.4, the difference between the current state vector code and the previous state and the next state vector code is taken as the intrinsic reward value 2, and the calculation expression is as follows:
[0051] ;
[0052] in, represents the intrinsic reward value 2, Describes the current step visual state The vectorized coding expression obtained after ClipCap model and Bert model, Describes the visual state of the previous step The vectorized coding expression obtained after ClipCap model and Bert model, Describe the next visual state The vectorized coding expression obtained after ClipCap model and Bert model, Represents a vector All elements in are processed by absolute value. Represents vector With vector The maximum value in the same position, Representation vector of 1 norm.
[0053] Among them, the pre-trained ClipCap model and Bert model in step S3.1 and step S3.2 do not update parameters during the reinforcement learning process.
[0054] Step S4: Intrinsic reward value 1 and intrinsic reward value 2 are merged to obtain the final intrinsic reward value of the current step. Specifically, a pooling operation is adopted, and the calculation method is as follows:
[0055] ;
[0056] In the formula, represents the final intrinsic reward value of the current step, For a reinforcement learning training batch data The maximum value of For a reinforcement learning training batch data The maximum value of .
[0057] Step S5, the final intrinsic reward value of the current step is combined with the environmental feedback value to perform reinforcement learning strategy training to obtain the final reinforcement learning strategy, wherein the reinforcement learning strategy training is performed using the value method or the policy gradient method. The calculation method of the final intrinsic reward value of the current step combined with the environmental feedback value during the training process is:
[0058] ;
[0059] in, is the environmental feedback value, is a hyperparameter whose value decreases linearly over time.
[0060] In addition, during the training of the reinforcement learning strategy, the parameters of the learnable ResNet neural network in step S2.2 are updated at fixed time steps to update the objective function for:
[0061] ;
[0062] In the formula, To learn the parameters of the ResNet neural network, is the set of sampled states, when When sampling the same state , The value of is 1, when , sampling different states, The value of is 0. Indicates a positive value or 0. and is a hyperparameter with values of 0.2 and 0.9 respectively.
[0063] Example
[0064] The following is a detailed description of the specific implementation method using the CLEVER-Robot environment as the implementation environment. CLEVER-Robot is a classic environment for verifying the ability of reinforcement learning agents to manipulate objects in a 3D environment. The input of the interaction between the agent and the environment is an image state description, and the output is a 40-dimensional discrete vector, representing the probability of a certain color ball moving in 8 directions in the environment. The reinforcement learning algorithm of this embodiment selects the PPO algorithm.
[0065] Step S1: Extract the visual state image description of the current step of the interaction between the PPO algorithm and the CLEVER-Robot environment ;
[0066] Step S2.1, using the ResNet neural network to construct a target ResNet neural network for image description; wherein the target ResNet neural network uses the network parameters pre-trained by ImageNet and remains fixed during the reinforcement learning process;
[0067] Step S2.2, using the same ResNet neural network as step S2.1 to construct a learnable ResNet neural network for image description;
[0068] Step S2.3, the difference in state representation between the target ResNet neural network and the learnable ResNet neural network is taken as the intrinsic reward value 1, and the calculation expression is as follows:
[0069] ;
[0070] In the formula, represents an intrinsic reward value of 1, and They are the target ResNet neural network and the learnable ResNet neural network, It is the description of the current step visual state. is a hyperparameter with a value range of , the initial value is 1, and it decreases linearly to 0 with reinforcement learning training. Representation vector of 2The square of the norm.
[0071] Step S3.1: Use the pre-trained ClipCap model to classify the state Provide textual description;
[0072] Step S3.2, using a pre-trained Bert model to vectorize and encode the above text description;
[0073] Step S3.3: Repeat steps S3.1 and S3.2 to perform the next state after reinforcement learning The previous state Vectorized encoding is performed, and the action selection is obtained based on PPO strategy network sampling;
[0074] Step S3.4, the difference between the current state vector code and the previous state and the next state vector code is taken as the intrinsic reward value 2, and the calculation expression is as follows:
[0075] ;
[0076] in, represents the intrinsic reward value 2, Describes the current step visual state The vectorized coding expression obtained after ClipCap model and Bert model, Describes the visual state of the previous step The vectorized coding expression obtained after ClipCap model and Bert model, Describe the next visual state The vectorized coding expression obtained after ClipCap model and Bert model, Represents a vector All elements in are processed by absolute value. Represents vector With vector The maximum value in the same position, Representation vector of 1 norm.
[0077] Step S4: Intrinsic reward value 1 and intrinsic reward value 2 are merged to obtain the final intrinsic reward value of the current step. Specifically, a pooling operation is adopted, and the calculation method is as follows:
[0078] ;
[0079] In the formula, Represents the final intrinsic reward value of the current step. The PPO algorithm updates the strategy every time it collects data. Collecting data for this iteration of PPO The maximum value of Collecting data for this iteration of PPO The maximum value of .
[0080] Step S5, the final intrinsic reward value of the current step is combined with the environmental feedback value to perform reinforcement learning strategy training to obtain the final reinforcement learning strategy. The training process of the PPO algorithm adopts the standard PPO training method, that is, collecting trajectories and updating the PPO strategy network parameters. The calculation method of the final intrinsic reward value of the current step combined with the environmental feedback value during the training process is:
[0081] ;
[0082] in, is the environmental feedback value, It is a hyperparameter, and its value decreases linearly with time. The initial value is 1, and the value decreases every 10 seconds. 6 The number of decision steps is halved. The value is 100 for completing the task and 0 for failing to complete the task.
[0083] During the reinforcement learning strategy training process, the PPO algorithm updates the strategy every time it collects data. Every 5 iterations, the parameters of the learnable ResNet neural network in step S2.2 are updated to update the objective function. for:
[0084] ;
[0085] In the formula, To learn the parameters of the ResNet neural network, is the set of sampled states, when When sampling the same state , The value of is 1, when , sampling different states, The value of is 0. Indicates a positive value or 0. and is a hyperparameter with values of 0.2 and 0.9 respectively.
[0086] Therefore, the present invention adopts the above-mentioned reinforcement learning method based on state semantic representation learning intrinsic rewards, considers the intrinsic rewards under state image description and state image text representation, realizes effective estimation of the current state value or novelty, and improves the performance and efficiency of reinforcement learning to solve environmental sparse rewards.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A reinforcement learning method based on state semantic representation to learn intrinsic rewards, characterized in that: The specific steps are as follows: Step S1, extracting the visual state image description of the current step of the interaction between the reinforcement learning algorithm and the environment; Step S2, constructing an intrinsic reward value 1 based on image information based on the visual state image description; Step S3, converting the visual state image description in step S1 into a visual state image text representation using a pre-training method, and constructing an intrinsic reward value 2 based on semantic information based on the visual state image text representation; Step S4, fusing the intrinsic reward value 1 and the intrinsic reward value 2 to obtain the final intrinsic reward value of the current step; Step S5, the final intrinsic reward value of the current step is combined with the environmental feedback value to perform reinforcement learning strategy training to obtain the final reinforcement learning strategy; The specific process of step S2 is as follows: Step S2.1, using the ResNet neural network to construct a target ResNet neural network for image description; Step S2.2, using the same ResNet neural network as step S2.1 to construct a learnable ResNet neural network for image description; Step S2.3, the difference in state representation between the target ResNet neural network and the learnable ResNet neural network is taken as the intrinsic reward value 1, and the calculation expression is as follows: ; In the formula, represents an intrinsic reward value of 1, and They are the target ResNet neural network and the learnable ResNet neural network, It is the description of the current step visual state. is a hyperparameter with a value range of , the initial value is 1, and it decreases linearly to 0 with reinforcement learning training. Representation vector of The square of the norm; The specific process of step S3 is as follows: Step S3.1, using the pre-trained ClipCap model to describe the state in text form; Step S3.2, using a pre-trained Bert model to vectorize and encode the above text description; Step S3.3, repeating steps S3.1 and S3.2 to vectorize the next state after the reinforcement learning action is performed and the previous state of the state; Step S3.4, the difference between the current state vector code and the previous state and the next state vector code is taken as the intrinsic reward value 2, and the calculation expression is as follows: ; in, represents the intrinsic reward value 2, Describes the current step visual state The vectorized coding expression obtained after ClipCap model and Bert model, Describes the visual state of the previous step The vectorized coding expression obtained after ClipCap model and Bert model, Describe the next visual state The vectorized coding expression obtained after ClipCap model and Bert model, Represents a vector All elements in are processed by absolute value. Represents vector With vector The maximum value in the same position, Representation vector of Norm.
2. A reinforcement learning method based on state semantic representation learning intrinsic reward according to claim 1, characterized in that: The target ResNet neural network in step S2.1 uses the network parameters pre-trained by ImageNet and remains fixed during the reinforcement learning process.
3. A reinforcement learning method based on state semantic representation learning intrinsic reward according to claim 2, characterized in that: The pre-trained ClipCap model and Bert model in step S3.1 and step S3.2 do not update parameters during the reinforcement learning process.
4. A reinforcement learning method based on state semantic representation learning intrinsic reward according to claim 3, characterized in that: In step S4, the intrinsic reward value 1 and the intrinsic reward value 2 are merged to obtain the final intrinsic reward value of the current step. The specific pooling operation method is as follows: ; In the formula, represents the final intrinsic reward value of the current step, For a reinforcement learning training batch data The maximum value of For a reinforcement learning training batch data The maximum value of .
5. The reinforcement learning method based on state semantic representation learning intrinsic reward according to claim 4, characterized in that: In step S5, the value method or the policy gradient method is used to perform reinforcement learning strategy training. The final intrinsic reward value of the current step in the training process is calculated in combination with the environmental feedback value as follows: ; in, is the environmental feedback value, is a hyperparameter whose value decreases linearly over time.
6. A reinforcement learning method based on state semantic representation learning intrinsic reward according to claim 5, characterized in that: During the reinforcement learning strategy training in step S5, the parameters of the learnable ResNet neural network in step S2.2 are updated at fixed time steps, and the updated objective function is: ; In the formula, To learn the parameters of the ResNet neural network, is the set of sampled states, when When sampling the same state , The value of is 1, when , sampling different states, The value of is 0. Indicates a positive value or 0. and is a hyperparameter with values of 0.2 and 0.9 respectively.
Citation Information
Patent Citations
Method and system for generating image model by optimizing text through artificial feedback reinforcement learning
CN116955972A
Deep reinforcement learning-based captioning with embedding reward
US10467274B1
