An Adaptive Approach to Robot Reward Functions Based on Reinforcement Learning

Abstract

This invention discloses an adaptive method for robot reward functions based on reinforcement learning. It can learn reward values ​​based on the interaction trajectory between the robot and the environment, thereby guiding the reinforcement learning algorithm to optimize the control strategy. This avoids the need for manual design intervention in the reward model and can improve the efficiency of walking control in different scenarios through an adaptive reward model.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, specifically to an adaptive method for robot reward functions based on reinforcement learning. Background Technology

[0002] Currently, robot walking control is mainly based on traditional control technologies. However, traditional control technologies offer limited options for planning robot walking paths, resulting in inflexible path planning and a lack of strategies for handling complex scenarios. With the rapid development of deep learning and reinforcement learning technologies, the powerful feature learning capabilities of deep neural networks can be leveraged to learn a large number of walking control-related features from the robot's interaction data with its external environment. Combined with reinforcement learning to model the robot's walking problem, obstacle avoidance during walking can be achieved. However, the reward problem in the interaction process still requires manual design. Summary of the Invention

[0003] In order to at least overcome the above-mentioned shortcomings in the prior art, the purpose of this application is to provide an adaptive method for robot reward functions based on reinforcement learning.

[0004] This application provides an adaptive method for robot reward functions based on reinforcement learning, including:

[0005] The Actor strategy in the Actor-Critic network is used to control the robot to interact with the external environment and obtain trajectory data, and the trajectory data is stored in the environment buffer pool;

[0006] The reward model and the Actor-Critic network are updated using data from the environmental buffer pool; the reward model is constructed based on the amount of information generated during state transitions when the robot interacts with the external environment.

[0007] The updated Actor-Critic network controls the robot's interaction with the external environment and acquires new trajectory data to update the reward model and the Actor-Critic network.

[0008] In existing technologies, modeling robot walking problems such as obstacle avoidance through reinforcement learning heavily relies on the evaluation scheme of the robot's walking strategy, mainly reflected in the calculation of the reward value when the robot executes the strategy. Currently, the reward value of the robot's execution strategy often needs to be manually designed, which requires a lot of human and material resources to adjust the reward value design scheme. In the implementation of the embodiments of this application, the reward model operates independently of the Actor-Critic network that makes decisions. The Actor-Critic network is essentially a model used to provide strategies, and the purpose of the reward model is to provide guidance for the optimization of the strategy. In actual operation, when the robot interacts with the external environment through the Actor strategy, it generates certain information data, namely trajectory data. The data in the environment buffer pool is updated as the trajectory data is updated. At this time, the reward model can be updated by training the neural network, so that the reward model can learn the reward value based on the interaction trajectory between the robot and the environment, thereby guiding the reinforcement learning algorithm to optimize the control strategy. It is worth noting that the reward model designed in this invention does not require manually designed reward values ​​as signals during supervised learning. It can complete the learning and calculation of rewards using only the information existing in the state transition, and provide guidance when using the reinforcement learning algorithm to control the robot to complete the actual task.

[0009] In one possible implementation, the reward function includes an encoder and a decoder;

[0010] Updating the return model using data from the environmental buffer pool includes:

[0011] The current state and actions in the environment buffer pool are used as the first input data to the encoder. The information in the first input data is compressed to ensure information integrity through the fully connected layer and activation layer of the neural network configured in the reward model. The mean and variance of the multi-dimensional Gaussian distribution are output by the last layer of the encoder as the first output data.

[0012] The second input data is sampled from the first output data using the reparameter method and input into the decoder. The mean and variance of the state at the next time step are output by the last fully connected layer of the decoder as the second output data.

[0013] The decoder and encoder are trained using the next-time state sampled from the environment buffer pool and the second output data.

[0014] In one possible implementation, the information in the first input data is compressed to ensure information integrity through the fully connected layers and activation layers of the neural network configured in the reward model, using the following formula:

[0015]

[0016]

[0017] In the formula, z is the compressed information, KL is the KL divergence, q(z) is the prior probability of the compressed information, s is the state value, a is the action value, p(z|s,a) is the posterior probability of compressing to z through s and a, μ1 is the mean of the encoder output, and σ1 is the variance of the encoder output.

[0018] In one possible implementation, the mean and variance of the next-time state output by the final fully connected layer of the decoder are used as the second output data, calculated using the following formula:

[0019]

[0020] In the formula, s′ represents the state at the next moment. i σ² represents the state sampled from the buffer pool for the next time step, μ² is the mean of the output of the last fully connected layer in the decoder, and σ² is the variance of the output of the last fully connected layer in the decoder.

[0021] In one possible implementation, the optimization function of the reward model is implemented based on the encoder and the decoder, and the optimization function adopts the following formula:

[0022]

[0023] In the formula, μ1 is the mean of the encoder output, σ1 is the variance of the encoder output; μ2 is the mean of the next time step state output by the decoder, and σ2 is the variance of the next time step state output by the decoder.

[0024] In one possible implementation, using the Actor policy in an Actor-Critic network to control the robot's interaction with the external environment to acquire trajectory data, and storing the trajectory data in an environment buffer pool includes:

[0025] The current state of the robot in its real-world environment is transmitted to the Actor-Critic network, which then calculates and outputs the mean and variance of the actions; the mean and variance are multidimensional Gaussian distributions.

[0026] Motion values ​​are sampled from a multidimensional Gaussian distribution and sent to the robot. Relevant parameters are adjusted, and the robot executes the corresponding instructions to enter a new environmental state.

[0027] The current state, current action, and next state are simultaneously input into the reward model to calculate the reward value corresponding to the current state transition.

[0028] The current state, current action, calculated reward value, and the state at the next moment are stored as trajectory data in the environment buffer pool.

[0029] In one possible implementation, the reward value corresponding to the current state transition is calculated using the following formula:

[0030]

[0031] In the formula, s is the state value of the input encoder, a is the action value of the input encoder, μ1 is the mean of the encoder output, σ1 is the variance of the encoder output, z is the value sampled from the encoder output and is the input of the decoder, z = μ1 + σ1, μ2 is the mean of the next time step state of the decoder output, σ2 is the variance of the next time step state of the decoder output, and ∈ is the value sampled from the mean of 0 and the variance of 1.

[0032] In one possible implementation, the Critic network of the Actor-Critic network includes a target Q-function network and at least two current Q-function networks;

[0033] Updating the Actor-Critic network using data from the environmental buffer pool includes:

[0034] When evaluating the action selected in the current state, all current Q-function networks are used for calculation, and the current Q-function network with the smallest current Q-function value is selected from the calculation results to update the Actor network in the Actor-Critic network;

[0035] The current Q-function value from the calculation results is combined with the target Q-function network to form the MSE loss function, which is then used to update the current Q-function network.

[0036] When updating the target Q-function network, the momentum principle is used to update the parameters based on the updated parameters of the current Q-function network and the parameters of the target Q-function network.

[0037] In one possible implementation, the parameters of the updated current Q-function network and the target Q-function network are updated using the momentum principle, as follows:

[0038]

[0039] In the formula, Q θ For the updated target Q-function network parameters, To update the previous target Q-function network parameters, Q δ ε represents the current parameters of the Q-function network, and ε represents the momentum parameter value during momentum update, which is between 0 and 1.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] This invention presents an adaptive method for robot reward functions based on reinforcement learning. It can learn reward values ​​based on the robot's interaction trajectory with the environment, thereby guiding the reinforcement learning algorithm to optimize control strategies. This avoids manual design intervention of the reward model and can improve the efficiency of walking control in different scenarios through an adaptive reward model. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a schematic diagram of the method steps in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of the return model in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the encoder and decoder network structure according to an embodiment of this application;

[0046] Figure 4 This is a diagram of the Actor policy network structure in an embodiment of this application;

[0047] Figure 5 This is a diagram of the Critic network structure in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0049] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0050] Please refer to the following: Figure 1 This is a flowchart illustrating the adaptive method for robot reward function based on reinforcement learning provided in this embodiment of the invention. Further, the adaptive method for robot reward function based on reinforcement learning may specifically include the content described in steps S1-S3.

[0051] S1: Use the Actor strategy in the Actor-Critic network to control the robot to interact with the external environment to obtain trajectory data, and store the trajectory data in the environment buffer pool;

[0052] S2: Update the reward model and the Actor-Critic network using data from the environmental buffer pool; the reward model is constructed based on the information generated during state transitions when the robot interacts with the external environment;

[0053] S3: Control the robot to interact with the external environment and obtain new trajectory data according to the updated Actor-Critic network, and update the reward model and the Actor-Critic network.

[0054] In existing technologies, modeling robot walking problems such as obstacle avoidance through reinforcement learning heavily relies on the evaluation scheme of the robot's walking strategy, mainly reflected in the calculation of the reward value when the robot executes the strategy. Currently, the reward value of the robot's execution strategy often needs to be manually designed, which requires a lot of human and material resources to adjust the reward value design scheme. In the implementation of the embodiments of this application, the reward model operates independently of the Actor-Critic network that makes decisions. The Actor-Critic network is essentially a model used to provide the strategy, and the purpose of the reward model is to optimize the strategy. In actual operation, when the robot interacts with the external environment through the Actor strategy, it generates certain information data, namely trajectory data. The data in the environment buffer pool is updated as the trajectory data is updated. At this time, the reward model can be updated by training the neural network, so that the reward model can learn the reward value based on the interaction trajectory between the robot and the environment, thereby guiding the reinforcement learning algorithm to optimize the control strategy. It is worth noting that the reward model designed in this invention does not require manually designed reward values ​​as signals during supervised learning; it can complete the learning and calculation of rewards using only the information of state transitions.

[0055] In one possible implementation, the reward function includes an encoder and a decoder;

[0056] Updating the return model using data from the environmental buffer pool includes:

[0057] The current state and actions in the environment buffer pool are used as the first input data to the encoder. The information in the first input data is compressed to ensure information integrity through the fully connected layer and activation layer of the neural network configured in the reward model. The mean and variance of the multi-dimensional Gaussian distribution are output by the last layer of the encoder as the first output data.

[0058] The second input data is sampled from the first output data using the reparameter method and input into the decoder. The mean and variance of the state at the next time step are output by the last fully connected layer of the decoder as the second output data.

[0059] The decoder and encoder are trained using the next-time state sampled from the environment buffer pool and the second output data.

[0060] In the implementation of this application, the optimization function of the reward model is a unique design, derived from the information generated during state transitions when the robot interacts with the external environment. Specifically, the optimization function of the reward model learns the information present during environmental state transitions and then guides the optimization of the Actor-Critic model. The reward model uses a neural network to learn the reward information during state transitions, comprising two sequentially configured modules: an encoder and a decoder. Both the encoder and decoder contain fully connected layers and activation layers.

[0061] Specifically, the encoder in the reward model takes the current state and action pair as input and compresses the information contained therein to the maximum extent possible through the fully connected and activation layers of the neural network, ensuring the integrity of the information while compressing it. The last layer of the encoder outputs a mean and variance with multiple dimensions. The decoder in the reward model reconstructs the next state from the information compressed by the encoder. Specifically, the decoder's input is sampled from the multidimensional Gaussian distribution of the encoder output using a reparameterization method. The decoder's final output is also the mean and variance of a multidimensional Gaussian distribution, that is, the mean and variance of the next state output by the last fully connected layer in the decoder. The use of a multidimensional Gaussian distribution increases the model's adaptability to complex environments.

[0062] In one possible implementation, the information in the first input data is compressed to ensure information integrity through the fully connected layers and activation layers of the neural network configured in the reward model, using the following formula:

[0063]

[0064]

[0065] In the formula, z is the compressed information, KL is the KL divergence, q(z) is the prior probability of the compressed information, s is the state value, a is the action value, p(z|s,a) is the posterior probability of compressing to z through s and a, μ1 is the mean of the encoder output, and σ1 is the variance of the encoder output.

[0066] In the implementation of this application embodiment, the reward model requires encoding by the encoder and decoding by the decoder before the reward value during the transition process can be calculated. Therefore, in order to compress the information of the state-action pair, the encoder part of the reward model needs to use a variational inference method to estimate the information between the input and the compression, so the above formula is used to calculate the compressed information.

[0067] In one possible implementation, the mean and variance of the next-time state output by the final fully connected layer of the decoder are used as the second output data, calculated using the following formula:

[0068]

[0069] In the formula, s′ represents the state at the next moment. i σ² represents the state sampled from the buffer pool for the next time step, μ² is the mean of the output of the last fully connected layer in the decoder, and σ² is the variance of the output of the last fully connected layer in the decoder.

[0070] In the implementation of this application embodiment, the decoder part of the reporting model is to decode the information compressed by the encoder to obtain the state of the next time step. For this purpose, the log-maximum likelihood estimation can be used to decode the state of the next time step from the compressed information.

[0071] In one possible implementation, the optimization function of the reward model is implemented based on the encoder and the decoder, and the optimization function adopts the following formula:

[0072]

[0073] In the formula, μ1 is the mean of the encoder output, σ1 is the variance of the encoder output; μ2 is the mean of the next time step state output by the decoder, and σ2 is the variance of the next time step state output by the decoder.

[0074] In the implementation of this application embodiment, the encoder and decoder are combined based on the above-mentioned optimization function to form the optimization function of the above formula.

[0075] In one possible implementation, using the Actor policy in an Actor-Critic network to control the robot's interaction with the external environment to acquire trajectory data, and storing the trajectory data in an environment buffer pool includes:

[0076] The current state of the robot in its real-world environment is transmitted to the Actor-Critic network, which then calculates and outputs the mean and variance of the actions; the mean and variance are multidimensional Gaussian distributions.

[0077] Motion values ​​are sampled from a multidimensional Gaussian distribution and sent to the robot. Relevant parameters are adjusted, and the robot executes the corresponding instructions to enter a new environmental state.

[0078] The current state, current action, and next state are simultaneously input into the reward model to calculate the reward value corresponding to the current state transition.

[0079] The current state, current action, calculated reward value, and the state at the next moment are stored as trajectory data in the environment buffer pool.

[0080] In the implementation of this application embodiment, the robot walking problem in the external environment is constructed as a Markov decision process. During the walking process, the robot observes the external environment state and inputs it into the Actor network. The Actor network returns the action value to the robot. After the robot performs the action, it continues to walk and then enters a new external state. During this transition process, a reward value is obtained. The reward value calculated by the reward model guides the optimization of the Critic network and the Actor network, thereby obtaining an algorithm for controlling robot walking based on deep reinforcement learning.

[0081] In one possible implementation, the reward value corresponding to the current state transition is calculated using the following formula:

[0082]

[0083] In the formula, s is the state value of the input encoder, a is the action value of the input encoder, μ1 is the mean of the encoder output, σ1 is the variance of the encoder output, z is the value sampled from the encoder output and is the input of the decoder, z = μ1 + σ1, μ2 is the mean of the next time step state of the decoder output, σ2 is the variance of the next time step state of the decoder output, and ∈ is the value sampled from the mean of 0 and the variance of 1.

[0084] In the implementation of this application, the optimization function of the reward model is a unique design. This optimization function is derived from the information generated during state transitions when the robot interacts with the external environment. Specifically, the optimization function of the reward model learns the information present during environmental state transitions and then guides the optimization of the Actor-Critic model. The reward model uses a neural network to learn the reward information during state transitions, including two sequentially arranged modules: an encoder and a decoder. Both the encoder and decoder contain fully connected layers and activation layers. The encoder in the reward model takes the current state and action pair as input and maximizes the compression of the information contained therein through the fully connected and activation layers of the neural network, ensuring the integrity of the information while compressing it. The last layer of the encoder outputs a multi-dimensional mean and variance. The decoder in the reward model reconstructs the next-time state from the information compressed by the encoder. Specifically, the decoder's input is sampled from the multidimensional Gaussian distribution of the encoder output using a reparameterization method. The decoder's final output is also the mean and variance of a multidimensional Gaussian distribution, representing the mean and variance of the next-time state output from the final fully connected layer in the decoder. The use of a multidimensional Gaussian distribution increases the model's adaptability to complex environments. Through the combined operation of the encoder and decoder, the reward value of the robot after transitioning to the next time step can be calculated.

[0085] In one possible implementation, the Critic network of the Actor-Critic network includes a target Q-function network and at least two current Q-function networks;

[0086] Updating the Actor-Critic network using data from the environmental buffer pool includes:

[0087] When evaluating the action selected in the current state, all current Q-function networks are used for calculation, and the current Q-function network with the smallest current Q-function value is selected from the calculation results to update the Actor network in the Actor-Critic network;

[0088] The current Q-function value from the calculation results is combined with the target Q-function network to form the MSE loss function, which is then used to update the current Q-function network.

[0089] When updating the target Q-function network, the momentum principle is used to update the parameters based on the updated parameters of the current Q-function network and the parameters of the target Q-function network.

[0090] In the implementation of this application embodiment, the Actor-Critic network includes one Actor policy network and two Critic networks, each comprising a fully connected layer and an activation layer arranged sequentially. The Actor policy network determines the mean and variance of the action in the current state based on the output of the last fully connected layer in the neural network structure. Specifically, the Actor policy network samples the final action value using the mean and variance of a multidimensional Gaussian distribution obtained from the output of its fully connected layers. A neural network model learns the amount of information present in the real-world environmental state transition process as the actual reward value. The essence of this reward model is its ability to capture the hidden information in the dynamic transitions of the real environment. Using this ability as the reward value for the corresponding environmental state transition process, these reward values ​​can reflect the various uncertainties present during state transitions to the greatest extent possible, which is beneficial to Actor-Critic. The Critic network is used for learning. When the Actor policy network outputs the mean and variance of its final fully connected layer, it performs a nonlinear mapping using the tanh function after sampling from a multidimensional Gaussian distribution to ensure that the final action values ​​are within the effective range. The Critic network contains a target Q-function network and two current Q-function networks. When evaluating the action selected in the current state, the current Q-function networks are used for calculation. The network with the smaller Q-function value is selected to update the Actor policy network. The two calculated current Q-function values ​​are combined with the target Q-function network to form the MSE loss function, thereby updating the two current Q-function networks.

[0091] In one possible implementation, the parameters of the updated current Q-function network and the target Q-function network are updated using the momentum principle, as follows:

[0092]

[0093] In the formula, Q θ For the updated target Q-function network parameters, To update the previous target Q-function network parameters, Q δ ε represents the current parameters of the Q-function network, and ε represents the momentum parameter value during momentum update, which is between 0 and 1.

[0094] In the implementation of this application embodiment, the update of the target Q-function network uses the momentum principle, that is, the momentum principle is used based on the updated parameters of the current Q-function network and the parameters of the target Q-function network.

[0095] In one possible implementation, the loss function of the Actor network in the Actor-Critic network consists of two parts. The first part is the evaluation value of the Critic network when the policy chooses a certain action in a certain state, i.e., the state-action value function. The second part is the entropy of the policy choosing that action. The parameter update of the Critic network uses TD-error as the loss function.

[0096] In one possible implementation, please refer to Figure 2 The diagram illustrates the neural network structure of the reward model, which consists of four fully connected layers and three activation functions. Each fully connected layer has 256 hidden layers, and the activation function used is swish. Figure 3 As shown, the encoder and decoder of the reward model each consist of a neural network model. In the reward model, the encoder's input is a randomly given state and the action in that state, and its output is the compressed mean and variance of the information. In the reward model, the decoder's input is the value sampled from the mean and variance of the encoder's output, and its output is the mean and variance of the state at the next time step. Figure 4 As shown, the Actor policy network in this embodiment includes three fully connected layers and a rule layer arranged sequentially. Each fully connected layer contains 256 neurons. The input is the current or given state, and the output is the action to be taken in the face of this state. Figure 5 As shown, the two Critic network structures in this embodiment are completely identical, including three fully connected layers and a rule layer arranged sequentially. Each layer of the fully connected network contains 256 neurons. The input is a given state and its corresponding action, and the input is the evaluation value for this situation.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0099] The units described as separate components may or may not be physically separate. As will be apparent to those skilled in the art, the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0100] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or grid device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive method for robot reward functions based on reinforcement learning, characterized in that, include: The Actor strategy in the Actor-Critic network is used to control the robot to interact with the external environment and obtain trajectory data, and the trajectory data is stored in the environment buffer pool; The reward model and the Actor-Critic network are updated using data from the environmental buffer pool; the reward model is constructed based on the amount of information generated during state transitions when the robot interacts with the external environment. The updated Actor-Critic network controls the robot's interaction with the external environment and acquires new trajectory data to update the reward model and the Actor-Critic network. The reward function includes an encoder and a decoder; Updating the return model using data from the environmental buffer pool includes: The current state and actions in the environment buffer pool are used as the first input data to the encoder. The information in the first input data is compressed to ensure information integrity through the fully connected layer and activation layer of the neural network configured in the reward model. The mean and variance of the multi-dimensional Gaussian distribution are output by the last layer of the encoder as the first output data. The second input data is sampled from the first output data using the reparameter method and input into the decoder. The mean and variance of the state at the next time step are output by the last fully connected layer of the decoder as the second output data. The decoder and the encoder are trained using the next-time state sampled from the environment buffer and the second output data; The information in the first input data is compressed to ensure information integrity by means of the fully connected layers and activation layers of the neural network configured in the reward model, using the following formula: In the formula, z is the compressed information, KL is the KL divergence, q(z) is the prior probability of the compressed information, s is the state value, a is the action value, p(z|s,a) is the posterior probability of compressing to z through s and a, μ1 is the mean of the encoder output, and σ1 is the variance of the encoder output. The mean and variance of the next time-step state output from the final fully connected layer of the decoder are used as the second output data, calculated using the following formula: In the formula, s ′ For the state at the next moment, s ′ i σ² represents the next time step state sampled from the buffer pool, μ² is the mean of the output of the last fully connected layer in the decoder, and σ² is the variance of the output of the last fully connected layer in the decoder. The optimization function of the reward model is implemented based on the encoder and the decoder, and the optimization function adopts the following formula: In the formula, μ1 is the mean of the encoder output, σ1 is the variance of the encoder output; μ2 is the mean of the next time step state output by the decoder, and σ2 is the variance of the next time step state output by the decoder. Using the Actor policy in the Actor-Critic network to control the robot to interact with the external environment and acquire trajectory data, and storing the trajectory data in an environment buffer pool includes: The current state of the robot in its real-world environment is transmitted to the Actor-Critic network, which then calculates and outputs the mean and variance of the actions; the mean and variance are multidimensional Gaussian distributions. Motion values ​​are sampled from a multidimensional Gaussian distribution and sent to the robot. Relevant parameters are adjusted, and the robot executes the corresponding instructions to enter a new environmental state. The current state, current action, and next state are simultaneously input into the reward model to calculate the reward value corresponding to the current state transition. The current state, current action, calculated reward value, and the state at the next moment are stored as trajectory data in the environment buffer pool; The reward value corresponding to the current state transition is calculated using the following formula: In the formula, s is the state value of the input encoder, a is the action value of the input encoder, μ1 is the mean of the encoder output, σ1 is the variance of the encoder output, z is the value sampled from the encoder output and is the input of the decoder, z = μ1 + σ1, μ2 is the mean of the next time step state of the decoder output, σ2 is the variance of the next time step state of the decoder output, and ∈ is the value randomly sampled from the mean of 0 and the variance of 1.

2. The adaptive method for robot reward function based on reinforcement learning according to claim 1, characterized in that, The Actor-Critic network's Critic network includes a target Q-function network and at least two current Q-function networks; Updating the Actor-Critic network using data from the environmental buffer pool includes: When evaluating the action selected in the current state, all current Q-function networks are used for calculation, and the current Q-function network with the smallest current Q-function value is selected from the calculation results to update the Actor network in the Actor-Critic network; The current Q-function value from the calculation results is combined with the target Q-function network to form the MSE loss function, which is then used to update the current Q-function network. When updating the target Q-function network, the momentum principle is used to update the parameters based on the updated parameters of the current Q-function network and the parameters of the target Q-function network.

3. The adaptive method for robot reward function based on reinforcement learning according to claim 2, characterized in that, The parameters of the updated current Q-function network and the target Q-function network are updated using the momentum principle, as shown in the following formula: In the formula, Q θ For the updated target Q-function network parameters, To update the previous target Q-function network parameters, Q δ ε represents the current parameters of the Q-function network, and ε represents the momentum parameter value during momentum update, which is between 0 and 1.

Images