Intelligent agent action control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an agent action control method and device, electronic equipment and a storage medium, when the action of an agent is controlled, a deep reinforcement learning algorithm combining an auxiliary task, experience double sampling and value function dynamic correction is combined, the auxiliary task is introduced, the quality of feature extraction is improved, and the accuracy of feature extraction is improved. The understanding capability of the model on the state is enhanced; meanwhile, double sampling is performed through experience, so that the utilization efficiency of samples is improved, and the learning process is accelerated; value function dynamic correction is helpful for stabilizing the learning process and preventing the model from falling into local optimum.

Description

Technical Field

[0001] The present invention relates to the field of intelligent body motion control technology, and in particular to an intelligent body motion control method, an intelligent body motion control device, an electronic device and a computer-readable storage medium. Background Art

[0002] Reinforcement learning is one of the machine learning methods in artificial intelligence, which is used to describe and solve the problem of maximizing rewards or achieving specific goals by learning action strategies during the interaction between an intelligent agent and the environment.

[0003] In the actual tasks of deep reinforcement learning, the idea of ​​sampling and reusing historical experience has been applied, but currently sampling is performed based on the correlation or differential error between the experience and the current action strategy distribution.

[0004] Therefore, how to expand auxiliary tasks to general problem areas and improve the training efficiency and sample utilization of reinforcement learning methods has become a major problem that technicians in this field need to solve. Summary of the invention

[0005] An embodiment of the present invention provides an intelligent body motion control method, device, electronic device and computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0006] The embodiment of the present invention discloses an intelligent body motion control method, comprising:

[0007] Select experience samples from the experience pool;

[0008] Obtaining encoding and decoding errors and estimation errors for actual states and steps based on the experience samples;

[0009] Generate a feature extraction network through encoding and decoding errors, and generate a state step network through estimation errors;

[0010] Outputting state feature errors through the feature extraction network and outputting step count estimates through the state step count network;

[0011] Generate a state incentive value based on the state feature error and the step number estimation calculation;

[0012] Determine the sampling probability by the state incentive value to perform double sampling on the experience samples in the experience pool, and combine the sampling results with the historical experience of the pre-built experience buffer to generate training samples;

[0013] generating an action strategy network based on the training samples;

[0014] When the target intelligent agent interacts with the environment, the action decision result is output through the action strategy network, and based on the action decision result, the target intelligent agent is controlled to perform an action corresponding to the action decision result.

[0015] The embodiment of the present invention further discloses an intelligent body motion control device, comprising:

[0016] An experience sample selection module is used to select experience samples from the experience pool;

[0017] A coding and decoding error and estimation error acquisition module, used to obtain the coding and decoding error and the estimation error for the actual state and the number of steps based on the experience sample;

[0018] A network training module is used to generate a feature extraction network through encoding and decoding errors, and to generate a state step number network through estimation errors;

[0019] A network output module, configured to output a state feature error through the feature extraction network and output a step count estimate through the state step count network;

[0020] A state incentive value calculation module, used for calculating and generating a state incentive value based on the state feature error and the step number estimation;

[0021] A training sample acquisition module, used to perform double sampling on the experience samples in the experience pool by determining the sampling probability through the state incentive value, and to merge the sampling results with the historical experience of the pre-built experience buffer to generate a training sample;

[0022] An action strategy network training module, used for generating an action strategy network based on the training samples;

[0023] The action decision result output module is used to output the action decision result through the action strategy network when the target intelligent agent interacts with the environment, and control the target intelligent agent to perform an action corresponding to the action decision result based on the action decision result.

[0024] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0025] The memory is used to store computer programs;

[0026] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0027] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0028] The embodiments of the present invention include the following advantages:

[0029] In the embodiment of the present invention, experience samples in the experience pool are selected; based on the experience samples, encoding and decoding errors and estimation errors for actual states and steps are obtained; a feature extraction network is generated through the encoding and decoding errors, and a state step network is generated through the estimation errors; state feature errors are output through the feature extraction network, and step estimates are output through the state step network; state incentive values ​​are generated based on the state feature errors and step estimates; sampling probability is determined through the state incentive value to perform double sampling on the experience samples in the experience pool, and the sampling results are combined with the historical experience of the pre-constructed experience buffer to generate training samples; an action strategy network is generated based on the training samples; when the target intelligent agent interacts with the environment, an action decision result is output through the action strategy network, and based on the action decision result, the target intelligent agent is controlled to perform an action corresponding to the action decision result. By introducing technologies such as auxiliary tasks, experience double sampling and state incentive value, the algorithm can better utilize data, improve the generalization ability and learning efficiency of the model, and thus improve the action control efficiency of the intelligent agent. By continuously interacting with the environment, collecting data, training models, and using the trained models to guide the behavior of the intelligent agent, the autonomous learning of the intelligent agent is finally achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of a method for controlling an intelligent body motion provided in an embodiment of the present invention;

[0031] Figure 2 is a structural block diagram of a motion control system provided in an embodiment of the present invention;

[0032] Figure 3 is a flow chart of an intelligent body motion control method provided in an embodiment of the present invention;

[0033] Figure 4 is a structural block diagram of an auxiliary task module provided in an embodiment of the present invention;

[0034] Figure 5 is a workflow diagram of an auxiliary task module provided in an embodiment of the present invention;

[0035] Figure 6 is a schematic flow chart of a double sampling method provided in an embodiment of the present invention;

[0036] Figure 7 It is a schematic diagram of a flow chart of a method for dynamic correction of a value function provided in an embodiment of the present invention;

[0037] Figure 8 is a structural block diagram of an intelligent body motion control device provided in an embodiment of the present invention;

[0038] Fig. 9 is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention;

[0039] Fig.10 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] In order to enable those skilled in the art to better understand the embodiments of the present invention, some technical terms involved in the embodiments of the present invention are explained below.

[0042] Reinforcement learning: It is one of the machine learning methods in artificial intelligence, which is used to describe and solve the problem of maximizing rewards or achieving specific goals by learning action strategies during the interaction between the intelligent agent and the environment.

[0043] Actor-Critic Framework: One of the training frameworks for reinforcement learning. Mainstream methods are usually built on the actor-critic framework. The actor has an action strategy and can choose actions based on the current state. The critic maintains a state-value function or an action-value function and estimates the value of the input state and action.

[0044] Proximal Policy Optimization (PPO): A reinforcement learning method based on the policy gradient method, with importance sampling and action ratio clipping added to further improve performance.

[0045] Intelligent agent: In terms of hardware, it can be a machine that is optimized and trained to observe the state of the environment and can select actions based on the state to complete action interaction. In terms of software, it can be seen as a combination of an actor and a critic, for example, a multi-joint robot that can obtain environmental information.

[0046] Environment: The task environment of the sequential decision-making problem, which can display the current state, transfer states according to the actions performed, and give reward feedback.

[0047] Action strategy: The agent’s action selection scheme based on the current state, usually a neural network function mapping, which gives the corresponding action selection based on the input current state value.

[0048] Return: Also known as reward, it is the immediate reward feedback given by the environment. Maximizing the return is also the goal of decision optimization.

[0049] Cumulative return: also known as cumulative reward, the sum of the rewards obtained from the current state to the terminal state in a single round.

[0050] Experience buffer: used to save the experience data obtained by the interaction between the agent and the environment. For policy reinforcement learning methods, such as PPO, it is recommended to use experience training based on the interaction between the current action strategy and the environment. These experiences are stored in the buffer and will be cleared and resampled after each training batch.

[0051] On-policy reinforcement learning: Reinforcement learning methods can be divided into two categories: on-policy reinforcement learning and off-policy reinforcement learning. On-policy reinforcement learning is usually optimized in the form of policy gradients, using the current action strategy to interact with the environment and calculate the gradient of the reward to the action, optimizing the action strategy in the direction of obtaining more cumulative rewards, so there is less experience available for training. Off-policy reinforcement learning calculates the action value function, gives an estimated value of each action in each state, and uses this to select the optimal action. It is optimized based on the Bellman equation and can use more historical experience, but the training speed is slower.

[0052] Catastrophic forgetting: Catastrophic forgetting is one of the problems that affect the performance of reinforcement learning methods. It often occurs in policy reinforcement learning methods because it is impossible to maintain the value of all state actions. As the number of training batches continues to increase, the estimation error of early samples increases, resulting in "forgetting" earlier training experience, which will cause the agent to fall into the local optimum prematurely.

[0053] In practical applications, with the continuous development of the field of artificial intelligence, deep reinforcement learning has demonstrated remarkable effects in many fields by combining the efficient function representation and perception capabilities of deep learning with the sequential decision-making capabilities of reinforcement learning. For example, it can simulate human actions in the field of robot control, defeat world-class professional players in the field of Go, and achieve a level beyond humans in pixel-input arcade games.

[0054] However, deep reinforcement learning decision optimization solutions still face many problems in practical applications:

[0055] 1) Reinforcement learning methods require a large number of action interaction sampling times and computing time to solve action decision-making problems, and the training efficiency is low, which makes it difficult to meet the needs of actual task scenarios.

[0056] 2) Reinforcement learning methods have the problem of catastrophic forgetting of experience, which causes them to fall into local optimal solutions prematurely, especially in "on-policy reinforcement learning methods", such as A3C, TRPO, and PPO methods.

[0057] 3) In the policy reinforcement learning method, the current action strategy and the experience obtained by environment sampling need to be used for training. The action distribution differences and noise in the historical experience are too large to be effectively utilized, resulting in low sample efficiency.

[0058] 4) The actual decision-making task scenarios are more complex. The training instability caused by falling into the local optimum too early, the adjustment of environmental rewards and hyperparameters have seriously increased the cost of practical application of reinforcement learning methods. Improving training performance and stability is also more important.

[0059] The problem with sample efficiency is that reinforcement learning is essentially a sequential decision-making problem that requires learning the optimal strategy through interaction with the environment. However, unlike supervised learning, the data for reinforcement learning is generated through the interaction between the agent and the environment, which makes data acquisition more expensive. In addition, the reward signals for reinforcement learning are often sparse, which further increases the difficulty of learning.

[0060] The problem of balancing exploration and exploitation is that reinforcement learning needs to find a balance between exploring new state-action pairs and exploiting existing experience. Too much exploration will lead to inefficient learning, while too much exploitation may cause the algorithm to fall into a local optimum.

[0061] The problem of function approximation error is that although deep neural networks, as function approximators for reinforcement learning, have powerful expressive power, they also introduce certain errors. This error will affect the estimation of the value function and the policy function, thus affecting the performance of the algorithm.

[0062] Although the current reinforcement learning methods based on auxiliary tasks have made some progress, they still have some limitations:

[0063] Task dependency: The design of auxiliary tasks often depends on the specific task environment and lacks generality.

[0064] Difficulty in reward construction: How to design effective auxiliary rewards is a challenging problem.

[0065] Complex parameter tuning: The introduction of auxiliary tasks will increase the complexity of the model and require a lot of parameter tuning.

[0066] In the actual tasks of deep reinforcement learning, the idea of ​​sampling and reusing historical experience has been applied, but currently sampling is performed based on the correlation or differential error between the experience and the current action strategy distribution.

[0067] Therefore, how to expand auxiliary tasks to the field of general problems, improve the training efficiency and sample utilization of reinforcement learning methods, alleviate experience forgetting and avoid falling into local optimality too early has become the main problem to be solved by this solution.

[0068] The embodiment of the present invention introduces methods of auxiliary task feature optimization, empirical double sampling utilization and dynamic correction of value functions to more fully integrate auxiliary tasks into the training process; by increasing the amount of calculation in the training process, the sample efficiency and training efficiency of the reinforcement learning method are improved within a limited number of interactive sampling times, the problem of catastrophic forgetting is alleviated, and falling into local optimality too early is avoided, ultimately obtaining an action selection strategy with better performance.

[0069] By determining the decision result in the above manner, the following beneficial effects can be achieved.

[0070] More efficient sample utilization:

[0071] Experience replay: By storing historical experience and randomly sampling from it to train the model, the utilization rate of samples can be improved.

[0072] Importance sampling: weighting historical experience to mitigate the impact of distribution shift.

[0073] Model-based reinforcement learning: Use models to predict future states and generate more training data.

[0074] Better exploration strategies:

[0075] Intrinsic rewards: Design some intrinsic rewards to encourage the agent to explore new states.

[0076] Curiosity-driven: Let the intelligent agent become curious about unknown things and actively explore them.

[0077] More robust function approximators:

[0078] Improve neural network structure: Design a neural network structure that is more suitable for reinforcement learning tasks.

[0079] Regularization techniques: Use techniques such as L1 / L2 regularization to prevent overfitting.

[0080] More effective auxiliary task design:

[0081] General auxiliary tasks: Design some auxiliary tasks that are not related to specific tasks, such as predicting the next state, estimating the value function, etc.

[0082] Adaptive auxiliary tasks: Dynamically adjust auxiliary tasks according to the learning process.

[0083] Reference Figure 1 , shows a flow chart of the steps of an intelligent body motion control method provided in an embodiment of the present invention, which may specifically include the following steps:

[0084] Step 101, selecting an experience sample from an experience pool;

[0085] Step 102, obtaining a coding and decoding error and an estimated error for an actual state and a step number based on the empirical sample;

[0086] Step 103, generating a feature extraction network through encoding and decoding errors, and generating a state step number network through estimation errors;

[0087] Step 104, outputting a state feature error through the feature extraction network, and outputting a step count estimate through the state step count network;

[0088] Step 105, generating a state incentive value based on the state feature error and the step number estimation calculation;

[0089] Step 106, performing double sampling on the experience samples in the experience pool by determining the sampling probability through the state incentive value, and merging the sampling results with the historical experience of the pre-built experience buffer to generate training samples;

[0090] Step 107, generating an action strategy network based on the training samples;

[0091] Step 108, when the target intelligent agent interacts with the environment, the action decision result is output through the action strategy network, and based on the action decision result, the target intelligent agent is controlled to perform an action corresponding to the action decision result.

[0092] The embodiment of the present invention can be applied to an action control system for an intelligent agent, and the action control system can be an on-policy reinforcement learning optimization architecture based on auxiliary tasks and experience enhancement. Figure 2 , Figure 2 It is a structural block diagram of a motion control system provided in an embodiment of the present invention; the motion control system may include: an environment interaction module, an experience storage module, an auxiliary task module, an experience sampling module and an intelligent agent training module.

[0093] 1. Environmental interaction module:

[0094] Function: Interact with the task environment and collect experience data.

[0095] Process: The agent perceives the state of the environment, selects an action based on the current strategy, and executes the action. The environment will give corresponding feedback (reward or punishment) and a new state based on the action. The four-tuple (s, a, r, s') consisting of these states, actions, rewards, and next states is an experience, which is stored in the experience buffer.

[0096] 2. Experience storage module:

[0097] Function: Store and manage experience data.

[0098] composition:

[0099] Experience buffer: Stores recent interaction experiences that are highly relevant to the current strategy and have high training value.

[0100] Experience pool: stores the experience of historical batches and screens them through sampling strategies to improve sample utilization.

[0101] Function: The experience buffer provides recent, highly relevant data, while the experience pool provides diverse historical data, which together support model training.

[0102] 3. Auxiliary task module:

[0103] Function: Improve feature extraction quality and enhance exploration capabilities.

[0104] composition:

[0105] Feature extractor: Through self-supervised learning, it extracts more discriminative features from the original state for subsequent model training.

[0106] State Step Estimator: Predicts the number of transition steps between states, helping the agent better understand the relationship between states.

[0107] effect:

[0108] Feature optimization: The extracted features can better reflect the essence of the state and help the model learn better.

[0109] Exploration incentive: State-step estimates can guide the agent to explore actions that may lead to larger state changes, thereby discovering new sources of rewards.

[0110] 4. Experience sampling module:

[0111] Function: Sample data from the experience pool and experience buffer for training the model.

[0112] method:

[0113] Double sampling: Combine the state incentive value (determined by feature extraction error and step number estimation) and action probability to perform weighted sampling of experience.

[0114] Purpose: To improve sample utilization and accelerate the learning process.

[0115] 5. Agent training module:

[0116] Function: Update model parameters and improve the decision-making ability of the intelligent agent.

[0117] process:

[0118] Action policy network: Outputs action probability distribution based on the current state.

[0119] Critic Networks: Estimating the value of state-action pairs.

[0120] Value Corrector: Corrects value estimates to improve training stability.

[0121] Optimization goal: maximize long-term cumulative rewards.

[0122] In a specific implementation, the embodiment of the present invention can select experience samples in the experience pool to randomly extract a portion of samples from the historical interaction data as data for training the model, thereby providing a data basis for subsequent model training.

[0123] In reinforcement learning, the experience pool (Experience Replay Buffer) is like a memory bank that stores the experience data generated during the interaction between the agent and the environment. These experience data can be used repeatedly to help the agent learn better and improve strategies.

[0124] An experience sample is a single data stored in the experience pool. It usually contains the following information:

[0125] State: The state of an agent at a certain moment.

[0126] Action: The action taken by the agent in this state.

[0127] Reward: The reward the agent receives after performing the action.

[0128] Next state: The new state that the agent enters after executing an action.

[0129] Terminated (done): Indicates whether the current round is finished.

[0130] In a specific implementation, an embodiment of the present invention can obtain the encoding and decoding error and the estimation error for the actual state and step number based on the empirical samples to calculate the loss function of the auxiliary task and provide a feedback signal for optimizing the feature extraction network and the state step number network.

[0131] The encoding and decoding error of a state refers to the difference between the encoding obtained after inputting the state into the feature extraction network and reconstructing the original state through the decoder. The larger this difference is, the worse the feature extraction network's ability to represent the state is.

[0132] Relationship with experience samples: Experience samples contain a large number of state-action-reward-next state quadruples. During the auxiliary task training phase, we randomly extract some samples from the experience pool. For each state in the sample, we input it into the feature extraction network to obtain an encoding. Then, we try to use this encoding to reconstruct the original state. The difference between the reconstructed state and the original state is the encoding-decoding error.

[0133] The specific process is as follows:

[0134] 1. Randomly sample a batch of experience samples from the experience pool.

[0135] 2. Extract the state in each sample and input it into the feature extraction network to obtain the encoding.

[0136] 3. Use the decoder to reconstruct the code into state.

[0137] 4. Calculate the error (e.g., mean square error) between the reconstructed state and the original state.

[0138] 5. Use the calculated error to update the parameters of the feature extraction network and decoder.

[0139] By minimizing the encoding and decoding error, the feature extraction network can be forced to learn more discriminative features, thereby improving the representation ability of the model.

[0140] The actual state at two moments and the estimated error of the step calculation mean that for the states at two consecutive moments in the experience sample, we use the state-step network to predict the number of steps required for the state change. Then, we compare the predicted number of steps with the actual number of steps and calculate the error between the two.

[0141] Relationship with experience samples: Similarly, we randomly extract some samples from the experience pool. For the states of two consecutive moments in each sample, we input them into the state-step network to get a predicted number of steps. Then, we compare this predicted number of steps with the actual number of steps (that is, the number of steps between the two states) and calculate the error between the two.

[0142] The specific process is as follows:

[0143] 1. Randomly sample a batch of experience samples from the experience pool.

[0144] 2. For each sample, extract the states of two consecutive moments.

[0145] 3. Input these two states into the state-step network to get a predicted number of steps.

[0146] 4. Calculate the error (e.g., mean square error) between the predicted number of steps and the actual number of steps.

[0147] 5. Use the calculated error to update the parameters of the state step network.

[0148] By minimizing the step estimation error, we can improve the prediction accuracy of the state-step network, thereby helping the model better understand the relationship between states and provide a more accurate reference for subsequent action decisions.

[0149] Whether it is the encoding and decoding error of the state or the step estimation error, it is calculated by comparing the difference between the network output and the true value. The calculation of these differences depends on samples randomly drawn from the experience pool. By minimizing these errors, the parameters of the network can be continuously optimized and the performance of the model can be improved.

[0150] The functions of these two auxiliary tasks are:

[0151] Improve the quality of feature extraction: The encoding and decoding tasks force the feature extraction network to learn more discriminative features.

[0152] Enhance the model’s understanding of state changes: The step estimation task helps the model better understand the relationship between states and thus make more accurate predictions.

[0153] These two auxiliary tasks work together to help the model better understand the environment, thereby improving the performance of the reinforcement learning algorithm.

[0154] In a specific implementation, the embodiment of the present invention can generate a feature extraction network through encoding and decoding errors, and generate a state step network through estimation errors, so as to update the parameters of the feature extraction network and the state step network by using the error calculated in step 102, thereby generating a feature extraction network and a state step network, thereby achieving the effect of improving the feature extraction network's ability to represent the state and improving the state step network's prediction accuracy for state transitions.

[0155] In a specific implementation, an embodiment of the present invention can output a state feature error through the feature extraction network, and output a step estimate through the state step network, so as to calculate a new feature error and step estimate using the updated network, thereby providing basic data for calculating the state incentive value.

[0156] State feature error refers to the fact that in reinforcement learning, we usually encode the state of the environment and convert it into a numerical representation that the neural network can process. This process is feature extraction. The state feature error refers to the difference between the extracted feature vector and the original state.

[0157] The reasons for introducing state characteristic errors include:

[0158] Feature quality assessment: By calculating the feature error, we can evaluate whether the extracted features can accurately reflect the information of the original state. If the error is large, it means that the extracted features may have lost important information.

[0159] Drive feature learning: By minimizing feature errors, the neural network can be driven to learn better feature representations.

[0160] Step estimation refers to trying to predict how many actions are needed to get from the current state to the target state in reinforcement learning. This predicted value is the step estimate.

[0161] Reasons for introducing step count estimation include:

[0162] Exploration guidance: Step estimation can help the agent explore the environment better. If the step estimation value of a state is large, it means that it may take more state transitions to reach the target state. This state may contain more information and is worth further exploration.

[0163] Intrinsic Reward: Step count estimates can be used as an intrinsic reward to encourage the agent to explore actions that lead to larger changes in state.

[0164] The state feature error and step number estimation are based on the output results of the auxiliary task in step 103.

[0165] State feature error: In step 103, the embodiment of the present invention reconstructs the state by inputting the state into the feature extraction network and then calculates the reconstruction error. This reconstruction error represents the quality of the feature extraction network's ability to represent the state. The smaller the error, the more accurate the feature extraction network's representation of the state, indicating that the state may contain more information and is more valuable for learning.

[0166] Step number estimation: Also in step 103, the embodiment of the present invention predicts the number of steps required for state transition through the state step number network, and calculates the error between the predicted value and the true value. This error reflects the predictive ability of the state step number network for state transition. The smaller the error, the more accurate the prediction of the state step number network for state transition, indicating that the state may be in an important state transition area and deserves more exploration.

[0167] In a specific implementation, an embodiment of the present invention can calculate and generate a state incentive value based on the state feature error and the step number estimation, and can assign different importances to experience samples to guide the model to preferentially learn more valuable experiences. By calculating the state incentive value, the potential value of each state can be measured. In the subsequent sampling process, states with higher incentive values ​​will be sampled preferentially, thereby accelerating the learning process. Calculating and generating the state incentive value can quantify the value of each state, provide a basis for subsequent sampling, guide the intelligent agent to preferentially explore more valuable states, and accelerate the learning process.

[0168] In practical applications, performing experience double sampling improves sample utilization efficiency and prevents the model from falling into local optimum too early. In a specific implementation, the embodiment of the present invention can determine the sampling probability by the state incentive value to perform double sampling on the experience samples in the experience pool, and merge the sampling results with the historical experience of the pre-constructed experience buffer to generate training samples.

[0169] For example, two samplings can be performed: on the one hand, the experience is screened according to the state incentive value, and on the other hand, it is screened according to the action probability. This dual sampling mechanism can ensure that the sampled samples are both representative and diverse, thereby improving the generalization ability of the model; further, the samples are weighted according to the state incentive value to improve the utilization efficiency of the samples, thereby improving the generalization ability of the model and accelerating convergence.

[0170] In a specific implementation, the embodiment of the present invention can generate an action strategy network based on the training samples.

[0171] Exemplarily, the actor action strategy network and the critic value estimation network can be trained based on training samples to update the parameters of the action strategy network so that the model can select better actions according to the current state, thereby improving the decision-making ability of the intelligent agent.

[0172] In a specific implementation, the embodiment of the present invention can output an action decision result through the action strategy network when the target intelligent agent interacts with the environment, and control the target intelligent agent to perform an action corresponding to the action decision result based on the action decision result, so as to apply the trained model to the actual environment, thereby verifying the performance of the model and continuously improving the model through interaction with the environment.

[0173] refer to Figure 3 , Figure 3 is a flow chart of an intelligent body motion control method provided in an embodiment of the present invention;

[0174] The sequential decision task solution process can be executed by the following example.

[0175] Step 1: Initialization of each module. Initialize the interfaces and parameters of each module, and initialize the structure and parameters of the value estimation network, action decision network, feature extraction network, and state step network. The four networks share the feature extractor. Initialize the structure and specifications of the experience pool and buffer, and initialize the correction value generator. In addition, when each module is initialized, preset hyperparameters such as training termination conditions, action constraint size, and historical sample expiration mode are loaded.

[0176] Step 2: In the environmental interaction phase, the actor obtains the environmental state, makes action choices based on the output of the action decision network, acts on the environment to obtain reward values, obtains round experience and stores it in the experience buffer, and the experience buffer stores historical experience in the experience pool.

[0177] Step 3: During the auxiliary task training phase, the auxiliary task module selects experience samples from the experience pool and trains a feature extraction network based on the state encoding and decoding errors.

[0178] The estimated error is calculated based on the actual state and number of steps at two moments, and the state-step network is trained to optimize.

[0179] This enables the feature extraction network to have the function of encoding and decoding the state, and the state step network can estimate the number of steps required for state changes.

[0180] Step 4: In the state incentive value calculation phase, the state incentive value in the historical experience sample is calculated according to the incentive value calculation function for the subsequent double sampling and training process.

[0181] The state incentive value is calculated by the state feature error and step number estimation in the auxiliary task to encourage the agent to explore and train in the direction of more potential value.

[0182] Step 5: In the experience double sampling phase, the historical samples in the experience pool are double sampled. The first sampling is performed based on the sampling probability divided according to the state incentive value, and then the second sampling is performed based on the selection probability of the current state action given by the actor action strategy network. The sampling results are merged with the historical experience of the experience buffer as training samples for subsequent actors and critics.

[0183] Step 6: The actor action strategy network training needs to use the critic to calculate the action valuation based on the experience samples, calculate the cumulative reward based on the experience samples, use the corrector to generate the value correction, and then get the action advantage value, and perform gradient training optimization and update through back propagation. The advantage value is a measure of the action value, and the actor strategy network will optimize the parameters according to the gradient training to increase or decrease the probability of the current action selection.

[0184] Step 7: Critic value estimation network training requires calculating the output of the critic value estimation network based on the experience samples, and performing gradient training optimization and update with the cumulative true value of the experience samples as the target.

[0185] Step 8: Verify whether the training termination conditions are met. After the current training batch is finished, measure whether the termination conditions are met based on indicators such as action strategy performance and number of training rounds. If the termination conditions are not met, go to step 2 for the next batch of environment interaction and training. If the termination conditions are met, the training process is completed.

[0186] Step 9: Use the final action policy network, or the policy network with the highest reward saved in the middle, as the output of the agent, interact the action policy network with the environment, and perform more policy performance evaluations.

[0187] In the embodiment of the present invention, experience samples in the experience pool are selected; based on the experience samples, encoding and decoding errors and estimation errors for actual states and steps are obtained; a feature extraction network is generated through the encoding and decoding errors, and a state step network is generated through the estimation errors; state feature errors are output through the feature extraction network, and step estimates are output through the state step network; state incentive values ​​are generated based on the state feature errors and step estimates; sampling probability is determined through the state incentive value to perform double sampling on the experience samples in the experience pool, and the sampling results are combined with the historical experience of the pre-constructed experience buffer to generate training samples; an action strategy network is generated based on the training samples; when the target intelligent agent interacts with the environment, an action decision result is output through the action strategy network, and based on the action decision result, the target intelligent agent is controlled to perform an action corresponding to the action decision result. By introducing technologies such as auxiliary tasks, experience double sampling and state incentive value, the algorithm can better utilize data, improve the generalization ability and learning efficiency of the model, and thus improve the action control efficiency of the intelligent agent. By continuously interacting with the environment, collecting data, training models, and using the trained models to guide the behavior of the intelligent agent, the autonomous learning of the intelligent agent is finally achieved.

[0188] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0189] In an optional embodiment of the present invention, the auxiliary task module may include the following network:

[0190] 1. The value estimation network is responsible for evaluating the long-term value of the current state or state-action pair and is one of the core components of the reinforcement learning algorithm.

[0191] 2. The action decision network is used to select the next action to be performed based on the current state and the learned strategy.

[0192] 3. The feature extraction network is used as a preprocessing module to convert the original state into a feature representation that can be processed by the neural network.

[0193] 4. The state-step network is used as an auxiliary task network to provide additional information for the reinforcement learning algorithm by predicting the number of steps required to reach the target state.

[0194] Specifically,

[0195] The feature extraction network is the basis, which transforms the raw state into features that can be processed by the neural network.

[0196] The state-step network and the feature extraction network jointly provide support for step estimation.

[0197] The results of step number estimation can provide reference for the value estimation network and action decision network.

[0198] The value estimation network and the action decision network are the core components of the reinforcement learning algorithm, which together determine the behavior of the agent.

[0199] In an optional embodiment of the present invention, before the step of selecting the experience sample in the experience pool, the method further includes:

[0200] Determine the initial value estimation network, initial action decision network, initial feature extraction network and initial state step number network;

[0201] Initialization operations are performed on the structures and parameters of the initial value estimation network, the initial action decision network, the initial feature extraction network, and the initial state step number network.

[0202] Exemplarily, the initialization operation may be performed in the following manner.

[0203] 1.1 Module initialization:

[0204] Before the reinforcement learning task begins, the embodiments of the present invention can initialize each module in the system, including defining the structure of the module, setting parameters, and loading some preset hyperparameters.

[0205] 1.2 Network structure and parameter initialization:

[0206] Value Estimation Network:

[0207] Structure: Usually a fully connected neural network, the input is the output of the feature extractor (i.e. the feature representation of the state), and the output is the value estimate of the state.

[0208] Parameter initialization: Parameters are usually initialized randomly, such as Xavier initialization or He initialization.

[0209] Action Decision Network:

[0210] Structure: It is also a fully connected neural network. The input is the output of the feature extractor, and the output is the probability distribution of each action.

[0211] Parameter initialization: Similar to the value estimation network.

[0212] Feature extraction network:

[0213] Structure: This can be a convolutional neural network (CNN), a recurrent neural network (RNN), or other networks suitable for processing specific types of states.

[0214] Parameter initialization: Parameter initialization method is the same as other networks.

[0215] State-step network:

[0216] Structure: Usually a fully connected neural network, the input is the output of the feature extractor, and the output is the estimate of the number of steps required to reach the target state.

[0217] Parameter initialization: Parameter initialization method is the same as other networks.

[0218] 1.3 Shared Feature Extractor:

[0219] Reason: In order to improve computational efficiency and generalization ability of the model, the four networks share the same feature extractor. This shows that the four networks all believe that these features extracted from the original state are important and can be used for different tasks (value estimation, action decision, step estimation, etc.).

[0220] 1.4 Experience pool and buffer initialization:

[0221] Experience Pool:

[0222] Structure: Usually a queue or list of fixed size.

[0223] Specifications: Set the size of the experience pool based on the complexity of the task and the computing resources.

[0224] Experience Buffer:

[0225] Structure: Similar to the experience pool.

[0226] Specifications: The size of the experience buffer is usually smaller than the experience pool.

[0227] 1.5 Correction value generator initialization:

[0228] Function: The correction value generator is used to correct the value estimation and improve the stability of training.

[0229] Initialization: The structure and parameters of the correction value generator can vary depending on the specific algorithm.

[0230] 1.6 Hyperparameter settings:

[0231] Training termination conditions: define the conditions for the end of training, such as reaching a certain number of training steps, the average reward reaching a certain threshold, etc.

[0232] Action constraint size: Limit the range of actions that the agent can perform to prevent the agent from making overly aggressive actions.

[0233] Historical sample expiration mode: determines how to handle old samples in the experience pool, such as FIFO (first in, first out), random replacement, etc.

[0234] In the initialization phase, the embodiment of the present invention builds a basic framework for the reinforcement learning system. By initializing the structure and parameters of each module and setting hyperparameters, we lay the foundation for subsequent training and learning. The settings at this stage directly affect the performance and training efficiency of the model.

[0235] In an optional embodiment of the present invention, the experience pool includes an experience buffer zone, and before the step of selecting experience samples in the experience pool, the step further includes:

[0236] When the target agent interacts with the environment, it generates round experience based on the reward value obtained by selecting actions to act on the environment;

[0237] The round experience is stored in the experience buffer.

[0238] For example,

[0239] 1. Obtaining the state of the environment: The agent senses the state of the current environment through sensors or other means. This state can be any form of data, such as images, values, text, etc. For example: when playing Atari games, the agent obtains the current game screen as the state through screenshots; in robot control, the agent obtains information such as the robot's joint angles and speeds as the state through sensors.

[0240] 2. Action decision: The agent selects an action based on the current state through an action decision network (usually a neural network). The output of the action decision network is usually a probability distribution, indicating the probability of the agent performing each action. For example: when playing Atari games, the action decision network may output the probabilities of actions such as up, down, left, and right.

[0241] 3. Execute the action: The agent passes the selected action to the environment, and the environment changes accordingly. For example, when playing an Atari game, if the agent chooses the right action, the game screen will move to the right.

[0242] 4. Get rewards: The environment will give a reward value based on the agent's actions. The reward value reflects how well the agent performs the action. A positive reward means that the action is conducive to achieving the goal, and a negative reward means that the action is not conducive to achieving the goal. For example: when playing Atari games, if the agent successfully eats a prop, it will get a positive reward; if the agent hits an obstacle, it will get a negative reward.

[0243] 5. Update state: The state of the environment will change with the actions of the agent. For example, when playing an Atari game, the game screen will update after the agent performs an action.

[0244] 6. Store experience: Store the current state, action performed, reward obtained, next state, and whether the end point has been reached (whether the episode has ended) as an experience sample in the experience buffer. An experience sample is usually a four-tuple (s, a, r, s', done), where: s: current state; a: action performed; r: reward obtained; s': next state; done: whether the end point has been reached;

[0245] 7. Experience buffer and experience pool: Experience buffer: usually stores the most recent experiences. These experiences are more timely and more helpful for the current learning process. Experience pool: stores historical experience samples, which can provide more diversity and help improve the generalization ability of the model. Samples in the experience pool are usually screened to ensure their quality.

[0246] The functions of the experience buffer include: Breaking data correlation: Samples in the experience buffer are randomly sampled, which helps to break the time correlation between data and improve the stability of learning. Improving sample utilization: Samples in the experience buffer can be reused, which improves the efficiency of data utilization. Stabilizing the training process: The experience buffer can smooth the training process and reduce fluctuations during training. In this way, the core loop of the reinforcement learning algorithm is realized. By constantly interacting with the environment, the agent can collect a large amount of experience data. This experience data is stored in the experience buffer and used to train the agent's model. As training progresses, the agent's strategy will continue to improve to better adapt to the environment.

[0247] Exemplarily, steps 102 to 105 are implemented in the following manner.

[0248] The embodiment of the present invention can train the initial value estimation network, the initial action decision network, the initial feature extraction network and the initial state step number network to obtain the value estimation network, the action decision network, the feature extraction network and the state step number network.

[0249] The embodiment of the present invention introduces a feature optimization and exploration incentive method based on auxiliary tasks. A feature extraction network and a state step number network are proposed, and the value estimation network and the action strategy network are expanded. The structure of each network is as follows: Figure 4 As shown, Figure 4 is a structural block diagram of an auxiliary task module provided in an embodiment of the present invention; the network workflow is as follows Figure 5 As shown, Figure 5 is a workflow diagram of an auxiliary task module provided in an embodiment of the present invention;

[0250] The embodiment of the present invention uses experience sample training auxiliary tasks, utilizes feature extraction networks to optimize the feature extraction performance of each network, improves training efficiency, and utilizes state incentive value to promote actors to perform interactive sampling and training optimization in the direction of more potential exploration value, so as to obtain better action strategies.

[0251] The calculation of state incentive value mainly involves the following four parts:

[0252] 1. The feature extraction network encodes and decodes the state input to obtain the state estimate, and optimizes the features with the goal of minimizing the feature error. This is used as the structure shared by the four networks to optimize the features.

[0253] 2. The state step network converts two states S n n steps apart according to the experience in the experience pool. t and S t+nAs input, through encoding and step estimation, the step estimation output is obtained, and the ability to measure the number of steps required for the transition between any two states is obtained, which is used to calculate the state incentive value.

[0254] 3. The value estimation network and action strategy network almost retain the original structure and only share the state encoder.

[0255] 4. The state incentive value uses the step value and state feature error, and combines them through a preset formula. Through a similar addition or multiplication scheme, the incentive value is positively correlated with the step value and feature error. This encourages the actor to explore in the direction of more potential exploration value.

[0256] Optionally, the step of performing double sampling on the experience samples in the experience pool by determining the sampling probability through the state incentive value, and merging the sampling results with the historical experience of the pre-built experience buffer to generate training samples includes:

[0257] Dividing the sampling probability according to the state incentive value, and sampling the experience sample for the first time based on the sampling probability to obtain a first sampling result;

[0258] Perform secondary sampling according to the selection probability of the current state action given by the action decision network to obtain a second sampling result;

[0259] The first sampling result and the second sampling result are combined with historical experience in the experience buffer to obtain training samples.

[0260] Double sampling is a method to improve sample utilization efficiency and optimize model training in reinforcement learning. It combines state value and action strategy information to perform biased sampling on samples in the experience pool, allowing the model to pay more attention to samples that contribute more to learning.

[0261] Illustratively, double sampling may be performed as follows.

[0262] 1. Sampling probability of state incentive value division:

[0263] Calculate state incentive value: For each experience sample, calculate an incentive value based on the value of its state (which can be a state value function or other evaluation indicators). States with higher values ​​usually contain more learning information.

[0264] Determine sampling probability: Normalize the incentive values ​​of all samples to get the sampling probability of each sample. The higher the incentive value of the sample, the greater the probability of being sampled.

[0265] 2. Action strategy network sampling probability:

[0266] Calculate the action selection probability: Input the current state into the action strategy network to obtain the selection probability of each action output by the network.

[0267] Combined with the state incentive value: take the weighted average of the action selection probability and the sampling probability calculated in the first step to get the final sampling probability.

[0268] 3. Double sampling:

[0269] Sampling according to sampling probability: According to the calculated final sampling probability, a batch of samples are randomly selected from the experience pool.

[0270] Merge samples: Merge the extracted samples with the latest samples in the experience buffer to form a new training data set.

[0271] Beneficial effects include:

[0272] Improve sample utilization: By focusing on sampling valuable samples, the utilization efficiency of samples can be improved and the convergence of the model can be accelerated.

[0273] Improve model performance: Focusing on samples that contribute more to learning can help the model learn better strategies faster.

[0274] Balance exploration and utilization: Combining the state value and action strategy information can balance exploration and utilization, so that the model can explore new states and utilize existing knowledge.

[0275] Solve the sample imbalance problem: If the number of samples of certain states in the experience pool is small, double sampling can increase the probability of these samples being sampled, thereby alleviating the sample imbalance problem.

[0276] In the embodiment of the present invention, by using the experience double sampling method, the experience samples in the experience pool can be screened and filtered before the actor and critic are trained, and samples with more potential value can be selected to improve the sample efficiency. The double sampling process can be referred to Figure 6 , Figure 6 It is a flowchart of a double sampling method provided in an embodiment of the present invention; this scheme introduces an experience double sampling utilization method, which has the advantage of being able to utilize more historical samples. For the "on-policy reinforcement learning method", the current experience in the experience buffer has the greatest training value. Too old experience will cause the negative interference of noise to be greater than its training value due to the large gap between the sample distribution and the action strategy distribution. The experience double sampling method can more effectively screen samples with training value from the experience pool, thereby improving sample utilization and training effects. It mainly involves the following three parts:

[0277] 1. Introduce the experience pool and put the historical batch experience eliminated by the experience buffer into the experience pool.

[0278] 2. Calculate the incentive value of experience based on auxiliary tasks, and sample the experience according to the probability of value division to achieve the purpose of selecting samples with greater potential value.

[0279] 3. Based on the actor strategy network, the selection probability of the corresponding action in the experience is calculated, and importance sampling is performed according to the action probability to further correct the state-action probability distribution.

[0280] Optionally, it also includes:

[0281] The training samples are input into a value estimation network, and a value estimation is output.

[0282] Comparing the value estimate with the cumulative reward of the experience sample, calculating a mean square error between the value estimate and the cumulative reward of the experience sample, and determining the mean square error as a loss function;

[0283] A value estimation network is trained based on the loss function.

[0284] Exemplarily, calculate the network output: input the state in the training sample into the critic network to obtain the value estimate of the network output. Calculate the loss function: compare the output of the critic network (value estimate) with the cumulative reward of the experience sample, and calculate their mean square error as the loss function. Gradient update: use the back propagation algorithm to minimize the loss function and update the parameters of the critic network. Optimization training principle: Mean square error: The mean square error measures the difference between the output of the critic network and the true value. Gradient descent: By minimizing the mean square error, the parameters of the critic network are continuously adjusted to make the value estimate of the network output more accurate.

[0285] Optionally, the step of generating an action strategy network based on the training samples includes:

[0286] Determining an action valuation of a target action by the value estimation network;

[0287] Calculate the cumulative reward of each sample according to the reward information of the training sample;

[0288] determining a correction value based on the action valuation and the accumulated reward;

[0289] determining an action advantage value of the target action based on the correction value;

[0290] The action advantage value is used to train and generate an action strategy network.

[0291] Exemplarily, calculate action valuation: the critic network calculates the valuation of each action based on the state and action in the training sample. Calculate cumulative rewards: calculate the cumulative rewards of each sample based on the reward information in the training sample. Generate value correction: the corrector generates a value correction value based on the output of the critic network (action valuation) and the cumulative reward. This value is used to correct the output of the critic network so that it more accurately reflects the value of the action. Calculate action advantage value: subtract the action valuation from the value correction value to obtain the advantage value of each action. The advantage value indicates the superiority of the action relative to other actions. Gradient update: use the calculated advantage value as the target to update the parameters of the actor network through the back propagation algorithm. Optimization principle: Advantage value: The advantage value reflects the superiority of the current action relative to other actions. By maximizing the advantage value, the actor can be encouraged to choose more valuable actions. Gradient update: through the gradient descent method, the parameters of the actor network are continuously adjusted so that the action probability distribution output by the network can maximize the advantage value. The purpose of using advantage value is: Stability: using advantage value can improve the stability of training and prevent the network from falling into local optimum. Efficiency: Advantage values ​​can better reflect the relative value of actions, thus speeding up the learning process.

[0292] The embodiment of the present invention introduces a dynamic correction method for the value function and uses a correction value generator to optimize the action advantage value calculation process and prevent the actor from falling into a local optimum too early.

[0293] refer to Figure 7 , Figure 7 It is a schematic diagram of a flow chart of a method for dynamic correction of a value function provided in an embodiment of the present invention;

[0294] This solution introduces a dynamic correction method for the value function. The estimated value is corrected through a correction value generator and added to the action advantage value calculation. The action advantage value represents the evaluation of the current action. When the advantage value is positive, the actor will increase the probability of selecting the current action with gradient training, otherwise it will decrease.

[0295] However, in actual tasks, actors often fall into local optimal solutions. At this time, it is difficult to find actions that have a significant value improvement. The experience distribution also tends to be stable, resulting in the action advantage value close to 0, making it difficult to conduct more exploration in the action domain and gradually forgetting historical experience.

[0296] The correction value generator, as a function of (state -> value correction value), can periodically apply a correction value to each state to encourage the actor to further explore in the action domain. By making full use of the value of the critic, the actor believes that there are actions with higher value near the local optimal solution, and thus explores in the direction away from the current action.

[0297] The construction scheme of the correction value generator is relatively flexible. Here are two cases:

[0298] 1. The correction value generation function is only related to the training batch and gradually decreases with the training process, such as: v i =α(1-i / I), where α is a hyperparameter, i represents the current round, and I represents the total expected training rounds.

[0299] 2. The correction value is related to the training batch and the potential value, and gradually decreases with the training process, such as: v = α·(τ(s t ,s t+1 ))·(1-i / I), where, τ(s t ,s t+1 ) represents the state incentive value calculated by the auxiliary task according to the current state transition.

[0300] Optionally, it also includes:

[0301] When it is determined that the value estimation network and the action strategy network do not meet the preset termination conditions, the step of generating round experience according to the reward value obtained by selecting the action to act on the environment when the target agent interacts with the environment is re-executed.

[0302] For example,

[0303] End of training batch: A full training cycle is completed, that is, all processes from step 2 to step 7 are completed.

[0304] Performance evaluation:

[0305] Action strategy performance: Evaluate the performance of the current strategy on the validation set or test set, such as calculating average reward, success rate, and other indicators.

[0306] Number of training rounds: count how many rounds of training have been performed.

[0307] Other indicators: Depending on the specific task, other indicators can also be considered, such as the changing trend of the loss function, the update amplitude of the model parameters, etc.

[0308] Determine whether to terminate:

[0309] Set thresholds: Set thresholds for some performance indicators in advance.

[0310] Compare indicators: Compare the evaluated indicators with the set thresholds.

[0311] Satisfaction conditions: If the termination condition is met, stop training; otherwise, continue to the next round of training.

[0312] The choice of termination condition depends on the specific task and requirements. Here are some common examples of termination conditions:

[0313] The average reward reaches a certain threshold: When the average reward obtained by the agent in the environment reaches or exceeds the preset threshold, it means that the agent has learned how to obtain high rewards and training can be stopped.

[0314] Performance no longer improves after N consecutive training rounds: If the performance of the model no longer improves significantly after N consecutive training rounds, the model can be considered to have converged and training can be stopped.

[0315] The number of training rounds reaches the upper limit: When the number of training rounds reaches the preset maximum value, even if the model performance has not yet reached the optimal level, it may be necessary to stop training to save computing resources.

[0316] Loss function convergence: When the model's loss function converges to a smaller value, it can be considered that the model has learned enough information and training stops.

[0317] Meeting specific task requirements: For some specific tasks, there may be more specific termination conditions. For example, in a robot control task, the success rate of the robot completing a task can be set as the termination condition.

[0318] The purposes of setting termination conditions include:

[0319] Avoid overfitting: Overtraining may cause the model to overfit and reduce the generalization ability of the model.

[0320] Save computing resources: When the model has achieved good enough performance, it is unnecessary to continue training, which can save computing resources.

[0321] Ensure the effectiveness of training: By setting reasonable termination conditions, you can ensure that the training process is effective and avoid wasting time.

[0322] Termination conditions are a very important part of the reinforcement learning training process. By setting termination conditions reasonably, the training process can be effectively controlled and the performance of the model can be improved. In practical applications, it is necessary to select appropriate termination conditions based on the specific tasks and model characteristics.

[0323] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0324] Reference Figure 8, shows a structural block diagram of an intelligent body motion control device provided in an embodiment of the present invention, which may specifically include the following modules:

[0325] An experience sample selection module 801 is used to select experience samples from the experience pool;

[0326] A coding / decoding error and estimation error acquisition module 802, configured to obtain a coding / decoding error and an estimation error for an actual state and a number of steps based on the empirical sample;

[0327] A network training module 803 is used to generate a feature extraction network through encoding and decoding errors, and to generate a state step number network through estimation errors;

[0328] A network output module 804, configured to output a state feature error through the feature extraction network and output a step count estimate through the state step count network;

[0329] A state incentive value calculation module 805 is used to calculate and generate a state incentive value based on the state feature error and the step number estimation;

[0330] A training sample acquisition module 806 is used to perform double sampling on the experience samples in the experience pool by determining the sampling probability according to the state incentive value, and merge the sampling results with the historical experience of the pre-constructed experience buffer to generate a training sample;

[0331] An action strategy network training module 807, used to generate an action strategy network based on the training samples;

[0332] The action decision result output module 808 is used to output the action decision result through the action strategy network when the target agent interacts with the environment, and control the target agent to perform an action corresponding to the action decision result based on the action decision result.

[0333] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0334] In addition, an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0335] Memory 903, used for storing computer programs;

[0336] The processor 901 is used to implement any of the agent motion control methods described in the above embodiments when executing the program stored in the memory 903:

[0337] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0338] The communication interface is used for communication between the above terminal and other devices.

[0339] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0340] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0341] like Fig.10 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1001 is also provided, in which instructions are stored. When the computer-readable storage medium 1001 is run on a computer, the computer executes the intelligent body action control method described in the above embodiment.

[0342] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0343] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0344] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0345] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0346] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0347] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0348] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0349] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for controlling an intelligent agent's motion, characterized in that: include: Select experience samples from the experience pool; Obtaining encoding and decoding errors and estimation errors for actual states and steps based on the experience samples; Generate a feature extraction network through encoding and decoding errors, and generate a state step network through estimation errors; Outputting state feature errors through the feature extraction network and outputting step count estimates through the state step count network; Generate a state incentive value based on the state feature error and the step number estimation calculation; Determine the sampling probability by the state incentive value to perform double sampling on the experience samples in the experience pool, and combine the sampling results with the historical experience of the pre-built experience buffer to generate training samples; generating an action strategy network based on the training samples; When the target intelligent agent interacts with the environment, the action decision result is output through the action strategy network, and based on the action decision result, the target intelligent agent is controlled to perform an action corresponding to the action decision result.

2. The method according to claim 1, characterized in that Before the step of selecting the experience sample in the experience pool, the method further includes: Determine the initial value estimation network, initial action decision network, initial feature extraction network and initial state step number network; Initialization operations are performed on the structures and parameters of the initial value estimation network, the initial action decision network, the initial feature extraction network, and the initial state step number network.

3. The method according to claim 1, characterized in that The experience pool includes an experience buffer zone, and before the step of selecting experience samples from the experience pool, the step further includes: When the target agent interacts with the environment, it generates round experience based on the reward value obtained by selecting actions to act on the environment; The round experience is stored in the experience buffer.

4. The method according to claim 1, characterized in that: The step of performing double sampling on the experience samples in the experience pool by determining the sampling probability through the state incentive value, and merging the sampling results with the historical experience of the pre-built experience buffer to generate training samples comprises: Dividing the sampling probability according to the state incentive value, and sampling the experience sample for the first time based on the sampling probability to obtain a first sampling result; Perform secondary sampling according to the selection probability of the current state action given by the action decision network to obtain a second sampling result; The first sampling result and the second sampling result are combined with historical experience in the experience buffer to obtain training samples.

5. The method according to claim 1, characterized in that Also includes: Inputting the training sample into a value estimation network and outputting a value estimation; Comparing the value estimate with the cumulative reward of the experience sample, calculating a mean square error between the value estimate and the cumulative reward of the experience sample, and determining the mean square error as a loss function; A value estimation network is generated by training based on the loss function.

6. The method according to claim 5, characterized in that The step of generating an action strategy network based on the training sample comprises: Determining an action valuation of a target action by the value estimation network; Calculate the cumulative reward of each sample according to the reward information of the training sample; determining a correction value based on the action valuation and the accumulated reward; determining an action advantage value of the target action based on the correction value; The action advantage value is used to train and generate an action strategy network.

7. The method according to claim 5 or 6, characterized in that: Also includes: When it is determined that the value estimation network and the action strategy network do not meet the preset termination conditions, the step of generating round experience according to the reward value obtained by selecting the action to act on the environment when the target agent interacts with the environment is re-executed.

8. An intelligent body motion control device, characterized in that: include: An experience sample selection module is used to select experience samples from the experience pool; A coding and decoding error and estimation error acquisition module, used to obtain the coding and decoding error and the estimation error for the actual state and the number of steps based on the experience sample; A network training module is used to generate a feature extraction network through encoding and decoding errors, and to generate a state step number network through estimation errors; A network output module, configured to output a state feature error through the feature extraction network and output a step count estimate through the state step count network; A state incentive value calculation module, used for calculating and generating a state incentive value based on the state feature error and the step number estimation; A training sample acquisition module, used to perform double sampling on the experience samples in the experience pool by determining the sampling probability through the state incentive value, and to merge the sampling results with the historical experience of the pre-built experience buffer to generate a training sample; An action strategy network training module, used for generating an action strategy network based on the training samples; The action decision result output module is used to output the action decision result through the action strategy network when the target intelligent agent interacts with the environment, and control the target intelligent agent to perform an action corresponding to the action decision result based on the action decision result.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images