Target object control method and device, computing device, and storage medium

Abstract

The present application discloses a method for controlling a target object, comprising: obtaining a plurality of environmental interaction data from an intelligent agent network and determining a candidate sample data set based on the plurality of environmental interaction data, wherein the intelligent agent network comprises an executor network and an evaluation network; inputting each candidate sample data in the candidate sample data set into a plurality of sub-evaluation networks of the evaluation network respectively, and determining a sampling priority of the candidate sample data using the consistency of output values ​​of each self-evaluation network; sampling the candidate sample data set according to the sampling priority of each candidate sample data; training the intelligent agent network based on the sampled candidate sample data; and controlling the target object according to the executor network in the trained intelligent agent network.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a target object control method and apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] At present, deep reinforcement learning control algorithms have been widely used in controlling robots, and various neural networks for controlling robots have been developed based on this algorithm. Different neural networks have different performances in different application scenarios. In an environment with sparse rewards, how to improve the performance of neural networks for controlling robots has become an urgent problem to be solved.

[0003] In the related art, one technical solution is called Hindsight Experience Replay (HER), which randomly modifies the environmental interaction data and modifies the target value according to the environmental state that the robot has reached before to improve the training effect of the neural network that controls the robot, but the effect is not significant. Another technical solution is called Energy Based Prioritization (HEREBP), which is applied in the three-dimensional environment of controlling the manipulator. It uses physical formulas to calculate the kinetic energy difference, potential energy difference and rotational potential energy difference between two targets to prioritize the replay of training data, thereby improving the training effect of the neural network that controls the manipulator. However, this technical solution has too few applicable scenarios and poor generalization ability. Another technical solution is called Curriculum-guided (CHER), which proposes a function to calculate the priority of the replay data based on the characteristics of a specific scenario. It is also only applicable to the application of this specific scenario and has poor generalization ability. There is also a technical solution called Value Disagreement Sampling (VDS), which replays the training data by calculating the uncertainty of the target value of the environmental interaction data to improve the training effect of the neural network controlling the robot, but this requires that the environment can be modified or designed. The above technical solutions have poor generalization ability and require modification of the environment. Summary of the invention

[0004] In view of this, the present application provides a target object control method and apparatus, a computing device, a computer-readable storage medium, and a computer program product, in the hope of alleviating or overcoming some or all of the above-mentioned deficiencies and other possible deficiencies.

[0005] According to the first aspect of the present application, a target object control method is provided, comprising: obtaining multiple environmental interaction data from an intelligent agent network and determining a candidate sample data set based on the multiple environmental interaction data, wherein each of the multiple environmental interaction data includes at least an environmental state, a target value, and an execution action, the candidate sample data set includes at least a portion of the environmental interaction data, the intelligent agent network includes an executor network and an evaluation network, the executor network is used to determine the execution action based on the environmental state and the target value, the evaluation network includes multiple sub-evaluation networks, each sub-evaluation network is used to evaluate the executor network based on the environmental state, the target value and the execution action; each candidate sample data in the candidate sample data set is input into the multiple sub-evaluation networks of the evaluation network respectively, and the sampling priority of the candidate sample data is determined by using the consistency of the output values ​​of each self-evaluation network; the candidate sample data set is sampled according to the sampling priority of each candidate sample data; the intelligent agent network is trained based on the sampled candidate sample data; and the target object is controlled according to the executor network in the trained intelligent agent network.

[0006] According to a second aspect of the present application, a target object control device is provided, comprising: an acquisition module, configured to acquire multiple environmental interaction data from an intelligent agent network and determine a candidate sample data set based on the multiple environmental interaction data, wherein each of the multiple environmental interaction data includes at least an environmental state, a target value, and an execution action, the candidate sample data set includes at least a portion of the environmental interaction data, the intelligent agent network includes an executor network and an evaluation network, the executor network is used to determine the execution action based on the environmental state and the target value, the evaluation network includes multiple sub-evaluation networks, each sub-evaluation network is used to evaluate the executor network based on the environmental state, the target value and the execution action; a determination module, configured to input each candidate sample data in the candidate sample data set into multiple sub-evaluation networks of the evaluation network respectively, and determine the sampling priority of the candidate sample data by using the consistency of the output values ​​of each sub-evaluation network; a sampling module, configured to sample the candidate sample data set according to the sampling priority of each candidate sample data; a training module, configured to train the intelligent agent network based on the sampled candidate sample data; and a control module, configured to control the target object according to the executor network in the trained intelligent agent network.

[0007] According to a third aspect of the present application, a computing device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is prompted to perform the steps of the target object control method according to some embodiments of the present application.

[0008] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed, the target object control method according to some embodiments of the present application is implemented.

[0009] According to a fifth aspect of the present application, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the target object control method according to some embodiments of the present application.

[0010] In the target object control method and device according to some embodiments of the present application, since in a sparse reward environment, candidate sample data with high consistency also has low uncertainty, reflecting the high learning value of the execution action, such data can significantly improve the control performance of the intelligent network. The method and device design the intelligent network, use the consistency of the output values ​​of multiple sub-evaluation networks of the evaluation network of the intelligent network to measure the sampling priority of each candidate sample data, and give priority to using candidate sample data with high sampling priority to train the intelligent network, which can significantly improve the performance of the intelligent network. In addition, replaying the candidate sample data according to the consistency of the output value corresponding to the candidate sample data does not limit the application scope of the intelligent network. The training method has a high generalization ability and can adapt to various environments without requiring modification or design of the environment.

[0011] These and other advantages of the present application will become apparent from and be elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Embodiments of the present application will now be described in more detail and with reference to the accompanying drawings, in which:

[0013] Figure 1 The application scenarios of sparse rewards according to some embodiments of the present application are shown;

[0014] Figure 2 An exemplary application scenario of the target object control method according to some embodiments of the present application is shown;

[0015] Figure 3 A flow chart showing a target object control method according to some embodiments of the present application is shown;

[0016] Figure 4 A schematic diagram of an agent network according to some embodiments of the present application is shown;

[0017] Figure 5 A flowchart of determining sampling priority of candidate sample data according to some embodiments of the present application is shown;

[0018] Figure 6 A schematic diagram showing a target object control method according to some embodiments of the present application is shown;

[0019] Figures 7A-7J A schematic diagram showing the application of the target object control method according to an embodiment of the present application in various scenarios;

[0020] Figures 8A-8D A schematic diagram showing the effects of the target object control method in different test scenarios according to some embodiments of the present application is shown;

[0021] Fig. 9 An exemplary structural block diagram of a target object control device according to some embodiments of the present application is shown;

[0022] Fig.10 An example system is shown that includes an example computing device that represents one or more systems and / or devices that may implement the various methods described herein. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0024] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0027] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" and similar terms include all combinations of any, multiple, and all of the associated listed items.

[0028] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0029] Before describing the embodiments of the present application in detail, some related concepts are first explained for the sake of clarity.

[0030] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0031] Reinforcement learning: Usually used to solve sequential decision-making problems, it mainly includes two components: environment and agent. The agent predicts and executes actions based on the environmental state. The environment transforms to a new environmental state based on the agent's execution action and feeds back a numerical reward. The agent continuously optimizes its strategy based on the reward fed back by the environment.

[0032] Deterministic Deep Policy Gradient (DDPG): A common reinforcement learning algorithm that consists of two networks: the executor and the evaluator. The executor network determines the execution action based on the environment state, and the evaluator network takes the environment state and action as input, and predicts the reward value of taking this action under the environment state. As learning progresses, the strategy of the executor network becomes better and better, and the reward value predicted by the evaluator network becomes more and more accurate. In this application, the neural network designed according to this algorithm can be used as the basic structure of the agent network.

[0033] Environmental interaction data: data generated by the interaction between the environment and the target object, usually including environmental state, target value, execution action, reward value, and feedback environmental state.

[0034] Environmental state: refers to the description of the environment through some numerical values, such as the two-dimensional or three-dimensional coordinates, movement speed, rotation speed, movement direction and other numerical values ​​of the environmental state.

[0035] Target value: refers to the specific value that the target object is expected to reach in the environment. The value in the target value is part of the value in the environment state. For example, the target value can only be the specific horizontal and vertical coordinates that you want to reach in a two-dimensional environment, without including the speed value.

[0036] Execution action: The agent network can control the target object to perform various actions according to the environment state, and determine the action that is most conducive to achieving the target value. For example, in the environment state, the agent network can control the target object to turn left or move forward, and the execution action finally determined by calculation can be a left turn.

[0037] Feedback environment state: The new environment state that the environment feeds back to the agent network after receiving the execution action of the agent network.

[0038] Reward value: The reward value is the return value obtained by the target object from the environment after performing an action.

[0039] Hindsight Experience Replay (HER): A reinforcement learning algorithm that solves the problem of sparse rewards in goal-oriented environments. In a sparse reward environment, the target object will only receive a reward value of 0 after reaching the target value, and can only obtain a reward value of -1 in the remaining environmental states. In this environment, the environmental states that the target object receives feedback from are mostly environmental states with a reward value of -1. Negative reward values ​​make it difficult for the agent network to learn strategies. This method improves the efficiency of using failed data by modifying the target value of the data in the experience pool to the environmental state that has been reached.

[0040] Sampling: Extract a certain amount of sample data from a batch of sample data according to the probability of each sample data as the representative of this batch of sample data.

[0041] Figure 1 The application scenario 100 of sparse rewards according to some embodiments of the present application is shown. The application scenario 100 includes a target object R, a target location A, and an arrival location B.

[0042] In application scenario 100, the target object R is set to go to the target location A. The target value of the candidate sample data of the target object R can be the location coordinates of the target location A. The environmental state of the candidate sample data of the target object R can be the location coordinates of the target object R. Under the control of the intelligent agent network, the target object R performs several execution actions with the target location A as the target, such as executing fifty steps, and finally arrives at the destination B.

[0043] For example, in application scenario 100, if the target value is achieved, that is, the target location A is reached, the reward value is 0, otherwise it is -1. Then, the fifty interactions between the target object R and the environment, that is, the reward value of the environment for the fifty execution actions is -1. In other words, in application scenario 100, the cases where the target object R can obtain rewards are very few, and the cases where it cannot obtain rewards are very many. Such application scenario 100 is a sparse reward environment. The target object control method according to the embodiment of the present application can be applied to such a scenario, and the trained agent network has good performance.

[0044] Figure 2 An exemplary application scenario 200 of a target object control method according to some embodiments of the present application is shown. The application scenario 200 may include a target object 201, a network 202, and a server 203, wherein the server 203 is communicatively coupled with the target object 201 via the network 202. In this embodiment, only one target object 201 is shown, but this is not restrictive, and multiple target objects may exist at the same time and be communicatively coupled with the server 203. The target object 201 may be any type of robot.

[0045] In this embodiment, an agent network is pre-built, and the agent network includes an executor network and an evaluation network. The executor network is used to determine the execution action based on the environment state and the target value, and the evaluation network includes multiple sub-evaluation networks, each of which is used to evaluate the executor network based on the environment state, the target value and the execution action. These agent networks can have the same neural network structure, and the various parameters of the neural network can be randomly initialized.

[0046] In this embodiment, first, the server 203 obtains a plurality of environmental interaction data from the agent network and determines a candidate sample data set based on the plurality of environmental interaction data, wherein each of the plurality of environmental interaction data includes at least an environmental state, a target value, and an execution action, and the candidate sample data set includes at least a portion of the environmental interaction data. The plurality of environmental interaction data from the agent network may be environmental interaction data obtained by using one of the agent networks to control the target object to interact with the environment.

[0047] The server 203 may then input each candidate sample data in the candidate sample data set into a plurality of sub-evaluation networks of the evaluation network, and determine the sampling priority of the candidate sample data by using the consistency of the output values ​​of each sub-evaluation network.

[0048] The server 203 then samples the candidate sample data sets according to the sampling priority of each candidate sample data.

[0049] The server 203 trains the agent network based on the sampled candidate sample data.

[0050] After the training is completed, the server 203 controls the target object according to the actor network in the trained agent network. During the control process, for example, the server 203 gives instructions about specific execution actions according to the environmental state and target value of the target object 201, and the target object 201 performs actions according to the execution actions in the instructions, and transmits the environmental interaction data generated after the execution of the actions to the server 203. These environmental interaction data generated in the middle can be used by the server 203 to repeat the aforementioned training steps to achieve continuous optimization of the agent network.

[0051] exist Figure 2 In the application scenario shown, the target object control method according to some embodiments of the present application is implemented on the server 203, but this is only illustrative and not restrictive. The target object control method according to some embodiments of the present application can also be implemented on other entities with sufficient computing resources and computing capabilities, for example, it can be implemented on the target object 201 with sufficient computing resources and computing capabilities. Of course, it can also be partially implemented on the server 203 and partially implemented on the target object 201, which is not restrictive.

[0052] As understood by those skilled in the art, an instance of server 203 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communications, which is not limited in this application.

[0053] Figure 3 A flow chart of a target object control method according to some embodiments of the present application is shown. The method 300 shown can be performed on a server (for example, Figure 2 In some embodiments, when the target object 201 has sufficient computing resources and computing power, the target object control method according to some embodiments of the present application can be directly executed on the target object 201. In other embodiments, the target object control method according to some embodiments of the present application can also be executed by the server 203 and the target object 201 in combination. Figure 3 As shown, the target object control method according to some embodiments of the present application may include steps S301-S306.

[0054] In step S301, multiple environmental interaction data from the agent network are obtained and a candidate sample data set is determined based on the multiple environmental interaction data, wherein each of the multiple environmental interaction data includes at least an environmental state, a target value, and an execution action, and the candidate sample data set includes at least a portion of the environmental interaction data, and the agent network includes an executor network and an evaluation network, and the executor network is used to determine the execution action based on the environmental state and the target value, and the evaluation network includes multiple sub-evaluation networks, each of which is used to evaluate the executor network based on the environmental state, the target value, and the execution action. In this embodiment, in order to calculate the sampling priority of each candidate sample data, multiple sub-evaluation networks are built, and the output value of each sub-evaluation network is a factor in determining the sampling priority of the candidate sample data. Regarding the establishment of the agent network, for example, multiple sub-evaluation networks of the executor network and the evaluation network can be established according to the deterministic policy gradient algorithm (Deterministic Deep Policy Gradient, DDPG). The consistency of the output value indicates the similarity of each output value. For example, the consistency between output values ​​2 and 4 is higher than the consistency between output values ​​2 and 5.

[0055] In some embodiments, each environmental interaction data also includes a fed-back environmental state; and wherein obtaining multiple environmental interaction data from an intelligent agent network and determining a candidate sample data set based on the multiple environmental interaction data include: obtaining multiple environmental interaction data from an intelligent agent network; for each environmental interaction data of at least part of the multiple environmental interaction data, modifying a target value of the environmental interaction data based on the fed-back environmental state of the environmental interaction data; and updating the multiple environmental interaction data based on the modified at least part of the environmental interaction data to determine a candidate sample data set.

[0056] exist Figure 1 In the embodiment shown, if the value of the feedback environment state meets the target value, the reward value is 0, and if the value of the feedback environment state does not meet the target value, the reward value is -1. Figure 1 In the illustrated embodiment, for the generated environmental interaction data, the coordinate value of the environmental state point B reached can be directly used as the target value, so that the reward value of the candidate sample data becomes 0. In addition, the target value can be modified according to the environmental state of the feedback of the environmental interaction data corresponding to any step in the fifty-step execution operation from the environmental state of the target object 201 to the environmental state point B reached, so that the reward value of the environmental interaction data is also 0. These modified environmental interaction data are all used as candidate sample data. In order to reduce the number of candidate sample data, only one modification can be made for each environmental interaction data.

[0057] Since this embodiment can be applied to sparse reward environments, the learning efficiency of the agent network can be significantly improved by modifying the environment interaction data.

[0058] In step S302, each candidate sample data in the candidate sample data set is input into multiple sub-evaluation networks of the evaluation network, and the sampling priority of the candidate sample data is determined by using the consistency of the output values ​​of each sub-evaluation network. The output value corresponding to the candidate sample data has a higher consistency, which means that it also has a lower uncertainty. Using the candidate sample data for training can effectively improve the performance of the agent network. Therefore, the consistency of the output value corresponding to the candidate sample data is a good indicator for determining the sampling priority of the training data.

[0059] In step S303, the candidate sample data set is sampled according to the sampling priority of each candidate sample data. Because the consistency of the output value corresponding to the candidate sample data is a good indicator for determining the sampling priority of the training data, the candidate sample data with a high sampling priority is more likely to be replayed into the training process first. In practice, there may be other indicators for improving the performance of the intelligent agent network. Therefore, when selecting the candidate sample data for training the intelligent agent network, it is carried out in a sampling manner, that is, the candidate sample data with a low sampling priority is not excluded for the training of the intelligent agent.

[0060] In step S304, the agent network is trained based on the sampled candidate sample data. Since the evaluation network can evaluate the performer network, the evaluation network can be used to adjust the parameters of the performer network.

[0061] In step S305, the target object is controlled according to the actor network in the trained agent network. In practice, the evaluation function of the evaluation network is no longer needed. For the sake of simplicity, only the actor network can be used to control the target object.

[0062] In the target object control device according to some embodiments of the present application, since in a sparse reward environment, candidate sample data with high consistency also has low uncertainty, reflecting the high learning value of the execution action, such data can significantly improve the control performance of the intelligent agent network. This method designs the intelligent agent network, uses the consistency of the output values ​​of multiple sub-evaluation networks of the evaluation network of the intelligent agent network to measure the sampling priority of each candidate sample data, and preferentially uses candidate sample data with high sampling priority to train the intelligent agent network, which can significantly improve the performance of the intelligent agent network. In addition, replaying the candidate sample data according to the consistency of the output value corresponding to the candidate sample data does not limit the application scope of the intelligent agent network. The training method has a high generalization ability and can adapt to various environments without requiring modification or design of the environment.

[0063] In some embodiments, the executor network of the agent network includes multiple sub-executor networks; and each of the multiple sub-executor networks includes a feature extraction layer and a prediction layer, the feature special area layer is used to extract features of candidate sample data, and the prediction layer is used to determine the execution action according to the features of the candidate sample data; and the multiple sub-executor networks share the same feature extraction layer. Since the multiple sub-executor networks of the agent network share the same feature extraction layer, the prediction objects of the prediction layers of the multiple sub-executor networks of the agent network come from the same feature extraction layer, and during the training process of the multiple sub-executor networks, the parameters of the feature extraction layer are adjusted. In some embodiments, the feature extraction layer can be a two-layer fully connected layer.

[0064] In some embodiments, the output value of each sub-evaluation network is a predicted reward value, and each candidate sample data in the candidate sample data set is respectively input into multiple sub-evaluation networks of the evaluation network, and the sampling priority of the candidate sample data is determined by using the consistency of the output values ​​corresponding to the candidate sample data, including: for each candidate sample data in the candidate sample data set, the following steps are performed respectively: the candidate sample data is respectively input into multiple sub-evaluation networks of the evaluation network to obtain multiple predicted reward values; the sampling priority of the candidate sample data is determined by using the consistency of the multiple predicted reward values ​​corresponding to the candidate sample data.

[0065] Figure 4 A schematic diagram of an agent network according to some embodiments of the present application is shown. Figure 4 It shows a feature extraction layer 401 commonly used by n sub-executor networks, where n can be any natural number greater than 1, a prediction layer 402 of the first sub-executor network, a prediction layer 403 of the n-th sub-executor network, a feature extraction layer 404 common to n sub-evaluation networks, a prediction layer 405 of the first sub-evaluation network, and a prediction layer 406 of the n-th sub-evaluation network.

[0066] In this embodiment, the environment state s and the target value g can be input into the feature extraction layer 401 common to the n sub-executor networks. The feature extraction layer 401 extracts the features of these data and transmits the extracted features to the prediction layers 401, 402, etc. of the executor network of the intelligent agent network. The prediction layer 402 of the first sub-executor network determines the execution action a1 according to the extracted features, and the prediction layer 403 of the nth sub-executor network determines the execution action an according to the extracted features.

[0067] In addition, the execution action a, environment state s, and target value g can be input into the feature extraction layer 404 common to the n sub-evaluation networks. The feature extraction layer 401 extracts the features of these data and transmits the extracted features to the prediction layers 405, 406, etc. of the evaluation network of the intelligent agent network. The prediction layer 405 of the first sub-evaluation network predicts the reward value r1 based on the extracted features, and the prediction layer 406 of the nth sub-evaluation network predicts the reward value rn based on the extracted features.

[0068] Regarding how to obtain multiple environmental interaction data, in some embodiments, the execution action of each environmental interaction data is determined by the actor network in the agent network according to the environmental state and target value of the environmental interaction data. Figure 4Randomly select a sub-executor network from the executor networks of the agent network shown. For example, select the k-th sub-executor network, where 1 < k < n. Use the k-th sub-executor network to interact with the environment by controlling the target object according to the input environmental state and target value, and continuously make a number of execution actions. A number of environmental interaction data generated in this process can be directly used as the multiple environmental interaction data obtained in step S301.

[0069] Determining the sampling priority of candidate sample data can have different implementation manners. In some embodiments, the consistency of multiple predicted reward values corresponding to candidate sample data is used to determine the sampling priority of candidate sample data, including: determining the variance of multiple predicted reward values of candidate sample data as the variance of candidate sample data; determining the sampling priority of candidate sample data according to the variance of candidate sample data, where the sampling priority of candidate sample data is negatively correlated with the corresponding variance. In this embodiment, since multiple sub-evaluation networks of the evaluation network of the agent network are randomly initialized, the reward values predicted by each sub-evaluation network for the same candidate sample data are also inconsistent. Therefore, the sampling priority of candidate sample data can be determined by the consistency of these predicted reward values.

[0070] Further, in some embodiments, determining the sampling priority of candidate sample data according to the variance of candidate sample data includes: taking the difference between the maximum variance among the variances of each candidate sample data in the candidate sample data set and the variance of candidate sample data as the intermediate sampling priority of candidate sample data; taking the sum of the intermediate sampling priorities of each candidate sample data in the candidate sample data set as the total sampling priority; determining the sampling priority of candidate sample data based on the quotient of the intermediate sampling priority of candidate sample data and the total sampling priority. As a mathematical tool, variance can better reflect the consistency of data. Therefore, for multiple predicted reward values corresponding to each candidate sample data, the variance of these multiple predicted reward values can be calculated to more accurately determine the sampling priority of candidate sample data. The larger the variance, the greater the deviation of multiple predicted reward values from the average value and the more inconsistent they are. The smaller the variance, the smaller the deviation of multiple predicted reward values from the average value and the more consistent they are. Therefore, a negative correlation should be established between the sampling priority of candidate sample data and the variance of multiple predicted reward values of candidate sample data. As an example, the reciprocal of the corresponding variance can be used as the sampling priority of candidate sample data.

[0071] Further, Figure 5 The flowchart shows the determination of the sampling priority of candidate sample data according to some embodiments of the present application, including steps S501 - S503. This embodiment is an extension of step S302.

[0072] In step S501, the difference between the maximum variance of the variances of the candidate sample data set and the variance of the candidate sample data is taken as the intermediate sampling priority of the candidate sample data. Assuming that the maximum variance of the variances of the candidate sample data set is a, for the variance b of any candidate sample data m in the candidate sample data set, the intermediate sampling priority of the candidate sample data m is ab. The coefficient of the variance of each candidate sample data is -1, so the intermediate sampling priority of each candidate sample data is negatively correlated with the corresponding variance. Adding the variance of the candidate sample data to the maximum variance of the variance of the candidate sample data set can ensure that the intermediate sampling priority of each candidate sample data is greater than or equal to zero, which can ensure that the sampling probability is a positive value.

[0073] In step S502, the sum of the intermediate sampling priorities of the plurality of candidate sample data is taken as the total sampling priority, which indicates the overall consistency of the plurality of candidate sample data.

[0074] In step S503, the sampling priority of the candidate sample data is determined based on the quotient of the intermediate sampling priority of the candidate sample data and the total sampling priority. In this way, the sampling priority of each candidate sample data is determined as the percentage of the intermediate sampling priority to the total sampling priority, and its value is between 0 and 100%, and its sum is 1, so it can be directly used as the sampling probability.

[0075] Further, in some embodiments, the sampling priority of the candidate sample data is determined based on the quotient of the intermediate sampling priority of the candidate sample data and the total sampling priority, including: using a preset parameter as a power, performing a power operation on the quotient of the intermediate sampling priority of the candidate sample data and the total sampling priority as the sampling priority of the candidate sample data, wherein the preset parameter is greater than zero. The preset parameter plays a role in adjusting the discrimination degree. Specifically, when the preset parameter is set to a value between 0 and 1, the discrimination degree of the sampling priorities between the candidate sample data is not obvious, and the gaps between the sampling priorities are relatively close. When the preset parameter is set to a value greater than 1, the discrimination degree between the sampling priorities of the candidate sample data is obvious, and the gaps between the sampling priorities are relatively large. Different preset parameters can be selected according to specific circumstances to adjust the discrimination degree between the sampling priorities of the candidate sample data.

[0076] In some embodiments, the sampling priority of the candidate sample data may be determined according to formula (1):

[0077] (1)

[0078] in is the sampling priority of candidate sample data i, A is the candidate sample data set, is the maximum variance among the variances of the candidate sample data sets. is the variance of candidate sample data i, is a minimum value, used to prevent the numerator from being zero. is the sum of the intermediate sampling priorities of each candidate sample data in the candidate sample data set A and the minimum value The sum of is used as the total sampling priority.

[0079] After determining the sampling priorities of multiple candidate sample data, the sum of the sampling priorities of the multiple candidate sample data may be greater than 1, or may be between 0 and 1, which is defective for sampling and may result in the failure to extract any candidate sample data. In some embodiments, the candidate sample data set is sampled according to the sampling priority of each candidate sample data, including: normalizing the sampling priority of the candidate sample data set to obtain the normalized result of each candidate sample data; using the normalized result of each candidate sample data as the sampled probability of the candidate sample data; sampling the candidate sample data set according to the sampled probability of each candidate sample data. After the normalization operation, the sampling priorities of each candidate sample data are between 0 and 1, and their sum is 1, which can be directly used as the sampling probability.

[0080] In some embodiments, each environmental interaction data also includes a reward value; and wherein the agent network is trained based on the sampled candidate sample data, including: for each agent network, the following steps are performed respectively: according to the environmental state and target value of the sampled candidate sample data, the agent network's actor network is used to calculate the agent network's execution action; according to the agent network's execution action, the sampled candidate sample data's environmental state and target value, the agent network's evaluation network's multiple sub-evaluation networks are used to predict the reward value of the agent network's execution action; based on the predicted reward value and the sampled candidate sample data's reward value, the evaluation network's parameters and the corresponding actor network's parameters are adjusted. In this embodiment, although the sampled candidate sample data includes an execution action, the execution action is not used to be input into any sub-evaluation network to predict the reward value, but the execution action determined by the actor network is input into the sub-evaluation network.

[0081] Figure 6 A schematic diagram of a target object control method according to some embodiments of the present application is shown.

[0082] In this embodiment, first, the executor network in the agent network 604 controls the target object to interact with the environment according to the environment state and the target value, thereby obtaining a set of environment interaction data, which form multiple environment interaction data 602. Then, part of the environment interaction data in the environment interaction data 602 is modified to obtain a candidate sample data set 603. Then, the sampling priority of each candidate sample data in the candidate sample data set 603 is calculated and sampled according to the sampling priority, and the sampled candidate sample data is used to train the agent network 604. During the training process, the execution action generated by the executor network of the agent network 604 can be used to control the target object 601, thereby generating multiple environment interaction data, which can be used to repeat the above process to achieve continuous optimization of the agent network 604.

[0083] Figures 7A-7J A schematic diagram showing the application of the target object control method according to an embodiment of the present application in various scenarios is shown, wherein: Fig. 7A As shown in FIG. 1 , it is a scene of a robot hand reaching (HandReach-v0), in which the actor network controls the robot hand to use its thumb and a selected robot finger to touch the shadow 701, where the target object is the robot hand. Figure 7B The figure shows a schematic diagram of a robot controlling a block. The target object here is the robot. If the scene has a moving position set (HandManipulateBlockFull-v0), the executor network controls the robot to manipulate a block 702 until it reaches a desired moving position and rotation angle. If the scene has not set a moving position and is set to rotate only in the vertical direction (HandManipulateBlockZ-v0), the executor network controls the robot to manipulate a block 702 to rotate to the target angle in the vertical direction. If the scene has not set a moving position and is set to rotate only in the parallel direction (HandManipulateBlockParallel-v0), the executor network controls the robot to manipulate a block 702 to rotate to the target angle in the horizontal direction. If the scene has not set a moving position and is set to rotate in three dimensions (HandManipulateBlockXYZ-v0), the executor network controls the robot to manipulate a block 702 to rotate to the target angle in any direction. Figure 7CAs shown in FIG. 1 , it is a schematic diagram of a robot hand manipulating an egg or a sphere. The target object here is the robot hand. If the scene with a specified moving position (HandManipulateEggFull-v0), the executor network controls the robot hand to manipulate an egg or a sphere 703 until it reaches an ideal moving position and rotation angle. If the scene with no specified moving position (HandManipulateEggRotate-v0), it only manipulates an egg or a sphere 703 to reach an ideal rotation angle. Fig.7D , is a schematic diagram of a robot hand manipulating a pen or a stick, where the target object is the robot hand. If the scene is a specified moving position (HandManipulatePenFull-v0), the executor network controls the robot hand to manipulate a pen or a stick 704 until it reaches an ideal moving position and rotation angle. If the scene is not a specified moving position (HandManipulatePenRotate-v0), only the pen or the stick 704 is manipulated to reach an ideal rotation angle. Fig. 7E This is a scene where the robot reaches a certain position (FetchReach), where the target object is the robot, and the executor network controls the robot's end effector 705 to move to the required moving position. Figure 7F This is a robot sliding scenario (FetchSlide). The executor network controls the robot to move in a certain direction and rest on the desired target, where the target object is the robot. Figure 7G This is a robot push scenario (FetchPush), where the executor network controls the robot to move a box 706 until the box 706 reaches the desired moving position, and the target object here is the robot. Figure 7H This is a robot picking scenario (FetchPickAndPlace), where the actor network controls the robot to pick up a box 707 from the table with its gripper and move the box 707 to a moving position above the table. The target object here is the robot. Fig.7I It is a two-dimensional scene without walls (PointMassEmptyEnv-v1). The robot can reach any point in the two-dimensional scene. The target point is a specified point in the two-dimensional scene. The executor network controls the robot to reach the target point. The target object here is the robot. Figure 7J It is a two-dimensional scene with a middle wall (PointMassEmptyEnv-v1). There is a wall in the middle of the two-dimensional scene. The executor network controls the robot to reach the target point, but cannot control the robot to directly cross the wall to reach the target point. The target object here is the robot.

[0084] Fig. 8A-D shows a schematic diagram of the effect of the target object control method in different test scenarios according to some embodiments of the present application. Fig. 8A -D, each sub-graph represents the effect curve graph under a test scenario. Test scenario 1 corresponds to Fig.7I Test scenario 2 corresponds to Figure 7J Test scenario 3 is a robot arm scenario (Reacher-v2), where the executor network controls the robot arm to reach the target position. Test scenario 4 corresponds to Fig. 7E Test scenario 5 corresponds to Figure 7G Test scenario 6 corresponds to Figure 7F Test scenario 7 corresponds to Figure 7H Test scenario 8 corresponds to Fig. 7A Test scenario 9 corresponds to Fig.7D The scene without moving position (HandManipulatePenRotate-v0). Test scene 10 corresponds to Figure 7C The scene without moving position (HandManipulateEggRotate-v0). Test scene 11 corresponds to Figure 7C The scene with moving position (HandManipulatePenFull-v0). Test scene 12 corresponds to Fig.7D The scene with moving position (HandManipulateEggFull-v0). Test scene 13 corresponds to Figure 7B The scene with moving position (HandManipulateBlockFull-v0). Test scene 14 corresponds to Figure 7B Test scene 15 corresponds to the test scene with no moving position and only vertical rotation (HandManipulateBlockZ-v0). Figure 7B The scene with no moving position and three-dimensional rotation (HandManipulateBlockXYZ-v0). Test scene 16 corresponds to Figure 7BThe scene with no moving position and only parallel rotation (HandManipulateBlockParallel-v0). In test scenarios 1-16, the ordinate of each subgraph represents the success rate, the abscissa represents the number of steps of the target object, the effect marked by the solid line is the effect of the executor network of the present application, and the effects marked by other dotted lines are the effects of the HER, HEREBP, CHER, and VDS technical solutions. It can be seen from the figure that the executor network obtained by the target object control method according to some embodiments of the present application has good performance in various test scenarios compared with other technical solutions, has strong generalization ability, and does not require modification or design of the environment.

[0085] Fig. 9 An exemplary structural block diagram of a target object control device 900 according to some embodiments of the present application is shown. There is a changeable association relationship between the device 900 and at least one slot, each of the at least one slot has a corresponding client, the device stores version information for each slot, and allocation information related to the allocation of the serial number corresponding to each slot, and the device 900 includes: an acquisition module 901, a determination module 902, a sampling module 903, a training module 904, and a control module 905. The acquisition module 901 is configured to acquire multiple environmental interaction data from an intelligent agent network and determine a candidate sample data set based on the multiple environmental interaction data, wherein each of the multiple environmental interaction data includes at least an environmental state, a target value, and an execution action, and the candidate sample data set includes at least a portion of the environmental interaction data, the intelligent agent network includes an executor network and an evaluation network, the executor network is used to determine the execution action based on the environmental state and the target value, and the evaluation network includes multiple sub-evaluation networks, each sub-evaluation network is used to evaluate the executor network based on the environmental state, the target value, and the execution action. The determination module 902 is configured to input each candidate sample data in the candidate sample data set into multiple sub-evaluation networks of the evaluation network, and determine the sampling priority of the candidate sample data using the consistency of the output values ​​corresponding to the candidate sample data. The sampling module 903 is configured to sample the candidate sample data set according to the sampling priority of each candidate sample data. The training module 904 is configured to train the agent network based on the sampled candidate sample data. The control module 905 is configured to control the target object according to the executor network in the trained agent network.

[0086] It should be noted that the various modules described above can be implemented in software or hardware or a combination of the two. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0087] In the target object control device according to some embodiments of the present application, since in a sparse reward environment, candidate sample data with high consistency also has low uncertainty, reflecting a high learning value of the execution action, such data can significantly improve the control performance of the intelligent network. The device designs the intelligent network, uses the consistency of the output values ​​of multiple sub-evaluation networks of the evaluation network of the intelligent network to measure the sampling priority of each candidate sample data, and preferentially uses candidate sample data with high sampling priority to train the intelligent network, which can significantly improve the performance of the intelligent network. In addition, replaying the candidate sample data according to the consistency of the output value corresponding to the candidate sample data does not limit the application scope of the intelligent network. The training method has a high generalization ability and can adapt to various environments without requiring modification or design of the environment.

[0088] Fig.10 An example system 1000 is illustrated, which includes an example computing device 1010 that represents one or more systems and / or devices that can implement the various methods described herein. Computing device 1010 can be, for example, a server of a service provider, a device associated with a server, a system on a chip, and / or any other suitable computing device or computing system. Fig. 9 The depicted target object control apparatus 900 may take the form of a computing device 1010. Alternatively, the target object control apparatus 900 may be implemented as a computer program in the form of an application 1016.

[0089] The example computing device 1010 as shown includes a processing system 1011, one or more computer-readable media 1012, and one or more I / O interfaces 1013 that are communicatively coupled to each other. Although not shown, the computing device 1010 may also include a system bus or other data and command transmission system that couples various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures. Various other examples are also contemplated, such as control and data lines.

[0090] Processing system 1011 represents the functionality of performing one or more operations using hardware. Thus, processing system 1011 is illustrated as including hardware elements 1014 that may be configured as processors, functional blocks, and the like. This may include other logic devices implemented in hardware as application specific integrated circuits or formed using one or more semiconductors. Hardware elements 1014 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0091] Computer-readable media 1012 is illustrated as including memory / storage 1015. Memory / storage 1015 represents memory / storage capacity associated with one or more computer-readable media. Memory / storage 1015 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 1015 may include fixed media (e.g., RAM, ROM, fixed hard drives, etc.) and removable media (e.g., flash memory, removable hard drives, optical disks, etc.). Computer-readable media 1012 may be configured in various other ways as further described below.

[0092] One or more I / O interfaces 1013 represent functionality that allows a user to input commands and information to the computing device 1010 using various input devices and optionally also allows information to be presented to the user and / or other components or devices using various output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, a touch function (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., which can detect motion that does not involve touch as a gesture using visible or invisible wavelengths (such as infrared frequencies)), and the like. Examples of output devices include a display device, a speaker, a printer, a network card, a tactile response device, and the like. Thus, the computing device 1010 can be configured in various ways as further described below to support user interaction.

[0093] The computing device 1010 also includes an application 1016. The application 1016 may be, for example, a software instance of the target object control apparatus 900 and, in combination with other elements in the computing device 1010, implement the techniques described herein.

[0094] The present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes the time-shifted playback method implemented by the first party or the second party provided in the above various optional implementations.

[0095] Various techniques may be described herein in the general context of software hardware elements or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module", "function" and "component" as used herein generally represent software, firmware, hardware or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0096] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1010. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media”.

[0097] As opposed to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing the desired information and accessible by a computer.

[0098] "Computer-readable signal media" refers to signal-bearing media that is configured to send instructions to the hardware of the computing device 1010, such as via a network. Signal media typically can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed so as to encode information into the signal. By way of example and not limitation, communication media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared, and other wireless media.

[0099] As before, hardware elements 1014 and computer-readable media 1012 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements may include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices that perform program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0100] The aforementioned combination may also be used to implement various techniques and modules herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1014. Computing device 1010 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 1014 of a processing system, a module may be implemented as a module executable by computing device 1010 as software, at least in part, in hardware. Instructions and / or functions may be executable / operable by one or more articles (e.g., one or more computing devices 1010 and / or processing systems 1011) to implement the techniques, modules, and examples described herein.

[0101] In various embodiments, computing device 1010 can be implemented in various configurations. For example, computing device 1010 can be implemented as a computer-type device including a personal computer, a desktop computer, a multi-screen computer, a laptop computer, a netbook, etc. Computing device 1010 can also be implemented as a mobile device-type device including mobile devices such as mobile phones, portable music players, portable game devices, tablet computers, multi-screen computers, etc. Computing device 1010 can also be implemented as a television-type device, which includes a device with or connected to a generally larger screen in a casual viewing environment. These devices include televisions, set-top boxes, game consoles, etc.

[0102] The techniques described herein may be supported by these various configurations of computing device 1010 and are not limited to the specific examples of the techniques described herein. Functionality may also be implemented in whole or in part on the "cloud" 1020 by using a distributed system, such as through platform 1022 as described below.

[0103] Cloud 1020 includes and / or represents a platform 1022 for resources 1024. Platform 1022 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 1020. Resources 1024 may include applications and / or data that may be used when performing computer processing on a server remote from computing device 1010. Resources 1024 may also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network.

[0104] The platform 1022 can abstract resources and functionality to connect the computing device 1010 with other computing devices. The platform 1022 can also be used to abstract the hierarchy of resources to provide a hierarchy of corresponding levels of demand encountered for resources 1024 implemented via the platform 1022. Therefore, in an interconnected device embodiment, the implementation of the functionality described herein can be distributed throughout the system 1000. For example, the functionality can be implemented partially on the computing device 1010 and through the platform 1022 that abstracts the functionality of the cloud 1020.

[0105] The present disclosure provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed, any of the above methods is implemented.

[0106] The present disclosure provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes any method provided in the above-mentioned various optional implementations.

[0107] It should be understood that, for the sake of clarity, embodiments of the present application are described with reference to different functional units. However, it will be apparent that, without departing from the present application, the functionality of each functional unit can be implemented in a single unit, in multiple units or as a part of other functional units. For example, the functionality that is described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present application can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0108] Although the present application has been described in conjunction with some embodiments, it is not intended to be limited to the specific form set forth herein. On the contrary, the scope of the present application is limited only by the appended claims. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and the inclusion in different claims does not imply that the combination of features is not feasible and / or advantageous. The order of features in the claims does not imply any specific order in which the features must work. In addition, in the claims, the word "comprising" does not exclude other elements, and the term "one" or "an" does not exclude a plurality. The figure marks in the claims are provided only as clear examples and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. A target object control method, comprising: Acquire multiple environmental interaction data from an agent network and determine a candidate sample data set based on the multiple environmental interaction data, wherein each of the multiple environmental interaction data includes at least an environmental state, a target value, and an execution action, and the candidate sample data set includes at least a portion of the environmental interaction data, the agent network includes an actor network and an evaluation network, the actor network is used to determine the execution action based on the environmental state and the target value, and the evaluation network includes multiple sub-evaluation networks, each sub-evaluation network is used to evaluate the actor network based on the environmental state, the target value, and the execution action; Input each candidate sample data in the candidate sample data set into a plurality of sub-evaluation networks of the evaluation network respectively, and determine the sampling priority of the candidate sample data by using the consistency of the output values ​​of each sub-evaluation network; Sampling the candidate sample data set according to the sampling priority of each candidate sample data; Training the agent network based on the sampled candidate sample data; The target object is controlled according to the actor network in the trained agent network.

2. The method according to claim 1, wherein each of the plurality of environmental interaction data further includes a feedback environmental state; And wherein the step of acquiring a plurality of environmental interaction data from the agent network and determining a candidate sample data set based on the plurality of environmental interaction data comprises: Obtaining multiple environment interaction data from the agent network; For each environmental interaction data of at least part of the plurality of environmental interaction data, modifying a target value of the environmental interaction data based on an environmental state fed back by the environmental interaction data; The plurality of environment interaction data are updated based on the modified at least part of the environment interaction data to determine the candidate sample data set.

3. The method according to claim 1 or 2, wherein the output value of each sub-evaluation network is a predicted reward value, and wherein each candidate sample data in the candidate sample data set is input into a plurality of sub-evaluation networks of the evaluation network respectively, and the sampling priority of the candidate sample data is determined by using the consistency of the output values ​​of each sub-evaluation network, comprising: For each candidate sample data in the candidate sample data set, perform the following steps respectively: Inputting the candidate sample data into multiple sub-evaluation networks of the evaluation network respectively to obtain multiple predicted reward values; The sampling priority of the candidate sample data is determined by utilizing the consistency of multiple predicted reward values ​​corresponding to the candidate sample data.

4. The method according to claim 3, wherein the determining the sampling priority of the candidate sample data by using the consistency of multiple predicted reward values ​​corresponding to the candidate sample data comprises: Determining a variance of a plurality of predicted reward values ​​of the candidate sample data as a variance of the candidate sample data; The sampling priority of the candidate sample data is determined according to the variance of the candidate sample data, wherein the sampling priority of the candidate sample data is negatively correlated with the corresponding variance.

5. The method according to claim 4, wherein determining the sampling priority of the candidate sample data according to the variance of the candidate sample data comprises: taking the difference between the maximum variance among the variances of the candidate sample data in the candidate sample data set and the variance of the candidate sample data as the intermediate sampling priority of the candidate sample data; The sum of the intermediate sampling priorities of the candidate sample data of the candidate sample data set is taken as the total sampling priority; The sampling priority of the candidate sample data is determined based on the quotient of the intermediate sampling priority of the candidate sample data and the total sampling priority.

6. The method according to claim 5, wherein determining the sampling priority of the candidate sample data based on the quotient of the intermediate sampling priority of the candidate sample data and the total sampling priority comprises: A preset parameter is used as a power, and a power operation is performed on the quotient of the intermediate sampling priority and the total sampling priority of the candidate sample data to obtain the sampling priority of the candidate sample data, wherein the preset parameter is greater than zero.

7. The method according to claim 1 or 2, wherein sampling the candidate sample data set according to the sampling priority of each candidate sample data comprises: Normalizing the sampling priorities of the candidate sample data sets to obtain a normalized result of each candidate sample data; The normalized result of each candidate sample data is used as the sampling probability of the candidate sample data; The candidate sample data set is sampled according to the sampling probability of each candidate sample data.

8. The method according to claim 3, wherein each environmental interaction data further includes a reward value; And wherein the agent network is trained based on the sampled candidate sample data, comprising: For each agent network, perform the following steps: According to the environment state and target value of the sampled candidate sample data, using the actor network of the agent network to calculate the execution action of the agent network; According to the execution action of the agent network, the environmental state and the target value of the sampled candidate sample data, using multiple sub-evaluation networks of the evaluation network of the agent network to predict the reward value of the execution action of the agent network; Based on the predicted reward value and the reward value of the sampled candidate sample data, the parameters of the evaluation network and the parameters of the corresponding performer network are adjusted.

9. A method according to claim 1 or 2, wherein the execution action of each environmental interaction data is determined by an executor network in the agent network according to the environmental state and target value of the environmental interaction data.

10. The method according to claim 1 or 2, wherein the performer network comprises a plurality of sub-performer networks; The plurality of sub-executor networks each include a feature extraction layer and a prediction layer, the feature extraction layer is used to extract features of the candidate sample data, and the prediction layer is used to determine an execution action according to the features of the candidate sample data; And wherein the multiple sub-executor networks share the same feature extraction layer. The method according to claim 10 , wherein the commonly used feature extraction layer comprises two fully connected layers.

12. A target object control device, comprising: an acquisition module configured to acquire a plurality of environmental interaction data from an agent network and determine a candidate sample data set based on the plurality of environmental interaction data, wherein each of the plurality of environmental interaction data at least includes an environmental state, a target value, and an execution action, the candidate sample data set includes at least a portion of the environmental interaction data, the agent network includes an actor network and an evaluation network, the actor network is used to determine an execution action based on the environmental state and the target value, the evaluation network includes a plurality of sub-evaluation networks, each sub-evaluation network is used to evaluate the actor network based on the environmental state, the target value, and the execution action; A determination module, configured to input each candidate sample data in the candidate sample data set into a plurality of sub-evaluation networks of the evaluation network, and determine the sampling priority of the candidate sample data by using the consistency of the output values ​​of each sub-evaluation network; A sampling module, configured to sample the candidate sample data set according to the sampling priority of each candidate sample data; A training module, configured to train the agent network based on the sampled candidate sample data; A control module is configured to control the target object according to the actor network in the trained agent network.

13. A computing device comprising: Memory and processor, A computer program is stored in the memory, and when the computer program is executed by the processor, the processor is prompted to perform the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-readable instructions, wherein the computer-readable instructions implement the method of any one of claims 1 to 11 when executed.

15. A computer program product comprising computer instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images