Network training, robot control method and device, equipment and storage medium

Abstract

The present disclosure provides a network training, robot control method and device, equipment and storage medium, wherein the training method includes: obtaining environmental state information in a target application scenario; obtaining action sequence information according to the environmental state information and a pre-trained reinforcement learning network, and determining the total reward value corresponding to the action sequence information, wherein the action sequence information is used to indicate at least two consecutive execution actions within a preset time period in the future; adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain a trained reinforcement learning network. In the present disclosure, the reinforcement learning network adjusted by the total reward value can better adapt to the generation of action sequences, so that as the reinforcement learning network is adjusted, the generated action sequence becomes better and better. When the trained reinforcement learning network is applied to complex scenarios such as those requiring continuous control, it can also have better control performance.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a network training, robot control method and apparatus, device and storage medium. Background Art

[0002] At present, deep reinforcement learning integrates the powerful understanding ability of deep learning in perceptual problems such as vision and the decision-making ability of reinforcement learning, enabling some complex problems in real-life scenarios to be solved.

[0003] Taking robot control as an example, the decision-making ability of reinforcement learning can help the robot adaptively adjust the output actions according to the surrounding environmental information. Each time the robot interacts with the environment, it can be controlled to perform a corresponding action. As the number of environmental interactions increases, the actions performed become more accurate and more in line with the needs of the scene.

[0004] However, for some more complex control scenarios, such as controlling a robot to perform a set of continuous actions, the control performance corresponding to the above reinforcement learning method is poor. Summary of the invention

[0005] The embodiments of the present disclosure at least provide a network training, robot control method and apparatus, equipment and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a network training method, comprising:

[0007] Obtain environmental status information in the target application scenario;

[0008] Obtaining action sequence information according to the environment state information and the pre-trained reinforcement learning network, and determining a total reward value corresponding to the action sequence information, wherein the action sequence information is used to indicate at least two consecutive execution actions within a preset time period in the future;

[0009] The network parameter values ​​of the reinforcement learning network are adjusted based on the total reward value to obtain a trained reinforcement learning network; wherein the trained reinforcement learning network is used to obtain a target action sequence for continuous control of a target object, and the total reward value corresponding to the target action sequence is greater than a preset threshold.

[0010] By adopting the above network training method, the network parameter value of the reinforcement learning network can be adjusted based on the total reward value. Since the generated total reward value comprehensively evaluates the action relationship between at least two consecutive execution actions, the reinforcement learning network adjusted by the total reward value can better adapt to the generation of action sequences. In this way, as the reinforcement learning network is adjusted, the generated action sequence becomes better and better. When the trained reinforcement learning network is applied to complex scenarios such as those requiring continuous control, it can also have better control performance.

[0011] In an optional implementation manner, determining the total reward value corresponding to the action sequence information includes:

[0012] Applying the action sequence information to the target application scenario to obtain environment state sequence information and a reward value set corresponding to the action sequence information; the reward value set includes a reward value corresponding to each execution action when the at least two consecutive execution actions are executed sequentially;

[0013] Based on the environment state sequence information and the reward value set, the total reward value is obtained.

[0014] In this implementation, the corresponding environment state can be changed under the execution of each execution action, and the changed environment state can in turn act on the generation of the next execution action, and the cycle continues. Under the interaction between the environment and the action, the action sequence information composed of at least two actions and the environment state corresponding to the action and the corresponding environment state sequence information can make the total value of the generated return reward value comprehensively consider the interactive relationship between each action and each state, which makes the subsequent network adjustment more accurate.

[0015] In an optional implementation, obtaining the total reward value based on the environment state sequence information and the reward value set includes:

[0016] For each execution action included in the at least two consecutive execution actions, determine the environment state corresponding to the execution action and the reward value generated when the execution action is executed from the environment state sequence information and the reward value set; and determine the reward and value for the execution action based on the reward value and the environmental impact value under the environment state;

[0017] Based on the reward and value determined for each performed action respectively, the total reward value is determined.

[0018] In this implementation, for each execution action, a corresponding reward and value can be determined based on the corresponding reward value and the environmental impact value, and the reward and value can evaluate the quality of the execution action to a certain extent. In addition, considering that the quality of the current execution action will further affect the next environmental state and its corresponding next execution action, that is, the multiple continuous execution actions included in the entire action sequence influence each other. Under the mutual influence, the total reward value corresponding to each determined reward and value can determine a more accurate next action sequence, thereby achieving better and better control performance.

[0019] In an optional implementation, determining the total reward value based on the reward and value determined for each execution action includes:

[0020] Get the action weight value assigned to each execution action;

[0021] The total reward value is determined based on the reward and value determined for each executed action and the action weight value assigned to each action.

[0022] In this implementation, the action weight value is used to determine the reward and value. In this way, when determining the total reward value, the influence of different execution actions is taken into account, and the determined total reward value is more in line with the needs of the actual scene.

[0023] In an optional implementation, the action sequence information is used to indicate N consecutive execution actions within a future preset time period, where N is an integer greater than or equal to 2, and the action sequence information is obtained according to the environmental state information and the pre-trained reinforcement learning network, including:

[0024] Determine an nth execution action according to the pre-trained reinforcement learning network and the environmental state of the target application scenario at the n-1th moment, where n is an integer and 0<n≤N;

[0025] Among them, when n is 1, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state information; when 1<n≤N, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state of the target application scenario at the n-2th moment and the n-1th execution action.

[0026] In this implementation, when determining the action sequence information, the mutual influence between each environmental state and each execution action is taken into consideration, so that each execution action determined can better adapt to the environment.

[0027] In an optional implementation, adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain a trained reinforcement learning network includes:

[0028] The following steps are executed repeatedly until the total reward value corresponding to the target action sequence output by the trained reinforcement learning network is greater than a preset threshold;

[0029] Adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain an adjusted reinforcement learning network; and applying the action sequence information to the target application scenario to obtain environment state sequence information corresponding to the action sequence information;

[0030] The last environmental state information included in the environmental state sequence information is input into the adjusted reinforcement learning network to obtain the action sequence information output by the reinforcement learning network for executing multiple continuous execution actions within a preset time period in the future, and the total value of the return reward generated when executing the action sequence information.

[0031] In this implementation, through multiple rounds of network training, the action sequence information output by the reinforcement learning network can be made increasingly close to the needs of the actual scene, so that the output action sequence provides a basis for ensuring accurate continuous control.

[0032] In a second aspect, an embodiment of the present disclosure provides a robot control method, comprising:

[0033] Obtain the current environment status information of the target robot;

[0034] The current environment state information is input into a reinforcement learning network trained by any of the network training methods described in the first aspect to obtain a target action sequence for continuously controlling the target robot.

[0035] By adopting the above robot control method, the trained reinforcement learning network can be used to perform fast, accurate and continuous control on the robot, and the control performance is better.

[0036] In an optional implementation manner, the method further includes:

[0037] When receiving an execution success instruction for a current execution action included in the target action sequence sent by the target robot, an action instruction for executing the next execution action of the current execution action is sent to the target robot.

[0038] In this implementation, in order to minimize the adverse effects of command confusion on the target robot, an instruction to execute the next action can be sent to the robot when the previous action is successfully executed, thereby ensuring the stability of robot control.

[0039] In a third aspect, the present disclosure also provides a network training device, including:

[0040] An acquisition module is used to obtain environmental status information in a target application scenario;

[0041] A training module, used to obtain action sequence information according to the environmental state information and the pre-trained reinforcement learning network, and determine the total reward value corresponding to the action sequence information, wherein the action sequence information is used to indicate at least two consecutive execution actions within a preset time in the future;

[0042] An adjustment module is used to adjust the network parameter values ​​of the reinforcement learning network based on the total reward value to obtain a trained reinforcement learning network; wherein the trained reinforcement learning network is used to obtain a target action sequence for continuous control of a target object, and the total reward value corresponding to the target action sequence is greater than a preset threshold.

[0043] In an optional implementation, the training module is used to determine the total reward value corresponding to the action sequence information according to the following steps:

[0044] Applying the action sequence information to the target application scenario to obtain environment state sequence information and a reward value set corresponding to the action sequence information; the reward value set includes a reward value corresponding to each execution action when the at least two consecutive execution actions are executed sequentially;

[0045] Based on the environment state sequence information and the reward value set, the total reward value is obtained.

[0046] In an optional implementation, the training module is used to obtain the total reward value based on the environment state sequence information and the reward value set according to the following steps:

[0047] For each execution action included in the at least two consecutive execution actions, determine the environment state corresponding to the execution action and the reward value generated when the execution action is executed from the environment state sequence information and the reward value set; and determine the reward and value for the execution action based on the reward value and the environmental impact value under the environment state;

[0048] Based on the reward and value determined for each performed action respectively, the total reward value is determined.

[0049] In an optional implementation, the training module is used to determine the total reward value based on the reward and value determined for each execution action according to the following steps:

[0050] Get the action weight value assigned to each execution action;

[0051] The total reward value is determined based on the reward and value determined for each execution action and the action weight value assigned to each execution action.

[0052] In an optional implementation, the action sequence information is used to indicate N consecutive execution actions within a future preset time period, where N is an integer greater than or equal to 2, and the training module is used to obtain the action sequence information according to the environment state information and the pre-trained reinforcement learning network in accordance with the following steps:

[0053] Determine an nth execution action according to the pre-trained reinforcement learning network and the environmental state of the target application scenario at the n-1th moment, where n is an integer and 0<n≤N;

[0054] Among them, when n is 1, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state information; when 1<n≤N, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state of the target application scenario at the n-2th moment and the n-1th execution action.

[0055] In an optional implementation, the adjustment module is used to adjust the network parameter value of the reinforcement learning network based on the total reward value according to the following steps to obtain a trained reinforcement learning network:

[0056] The following steps are executed repeatedly until the total reward value corresponding to the target action sequence output by the trained reinforcement learning network is greater than a preset threshold;

[0057] Adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain an adjusted reinforcement learning network; and applying the action sequence information to the target application scenario to obtain environment state sequence information corresponding to the action sequence information;

[0058] The last environmental state information included in the environmental state sequence information is input into the adjusted reinforcement learning network to obtain the action sequence information output by the reinforcement learning network for executing multiple continuous execution actions within a preset time period in the future, and the total value of the return reward generated when executing the action sequence information.

[0059] In a fourth aspect, the present disclosure also provides a robot control device, including:

[0060] An acquisition module is used to obtain the current environment status information of the target robot;

[0061] A control module is used to input the current environment state information into a reinforcement learning network trained by any network training method described in the first aspect to obtain a target action sequence for continuously controlling the target robot.

[0062] In an optional implementation, the device further includes:

[0063] The sending module is used to send an action instruction for executing the next execution action of the current execution action to the target robot when receiving an execution success instruction for the current execution action included in the target action sequence sent by the target robot.

[0064] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps in the above-mentioned first aspect, or any possible implementation of the first aspect, or the above-mentioned second aspect, or any possible implementation of the second aspect are performed.

[0065] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect, or the above-mentioned second aspect, or any possible implementation of the second aspect.

[0066] For a description of the effects of the above-mentioned network training device, robot control device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned network training method and robot control method, which will not be repeated here.

[0067] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0069] Figure 1 A flow chart of a network training method provided by an embodiment of the present disclosure is shown;

[0070] Figure 2 A flowchart showing a specific method for determining the total value of a return reward in the network training method provided by an embodiment of the present disclosure;

[0071] Figure 3 A flow chart of a robot control method provided by an embodiment of the present disclosure is shown;

[0072] Figure 4 A schematic diagram of a network training device provided by an embodiment of the present disclosure is shown;

[0073] Figure 5 A schematic diagram of the architecture of a robot control device provided by an embodiment of the present disclosure is shown;

[0074] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0076] Research has found that deep reinforcement learning integrates deep learning's powerful understanding of visual and other perceptual problems, as well as reinforcement learning's decision-making ability, to solve some complex problems in real-world scenarios. Taking robot control as an example, the decision-making ability of reinforcement learning can help robots adaptively adjust the output actions based on the surrounding environmental information, and each time the environment interacts, the robot can be controlled to perform a corresponding action. As the number of environmental interactions increases, the actions performed are more accurate and more in line with the needs of the scene.

[0077] However, for some more complex control scenarios, such as controlling a robot to perform a set of continuous actions, the control performance corresponding to the above reinforcement learning method is poor.

[0078] Based on the above research, the present disclosure provides a network training, robot control method and device, equipment and storage medium, which has better network training performance and can be well applied to complex scenarios such as the need for continuous control of the robot, and has better control performance.

[0079] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0080] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0081] To facilitate understanding of this embodiment, a network training method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the network training method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities, and the electronic device includes, for example: a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, or various types of robots such as industrial robots and mobile service robots. In some possible implementations, the network training method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0082] The network training method provided by the embodiment of the present disclosure is described below by taking a robot as an example.

[0083] See also Figure 1 FIG. 1 is a flowchart of a network training method provided by an embodiment of the present disclosure, and the method includes steps S101 to S103, wherein:

[0084] S101, obtaining environmental status information in a target application scenario;

[0085] S102, obtaining action sequence information according to the environment state information and the pre-trained reinforcement learning network, and determining the total reward value corresponding to the action sequence information, where the action sequence information is used to indicate at least two consecutive execution actions within a preset time in the future;

[0086] S103. Adjust the network parameter values ​​of the reinforcement learning network based on the total reward value to obtain a trained reinforcement learning network; wherein the trained reinforcement learning network is used to obtain a target action sequence for continuous control of the target object, and the total reward value corresponding to the target action sequence is greater than a preset threshold.

[0087] In order to facilitate understanding of the network training method provided by the embodiment of the present disclosure, the application scenario of the method is briefly described below. The above network training method can be mainly applied to the field of robot control, and is particularly suitable for control in complex scenarios such as continuous robot control. Here, the robot can be continuously controlled well using a trained reinforcement learning network. For example, the robot can be controlled to perform a series of actions including moving to position A, grabbing a target object, and moving the object to position B, with better control performance.

[0088] It should be noted that the embodiments of the present disclosure can be applied not only to the above-mentioned robot control field, but also to other decision-making intelligent technology fields that require continuous control, and no specific limitation is made here. Considering the wide application of robot technology, the following examples are mainly based on robot control application scenarios.

[0089] The target application scenarios in the embodiments of the present disclosure may be various application scenarios related to continuous control. Taking robot control as an example, different robots may correspond to different application scenarios. For example, it may be a scenario in which a transport robot performs cargo transport, or a scenario in which a comfort robot performs hugging, caressing, and the like. No specific limitation is made here.

[0090] In order to better train the reinforcement learning network, the environmental state in the target application scenario can be obtained in advance. As the environmental state is updated, the actions that the reinforcement learning network can control and execute will also change. This change can act on the entire environment and cause the environmental state to change. This cycle repeats in turn. During the training of the reinforcement learning network, the environment also changes accordingly.

[0091] Here, in order to achieve relevant continuous control, based on the acquired environmental state information, it can be determined that the reinforcement learning network outputs relevant action sequence information, which is for multiple continuous execution actions within a preset time in the future. As the environmental state changes, multiple continuous actions affect each other, so the total reward value determined can evaluate the quality of the action sequence including multiple continuous actions. The higher the total reward value, to a certain extent, it means that the output action sequence is more in line with the environmental requirements, and the guiding role of such action sequences can be strengthened in the future. On the contrary, the lower the total reward value, to a certain extent, it means that the output action sequence does not meet the environmental requirements, and the guiding role of such action sequences can be weakened in the future.

[0092] Here, the reinforcement learning network can be adjusted once or multiple times based on the total reward value. After multiple adjustments, the output target action sequence can obtain a larger total reward value. In this way, when the trained reinforcement learning network group is applied to specific robot control and other related control fields, better continuous control performance can be achieved.

[0093] It should be noted that in the process of adjusting the network parameter values ​​of the reinforcement learning network based on the total reward value, the embodiment of the present disclosure aims at optimizing each action in the entire action sequence. Each action influences each other, which ultimately makes the determined action sequence more in line with actual environmental requirements and more applicable.

[0094] The total reward value here takes into account the reward value corresponding to each execution action and the environmental state related to the execution action. It can be further explained through the following steps:

[0095] Step 1: Apply the action sequence information to the target application scenario to obtain the environment state sequence information and the reward value set corresponding to the action sequence information; the reward value set includes the reward value corresponding to each execution action when at least two consecutive execution actions are executed sequentially;

[0096] Step 2: Based on the environment state sequence information and the reward value set, the total reward value is obtained.

[0097] The action sequence information in the disclosed embodiment can be determined by continuous action prediction of the initial environmental state information based on a pre-trained reinforcement learning network, and is used to indicate N consecutive execution actions within a preset time period in the future, where N is an integer greater than or equal to 2.

[0098] In practical applications, the execution action at the current moment can be determined by the pre-trained reinforcement learning network and the environmental state of the target application scenario at the previous moment. That is, the nth execution action can be determined here based on the reinforcement learning network and the environmental state of the target application scenario at the n-1th moment, where n is an integer and 0<n≤N.

[0099] When n is 1, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state information; when 1<n≤N, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state of the target application scenario at the n-2th moment and the n-1th execution action.

[0100] In practical applications, the current environment state can be input into the reinforcement learning network, and the reinforcement learning network outputs an execution action; then, when the current execution action is executed, the next environment state is obtained; then the environment state is input into the reinforcement learning network, and the reinforcement learning network outputs the next execution action. By repeating the above process, multiple consecutive execution actions can be obtained, and these consecutive execution actions constitute an action sequence information.

[0101] Here, executing an action can change the environment state, and the changed environment state can in turn affect the generation of the next execution action. That is, based on the action execution, corresponding multiple environment states can be obtained, which can constitute an environment state sequence information.

[0102] Different reward values ​​may be determined for different execution actions, and the reward values ​​corresponding to the various execution actions may be combined to obtain a reward value set.

[0103] In practical applications, the disclosed embodiment can use the Soft Actor-Critic (SAC) reinforcement learning algorithm to achieve the above-mentioned optimization of the action sequence. The SAC here mainly includes an environmental model and an action network for action prediction. In practical applications, the real samples obtained by the interaction between the Actor and the environment can be used to periodically train the environmental model. The sample is defined as a four-tuple (s_t, a_t, r_t+1, s_t+1), which respectively correspond to the current environmental state information, the execution action determined under the current environmental state information, the return reward value brought by the execution action, and the next environmental state. The environmental model learns the mapping from (s_t, a_t) to (r_t+1, s_t+1).

[0104] In addition, the environment model can be used to predict hypothetical samples periodically. That is, in the process of determining the execution action of each output, the state s_t in the sampled real sample is input into the current strategy, and the action a_t can be obtained by sampling according to the current strategy. When (s_t, a_t) is input into the environment model to obtain (r_t+1, s_t+1), a four-tuple hypothetical sample (s_t, a_t, r_t+1, s_t+1) is formed.

[0105] In the disclosed embodiment, as the environment model is trained, the action network can be trained accordingly. In the process of training the action network, the environment model is used to predict the environment state sequence s_t+1, ..., s_t+h under the action sequence a_t, ..., a_t+h-1 sampled by the current strategy in the future h moments, as well as the reward set r_t+1, ..., r_t+h. Then, based on a_t, ..., a_t+h-1, s_t+h, and r_t+1, ..., r_t+h, the optimization target (corresponding to the total reward value) can be obtained to optimize each execution action indicated by the action sequence a_t, ..., a_t+h-1.

[0106] Here, after obtaining the environment state sequence information and the reward value set, we can follow the following steps: Figure 2 The steps shown here give a total reward value:

[0107] S201, for each execution action included in at least two consecutive execution actions, determine the environment state corresponding to the execution action and the reward value generated when the execution action is executed from the environment state sequence information and the reward value set; and determine the reward and value for the execution action based on the reward value and the environmental impact value under the environment state;

[0108] S202: Determine a total reward value based on the reward and value determined for each execution action.

[0109] Here, for each execution action, the corresponding reward and value can be jointly determined based on the reward value corresponding to the execution action and the environmental impact value under the environmental state, so as to better evaluate the quality of the execution action.

[0110] Among them, the above return rewards and values ​​can be calculated by the following calculation formula:

[0111] R t+1 =r t+1 +e t

[0112] Among them, e t =-alpha*log(p(a_t|s_t)), used to represent the environmental impact value at time t, R t+1 represents the reward and value at time t+1, r t+1 Represents the return reward value at time t+1.

[0113] From the above formula, we can see that the current reward value is obtained by adding the current reward value and the environmental impact value of the previous moment. Under the joint action of the reward value and the environmental impact value, the reward value of the determined execution action is closer to the corresponding execution action, further optimizing the action sequence.

[0114] In practical applications, it is considered that different execution actions have different degrees of influence on the entire action sequence, that is, in an entire action sequence, some actions are not executed accurately enough to affect the accuracy of the entire action sequence, while some execution actions are not accurate enough to affect the accuracy of the entire action sequence. Therefore, in order to further optimize the action sequence, here, each execution action can be assigned an action weight value, and based on the reward and value of each reward and the action weight value, the total reward value can be obtained. The total reward value obtained in this way can take into account the influence of each execution action on the reward, which is more in line with the needs of actual scenarios.

[0115] In the embodiment of the present disclosure, the total reward value corresponding to multiple consecutive execution actions from the current time t to the next h time points can be calculated according to the following formula:

[0116] R 总 =w 0 R t+1 +w 1 R t+2 +w 2 R t+3 +…+w h-1 R t+h-1 +V t+h

[0117] In the above formula, R 总 Represents the total reward value corresponding to multiple consecutive actions from the current time t to the next h time points, R t+1 , R t+2 , R t+3 , …, R t+h-1 Represent the reward and value at time t+1, t+2, t+3, …, t+h-1 respectively, w 0 , w 1 , w 2 , w h-1 Respectively represent the weight value corresponding to each action, V t+h Represents the value of the environmental state at time t+h.

[0118] In the above formula, the value of the environment state at time t+h is added to the weighted sum of each reward and value. The larger the value of the environment state, the more accurate the execution action predicted based on the state.

[0119] After obtaining the total reward value, the network parameter value of the reinforcement learning network can be adjusted to obtain a trained reinforcement learning network. In the process of adjusting the reinforcement learning network, the following steps can be executed repeatedly until the total reward value corresponding to the target action sequence output by the trained reinforcement learning network is greater than the preset threshold.

[0120] Step 1: adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain the adjusted reinforcement learning network; and applying the action sequence information to the target application scenario to obtain the environment state sequence information corresponding to the action sequence information;

[0121] Step 2: Input the last environmental state information included in the environmental state sequence information into the adjusted reinforcement learning network to obtain the action sequence information output by the reinforcement learning network for executing multiple consecutive execution actions within a preset time in the future, as well as the total reward value generated when executing the action sequence information.

[0122] Here, when adjusting the reinforcement learning network, the network parameter values ​​of the reinforcement learning network can be adjusted based on the total reward value to output an action sequence with a larger total reward value. The current environment state information is input into the reinforcement learning network, and the reinforcement learning network can optimize each execution action in the output action sequence. That is, when the total reward value of the current output is relatively small, it means that the performance of the currently executed action sequence is not good. In the next adjustment process, the generation probability of such action sequences can be reduced. On the contrary, when the total reward value of the current output is relatively large, it means that the performance of the currently executed action sequence is better. In the next adjustment process, the generation probability of such action sequences can be enhanced. By training in sequence, a target action sequence with good enough performance can be obtained.

[0123] In practical applications, in addition to cyclically adjusting the next round of network based on the last environmental state information included in the environmental state sequence information, the environmental state information can also be directly selected from the real samples / hypothetical samples obtained from the above environmental model for the next round of network adjustment. No specific restrictions are made here.

[0124] During an adjustment process, each execution action in the action sequence corresponds to a reward value. At the same time, as the execution action proceeds, the environment state information will also be updated. The updated environment state information is then input into the reinforcement learning network, and the reinforcement learning network outputs a new execution action, thereby obtaining action sequence information including multiple consecutive actions.

[0125] Here, we take the robot walking scene as an example. When the robot is walking, the current state of the environment is s 1(corresponding to the first environmental state), s 1 Input network, the robot performs the action of lifting the left foot a 1 (corresponding to the first execution action), get the reward value r corresponding to the action 2 (corresponding to the reward for the first action) and the environment state s 2 (corresponding to the second environmental state), s 2 Input network, the robot performs the action of putting down the left foot a 2 (corresponding to the second execution action), get the reward value r corresponding to the action 3 (corresponding to the reward for the second action) and the environment state s 3 (corresponding to the third environmental state), s 3 Input network, the robot performs the action of lifting the right foot a 3 (corresponding to the third execution action), get the reward value r corresponding to this action 4 (corresponding to the reward for the third action) and the environment state s 4 (corresponding to the fourth environmental state), s 4 Input network, the robot performs the action of putting down the right foot a 4 (corresponding to the fourth execution action), get the reward value r corresponding to this action 5 (corresponding to the reward for the fourth action) and the environment state s 5 (Corresponding to the fifth environmental state).

[0126] Here a 1 , a 2 , a 3 , a 4 Can constitute an action sequence information, s 1 ,s 2 ,s 3 ,s 4 ,s 5 Can constitute an environmental state sequence information, s 1 ,s 2 ,s 3 ,s 4 ,s 5 The corresponding environmental impact value is e 1 , e 2 , e 3 , e 4 , e 5 Here s 1 Will affect a 1 , a 1 It will also affect 2 ,s 2 Will affect a 2 , a2 It will also affect 3 ,s 3 Will affect a 3 , a 3 It will also affect 4 ,s 4 Will affect a 4 , a 4 It will also affect 5 .

[0127] It can be seen that each execution action affects each other, and through the mutual influence between each execution action, the environmental state information is continuously updated, so that each execution action can better adapt to the environment. Substituting each reward value obtained into the calculation formula of the total reward value, the total reward value can be calculated. If the calculated total reward value is greater than the preset threshold, it means that the action sequence information can better adapt to the current environment, and the reinforcement learning network can be adjusted by enhancing the execution probability of each continuous execution action in the action sequence information.

[0128] Based on the above network training method, the present disclosure also provides a robot control method, referring to Figure 3 As shown, the specific execution steps are:

[0129] S301, obtaining the current environment status information of the target robot;

[0130] S302: Input the current environment state information into the reinforcement learning network trained by the network training method to obtain a target action sequence for continuously controlling the target robot.

[0131] The robot control method here is carried out by using the reinforcement learning network trained by the above-mentioned network training method, that is, the current environmental state information of the target robot is input into the trained reinforcement learning network, and an entire target action sequence can be obtained.

[0132] Among them, the training process of the reinforcement learning network can refer to the above content and will not be repeated here.

[0133] In practical applications, in order to avoid the adverse effects that the issuance of continuous action instructions may have on the target robot, the robot control method provided in the embodiment of the present disclosure can issue an action instruction for executing the next execution action of the current execution action to the target robot only when it receives an execution success instruction of the current execution action. That is, after each action is executed, the next execution action is issued. In this way, while ensuring that the target robot can execute a series of continuous actions according to the predicted target action sequence, the precise execution of each action can be controlled.

[0134] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0135] Based on the same inventive concept, a network training device corresponding to the network training method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned network training method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0136] Reference Figure 4 , which is a schematic diagram of the architecture of a network training device provided by an embodiment of the present disclosure, the device includes: an acquisition module 401, a training module 402, and an adjustment module 403; wherein,

[0137] The acquisition module 401 is used to acquire the environment status information in the target application scenario;

[0138] A training module 402 is used to obtain action sequence information according to the environment state information and the pre-trained reinforcement learning network, and determine the total reward value corresponding to the action sequence information, where the action sequence information is used to indicate at least two consecutive execution actions within a preset time period in the future;

[0139] The adjustment module 403 is used to adjust the network parameter values ​​of the reinforcement learning network based on the total reward value to obtain a trained reinforcement learning network; wherein the trained reinforcement learning network is used to obtain a target action sequence for continuous control of the target object, and the total reward value corresponding to the target action sequence is greater than a preset threshold.

[0140] The disclosed embodiment can adjust the network parameter value of the reinforcement learning network based on the total reward value. Since the generated total reward value comprehensively evaluates the action relationship between at least two consecutive execution actions, the reinforcement learning network adjusted by the total reward value can better adapt to the generation of action sequences. In this way, as the reinforcement learning network is adjusted, the generated action sequence becomes better and better. When the trained reinforcement learning network is applied to complex scenarios such as those requiring continuous control, it can also have better control performance.

[0141] In an optional implementation, the training module 402 is used to determine the total reward value corresponding to the action sequence information according to the following steps:

[0142] Applying the action sequence information to the target application scenario to obtain environment state sequence information and a reward value set corresponding to the action sequence information; the reward value set includes a reward value corresponding to each execution action when at least two consecutive execution actions are executed in sequence;

[0143] Based on the environment state sequence information and the reward value set, the total reward value is obtained.

[0144] In an optional implementation, the training module 402 is used to obtain the total reward value based on the environment state sequence information and the reward value set according to the following steps:

[0145] For each execution action included in at least two consecutive execution actions, determine the environment state corresponding to the execution action and the reward value generated when the execution action is executed from the environment state sequence information and the reward value set; and determine the reward and value for the execution action based on the reward value and the environmental impact value under the environment state;

[0146] Based on the reward and value determined for each performed action, a total reward value is determined.

[0147] In an optional implementation, the training module 402 is configured to determine the total reward value based on the reward and value determined for each execution action according to the following steps:

[0148] Get the action weight value assigned to each execution action;

[0149] The total reward value is determined based on the reward and value determined for each execution action and the action weight value assigned to each execution action.

[0150] In an optional implementation, the action sequence information is used to indicate N consecutive execution actions within a future preset time period, where N is an integer greater than or equal to 2. The training module 402 is used to obtain the action sequence information according to the environment state information and the pre-trained reinforcement learning network in accordance with the following steps:

[0151] Determine the nth execution action according to the pre-trained reinforcement learning network and the environmental state of the target application scenario at the n-1th moment, where n is an integer and 0<n≤N;

[0152] Among them, when n is 1, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state information; when 1<n≤N, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state of the target application scenario at the n-2th moment and the n-1th execution action.

[0153] In an optional implementation, the adjustment module 403 is used to adjust the network parameter values ​​of the reinforcement learning network based on the total reward value according to the following steps to obtain a trained reinforcement learning network:

[0154] The following steps are executed repeatedly until the total reward value corresponding to the target action sequence output by the trained reinforcement learning network is greater than the preset threshold;

[0155] Adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain an adjusted reinforcement learning network; and applying the action sequence information to the target application scenario to obtain the environment state sequence information corresponding to the action sequence information;

[0156] The last environmental state information included in the environmental state sequence information is input into the adjusted reinforcement learning network to obtain the action sequence information output by the reinforcement learning network for executing multiple consecutive execution actions within a preset time in the future, as well as the total reward value generated when executing the action sequence information.

[0157] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0158] Based on the same inventive concept, a robot control device corresponding to the robot control method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned robot control method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0159] Reference Figure 5 , which is a schematic diagram of the architecture of a robot control device provided by an embodiment of the present disclosure, the device includes: an acquisition module 501, a control module 502; wherein,

[0160] The acquisition module 501 is used to obtain the current environment state information of the target robot;

[0161] The control module 502 is used to input the current environment state information into the reinforcement learning network trained by the above network training method to obtain a target action sequence for continuously controlling the target robot.

[0162] The execution actions in the target action sequence obtained in the embodiment of the present disclosure are continuous, so that the robot can be better controlled.

[0163] In an optional implementation, the device further includes:

[0164] The sending module 503 is used to send an action instruction for executing the next action of the current action to the target robot when receiving an execution success instruction for the current action included in the target action sequence sent by the target robot.

[0165] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0166] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 6 As shown, it is a schematic diagram of the structure of the electronic device 600 provided in the embodiment of the present disclosure, including a processor 61, a memory 62, and a bus 63. Among them, the memory 62 is used to store execution instructions, including a memory 621 and an external memory 622; the memory 621 here is also called an internal memory, which is used to temporarily store the operation data in the processor 61, and the data exchanged with the external memory 622 such as a hard disk. The processor 61 exchanges data with the external memory 622 through the memory 621. When the electronic device 600 is running, the processor 61 communicates with the memory 62 through the bus 63, so that the processor 61 executes the following instructions:

[0167] Obtain environmental status information in the target application scenario;

[0168] Obtain action sequence information based on the environment state information and the pre-trained reinforcement learning network, and determine the total reward value corresponding to the action sequence information. The action sequence information is used to indicate at least two consecutive execution actions within a preset time in the future.

[0169] The network parameter values ​​of the reinforcement learning network are adjusted based on the total reward value to obtain a trained reinforcement learning network; wherein the trained reinforcement learning network is used to obtain a target action sequence for continuous control of the target object, and the total reward value corresponding to the target action sequence is greater than a preset threshold;

[0170] Alternatively, execute the following command:

[0171] Obtain the current environment status information of the target robot;

[0172] The current environment state information is input into the reinforcement learning network trained by the above network training method to obtain a target action sequence for continuous control of the target robot.

[0173] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the network training method or the steps of the robot control method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0174] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the network training method described in the above method embodiments, or the steps of the robot control method. For details, please refer to the above method embodiments, which will not be repeated here.

[0175] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0176] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0179] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0180] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A network training method, characterized in that: include: Obtain environmental status information in the target application scenario; Obtaining action sequence information according to the environment state information and the pre-trained reinforcement learning network, and determining a total reward value corresponding to the action sequence information, wherein the action sequence information is used to indicate at least two consecutive execution actions within a preset time period in the future; The network parameter value of the reinforcement learning network is adjusted based on the total reward value to obtain a trained reinforcement learning network; wherein the trained reinforcement learning network is used to obtain a target action sequence for continuously controlling a target object, and the total reward value corresponding to the target action sequence is greater than a preset threshold; The determining the total reward value corresponding to the action sequence information includes: Applying the action sequence information to the target application scenario to obtain environment state sequence information and a reward value set corresponding to the action sequence information; the reward value set includes a reward value corresponding to each execution action when the at least two consecutive execution actions are executed sequentially; based on the environment state sequence information and the reward value set, the total reward value is obtained; The obtaining of the total reward value based on the environmental state sequence information and the reward value set comprises: for each execution action included in the at least two consecutive execution actions, respectively determining the environmental state corresponding to the execution action and the reward value generated when the execution action is executed from the environmental state sequence information and the reward value set; and determining the reward and value for the execution action based on the reward value and the environmental impact value under the environmental state; and determining the total reward value based on the reward and value determined for each execution action; The determining the total reward value based on the reward and value determined for each execution action includes: obtaining an action weight value assigned to each execution action; determining the total reward value based on the reward and value determined for each execution action and the action weight value assigned to each execution action; The return reward and value are calculated by the following calculation formula: R t+1 =r t+1 +e t Among them, e t =-alpha*log(p(a_t|s_t)), used to represent the environmental impact value at time t, R t+1 represents the reward and value at time t+1, r t+1 represents the reward value at time t+1, s_t represents the current environment state information, and a_t represents the execution action determined under the current environment state information.

2. The method according to claim 1, characterized in that The action sequence information is used to indicate N consecutive execution actions within a preset time in the future, where N is an integer greater than or equal to 2. The action sequence information is obtained according to the environment state information and the pre-trained reinforcement learning network, including: Determine an nth execution action according to the pre-trained reinforcement learning network and the environmental state of the target application scenario at the n-1th moment, where n is an integer and 0<n≤N; Among them, when n is 1, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state information; when 1<n≤N, the environmental state of the target application scenario at the n-1th moment is determined according to the environmental state of the target application scenario at the n-2th moment and the n-1th execution action.

3. The method according to claim 1 or 2, characterized in that: The adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain a trained reinforcement learning network includes: The following steps are executed repeatedly until the total reward value corresponding to the target action sequence output by the trained reinforcement learning network is greater than a preset threshold; Adjusting the network parameter value of the reinforcement learning network based on the total reward value to obtain an adjusted reinforcement learning network; and applying the action sequence information to the target application scenario to obtain environment state sequence information corresponding to the action sequence information; The last environmental state information included in the environmental state sequence information is input into the adjusted reinforcement learning network to obtain the action sequence information output by the reinforcement learning network for executing multiple continuous execution actions within a preset time period in the future, and the total value of the return reward generated when executing the action sequence information.

4. A robot control method, characterized in that: include: Obtain the current environment status information of the target robot; The current environment state information is input into a reinforcement learning network trained by the network training method described in any one of claims 1 to 3 to obtain a target action sequence for continuously controlling the target robot.

5. The method according to claim 4, characterized in that The method further comprises: When receiving an execution success instruction for a current execution action included in the target action sequence sent by the target robot, an action instruction for executing the next execution action of the current execution action is sent to the target robot.

6. A network training device, characterized in that: include: An acquisition module is used to obtain environmental status information in a target application scenario; A training module, used to obtain action sequence information according to the environmental state information and the pre-trained reinforcement learning network, and determine the total reward value corresponding to the action sequence information, wherein the action sequence information is used to indicate at least two consecutive execution actions within a preset time in the future; An adjustment module, used to adjust the network parameter value of the reinforcement learning network based on the total reward value to obtain a trained reinforcement learning network; wherein the trained reinforcement learning network is used to obtain a target action sequence for continuously controlling the target object, and the total reward value corresponding to the target action sequence is greater than a preset threshold; The training module, when determining the total reward value corresponding to the action sequence information, is used to: Applying the action sequence information to the target application scenario to obtain environment state sequence information and a reward value set corresponding to the action sequence information; the reward value set includes a reward value corresponding to each execution action when the at least two consecutive execution actions are executed sequentially; based on the environment state sequence information and the reward value set, the total reward value is obtained; The training module, when obtaining the total reward value based on the environment state sequence information and the reward value set, is used to: for each execution action included in the at least two consecutive execution actions, respectively determine the environment state corresponding to the execution action and the reward value generated when the execution action is executed from the environment state sequence information and the reward value set; and determine the reward and value for the execution action based on the reward value and the environmental impact value under the environment state; determine the total reward value based on the reward and value determined for each execution action; The training module, when determining the total reward value based on the reward and value determined for each execution action, is used to: obtain an action weight value assigned to each execution action; determine the total reward value based on the reward and value determined for each execution action, and the action weight value assigned to each execution action; The return reward and value are calculated by the following calculation formula: R t+1 =r t+1 +e t Among them, e t =-alpha*log(p(a_t|s_t)), used to represent the environmental impact value at time t, R t+1 represents the reward and value at time t+1, r t+1 represents the reward value at time t+1, s_t represents the current environment state information, and a_t represents the execution action determined under the current environment state information.

7. A robot control device, characterized in that: include: An acquisition module is used to obtain the current environment status information of the target robot; A control module is used to input the current environment state information into a reinforcement learning network trained by the network training method described in any one of claims 1 to 3 to obtain a target action sequence for continuously controlling the target robot.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the network training method as described in any one of claims 1 to 3 or the steps of the robot control method as described in claim 4 or 5 are performed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the network training method according to any one of claims 1 to 3 or the steps of the robot control method according to claim 4 or 5.

Images