Executable strategy judgment method and robot

Abstract

The invention discloses an executable strategy judgment method and a robot, and the method comprises the steps: firstly inputting a natural language instruction sent by a user into a large language model, obtaining a plurality of continuous task descriptions, then determining that the plurality of continuous task descriptions belong to a preset task description set, and obtaining an executable strategy based on an image collected by the robot; and determining current state information of the robot corresponding to the target task description, and finally judging whether the multiple to-be-executed actions are robot executable strategies or not by adopting a multi-step time sequence difference reinforcement learning method. The continuous task descriptions are determined to belong to the task description set from the task description set, so that the continuous task descriptions can be determined to be actions which can be executed by the robot, and then the task description can be determined to be the actions which can be executed by the robot by combining the images acquired by the robot and adopting a multi-step time sequence difference reinforcement learning method. And whether the to-be-executed actions are robot executable strategies or not is judged, so that the unreasonable to-be-executed actions of the robot can be recognized, and the performance of the robot is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to an executable strategy judgment method and a robot. Background Art

[0002] With the development of Large Language Models (LLMs), they have been able to generate complex texts, answer questions, and even have conversations on various topics. However, LLMs are not rooted in the physical world, and it is impossible to observe the results of the generation of LLMs on any physical process. When LLMs are applied to robots, they may cause the robots to make unreasonable or funny mistakes after parsing instructions. It may even be impossible for the robot to complete the actions obtained by the large language model parsing instructions for specific physical situations.

[0003] In addition, usually, a robot needs to perform multiple actions to complete the user's instructions. The robot may not be able to complete the multiple actions obtained after parsing the instructions with a large language model.

[0004] In summary, the actions currently obtained by parsing user instructions using large language models may not be executed by robots. Summary of the invention

[0005] The present invention provides an executable strategy judgment method and a robot, which are used to solve the problem in the prior art that actions obtained by using a large language model cannot be executed by a robot.

[0006] In a first aspect, the present application provides an executable policy determination method, the method comprising:

[0007] Inputting the natural language instructions sent by the user into the large language model to obtain a plurality of continuous task descriptions, wherein the large language model is obtained through training, and the task descriptions are used to characterize the actions to be performed by the robot;

[0008] Traversing a preset task description set, and determining that the plurality of consecutive task descriptions all belong to the preset task description set;

[0009] Based on the image collected by the robot, determine the current state information of the robot corresponding to the target task description, wherein the target task description is from the first task description to the last task description in the multiple consecutive task descriptions;

[0010] A multi-step temporal difference reinforcement learning method is adopted to determine whether a plurality of actions to be executed are executable strategies for the robot based on the current state information of the robot and the target task description, wherein the plurality of actions to be executed correspond to the plurality of continuous task descriptions.

[0011] In a possible implementation, determining the current state information of the robot corresponding to the target task description based on the image collected by the robot includes:

[0012] Determine a first pose of the robot at an initial position based on the image collected by the robot, and use the first pose as current state information of the robot corresponding to a first task description;

[0013] For other task descriptions, based on the images collected by the robot, a second posture of the robot after executing the action to be executed corresponding to the previous task description is determined, and the second posture is used as the current state information of the robot corresponding to the current task description.

[0014] In a possible implementation, the reinforcement learning method using multi-step temporal difference determines whether multiple actions to be executed are executable strategies for the robot based on the current state information and target task description of the robot, including:

[0015] A multi-step temporal difference reinforcement learning method is used for the current state information and target task description of the robot to calculate the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information and the target task description of the robot;

[0016] Based on the calculated action reward values, it is determined whether the multiple actions to be executed are robot executable strategies.

[0017] In a possible implementation, the reinforcement learning method using multi-step temporal difference calculates the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information of the robot and the target task description, including:

[0018] A multi-step temporal difference reinforcement learning method is used to perform reinforcement learning on the data representing the robot's current state information, task description, and action reward value, and a multi-step temporal difference state-action value function is obtained;

[0019] For the current state information and target task description of the robot, based on the state-action value function of the multi-step temporal difference, the current state information of the robot and the target task description, the action reward value of the robot from the current state to the next action to be performed is calculated.

[0020] In a possible implementation, the determining whether the multiple actions to be executed are robot executable policies includes:

[0021] The calculated action reward values ​​are compared with the preset threshold value. When it is determined that each action reward value is greater than the preset threshold value, the multiple actions to be executed are determined to be executable strategies for the robot. When it is determined that at least one action reward value among the action reward values ​​is less than or equal to the preset threshold value, the multiple actions to be executed are determined to be non-executable strategies for the robot.

[0022] In a second aspect, the present application provides a robot, comprising a processor and an actuator;

[0023] The processor is used to input the natural language instructions sent by the user into the large language model to obtain multiple continuous task descriptions, wherein the large language model is obtained through training, and the task description is used to characterize the robot's to-be-performed actions; traverse a preset task description set to determine that the multiple continuous task descriptions all belong to the preset task description set; based on the image collected by the robot, determine the current state information of the robot corresponding to the target task description, wherein the target task description is from the first task description to the last task description in the multiple continuous task descriptions; adopt a multi-step temporal difference reinforcement learning method, based on the robot's current state information and the target task description, determine whether the multiple to-be-performed actions are robot executable strategies, wherein the multiple to-be-performed actions correspond to the multiple continuous task descriptions;

[0024] The execution mechanism is used to execute the multiple actions to be executed in sequence when the processor determines that the multiple actions to be executed are robot executable strategies.

[0025] In a possible implementation manner, the processor is specifically configured to:

[0026] Determine a first pose of the robot at an initial position based on the image collected by the robot, and use the first pose as current state information of the robot corresponding to a first task description;

[0027] For other task descriptions, based on the images collected by the robot, a second posture of the robot after executing the action to be executed corresponding to the previous task description is determined, and the second posture is used as the current state information of the robot corresponding to the current task description.

[0028] In a possible implementation manner, the processor is specifically configured to:

[0029] A multi-step temporal difference reinforcement learning method is used for the current state information and target task description of the robot to calculate the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information and the target task description of the robot;

[0030] Based on the calculated action reward values, it is determined whether the multiple actions to be executed are robot executable strategies.

[0031] In a possible implementation manner, the processor is specifically configured to:

[0032] A multi-step temporal difference reinforcement learning method is used to perform reinforcement learning on the data representing the robot's current state information, task description, and action reward value, and a multi-step temporal difference state-action value function is obtained;

[0033] For the current state information and target task description of the robot, based on the state-action value function of the multi-step temporal difference, the current state information of the robot and the target task description, the action reward value of the robot from the current state to the next action to be performed is calculated.

[0034] In a possible implementation manner, the processor is specifically configured to:

[0035] The calculated action reward values ​​are compared with the preset threshold value. When it is determined that each action reward value is greater than the preset threshold value, the multiple actions to be executed are determined to be executable strategies for the robot. When it is determined that at least one action reward value among the action reward values ​​is less than or equal to the preset threshold value, the multiple actions to be executed are determined to be non-executable strategies for the robot.

[0036] The beneficial effects of the present invention are as follows:

[0037] The present invention provides an executable strategy judgment method and robot. Since it is first determined from a task description set that multiple consecutive task descriptions all belong to the task description set, it can be determined that the multiple consecutive task descriptions obtained through a large language model are all actions executable by the robot. Then, in combination with the images collected by the robot, a multi-step temporal difference reinforcement learning method is adopted to judge whether multiple actions to be executed are executable strategies for the robot based on the current state information and task description of the robot, thereby identifying unreasonable actions to be executed for the robot and improving the performance of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A flowchart of an executable strategy determination method provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of a method for calculating an action reward value provided in an embodiment of the present application;

[0041] Figure 3 A network diagram of a reinforcement learning model provided in an embodiment of the present application;

[0042] Figure 4 A complete flow chart of an executable strategy determination method provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the layout of a robot, a user, a kettle and a cup provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of a path 1 determined by a robot provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a path 2 determined by a robot provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of the structure of a robot provided in an embodiment of the present application;

[0047] Fig. 9 A schematic diagram of the structure of another robot provided in an embodiment of the present application;

[0048] Fig.10 A schematic diagram of the structure of another robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] When the robot executes natural language instructions issued by the user, the large language model provides the robot with actions to be executed. The large language model may generate multiple reasonable robot actions to be executed based on the natural voice instructions, but it may not be applicable for specific tasks that need to be performed in a specific environment, resulting in the robot being unable to complete the actions to be executed.

[0051] For example, a user says to the robot, "How will you bring me an apple?" After the large language model in the robot receives the natural language instruction "How will you bring me an apple?", it parses the natural language instruction and outputs "The robot can go to a nearby store and buy you an apple." Although this output result is a reasonable answer, the robot cannot go to a nearby store to buy apples. Therefore, the robot cannot complete multiple actions corresponding to "The robot can go to a nearby store and buy you an apple."

[0052] Based on the above problems, an embodiment of the present application provides an executable strategy judgment method and a robot, which are described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 FIG. 1 is a flow chart of an executable strategy determination method provided in an embodiment of the present application, and the method specifically includes the following steps:

[0054] S101, inputting a natural language instruction sent by a user into a large language model to obtain a plurality of continuous task descriptions, wherein the large language model is obtained through training, and the task descriptions are used to characterize actions to be performed by the robot;

[0055] S102, traversing a preset task description set, and determining that the plurality of consecutive task descriptions all belong to the preset task description set;

[0056] S103, determining the current state information of the robot corresponding to the target task description based on the image collected by the robot, wherein the target task description is from the first task description to the last task description in the multiple consecutive task descriptions;

[0057] S104. Using a multi-step temporal difference reinforcement learning method, based on the current state information of the robot and the target task description, determine whether the multiple actions to be executed are executable strategies for the robot, wherein the multiple actions to be executed correspond to the multiple continuous task descriptions.

[0058] In the embodiment of the present application, since it is first determined from the task description set that multiple consecutive task descriptions all belong to the task description set, it can be determined that the multiple consecutive task descriptions obtained through the large language model are all actions executable by the robot, and then combined with the images collected by the robot, a multi-step temporal difference reinforcement learning method is adopted to judge whether multiple actions to be executed are robot executable strategies based on the current state information and task description of the robot, thereby identifying unreasonable actions to be executed for the robot and improving the performance of the robot.

[0059] In the specific implementation, the natural language instructions sent by the user are first input into the large language model, and the large language model outputs multiple continuous task descriptions. The task descriptions are used to characterize the robot's actions to be performed. The large language model is pre-trained.

[0060] For example, the natural language command sent by the user is "I am thirsty, do you have any water?", the large language model outputs 5 consecutive task descriptions, the first task description is task description 1, the second task description is task description 2, the third task description is task description 3, the fourth task description is task description 4, and the fifth task description is task description 5.

[0061] It should be noted that the task description represents the robot's action to be performed, that is, the content of the task description. For example, task description 1 is used to represent the robot's action to be performed as "find a cup and a kettle in the house". At the same time, "find a cup and a kettle in the house" is also task description 1.

[0062] The large language model in the embodiment of the present application is a pre-trained model. A large amount of data is used to pre-train the large language model for prompt engineering. The large amount of data includes the robot's current state information, task descriptions, and action reward values. The specific training process can refer to the training process provided by the relevant technology, which will not be repeated here.

[0063] After obtaining multiple consecutive task descriptions, traverse the preset task description set to determine that the multiple consecutive task descriptions all belong to the preset task description set. The task description set in the embodiment of the present application corresponds to the robot, that is, each robot corresponds to a task description set, and the task descriptions in the task description set represent the actions that can be executed by the robot. Determining that multiple consecutive task descriptions all belong to the preset task descriptions indicates that the actions to be executed represented by the determined multiple consecutive task descriptions are all actions that can be executed by the robot.

[0064] For example, the preset task description set includes {task description 1, task description 2, task description 3, task description 4, task description 5, task description 6, task description 7, task description 8, task description 9, task description 10}, if the determined multiple consecutive task descriptions are: task description 1, task description 2 and task description 3, traverse the preset task description set, determine that task description 1 belongs to the preset task description set, task description 2 also belongs to the preset task description set, and task description 3 also belongs to the preset task description set, that is, task description 1, task description 2 and task description 3 all belong to the preset task description set;

[0065] If the determined multiple consecutive task descriptions are: task description 1, task description 2 and task description 11, traverse the preset task description set to determine that task description 1 belongs to the preset task description set, task description 2 also belongs to the preset task description set, and task description 11 does not belong to the preset task description set.

[0066] In an embodiment of the present application, when it is determined that there are multiple consecutive task descriptions that do not belong to the preset task description set, that is, the multiple actions to be executed corresponding to the multiple consecutive task descriptions include actions that the robot cannot execute. For example, the actions to be executed corresponding to the above-mentioned task description 11 are actions that the robot cannot execute. At this time, the multiple actions to be executed are judged as strategies that the robot cannot execute.

[0067] In one embodiment, after determining that multiple consecutive task descriptions all belong to a preset task description set, the current state information of the robot corresponding to the target task description is determined based on the images collected by the robot, wherein the target task description is a task description other than the last task description among the multiple consecutive task descriptions, and the current state information includes the current posture of the robot.

[0068] It should be noted that the images captured by the robot in the embodiment of the present application may also be videos.

[0069] Specifically, the robot collects images through its own camera, can collect multiple images in multiple directions, and then analyzes the multiple images to determine the current position and posture of the robot.

[0070] In one embodiment, based on the image collected by the robot, a first pose of the robot at the initial position is determined, and the first pose is used as the current state information of the robot corresponding to the first task description.

[0071] For example, when the robot receives a natural language instruction sent by the user at position 1, the position of the robot at position 1 is used as the current state information of the robot. The target task description at this time is the first task description among multiple consecutive task descriptions output by the large language model.

[0072] For other task descriptions, based on the images collected by the robot, the second posture of the robot after executing the action to be executed corresponding to the previous task description is determined, and the second posture is used as the current state information of the robot corresponding to the current task description.

[0073] When the robot executes the action to be executed corresponding to the first task description and moves to position 2, the robot determines the current posture of the robot through the acquired image and uses the current posture as the current state information of the robot. The target task description at this time is the second task description among the multiple consecutive task descriptions output by the large language model.

[0074] When the robot executes the pending action corresponding to the second task description, its position does not change. The robot still determines its current posture through the collected images and uses the current posture as the current state information of the robot. At this time, the target task description is the third task description among the multiple consecutive task descriptions output by the large language model.

[0075] For example, there are 5 consecutive task descriptions, task description 1 is "find a water cup and a kettle in the room", task description 2 is "walk to the location of the water cup", task description 3 is "pick up the water cup and the kettle", task description 4 is "pour water into the water cup", and task description 5 is "give the water cup to the user", among which task description 1 is the first task description and task description 5 is the last task description;

[0076] The position and posture of the robot when receiving the natural voice command sent by the user is the current state information of the robot corresponding to task description 1. The position and posture of the robot determined after executing the action to be executed corresponding to task description 1 is the current state information of the robot corresponding to task description 2. The position and posture of the robot determined after executing the action to be executed corresponding to task description 2 is the current state information of the robot corresponding to task description 3. The position and posture of the robot determined after executing the action to be executed corresponding to task description 3 is the current state information of the robot corresponding to task description 4. The position and posture of the robot determined after executing the action to be executed corresponding to task description 4 is the current state information of the robot corresponding to task description 5.

[0077] After determining the current state information of the robot corresponding to the target task description, a multi-step temporal difference reinforcement learning method is used to judge whether multiple actions to be executed are executable strategies for the robot based on the target task description and the determined current state information of the robot corresponding to each target task description.

[0078] In the specific implementation, for the current state information and target task description of each group of robots, a multi-step temporal difference reinforcement learning method is adopted to calculate the action reward value of the robot from the current state to the action to be performed corresponding to the target task description. Based on the calculated action reward value, it is judged whether multiple actions to be performed are executable strategies for the robot.

[0079] Specifically, a multi-step temporal difference reinforcement learning method is first used to perform reinforcement learning on the data representing the robot's state information and task description, and a multi-step temporal difference state-action value function is obtained. Then, for each group of robots' current state information and target task description, the action reward value of the robot from the current state to the action to be performed corresponding to the target task description is calculated based on the robot's current state information, target task description and the multi-step temporal difference state-action value function.

[0080] The following first explains how to obtain the state-action value function of multi-step temporal difference.

[0081] The embodiment of the present application adopts the n-step Sarsa algorithm, which is obtained by combining the n-step TD estimation and the Sarsa algorithm. The goal of the n-step TD estimation is to learn the state or state-action value function (Sarsa function) (s, a), which represents the discounted sum of rewards obtained from the actions generated according to the strategy starting from state s and action a.

[0082] In the Sarsa algorithm, the update formula of the value function is:

[0083] Q(S t ,A t )←Q(S t ,A t )+α[R t+1 +γQ(S t+1 ,A t+1 )-Q(S t ,A t )]

[0084] Among them, "R t+1 +γQ(S t+1 ,A t+1 ” is the update target. To realize the strategy estimation of multi-step actions of the robot, it is replaced by the n-step TD estimation to obtain the n-step Sarsa algorithm.

[0085] The update target of the n-step TD estimate is:

[0086] G t：t+n =R t+1 +γR t+2 +…+γnQ t+1-1 (S t+n ,A t+n ), n≥1,0≤t<Tn

[0087] Thus, the update formula of the n-step Sarsa algorithm is obtained as follows:

[0088] Q t+n (S t ,A t )←Q t+n-1 (S t ,A t )+α[G t：t+n -Q t+n-1 (S t ,A t )]

[0089] Among them, "G t：t+n ” is the goal of the update.

[0090] In the specific implementation, a finite recursive Markov chain (FRMC) is first established. FRMC is a special Markov chain whose state space is finite and its transition probability matrix is ​​recursive. This recursive property means that the future state of a state only depends on the current state and a finite number of past states.

[0091] Finite Markov Decision Process (MDP), in are the state and action spaces, respectively. is the state transition probability function, is the reward function, γ is the discount factor, and the learning finite Markov decision process is the Sarsa algorithm.

[0092] Combining the n-step TD estimation and Sarsa algorithm, we get the update formula of the n-step Sarsa algorithm. Based on a large amount of data, we calculate α and G in the update formula of the n-step Sarsa algorithm. t：t+n , the calculated α and G t：t+n Substituting into the update formula of the n-stepSarsa algorithm, the state-action value function of the multi-step temporal difference in the embodiment of the present application is obtained.

[0093] In the embodiment of the present application, a TD-based method is used to learn the value function, and the learned value function is related to the natural language instructions, and they are used to determine whether a given command can be executed in a given state. In addition, in the case of undiscounted sparse rewards, if the robot can successfully complete the next action from the current state, it will receive a reward of 1.0, otherwise 0.0. The value function trained by RL corresponds to an ability function, indicating whether a skill can be performed in a given state.

[0094] After obtaining the state-action value function of multi-step temporal difference, the action reward value of the robot from the current state to the next action to be performed is calculated based on the state-action value function of multi-step temporal difference, the current state information of the robot and the target task description.

[0095] Specifically, the current state information and target task description of the robot are substituted into the state-action value function of the multi-step temporal difference, and the calculated value is the action reward value of the robot from the current state to the next action to be performed.

[0096] like Figure 2FIG. 1 is a flow chart of a method for calculating an action reward value provided in an embodiment of the present application, which specifically includes the following steps:

[0097] S201, using a multi-step temporal difference reinforcement learning method to perform reinforcement learning on a large amount of data to obtain a multi-step temporal difference state-action value function;

[0098] S202, determining the target task description and the current state information of the robot corresponding to the target task description;

[0099] S203: Substitute the current state information and target task description of the robot into the state-action value function of multi-step temporal difference for calculation to obtain the action reward value of the robot from the current state to the next action to be performed.

[0100] After obtaining the action reward value, compare the action reward value with the preset threshold. If each action reward value obtained is greater than the preset threshold, multiple actions to be executed are determined to be executable strategies for the robot. If there is at least one action reward value less than or equal to the preset threshold, multiple actions to be executed are determined to be non-executable strategies for the robot.

[0101] For example, there are 5 task descriptions, task description 1, task description 2, task description 3, task description 4 and task description 5. The current state information 1 of the robot corresponds to task description 1, the current state information 2 of the robot corresponds to task description 2, the current state information 3 of the robot corresponds to task description 3, the current state information 4 of the robot corresponds to task description 4, and the current state information 5 of the robot corresponds to task description 5.

[0102] Substitute the robot's current state information 1 and task description 1 into the state-action value function of multi-step temporal difference (Formula 3), and the calculated action reward value is the action reward value 1;

[0103] Substitute the robot's current state information 2 and task description 2 into the state-action value function of multi-step temporal difference, and the calculated action reward value is the action reward value 2;

[0104] Substitute the robot's current state information 3 and task description 3 into the state-action value function of multi-step temporal difference, and the calculated action reward value is the action reward value 3;

[0105] Substitute the current state information 4 and the task description 4 of the robot into the state-action value function of the multi-step temporal difference, and the calculated action reward value is the action reward value 4;

[0106] Substitute the robot's current state information 5 and task description 5 into the state-action value function of multi-step temporal difference, and the calculated action reward value is the action reward value 5.

[0107] The action reward value 1, action reward value 2, action reward value 3, action reward value 4 and action reward value 5 are compared with the preset thresholds respectively. If the five action reward values ​​are all greater than the preset thresholds, the multiple actions to be executed are determined to be executable strategies for the robot.

[0108] In a specific implementation, after comparing the action reward value with the preset threshold, if the action reward value is greater than the preset threshold, it is marked as 1; if the action reward value is less than or equal to the preset threshold, it is marked as 0. When judging whether multiple actions to be executed are executable strategies of the robot, judge whether the mark values ​​are all 1. If the mark values ​​are all 1, it is determined that the multiple actions to be executed are executable strategies of the robot; if the mark values ​​include at least one 0, it is determined that the multiple actions to be executed are non-executable strategies of the robot.

[0109] In a specific implementation, after the robot determines multiple consecutive task descriptions, starting from the first task description, the robot determines the current state information of the robot corresponding to the first task description based on the acquired image, and then substitutes the first task description and the current state information into the state-action value function of the multi-step temporal difference to calculate the action reward value, and compares the calculated action reward value with a preset threshold value. If it is greater than the preset threshold value, the action reward value is recorded as 1. After the robot recognizes "1", it is determined that the to-be-executed action corresponding to the first task description can be executed;

[0110] After the robot executes the pending action corresponding to the first task description, the robot determines the current state information of the robot corresponding to the second task description based on the image collected by the robot, and then substitutes the second task description and the current state information into the state-action value function of the multi-step temporal difference to calculate the action reward value, and compares the calculated action reward value with the preset threshold. If it is less than or equal to the preset threshold, the action reward value is recorded as 0. After the robot recognizes "0", it determines that the pending action corresponding to the second task description cannot be executed. At this time, the robot determines that multiple pending actions are robot unexecutable strategies;

[0111] After the robot executes the pending action corresponding to the first task description, the current state information of the robot corresponding to the second task description is determined based on the image collected by the robot, and then the second task description and the current state information are substituted into the state-action value function of the multi-step temporal difference to calculate the action reward value, and the calculated action reward value is compared with the preset threshold. If it is greater than the preset threshold, the action reward value is recorded as 1. After the robot recognizes "1", it determines that the pending action corresponding to the second task description is executable, and determines other task descriptions in turn according to the above steps. If the pending actions corresponding to each determined task description are all robot-executable actions, the robot determines multiple pending actions as robot-executable strategies.

[0112] In one embodiment, the robot receives a natural language instruction i provided by the user, which describes the task that the user requires the robot to perform. Before the specific implementation, a set of skills E is pre-set, that is, a set of task descriptions of actions that the robot can perform, where each skill expertise∈E performs a short task, such as picking up a specific object, and the skill is also an action that can be performed.

[0113] Each skill in skill group E includes a task description l expertise (e.g., “Find a sponge”), in addition to including the task description l expertise It can also include a capability function p(C expertis |s,l expertise ), which indicates the successful completion of a transition from state s to state s with description l expertise That is, p(C expertise |s,l expertise ) means "If I ask the robot to do l expertise , will it do it?" In RL terms, p(C expertise |s,l expertise ) is the value function of the skill, if we set the reward to 1 for successful completion and 0 otherwise.

[0114] It should be noted that the capability function is the state-action value function of multi-step temporal difference.

[0115] like Figure 3 As shown, this is a network diagram of a reinforcement learning method provided in an embodiment of the present application. Natural language instruction input is added to the reinforcement learning method. The image captured by the robot's camera is first processed by 6 convolutional layers. The natural language instruction is embedded by LLM, and the output is controlled by a sigmoid gate, so the value function is always in the range of [0, 1].

[0116] As mentioned above, expertiseThe task description representing the skill expertise can also be called a text label, p(C expertise |s,l expertise ) means if it is executed from state s, with task description l expertise The probability of successful completion of the skill expertise, where C expertise is a Bernoulli random variable.

[0117] In the implementation, planning is structured as a conversation between a user and a robot, where the user provides high-level instructions (e.g., “How would you get me a glass of water?”) and the large language model responds with an explicit sequence (“I would: 1. find a cup and a kettle in the house, 2. move to the cup and kettle, 3. pick up the cup and kettle, 4. pour water into the cup, 5. move to the user’s location and hand the cup to the user”).

[0118] The embodiment of the present application combines the world-based model of the large model with the physical perception of the robot. The large language model uses a large amount of knowledge learned from a large amount of text to decompose the natural language instructions input by the user into multiple continuous task descriptions, each task description corresponds to an action to be performed. Although the large language model can output multiple continuous task descriptions, the actions to be performed corresponding to multiple continuous task descriptions cannot guarantee that the robot can do them. For example, a language model is asked "How will the robot bring me an apple?" It may answer "The robot can go to a nearby store and buy you an apple." Although this response is a reasonable completion of the prompt, for the robot, "go to a nearby store" and "buy an apple" are operations that the robot cannot perform. In order to make the large language model adapt to the statement of human instructions, we must somehow tell the robot that the natural language instructions input by the user need to be decomposed into task descriptions that the robot can execute, that is, in the embodiment of the present application, traverse the preset task description set to determine whether the multiple continuous task descriptions output by the large language model are in the preset task description set.

[0119] After determining that multiple consecutive task descriptions are in the preset task description set, since the multiple pending actions are actions that the robot needs to perform continuously, it is also necessary to determine whether the robot can complete the next pending action from its current state. Finally, based on the judgment result, determine whether the multiple pending actions are executable strategies for the robot.

[0120] like Figure 4 FIG. 1 is a complete flow chart of an executable strategy determination method provided in an embodiment of the present application, which specifically includes the following steps:

[0121] S401, inputting the natural language instruction sent by the user into the large language model to obtain multiple continuous task descriptions;

[0122] S402, determining whether the plurality of consecutive task descriptions are all in a preset task description set, if so, executing S403, otherwise executing S409;

[0123] S403, determining the position and posture of the robot based on the image collected by the robot;

[0124] S404, using the determined position and posture of the robot as the current state information of the robot corresponding to the target task description;

[0125] S405, substituting the target language description and the current state information of the robot into the state-action value function of multi-step temporal difference to calculate the action reward value;

[0126] S406, determine whether the action reward values ​​are greater than the preset threshold, if so, return to execute S407, otherwise execute S409;

[0127] S407, determine whether the target task description is the last task description, if so, execute S408, otherwise execute S403;

[0128] S408, determining that a plurality of actions to be executed are robot executable strategies;

[0129] S409: Determine that the multiple actions to be executed are non-executable strategies for the robot.

[0130] For ease of understanding, the present application is described below with reference to specific embodiments.

[0131] For example, the user says to the robot, "I am thirsty, do you have any water?" After receiving the natural language instruction "I am thirsty, do you have any water?" sent by the user, the robot inputs the natural language instruction into the large language model, and the large language model outputs 5 task descriptions, namely, task description 1, task description 2, task description 3, task description 4, and task description 5;

[0132] Among them, task description 1 represents the action to be executed as "find a cup and a kettle in the house", task description 2 represents the action to be executed as "walk to the location of the cup and the kettle", task description 3 represents the action to be executed as "pick up the cup and the kettle", task description 4 represents the action to be executed as "pour water into the cup", and task description 5 represents the action to be executed as "give the cup to the user".

[0133] The preset task description set is {task description 1, task description 2, task description 3, task description 4, task description 5, task description 6, task description 7, task description 8, task description 9, task description 10}. Traverse the preset task description set to determine that task description 1, task description 2, task description 3, task description 4 and task description 5 all belong to the preset task description set.

[0134] The robot collects images, analyzes the images collected by the robot to determine the robot's posture, and uses the determined robot's posture as the current state information of the robot corresponding to task description 1, such as Figure 5 As shown, the initial position of the robot is position 1, the robot collects images, determines the posture 1 of the robot at the initial position, and uses the posture 1 as the current state information of the robot corresponding to the task description 1.

[0135] Substitute the posture 1 and the task description 1 into the state-action value function of the multi-step temporal difference, calculate the action reward value 1, and compare the calculated action reward value 1 with the preset threshold. If it is greater than the preset threshold, it is determined that the robot can execute the action to be executed corresponding to the task description 1;

[0136] The action to be executed corresponding to task description 1 is "find a cup and a kettle in the house". The action to be executed can be obtained by analyzing the images collected by the robot. The action to be executed is an executable action of the robot.

[0137] After the robot performs the action to be performed corresponding to task description 1, the robot's posture 2 is determined based on the collected image, and the posture 2 is used as the current state information of the robot corresponding to task description 2;

[0138] Substitute posture 2 and task description 2 into the state-action value function of multi-step temporal difference, calculate action reward value 2, compare the calculated action reward value 2 with the preset threshold, and if it is greater than the preset threshold, determine that the robot can execute the action to be executed corresponding to task description 2;

[0139] The action to be executed corresponding to task description 2 is "walk to the location of the cup and kettle". Specifically, after the robot determines that there are cups and kettles in the room through analysis, the robot first determines the location of the cups and kettles, and then plans a path based on the robot's own position, the position of the cups and the position of the kettle, and then determines whether the robot can reach the location of the cups and kettles through the path (assuming that the positions of the cups and kettles are the same). If yes, the robot executes the action to be executed corresponding to task description 2. If not, for example, Figure 6 As shown, there is an obstacle in the path 1 that the robot cannot cross. Due to the existence of the obstacle, the robot determines that it cannot move from position 1 to position 2. In this case, the robot determines that multiple actions to be executed are unexecutable strategies for the robot.

[0140] If the robot determines that it can execute the pending action corresponding to task description 2, after the robot executes the pending action corresponding to task description 2, the robot's posture 3 is determined based on the collected image, and the posture 3 is used as the current state information of the robot corresponding to task description 3, such as Figure 7As shown, at this time, the robot moves to position 2.

[0141] Substitute the posture 3 and the task description 3 into the state-action value function of the multi-step temporal difference, calculate the action reward value 3, and compare the calculated action reward value 3 with the preset threshold. If it is greater than the preset threshold, it is determined that the robot can execute the action to be executed corresponding to the task description 3;

[0142] The action to be executed corresponding to task description 3 is "pick up the cup and kettle". After determining that the action to be executed corresponding to task description 3 is an executable action for the robot, the robot executes the action to be executed corresponding to task description 3, and then determines the robot's posture 4 based on the collected image, and uses posture 4 as the current state information of the robot corresponding to task description 4.

[0143] Substitute the posture 4 and the task description 4 into the state-action value function of the multi-step temporal difference, calculate the action reward value 4, and compare the calculated action reward value 4 with the preset threshold. If it is greater than the preset threshold, it is determined that the robot can execute the action to be executed corresponding to the task description 4;

[0144] The action to be executed corresponding to task description 4 is "pour water into the cup". After the robot determines that the action to be executed corresponding to task description 4 is an executable action for the robot, the robot executes the action to be executed corresponding to task description 4, and then determines the robot's posture 5 based on the collected image, and uses posture 5 as the current state information of the robot corresponding to task description 5.

[0145] Substitute the posture 5 and the task description 5 into the state-action value function of the multi-step temporal difference, calculate the action reward value 5, and compare the calculated action reward value 5 with the preset threshold. If it is greater than the preset threshold, it is determined that the robot can execute the action to be executed corresponding to the task description 5;

[0146] The action to be executed corresponding to task description 5 is "bring the cup to the user". Specifically, after the robot determines the user's position by analyzing the collected image, the robot plans a path based on its own position and the user's position, and then determines whether the robot can reach the user's position through the path, such as Figure 7 As shown, the robot plans path 3 from position 2 to position 3. If possible, the robot executes the pending actions corresponding to task description 5, and determines that multiple pending actions are unexecutable strategies for the robot. If not, the robot determines that multiple pending actions are unexecutable strategies for the robot.

[0147] Based on the same inventive concept, an embodiment of the present application also provides a robot, the principle of which is similar to the principle of which the above-mentioned executable strategy judgment method solves technical problems. The implementation of the robot can refer to the implementation of the above-mentioned executable strategy judgment method, and the repeated parts will not be repeated.

[0148] like Figure 8 , which is a schematic diagram of the structure of a robot provided in an embodiment of the present application, the robot includes a processor 801 and an actuator 802;

[0149] Processor 801 is used to input the natural language instruction sent by the user into the large language model to obtain multiple continuous task descriptions, wherein the large language model is obtained through training, and the task description is used to characterize the robot's to-be-performed actions; traverse a preset task description set to determine that the multiple continuous task descriptions all belong to the preset task description set; based on the image collected by the robot, determine the current state information of the robot corresponding to the target task description, wherein the target task description is from the first task description to the last task description in the multiple continuous task descriptions; adopt a multi-step temporal difference reinforcement learning method, based on the robot's current state information and the target task description, determine whether the multiple to-be-performed actions are robot executable strategies, wherein the multiple to-be-performed actions correspond to the multiple continuous task descriptions;

[0150] The execution mechanism 802 is used to execute the multiple robot actions to be executed in sequence when the processor determines that the multiple robot actions to be executed are robot executable strategies.

[0151] In one embodiment, the processor 801 is specifically configured to:

[0152] Determine a first pose of the robot at an initial position based on the image collected by the robot, and use the first pose as current state information of the robot corresponding to a first task description;

[0153] For other task descriptions, based on the images collected by the robot, a second posture of the robot after executing the action to be executed corresponding to the previous task description is determined, and the second posture is used as the current state information of the robot corresponding to the current task description.

[0154] In one embodiment, the processor 801 is specifically configured to:

[0155] A multi-step temporal difference reinforcement learning method is used for the current state information and target task description of the robot to calculate the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information and the target task description of the robot;

[0156] Based on the calculated action reward values, it is determined whether the multiple actions to be executed are robot executable strategies.

[0157] In one embodiment, the processor 801 is specifically configured to:

[0158] A multi-step temporal difference reinforcement learning method is used to perform reinforcement learning on the data representing the robot's current state information, task description, and action reward value, and a multi-step temporal difference state-action value function is obtained;

[0159] For the current state information and target task description of the robot, based on the state-action value function of the multi-step temporal difference, the current state information of the robot and the target task description, the action reward value of the robot from the current state to the next action to be performed is calculated.

[0160] In one embodiment, the processor 801 is specifically configured to:

[0161] The calculated action reward values ​​are compared with the preset threshold value. When it is determined that each action reward value is greater than the preset threshold value, the multiple actions to be executed are determined to be executable strategies for the robot. When it is determined that at least one action reward value among the action reward values ​​is less than or equal to the preset threshold value, the multiple actions to be executed are determined to be non-executable strategies for the robot.

[0162] Based on the same inventive concept, the embodiment of the present application also provides another robot, the principle of which is similar to the principle of which the above-mentioned executable strategy judgment method is used to solve technical problems. The implementation of the robot can refer to the implementation of the above-mentioned executable strategy judgment method, and the repeated parts will not be repeated.

[0163] like Fig. 9 FIG. 1 is a schematic diagram of the structure of another robot provided in an embodiment of the present application, the robot comprising:

[0164] An input module 901 is used to input the natural language instructions sent by the user into the large language model to obtain a plurality of continuous task descriptions, wherein the large language model is obtained through training, and the task description is used to characterize the actions to be performed by the robot;

[0165] A traversal module 902 is used to traverse a preset task description set and determine that the plurality of consecutive task descriptions all belong to the preset task description set;

[0166] A determination module 903 is used to determine the current state information of the robot corresponding to the target task description based on the image collected by the robot, wherein the target task description is from the first task description to the last task description in the multiple consecutive task descriptions;

[0167] The judgment module 904 is used to use a multi-step temporal difference reinforcement learning method to judge whether multiple actions to be executed are robot executable strategies based on the current state information and target task description of the robot, wherein the multiple actions to be executed correspond to the multiple continuous task descriptions.

[0168] In one embodiment, the determination module 903 is specifically configured to:

[0169] Determine a first pose of the robot at an initial position based on the image collected by the robot, and use the first pose as current state information of the robot corresponding to a first task description;

[0170] For other task descriptions, based on the images collected by the robot, a second posture of the robot after executing the action to be executed corresponding to the previous task description is determined, and the second posture is used as the current state information of the robot corresponding to the current task description.

[0171] In one embodiment, the determination module 904 is specifically configured to:

[0172] A multi-step temporal difference reinforcement learning method is used for the current state information and target task description of the robot to calculate the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information and the target task description of the robot;

[0173] Based on the calculated action reward values, it is determined whether the multiple actions to be executed are robot executable strategies.

[0174] In one embodiment, the robot further includes a computing module 905, such as Fig.10 As shown;

[0175] The calculation module 905 is used to use a multi-step temporal difference reinforcement learning method to perform reinforcement learning on data representing the current state information, task description and action reward value of the robot to obtain a multi-step temporal difference state-action value function;

[0176] For the current state information and target task description of the robot, based on the state-action value function of the multi-step temporal difference, the current state information of the robot and the target task description, the action reward value of the robot from the current state to the next action to be performed is calculated.

[0177] In one embodiment, the determination module 904 is specifically configured to:

[0178] The calculated action reward values ​​are compared with the preset threshold value. When it is determined that each action reward value is greater than the preset threshold value, the multiple actions to be executed are determined to be executable strategies for the robot. When it is determined that at least one action reward value among the action reward values ​​is less than or equal to the preset threshold value, the multiple actions to be executed are determined to be non-executable strategies for the robot.

[0179] An executable strategy judgment method and robot provided in an embodiment of the present application first determine that multiple consecutive task descriptions belong to a task description set. Therefore, it can be determined that the multiple consecutive task descriptions obtained through a large language model are all actions executable by the robot. Then, in combination with the images collected by the robot, a multi-step temporal difference reinforcement learning method is adopted to determine whether multiple actions to be executed are executable strategies for the robot based on the current state information and task description of the robot, thereby identifying unreasonable actions to be executed for the robot and improving the performance of the robot.

[0180] The present application is described above with reference to the block diagrams and / or flow charts showing the methods, devices (systems) and / or computer program products according to the embodiments of the present application. It should be understood that a block of the block diagram and / or flow chart diagram and a combination of blocks of the block diagram and / or flow chart diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer and / or other programmable data processing device to produce a machine, so that the instructions executed by the computer processor and / or other programmable data processing device create a method for implementing the functions / actions specified in the block diagram and / or flow chart block.

[0181] Accordingly, the present application may also be implemented with hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, the present application may take the form of a computer program product on a computer-usable or computer-readable storage medium, which has a computer-usable or computer-readable program code implemented in the medium, for use by an instruction execution system or in conjunction with an instruction execution system. In the context of the present application, a computer-usable or computer-readable medium may be any medium that may contain, store, communicate, transmit, or convey a program for use by an instruction execution system, device, or apparatus, or in conjunction with an instruction execution system, device, or apparatus.

[0182] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for determining an executable strategy, characterized in that: The method includes: Inputting the natural language instructions sent by the user into the large language model to obtain a plurality of continuous task descriptions, wherein the large language model is obtained through training, and the task descriptions are used to characterize the actions to be performed by the robot; Traversing a preset task description set, and determining that the plurality of consecutive task descriptions all belong to the preset task description set; Based on the image collected by the robot, determine the current state information of the robot corresponding to the target task description, wherein the target task description is from the first task description to the last task description in the multiple consecutive task descriptions; A multi-step temporal difference reinforcement learning method is adopted to determine whether a plurality of actions to be executed are executable strategies for the robot based on the current state information of the robot and the target task description, wherein the plurality of actions to be executed correspond to the plurality of continuous task descriptions.

2. The method according to claim 1, characterized in that The determining of the current state information of the robot corresponding to the target task description based on the image collected by the robot includes: Determine a first pose of the robot at an initial position based on the image collected by the robot, and use the first pose as current state information of the robot corresponding to a first task description; For other task descriptions, based on the images collected by the robot, a second posture of the robot after executing the action to be executed corresponding to the previous task description is determined, and the second posture is used as the current state information of the robot corresponding to the current task description.

3. The method according to claim 1, characterized in that The reinforcement learning method using multi-step temporal difference determines whether multiple actions to be executed are executable strategies for the robot based on the current state information and target task description of the robot, including: A multi-step temporal difference reinforcement learning method is used for the current state information and target task description of the robot to calculate the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information and the target task description of the robot; Based on the calculated action reward values, it is determined whether the multiple actions to be executed are robot executable strategies.

4. The method according to claim 3, characterized in that The reinforcement learning method using multi-step temporal difference calculates the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information of the robot and the target task description, including: A multi-step temporal difference reinforcement learning method is used to perform reinforcement learning on the data representing the robot's current state information, task description, and action reward value, and a multi-step temporal difference state-action value function is obtained; For the current state information and target task description of the robot, based on the state-action value function of the multi-step temporal difference, the current state information of the robot and the target task description, the action reward value of the robot from the current state to the next action to be performed is calculated.

5. The method according to claim 3 or 4, characterized in that The step of determining whether the plurality of actions to be executed are robot executable strategies includes: The calculated action reward values ​​are compared with the preset threshold value. When it is determined that each action reward value is greater than the preset threshold value, the multiple actions to be executed are determined to be executable strategies for the robot. When it is determined that at least one action reward value among the action reward values ​​is less than or equal to the preset threshold value, the multiple actions to be executed are determined to be non-executable strategies for the robot.

6. A robot, characterized in that: including a processor and an actuator; The processor is used to input the natural language instructions sent by the user into the large language model to obtain multiple continuous task descriptions, wherein the large language model is obtained through training, and the task description is used to characterize the robot's to-be-performed actions; traverse a preset task description set to determine that the multiple continuous task descriptions all belong to the preset task description set; based on the image collected by the robot, determine the current state information of the robot corresponding to the target task description, wherein the target task description is from the first task description to the last task description in the multiple continuous task descriptions; adopt a multi-step temporal difference reinforcement learning method, based on the robot's current state information and the target task description, determine whether the multiple to-be-performed actions are robot executable strategies, wherein the multiple to-be-performed actions correspond to the multiple continuous task descriptions; The execution mechanism is used to execute the multiple actions to be executed in sequence when the processor determines that the multiple actions to be executed are robot executable strategies.

7. The robot according to claim 6, characterized in that: The processor is specifically used for: Determine a first pose of the robot at an initial position based on the image collected by the robot, and use the first pose as current state information of the robot corresponding to a first task description; For other task descriptions, based on the images collected by the robot, a second posture of the robot after executing the action to be executed corresponding to the previous task description is determined, and the second posture is used as the current state information of the robot corresponding to the current task description.

8. The robot according to claim 6, characterized in that: The processor is specifically used for: A multi-step temporal difference reinforcement learning method is used for the current state information and target task description of the robot to calculate the action reward value of the robot from the current action to be performed to the next action to be performed based on the current state information and the target task description of the robot; Based on the calculated action reward values, it is determined whether the multiple actions to be executed are robot executable strategies.

9. The robot according to claim 8, characterized in that The processor is specifically used for: A multi-step temporal difference reinforcement learning method is used to perform reinforcement learning on the data representing the robot's current state information, task description, and action reward value, and a multi-step temporal difference state-action value function is obtained; For the current state information and target task description of the robot, based on the state-action value function of the multi-step temporal difference, the current state information of the robot and the target task description, the action reward value of the robot from the current state to the next action to be performed is calculated.

10. The robot according to claim 8 or 9, characterized in that: The processor is specifically used for: The calculated action reward values ​​are compared with the preset threshold value. When it is determined that each action reward value is greater than the preset threshold value, the multiple actions to be executed are determined to be executable strategies for the robot. When it is determined that at least one action reward value among the action reward values ​​is less than or equal to the preset threshold value, the multiple actions to be executed are determined to be non-executable strategies for the robot.

Images