Robot control method and device based on human-robot collaboration and electronic equipment

Abstract

This application provides a robot control method, device, electronic device, and storage medium based on human-robot collaboration. The method includes: determining an initial condition plan set for the robot; determining an environmental state space and a value preference space; determining the joint action value of the robot and human when they collaborate, based on the environmental state space, value preference space, initial condition plan set, and a preset set of executable human actions; determining the predicted human action to be executed by the robot based on the joint action value; determining a first value for each condition plan based on the predicted execution action, and filtering multiple condition plans based on the first value to obtain a target condition plan set; and responding to a control trigger command to determine the robot's target execution action based on the target condition plan set. This application can improve the control efficiency of the robot and the accuracy of human-robot collaborative work.

Description

Technical Field

[0001] This application relates to the fields of reinforcement learning and value alignment, and in particular to a robot control method, device, electronic device and storage medium based on human-machine collaboration. Background Technology

[0002] With the development of artificial intelligence technology, the demand for intelligent robotic assistants is growing. People urgently hope that robots can understand human intentions to better achieve human-machine collaboration. In the field of reinforcement learning, a fixed reward function is typically pre-designed, and the robot continuously interacts with the environment to obtain rewards based on this function. The robot adjusts its behavior based on the feedback from these rewards, eventually learning a corresponding strategy. However, reward functions designed by humans based on experience are unreliable. Generally, it is necessary to wait for the training to stabilize, judge the rationality of the reward function through the robot's behavior, and then adjust the reward function. This method consumes a lot of time and computing power and cannot achieve rapid value alignment in real-time scenarios such as human-machine collaboration. To solve the difficulty of value alignment caused by fixed reward functions, a value preference parameter is introduced, and the reward function is mapped to a parameterized function of value preference. The robot dynamically estimates human value preferences through interaction with humans, and then adjusts its own reward function, ultimately learning a strategy that conforms to human value preferences.

[0003] In related technologies, there are generally two methods to solve robot policies: the first is to model the human-robot collaboration problem as a distributed Markov decision process and use a multi-agent inverse reinforcement learning algorithm to solve the policy; the second is to model the human, transforming the two-player game into a special single-agent partially observable Markov decision process problem for solution. For the first method, after modeling the human-robot collaboration problem as a distributed Markov decision process, humans and robots are independent decision-making entities with their own state spaces and action spaces. The joint action space is the Cartesian product of the human action space and the robot action space, making it difficult to solve. For the second method, because the modeled human behavior deviates from actual human behavior, this method may incorrectly predict human actions to some extent, thus incorrectly estimating the human's current value preferences, ultimately affecting the effectiveness of value alignment. Summary of the Invention

[0004] In view of this, this application provides a robot control method, device, electronic device and storage medium based on human-machine collaboration, which can improve the control efficiency of the robot and improve the accuracy of human-machine collaborative work.

[0005] A first aspect of this application provides a robot control method based on human-machine collaboration, comprising: determining an initial condition plan set for a robot, wherein the initial condition plan set includes multiple condition plans, the condition plans being used to characterize the executable actions of the robot; determining an environmental state space and a value preference space, wherein the environmental state space is used to characterize multiple environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of preference of the human for events; determining a joint action value when the robot and the human cooperate in action based on the environmental state space, the value preference space, the initial condition plan set, and a preset set of executable actions of the human; determining a predicted execution action of the human predicted by the robot based on the joint action value; determining a first value for each condition plan based on the predicted execution action, and filtering multiple condition plans based on the first value to obtain a target condition plan set; and responding to a control trigger command to determine a target execution action of the robot based on the target condition plan set.

[0006] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the joint action value of robot-human collaborative action based on the environmental state space, value preference space, initial condition plan set, and human executable action set, since the environmental state space is used to characterize the various environmental states that the robot and human may be in, and the value preference space is used to characterize the degree of human preference for events, the defined joint action value conforms to human decision-making patterns. By determining the robot's predicted human execution action based on the joint action value, and then determining the first value of each condition plan based on the predicted execution action, on the one hand, the influence of robot and human behavior on future human decisions is considered, mitigating the decrease in solution accuracy caused by the robot's unknown human preferences, thereby improving the accuracy of human-machine collaborative work; on the other hand, by having the robot determine the predicted human execution action, the traversal search of human action decisions is avoided, thereby compressing the human action space, achieving rapid solution, and thus improving the robot's control efficiency. In addition, by filtering multiple condition plans based on the first value, the solution efficiency for the target execution action can be improved, thereby improving the robot's control efficiency.

[0007] In one possible implementation, determining the joint action value of the robot and the human when they cooperate, based on the environmental state space, the value preference space, the initial condition plan set, and the preset set of executable actions of the human, includes: calculating the Cartesian product of the environmental state space and the value preference space, and using the Cartesian product as the joint state space; setting the human to make action decisions based on the joint action value in the set of executable actions of the human, and calculating the joint action value according to the following formula:

[0008] ;

[0009] in, For the value of the joint action, Let be an element of the environmental state space. It is a human decision action within the set of executable human actions. For the aforementioned conditional plan, For a robot, it is a decision action within its set of executable actions. A mapping from the next human action to the next conditional plan. For an element of the value preference space, Let be a joint state of the joint state space. For state The next joint state is transitioned to after the robot and human perform the action.

[0010] In one possible implementation, determining the predicted human action predicted by the robot based on the joint action value includes: calculating the predicted action according to the following formula: ;in, For the predicted execution action; determining the first value of each of the conditional plans based on the predicted execution action includes: calculating the first value according to the following formula:

[0011] ;in, Plan for the conditions in the joint state The first value below, for The reward function, This is the discount factor.

[0012] In one possible implementation, the conditional plan has different first values ​​under different joint states; the filtering of multiple conditional plans based on the first value includes: for each conditional plan's first value, detecting whether a dominant conditional plan exists among the multiple conditional plans, wherein, under the same joint state, the dominant first value of the dominant conditional plan is not less than the first value, and at least one joint state has a dominant first value greater than the first value; if the existence of the dominant conditional plan is detected, the conditional plan is retained; otherwise, the conditional plan is deleted.

[0013] In one possible implementation, the method further includes: calculating the value function scale of the initial conditional plan set based on the first value, and determining a pruning threshold based on the value function scale; arbitrarily selecting a first conditional plan and a second conditional plan from the initial conditional plan set, and calculating feature parameter values ​​according to the following formula:

[0014] M= Where M is the value of the characteristic parameter. The first value of the plan under the first condition. Let be the Euclidean norm of the first value of the first conditional plan. The first value of the second conditional plan. The first value of the second conditional plan is the Euclidean norm; the relationship between the feature parameter value and the pruning threshold is detected; if the feature parameter value is less than the pruning threshold, the first conditional plan and the second conditional plan are retained; if the feature parameter value is greater than or equal to the pruning threshold, the second conditional plan is deleted.

[0015] In one possible implementation, the control trigger command is the environmental state information of the robot's current environment; determining the robot's target action based on the target condition plan set includes: determining a first target value for each target condition plan in the target condition plan set based on the environmental state information; determining the robot's current belief state, wherein the belief state characterizes the degree of human preference for the event predicted by the robot; calculating a second value for the robot to execute the target condition plan in the belief state based on the belief state and the first target value, wherein each target condition plan corresponds to a second value; determining a final condition plan in the target condition plan set based on the second value, and determining the robot's target action based on the final condition plan.

[0016] In one possible implementation, calculating a second value for the robot to execute the target conditional plan in the belief state, based on the belief state and the first target value, includes: calculating the second value according to the following formula:

[0017] ;in, For the second value, This is the state of belief. The first value is the objective; determining the final condition plan from the set of objective condition plans based on the second value includes: determining the final condition plan according to the following formula:

[0018] ;in, For the final condition plan, The target condition plan set.

[0019] Secondly, embodiments of this application also provide a robot control device based on human-machine collaboration, comprising: a first determining module, a second determining module, a third determining module, a prediction module, a filtering module, and a response module; the first determining module is used to determine an initial condition plan set for the robot, wherein the initial condition plan set includes multiple condition plans, and the condition plans are used to characterize the executable actions of the robot; the second determining module is used to determine an environmental state space and a value preference space, wherein the environmental state space is used to characterize multiple environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of preference of the human for events; the third determining module is used to determine the joint action value when the robot and the human cooperate in action based on the environmental state space, the value preference space, the initial condition plan set, and a preset set of executable actions of the human; the prediction module is used to determine the predicted execution action of the human predicted by the robot based on the joint action value; the filtering module is used to determine a first value for each condition plan based on the predicted execution action, and to filter multiple condition plans based on the first value to obtain a target condition plan set; the response module is used to respond to a control trigger command and determine the target execution action of the robot based on the target condition plan set.

[0020] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the robot control method based on human-machine collaboration as described in the first aspect.

[0021] Fourthly, embodiments of this application also provide a storage medium that stores computer instructions, which, when executed on an electronic device, cause the electronic device to perform the human-machine collaborative robot control method as described in the first aspect.

[0022] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the implementation environment of a robot control system provided in an embodiment of this application.

[0024] Figure 2 This is a flowchart illustrating the steps of a human-machine collaborative robot control method provided in an embodiment of this application.

[0025] Figure 3 A flowchart illustrating the steps for determining the target condition plan set provided in an embodiment of this application.

[0026] Figure 4 This is another flowchart of a human-machine collaborative robot control method provided in an embodiment of this application.

[0027] Figure 5 A functional block diagram of a robot control device based on human-machine collaboration provided in an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0030] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0032] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0033] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0034] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0036] Value function scale: refers to the complexity or coverage of the state value function (V function) or action value function (Q function) in reinforcement learning, mainly reflected in the massiveness of the state space or action space.

[0037] Environmental state space: refers to the set of all possible states in an environment, used to describe the dynamic characteristics and evolution of the environment at a specific moment.

[0038] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the implementation environment of the robot control system based on human-machine collaboration provided in the embodiments of this application.

[0039] Those skilled in the art will understand that Figure 1The schematic diagram of the implementation environment of the human-machine collaborative robot system shown is merely an example. The implementation environment described in this application embodiment is for the purpose of more clearly illustrating the technical solution of this application embodiment and does not constitute a limitation on the technical solution provided by this application embodiment. As those skilled in the art will know, with the emergence of new business scenarios, the technical solution provided by this application embodiment is also applicable to similar technical problems.

[0040] exist Figure 1 In this application, the human-machine collaborative robot system provided in the embodiments may include a server 110, a human 120, and a robot 130. The server 110 may integrate the human-machine collaborative robot control device provided in the embodiments of this application.

[0041] Server 110 can communicate with human 120 and robot 130. For example, server 110 and robot 130 can be connected via a wired communication link or a wireless communication link, etc. This application embodiment does not limit this. In addition, server 110 provides an interface for interacting with human 120, and server 110 exchanges information with human 120 through the interface.

[0042] It is understood that server 110 can be a standalone server, or a server network or server cluster. For example, server 110 described in the embodiments of this application includes, but is not limited to, computers, network hosts, single network servers, multiple sets of network servers, or cloud servers composed of multiple servers. Among them, cloud servers are composed of a large number of computers or network servers based on cloud computing.

[0043] Server 110 can execute the robot control method provided in the embodiments of this application to determine the target action to be performed by robot 130.

[0044] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps of an embodiment of the human-machine collaboration-based robot control method of this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted. The human-machine collaboration-based robot control method of this application can be applied to human-machine collaboration-based robot control devices, but is not limited thereto, and the embodiments of this application do not limit this application.

[0045] The specific process of this embodiment is as follows: Figure 1 As shown, it includes the following steps:

[0046] Step 101: Determine the initial condition plan set for the robot, wherein the initial condition plan set includes multiple condition plans, which are used to characterize the robot's executable actions.

[0047] In some embodiments, a conditional plan can be viewed as a specific manifestation of a strategy at a certain stage, represented as... ,in It's the robot's action. It is the robot's observation of current human behavior. Mapping to future conditions plans.

[0048] Step 102: Determine the environmental state space and value preference space.

[0049] Specifically, the environmental state space is used to characterize the various environmental states that robots and humans may be in, while the value preference space is used to characterize the degree of human preference for events.

[0050] Step 103: Determine the value of the joint action when the robot and the human cooperate, based on the environmental state space, value preference space, initial condition plan set, and human executable action set.

[0051] In some embodiments, the joint action value is determined by: calculating the Cartesian product of the environmental state space and the value preference space, and using the Cartesian product as the joint state space; setting humans to make action decisions based on the joint action value within their set of executable actions, and calculating the joint action value according to the following formula:

[0052] ;

[0053] in, For the value of joint operations, As an element of the environmental state space, It refers to a human decision-making action within the set of executable human actions. For the conditional plan, For a robot, it is a decision action within its set of executable actions. A mapping from the next human action to the next conditional plan. As an element of the value preference space, Let a joint state be a joint state in the joint state space. For state The next joint state is transitioned to after the robot and human perform the action.

[0054] It is worth noting that by defining joint action value, we can predict human behavior to compress the action space, accelerate algorithm iteration, and minimize the impact of reduced solution accuracy caused by human modeling.

[0055] Step 104: Determine the predicted human action to be performed by the robot based on the combined action value.

[0056] In some embodiments, the predicted action is calculated according to the following formula:

[0057] ;in, The goal is to predict the action to be executed. In other words, the predicted action is the one that maximizes the value of the combined action state. By modeling humans as individuals who make decisions by choosing the action with the highest combined action state value, the robot can then compare the values ​​of different conditional plans after iterative convergence when calculating the value of the conditional plan to decide the next action.

[0058] Step 105: Determine the first value of each conditional plan based on the predicted execution action, and filter multiple conditional plans based on the first value to obtain the target conditional plan set.

[0059] Specifically, the first value is calculated according to the following formula:

[0060] ;in, The plan is to be in a joint state. The first value is... for The reward function, As a discount factor, For the environment state transition function, Indicates the performance of actions by humans And the robot's actions Under these circumstances, the current environmental state Transfer to environmental state The probability of.

[0061] In some embodiments, a condition plan has different first values ​​under different joint states. Filtering multiple condition plans based on the first value includes: for each condition plan's first value, detecting whether a dominant condition plan exists among the multiple condition plans, wherein, under the same joint state, the dominant first value of each dominant condition plan is not less than the first value, and at least one of the joint states has a dominant first value greater than the first value; if a dominant condition plan is detected, retaining the condition plan; otherwise, deleting the condition plan.

[0062] Specifically, for a newly generated conditional plan a, the value of conditional plan a is compared with that of all conditional plans in the current conditional plan set. If the dominance condition is met, that is, conditional plan a is the dominant conditional plan mentioned above, the dominated conditional plan in the current conditional plan set is removed, and the current conditional plan set is added to the current conditional plan set. If there is a conditional plan b in the current conditional plan set that dominates conditional plan a, conditional plan a is removed. In other cases, that is, conditional plan a does not dominate any conditional plan in the conditional plan set, and there is no conditional plan in the conditional plan set that dominates conditional plan a, then conditional plan a is directly added to the conditional plan set.

[0063] In some embodiments, to accelerate algorithm convergence, the first value of each conditional plan is standardized before filtering multiple conditional plans. Specifically, the standardization is performed using the following formula:

[0064] ;in, Let i be the first value of the i-th conditional plan in the initial set of conditional plans under joint state s. for The primary value after standardization and All are constants.

[0065] In some embodiments, the value function size of the initial conditional plan set is also calculated based on the first value, and the pruning threshold is determined based on the value function size; arbitrarily selecting a first conditional plan and a second conditional plan from the initial conditional plan set, the characteristic parameter value is calculated according to the following formula:

[0066] M= Where M is the characteristic parameter value, The first value of the first condition plan, The Euclidean norm of the first value of the first-condition plan. The first value of the second condition plan, The first value of the second conditional plan is the Euclidean norm; the relationship between the feature parameter value and the pruning threshold is detected; if the feature parameter value is less than the pruning threshold, the first and second conditional plans are retained; if the feature parameter value is greater than or equal to the pruning threshold, the second conditional plan is deleted.

[0067] For ease of understanding, the second screening condition plan method used in this embodiment will be explained in detail below:

[0068] 1. Order The value function representing the scale of all current first values ​​is defined as follows:

[0069] ;in, The size of the current conditional plan set, The Euclidean norm of the first value of the first conditional plan.

[0070] 2. Pruning threshold Based on the scale of the value function Adaptive adjustment:

[0071] ;in, and The initial condition plan set corresponds to the pruning threshold and value function size.

[0072] 3. Retain the second-condition plan if the following constraints are met:

[0073] .

[0074] It is worth noting that by detecting whether there is a dominant conditional plan among multiple conditional plans, first values ​​that do not conform to strict dominance relationships are pruned, thus improving the quality of conditional plans. By detecting the relationship between the characteristic parameter values ​​and the pruning threshold, first values ​​that are approximately dominant are pruned. At the same time, a pruning threshold with dynamic threshold adjustment is set, that is, the pruning threshold is set to be related to the value function size of the current conditional plan set, thereby achieving a balance between computational efficiency and conditional plan quality.

[0075] In some embodiments, such as Figure 3 As shown, a maximum step size is preset, and the initial first step size is set to 1. After removing condition plans from the initial condition plan set using the two methods described above, it is checked whether the current first step size is less than the preset maximum step size. If the current first step size is less than the preset maximum step size, the current first step size is incremented by 1 to generate a new condition plan set. Then, the new first value of each new condition plan in the new condition plan set is recalculated, and low-value condition plans in the new condition plan set are removed again. The above process is repeated until the final first step size equals the preset maximum step size. The final condition plan set is the target condition plan set of this embodiment.

[0076] Specifically, a new set of conditional plans can be generated based on the following formula:

[0077] ;in, For the new set of conditional plans, For the current conditional plan set, This is the set of actions for the robot.

[0078] Step 106: Respond to the control trigger command and determine the robot's target action based on the target condition plan set.

[0079] The method for determining the target action of the robot is described in detail in subsequent embodiments, and will not be repeated here to avoid repetition.

[0080] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the joint action value of robot-human collaborative action based on the environmental state space, value preference space, initial condition plan set, and human executable action set, since the environmental state space is used to characterize the various environmental states that the robot and human may be in, and the value preference space is used to characterize the degree of human preference for events, the defined joint action value conforms to human decision-making patterns. By determining the robot's predicted human execution action based on the joint action value, and then determining the first value of each condition plan based on the predicted execution action, on the one hand, the influence of robot and human behavior on future human decisions is considered, mitigating the decrease in solution accuracy caused by the robot's unknown human preferences, thereby improving the accuracy of human-machine collaborative work; on the other hand, by having the robot determine the predicted human execution action, the traversal search of human action decisions is avoided, thereby compressing the human action space, achieving rapid solution, and thus improving the robot's control efficiency. In addition, by filtering multiple condition plans based on the first value, the solution efficiency for the target execution action can be improved, thereby improving the robot's control efficiency.

[0081] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the steps of an embodiment of the human-machine collaboration-based robot control method of this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted. This human-machine collaboration-based robot control method can be applied to the aforementioned human-machine collaboration-based robot control device, but is not limited thereto, and the embodiments of this application do not limit it in this regard.

[0082] This embodiment is a detailed description of the foregoing embodiments, mainly illustrating how to determine the robot's target execution action based on the target condition plan set. This method ensures the accuracy of the target execution action, thereby further improving the precision of the robot control method.

[0083] The specific process of this embodiment is as follows: Figure 4 As shown, it includes the following steps:

[0084] Step 201: Determine the initial condition plan set for the robot, wherein the initial condition plan set includes multiple condition plans, which are used to characterize the robot's executable actions.

[0085] Step 202: Determine the environmental state space and value preference space.

[0086] Step 203: Determine the value of the joint action when the robot and the human cooperate, based on the environmental state space, value preference space, initial condition plan set, and human executable action set.

[0087] Step 204: Determine the predicted human action to be performed by the robot based on the combined action value.

[0088] Step 205: Determine the first value of each conditional plan based on the predicted execution action, and filter multiple conditional plans based on the first value to obtain the target conditional plan set.

[0089] Steps 201 to 205 in this embodiment are similar to steps 101 to 105 in the previous embodiment. To avoid repetition, they will not be described again here.

[0090] Step 206: Based on the environmental status information, determine the primary value of each target condition plan in the target condition plan set.

[0091] Specifically, the control trigger command is the environmental state information of the robot's current environment. After receiving the environmental state information sent by the environmental sensor, the robot determines the joint state s based on the environmental state information, and then obtains the first target value of each target condition plan in the target condition plan set under the joint state s based on the method of the aforementioned embodiment.

[0092] Step 207: Determine the robot's current belief state, where the belief state is used to characterize the degree of human preference for the event as predicted by the robot.

[0093] Step 208: Based on the belief state and the first value of the goal, calculate the second value of the robot executing the goal condition plan in the belief state.

[0094] Specifically, each target condition plan corresponds to a secondary value.

[0095] In some embodiments, the second value is calculated according to the following formula: ;in, As the second value, For the state of belief, The primary value is to achieve the goal.

[0096] Step 209: Determine the final condition plan in the target condition plan set based on the second value, and determine the robot's target execution action based on the final condition plan.

[0097] In some embodiments, the final condition plan is determined according to the following formula:

[0098] ;in, For the final conditional plan, The set of target conditions is the plan set.

[0099] It is worth noting that this embodiment also updates the robot's belief state based on the robot's target action and the actual action performed by the human, thereby improving the accuracy of the robot control method.

[0100] Specifically, belief state The update formula is:

[0101] ;in, This represents the new belief state. In other words, for each joint state s, if the action actually taken by the human does not maximize the corresponding joint action value Q, then the weight of b(s) is reduced, meaning the probability that the human is in the corresponding preference in joint state s is reduced.

[0102] In some embodiments, a second step size and a maximum number of interaction steps are also set, with the initial value of the second step size set to 1. After updating the robot's belief state, it is also checked whether the size of the second step size is greater than or equal to the maximum number of interaction steps. If the second step size is less than the maximum number of interaction steps, the second step size is incremented by 1, and new environmental state information is reacquired to determine new actions for both the human and the robot. The belief state is then updated again based on the new actions, and the above process is repeated until the second step size equals the maximum number of interaction steps.

[0103] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the joint action value of robot-human collaborative action based on the environmental state space, value preference space, initial condition plan set, and human executable action set, since the environmental state space is used to characterize the various environmental states that the robot and human may be in, and the value preference space is used to characterize the degree of human preference for events, the defined joint action value conforms to human decision-making patterns. By determining the robot's predicted human execution action based on the joint action value, and then determining the first value of each condition plan based on the predicted execution action, on the one hand, the influence of robot and human behavior on future human decisions is considered, mitigating the decrease in solution accuracy caused by the robot's unknown human preferences, thereby improving the accuracy of human-machine collaborative work; on the other hand, by having the robot determine the predicted human execution action, the traversal search of human action decisions is avoided, thereby compressing the human action space, achieving rapid solution, and thus improving the robot's control efficiency. In addition, by filtering multiple condition plans based on the first value, the solution efficiency for the target execution action can be improved, thereby improving the robot's control efficiency.

[0104] Based on the same idea as the human-machine collaboration-based robot control method in the above embodiments, this application also provides a human-machine collaboration-based robot control device, which can be used to execute the above-described human-machine collaboration-based robot control method. For ease of explanation, the structural schematic diagram of the human-machine collaboration-based robot control device embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0105] like Figure 5 As shown, the human-machine collaborative robot control device 50 includes a first determining module 501, a second determining module 502, a third determining module 503, a prediction module 504, a screening module 505, and a response module 506. In some embodiments, the above modules can be programmable software instructions stored in memory and executable by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware embedded in a processor.

[0106] The first determining module 501 is used to determine an initial condition plan set for the robot, wherein the initial condition plan set includes multiple condition plans, and the condition plans are used to characterize the executable actions of the robot.

[0107] The second determining module 502 is used to determine the environmental state space and the value preference space, wherein the environmental state space is used to characterize the various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of human preference for events.

[0108] The third determining module 503 is used to determine the value of the joint action when the robot and the human cooperate, based on the environmental state space, the value preference space, the initial condition plan set, and the preset set of executable actions of the human.

[0109] The prediction module 504 is used to determine the predicted human action predicted by the robot based on the value of the joint action;

[0110] The filtering module 505 is used to determine a first value for each of the condition plans based on the predicted execution action, and to filter multiple condition plans based on the first value to obtain a target condition plan set.

[0111] The response module 506 is used to respond to control trigger commands and determine the target action of the robot according to the target condition plan set.

[0112] Please refer to Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the electronic device of this application.

[0113] The electronic device 100 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. When the processor 30 executes the computer program 40, it implements the steps described in the human-machine collaborative robot control method embodiment above, for example... Figure 1 Steps 101 to 106 are shown.

[0114] For example, computer program 40 can also be divided into one or more modules / units, one or more of which are stored in memory 20 and executed by processor 30. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, the instruction segments describing the execution process of computer program 40 in electronic device 100. For example, it can be divided into the first determining module 501, the second determining module 502, the third determining module 503, the prediction module 504, the filtering module 505, and the response module 506 shown.

[0115] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device 100 may also include input / output devices, network access devices, buses, etc.

[0116] Processor 30 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors, single-chip microcomputers, or any conventional processor.

[0117] The memory 20 can be used to store computer programs 40 and / or modules / units. The processor 30 implements various functions of the electronic device 100 by running or executing the computer programs and / or modules / units stored in the memory 20 and by calling data stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0118] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the storage medium can be appropriately added or removed according to the requirements of patent practice. For example, according to patent practice, the storage medium does not include electrical carrier signals and telecommunication signals.

[0119] The above provides a detailed description of the robot control method, device, electronic device, and storage medium based on human-machine collaboration provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A human-robot collaboration based robot control method, characterized by, include: Determine an initial condition plan set for the robot, wherein the initial condition plan set includes multiple condition plans, which are used to characterize the executable actions of the robot; Define an environmental state space and a value preference space, wherein the environmental state space is used to characterize the various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of preference of the human for events; Based on the environmental state space, the value preference space, the initial condition plan set, and the preset set of executable actions for the human, the joint action value of the robot and the human is determined, including: calculating the Cartesian product of the environmental state space and the value preference space, and using the Cartesian product as the joint state space; setting the human to make action decisions based on the joint action value in the human's set of executable actions, and calculating the joint action value according to the following formula: ； in, For the value of the joint action, Let be an element of the environmental state space. It is a human decision action within the set of executable human actions. For the aforementioned conditional plan, For a robot, it is a decision action within its set of executable actions. A mapping from the next human action to the next conditional plan. For an element of the value preference space, Let be a joint state of the joint state space. For state The next joint state after the robot and human perform actions. Indicates the performance of actions by humans And the robot's actions Under these circumstances, the current environmental state Transfer to environmental state The probability of; The robot predicts the human's predicted action based on the value of the combined action; A first value is determined for each of the conditional plans based on the predicted execution action, and multiple conditional plans are filtered based on the first value to obtain a target conditional plan set; In response to a control trigger command, the robot determines its target action based on the target condition plan set. Determining the predicted human action predicted by the robot based on the joint action value includes: calculating the predicted action according to the following formula: ; wherein, is the predicted execution action; Determining the first value of each of the conditional plans based on the predicted execution action includes: calculating the first value according to the following formula: ; wherein, is the first value of the conditional plan in joint state under the condition, is a reward function, is a discount factor.

2. The human-robot collaboration based robot control method of claim 1, wherein, The conditional plan has different first values ​​under different joint states; The step of filtering the multiple conditional plans based on the first value includes: For each of the first values ​​of the conditional plans, it is detected whether there is a dominant conditional plan among the plurality of conditional plans, wherein, under the same joint state, the dominant first value of the dominant conditional plan is not less than the first value, and at least in one of the joint states, the dominant first value is greater than the first value; If the dominance condition plan is detected, the condition plan is retained; otherwise, the condition plan is deleted.

3. The human-robot collaboration based robot control method of claim 2, wherein, The method further includes: Calculate the value function size of the initial condition plan set based on the first value, and determine the pruning threshold based on the value function size; Arbitrarily select a first conditional plan and a second conditional plan from the initial conditional plan set, and calculate the characteristic parameter values ​​according to the following formula: M = 1 ; wherein M is the characteristic parameter value, is a first value of the first condition plan, is a Euclidean norm of the first value of the first condition plan, is a first value of the second condition plan, is a Euclidean norm of the first value of the second condition plan; The relationship between the feature parameter value and the pruning threshold is detected. If the feature parameter value is less than the pruning threshold, the first conditional plan and the second conditional plan are retained. If the feature parameter value is greater than or equal to the pruning threshold, the second conditional plan is deleted.

4. The human-robot collaboration based robot control method of claim 1, wherein, The control trigger command is the environmental status information of the robot's current environment; The step of determining the target action of the robot based on the target condition plan set includes: Based on the environmental state information, determine the primary target value for each target condition plan in the target condition plan set; Determine the robot's current belief state, wherein the belief state is used to characterize the degree of human preference for the event as predicted by the robot; Based on the belief state and the first value of the target, a second value is calculated for the robot to execute the target condition plan in the belief state, wherein each target condition plan corresponds to a second value; Based on the second value, a final condition plan is determined from the set of target condition plans, and the target action of the robot is determined based on the final condition plan.

5. The human-robot collaboration based robot control method of claim 4, wherein, The step of calculating the second value of the robot executing the target condition plan in the belief state, based on the belief state and the first value of the target, includes: The second value is calculated using the following formula: ; wherein, is the second value, is the belief state, is the target first value; The step of determining the final condition plan from the set of target condition plans based on the second value includes: determining the final condition plan according to the following formula: ; wherein, is the final condition plan, is the set of target condition plans.

6. A human-robot collaboration based robot control device, characterized by, include: The module comprises a first determination module, a second determination module, a third determination module, a prediction module, a filtering module, and a response module; The first determining module is used to determine an initial condition plan set for the robot, wherein the initial condition plan set includes multiple condition plans, and the condition plans are used to characterize the executable actions of the robot; The second determining module is used to determine the environmental state space and the value preference space, wherein the environmental state space is used to characterize the various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of preference of the human for events; The third determining module is used to determine the joint action value of the robot and the human when they cooperate, based on the environmental state space, the value preference space, the initial condition plan set, and the preset set of executable actions of the human. This includes: calculating the Cartesian product of the environmental state space and the value preference space, and using the Cartesian product as the joint state space; setting the human to make action decisions based on the joint action value in the set of executable actions of the human, and calculating the joint action value according to the following formula: ； in, For the value of the joint action, Let be an element of the environmental state space. It is a human decision action within the set of executable human actions. For the aforementioned conditional plan, For a robot, it is a decision action within its set of executable actions. A mapping from the next human action to the next conditional plan. For an element of the value preference space, Let be a joint state of the joint state space. For state The next joint state after the robot and human perform actions. Indicates the performance of actions by humans And the robot's actions Under these circumstances, the current environmental state Transfer to environmental state The probability of; The prediction module is used to determine the predicted human action to be performed by the robot based on the value of the joint action. The filtering module is used to determine a first value for each of the condition plans based on the predicted execution action, and to filter multiple condition plans based on the first value to obtain a target condition plan set. The response module is used to respond to control trigger commands and determine the target action to be performed by the robot based on the target condition plan set. Determining the predicted human action predicted by the robot based on the joint action value includes: calculating the predicted action according to the following formula: ; wherein, is the predicted execution action; Determining the first value of each of the conditional plans based on the predicted execution action includes: calculating the first value according to the following formula: ;in, Plan for the conditions in the joint state The first value below, for The reward function, This is the discount factor. 7.An electronic device comprising a processor and a memory, wherein The memory is used to store instructions, and the processor is used to call the instructions in the memory, causing the electronic device to execute the robot control method based on human-machine collaboration as described in any one of claims 1 to 5.

8. A storage medium, characterized by The storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the human-machine collaborative robot control method as described in any one of claims 1 to 5.

Images