Man-machine cooperative robot control method and device, electronic equipment and storage medium

Abstract

The invention provides a man-machine cooperative robot control method and device, electronic equipment and a storage medium. The method comprises the steps that an initial condition plan set of a robot is determined; determining an environment state space and a value preference space; according to the environment state space, the value preference space, the initial condition plan set and a preset human executable action set, determining a joint action value when the robot and the human cooperatively act; determining a predicted execution action of the human being predicted by the robot according to the joint action value; determining a first value of each condition plan according to the predicted execution action, and screening the plurality of condition plans according to the first values to obtain a target condition plan set; and in response to the control trigger instruction, determining a target execution action of the robot according to the target condition plan set. The control efficiency of the robot can be improved, and the precision of man-machine cooperative work can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reinforcement learning and value alignment, and in particular to a human-robot collaborative robot control method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, people's demand for intelligent agentized machine assistants is growing. People are eager for robots to understand human intentions in order to better achieve human-robot collaboration. In the field of reinforcement learning, a fixed reward function is generally designed in advance, and the robot interacts with the environment according to this reward function to obtain rewards, adjusts its behavior through the feedback of the rewards, and finally learns the corresponding strategy. However, the reward function designed by human experience is not reliable, and generally needs to be adjusted after the training is stable, and then the robot's behavior is used to judge whether the reward function is reasonable, which consumes a lot of time and computing power, and cannot achieve fast value alignment in real-time scenarios such as human-robot collaboration. In order to solve the value alignment difficulty problem caused by the fixed reward function, a value preference parameter is introduced, and the reward function is mapped to a parameterized function about the value preference. The robot dynamically estimates the human's value preference through interaction with the human, and then adjusts its reward function, and finally learns a strategy that meets the human's value preference.

[0003] In related technologies, there are generally two methods to solve the robot strategy: the first method models the human-robot collaboration problem as a decentralized Markov decision process, and uses a multi-agent inverse reinforcement learning algorithm to solve the strategy; the second method models the human, and converts the two-player game into a special single-agent partially observable Markov decision process problem for solving. For the first method, after modeling the human-robot collaboration problem as a decentralized Markov decision process, the human and the robot are independent decision-making individuals with their own state space and action space, and the joint action space is the Cartesian product of the human action space and the robot action space, which is difficult to solve. For the second method, since the modeled human behavior deviates from the actual human behavior, this method will incorrectly predict the human's action to some extent, and then incorrectly estimate the human's current value preference, ultimately affecting the value alignment effect. SUMMARY

[0004] Therefore, the present application provides a human-robot collaborative robot control method and device, electronic equipment and a storage medium, which can improve the control efficiency of the robot and improve the accuracy of human-robot collaboration.

[0005] The first aspect of the embodiment of the present application provides a human-robot collaborative robot control method, comprising: determining an initial condition plan set of a robot, wherein the initial condition plan set comprises a plurality of condition plans, and the condition plan is used to represent an executable action of the robot; determining an environment state space and a value preference space, wherein the environment state space is used to represent a plurality of environment states in which the robot and a human can be, and the value preference space is used to represent a preference degree of the human to an event; determining a joint action value when the robot and the human collaboratively act according to the environment state space, the value preference space, the initial condition plan set and a preset executable action set of the human; determining a predicted execution action of the human predicted by the robot according to the joint action value; determining a first value of each condition plan according to the predicted execution action, and screening a plurality of condition plans according to the first value to obtain a target condition plan set; and determining a target execution action of the robot according to the target condition plan set in response to a control trigger instruction.

[0006] Compared with the related art, the embodiment of the present application has at least the following advantages: by determining a joint action value when the robot and the human collaboratively act according to the environment state space, the value preference space, the initial condition plan set and the executable action set of the human, since the environment state space is used to represent a plurality of environment states in which the robot and the human can be, and the value preference space is used to represent a preference degree of the human to an event, the defined joint action value conforms to the human decision-making mode. By determining a predicted execution action of the human predicted by the robot according to the joint action value, and then determining a first value of each condition plan according to the predicted execution action, on the one hand, the influence of the robot and the human behavior on the future human decision-making is considered, the decrease of the solving accuracy caused by the unknown preference of the robot to the human is reduced, and thus the accuracy of the human-robot collaborative work is improved; on the other hand, by determining the predicted execution action of the human by the robot, the exhaustive search of the action decision of the human is avoided, and thus the action space of the human can be compressed, the fast solving is realized, and thus the control efficiency of the robot is improved. In addition, by screening a plurality of condition plans according to the first value, the solving efficiency of the target execution action can be improved, and thus the control efficiency of the robot is improved.

[0007] In a possible implementation, the determining the joint action value when the robot and the human perform actions cooperatively according to the environment state space, the value preference space, the initial condition plan set, and the preset executable action set of the human, comprises: calculating a Cartesian product of the environment state space and the value preference space, taking the Cartesian product as a joint state space; setting the human to make an action decision in the executable action set of the human according to the joint action value, and calculating the joint action value according to the following formula: ; wherein, is the joint action value, is an element of the environment state space, is a human decision action in the executable action set of the human, is the condition plan, is a robot decision action in the executable action set of the robot, is a mapping from a human next action to a next condition plan, is an element of the value preference space, is a joint state of the joint state space, is a state to which the robot and the human transfer after performing actions.

[0008] In a possible implementation, the determining the predicted execution action of the human predicted by the robot according to the joint action value comprises: calculating the predicted execution action according to the following formula: ; wherein, is the predicted execution action; the determining the first value of each condition plan according to the predicted execution action comprises: calculating the first value according to the following formula: ; wherein, is the first value of the condition plan in the joint state , is a reward function of , is a discount factor.

[0009] In a possible implementation, the conditional plan has different first values in different joint states; the screening of the plurality of conditional plans according to the first values comprises: for the first value of each conditional plan, detecting whether there is a dominated conditional plan in the plurality of conditional plans, wherein the first value of the dominated conditional plan is not less than the first value in the same joint state, and the first value of the dominated conditional plan is greater than the first value in at least one joint state; in a case where it is detected that there is the dominated conditional plan, retaining the conditional plan; otherwise, deleting the conditional plan.

[0010] In a possible implementation, the method further comprises: calculating a value function size of the initial conditional plan set according to the first values, and determining a pruning threshold according to the value function size; selecting a first conditional plan and a second conditional plan from the initial conditional plan set at random, and calculating a feature parameter value according to the following formula: M= ; wherein M is the feature parameter value, is the first value of the first conditional plan, is the Euclidean norm of the first value of the first conditional plan, is the first value of the second conditional plan, is the Euclidean norm of the first value of the second conditional plan; detecting a size relationship between the feature parameter value and the pruning threshold, and in a case where it is detected that the feature parameter value is less than the pruning threshold, retaining the first conditional plan and the second conditional plan; in a case where it is detected that the feature parameter value is greater than or equal to the pruning threshold, deleting the second conditional plan.

[0011] In a possible implementation, the control trigger instruction is environment state information of a current environment of the robot; and the determining of the target execution action of the robot according to the target conditional plan set comprises: determining a target first value of each target conditional plan in the target conditional plan set according to the environment state information; determining a current belief state of the robot, wherein the belief state is used to represent a predicted preference degree of the human to an event; calculating a second value of the robot in the belief state for executing the target conditional plan according to the belief state and the target first value, wherein each target conditional plan corresponds to a second value; determining a final conditional plan in the target conditional plan set according to the second value, and determining the target execution action of the robot according to the final conditional plan.

[0012] In a possible implementation, the calculating, according to the belief state and the target first value, of a second value of the robot performing the target conditional plan in the belief state comprises: calculating the second value according to the following formula: ; wherein, the second value is V2, the belief state is B, the target first value is V1; and the determining, according to the second value, of a final conditional plan in the target conditional plan set comprises: determining the final conditional plan according to the following formula: ; wherein, the final conditional plan is F, the target conditional plan set is T.

[0013] In a second aspect, the embodiments of the present application further provide a human-robot collaborative robot control device, comprising: a first determining module, a second determining module, a third determining module, a prediction module, a screening module, and a response module; the first determining module is configured to determine an initial conditional plan set of a robot, wherein the initial conditional plan set comprises a plurality of conditional plans, and the conditional plan is configured to represent an executable action of the robot; the second determining module is configured to determine an environment state space and a value preference space, wherein the environment state space is configured to represent a plurality of environment states in which the robot and a human can be located, and the value preference space is configured to represent a preference degree of the human for an event; the third determining module is configured to determine a joint action value of the robot and the human when performing a collaborative action according to the environment state space, the value preference space, the initial conditional plan set, and a preset executable action set of the human; the prediction module is configured to determine a predicted execution action of the human predicted by the robot according to the joint action value; the screening module is configured to determine a first value of each conditional plan according to the predicted execution action, and screen a plurality of conditional plans according to the first value to obtain a target conditional plan set; and the response module is configured to respond to a control trigger instruction, and determine a target execution action of the robot according to the target conditional plan set.

[0014] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a processor and a memory, wherein the memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device performs the human-robot collaborative robot control method as described in the first aspect.

[0015] In a fourth aspect, the embodiments of the present application further provide a storage medium, and the computer readable storage medium stores computer instructions, when the computer instructions run on an electronic device, the electronic device executes the human-robot collaborative robot control method as described in the first aspect.

[0016] The technical effects obtained by the second aspect, the third aspect and the fourth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An implementation environment schematic diagram of the robot control system provided by an embodiment of the present application.

[0018] Figure 2 A step flowchart of the human-robot collaborative robot control method provided by an embodiment of the present application.

[0019] Figure 3 A step flowchart of the target condition plan set determination provided by an embodiment of the present application.

[0020] Figure 4 Another step flowchart of the human-robot collaborative robot control method provided by an embodiment of the present application.

[0021] Figure 5 A functional module diagram of the human-robot collaborative robot control device provided by an embodiment of the present application.

[0022] Figure 6 A structure schematic diagram of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0024] In the following description, many specific details are set forth in order to fully understand the present application. The described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application.

[0026] It is further noted that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0027] In the present application, "at least one" means one or more, and "multiple" means two or more than two. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0028] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words such as "exemplary" or "for example" are intended to present the relevant concept in a specific manner.

[0029] For ease of understanding, some exemplary descriptions of concepts related to the embodiments of the present application are given for reference.

[0030] Value function size: refers to the complexity or coverage of the state value function (V function) or action value function (Q function) in reinforcement learning, mainly embodied in the huge amount of state space or action space.

[0031] Environment state space: refers to the set of all possible states in the environment, used to describe the dynamic characteristics and evolution law of the environment at a specific time.

[0032] Please refer to Figure 1 , Figure 1 An implementation environment schematic diagram of the human-robot collaborative robot control system provided by the embodiments of the present application is shown.

[0033] Those skilled in the art can understand that Figure 1The implementation environment schematic diagram of the illustrated human-robot collaborative robot system is merely an example, and the implementation environment described in the embodiments of the present application is used to more clearly illustrate the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0034] In Figure 1 , the human-robot collaborative robot system provided by the embodiments of the present application can include a server 110, a human 120, and a robot 130. The server 110 can be integrated with the human-robot collaborative robot control device provided by the embodiments of the present application.

[0035] The server 110 can be in communication connection with the human 120 and the robot 130. For example, the server 110 and the robot 130 can be connected by a wired communication link or a wireless communication link, and the embodiments of the present application do not limit this. In addition, the server 110 provides an interface for interacting with the human 120, and the server 110 and the human 120 exchange information through the interface.

[0036] It can be understood that the server 110 can be a stand-alone server, or a server network or a server cluster composed of servers. For example, the server 110 described in the embodiments of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0037] The server 110 can execute the robot control method provided by the embodiments of the present application to determine the target execution action of the robot 130.

[0038] Please refer to Figure 2 , Figure 2 is a step flowchart of an embodiment of the human-robot collaborative robot control method of the present application. The order of the steps in the flowchart can be changed according to different needs, and some steps can be omitted. The human-robot collaborative robot control method of the present application can be applied to a human-robot collaborative robot control device, but is not limited thereto, and the embodiments of the present application do not limit this.

[0039] The specific flow of the embodiment is as shown in Figure 1 , including the following steps: Step 101, determining an initial condition plan set of the robot, wherein the initial condition plan set includes a plurality of condition plans, and the condition plan is used to represent an executable action of the robot.

[0040] In some embodiments, the conditional plan can be regarded as a specific performance of the strategy at a certain stage, denoted as wherein is an action of the robot, is an observation of the robot on the current human behavior a mapping from the future conditional plan.

[0041] Step 102, determining an environment state space and a value preference space.

[0042] Specifically, the environment state space is used to represent various environment states in which the robot and the human can be, and the value preference space is used to represent the preference degree of the human on events.

[0043] Step 103, determining a joint action value when the robot and the human perform actions cooperatively according to the environment state space, the value preference space, an initial conditional plan set and a set of executable actions of the human.

[0044] In some embodiments, the joint action value is determined by calculating a Cartesian product of the environment state space and the value preference space, taking the Cartesian product as a joint state space; setting the human to make an action decision in the set of executable actions of the human according to the joint action value, and calculating the joint action value according to the following formula: ; wherein is the joint action value, is an element of the environment state space, is a human decision action in the set of executable actions of the human, is a conditional plan, is a robot decision action in the set of executable actions of the robot, is a mapping from the next action of the human to the next conditional plan, is an element of the value preference space, is a joint state of the joint state space, is a state to which the next joint state is transferred after the robot and the human perform actions.

[0045] It is worth noting that by defining the joint action value, the human behavior can be predicted to compress the action space, accelerate the algorithm iteration, and reduce the influence of the solution accuracy reduction caused by human modeling as much as possible.

[0046] Step 104, determining a predicted execution action of the human predicted by the robot according to the joint action value.

[0047] In some embodiments, the predicted execution action is calculated according to the following formula: ; wherein, is the predicted execution action. That is, the predicted execution action is an action that can maximize the joint action value. By modeling the human as an individual who makes decisions by selecting the action with the highest joint action state value, the subsequent robot can compare the values of different conditional plans when calculating the value of the conditional plan to make decisions about the next action after iterative convergence.

[0048] Step 105, determining a first value of each conditional plan according to the predicted execution action, and screening the plurality of conditional plans according to the first value to obtain a target conditional plan set.

[0049] Specifically, the first value is calculated according to the following formula: ; wherein, is the first value of the conditional plan in the joint state , is the reward function of , is a discount factor, is an environment state transition function, represents the probability that the current environment state transitions to the environment state when the human performs the action and the robot performs the action .

[0050] In some embodiments, the conditional plans have different first values in different joint states, and screening the plurality of conditional plans according to the first value includes: for the first value of each conditional plan, detecting whether there is a dominated conditional plan in the plurality of conditional plans, wherein the dominated first value of the dominated conditional plan is not less than the first value in the same joint state, and the dominated first value is greater than the first value in at least one of the joint states; in the case where it is detected that there is a dominated conditional plan, retaining the conditional plan; otherwise, deleting the conditional plan.

[0051] Specifically, for the newly generated conditional plan a, the values of the conditional plan a and all conditional plans in the current conditional plan set are compared, if the domination condition is met, that is, the conditional plan a is the above-mentioned dominated conditional plan, the dominated conditional plans in the current conditional plan set are removed, and the current conditional plan set is joined by the current conditional plan a; if there is a conditional plan b in the current conditional plan set that dominates the conditional plan a, the conditional plan a is removed; in other cases, that is, the conditional plan a does not dominate any conditional plan in the conditional plan set, and there is no conditional plan in the conditional plan set that dominates the conditional plan a, the conditional plan a is directly added to the conditional plan set.

[0052] In some embodiments, to accelerate algorithm convergence, the first value of each conditional plan is standardized before filtering multiple conditional plans. Specifically, the standardization is performed using the following formula: ;in, Let i be the first value of the i-th conditional plan in the initial set of conditional plans under joint state s. for The primary value after standardization and All are constants.

[0053] In some embodiments, the value function size of the initial conditional plan set is also calculated based on the first value, and the pruning threshold is determined based on the value function size; arbitrarily selecting a first conditional plan and a second conditional plan from the initial conditional plan set, the characteristic parameter value is calculated according to the following formula: M= Where M is the characteristic parameter value, The first value of the first condition plan, The Euclidean norm of the first value of the first-condition plan. The first value of the second condition plan, The first value of the second conditional plan is the Euclidean norm; the relationship between the feature parameter value and the pruning threshold is detected; if the feature parameter value is less than the pruning threshold, the first and second conditional plans are retained; if the feature parameter value is greater than or equal to the pruning threshold, the second conditional plan is deleted.

[0054] For ease of understanding, the second screening condition plan method used in this embodiment will be explained in detail below: 1. Order The value function representing the scale of all current first values ​​is defined as follows: ;in, The size of the current conditional plan set, The Euclidean norm of the first value of the first conditional plan.

[0055] 2. Pruning threshold Based on the scale of the value function Adaptive adjustment: ;in, and The initial condition plan set corresponds to the pruning threshold and value function size.

[0056] 3. Retain the second-condition plan if the following constraints are met: .

[0057] It is worth noting that by detecting whether there is a dominant conditional plan among multiple conditional plans, first values ​​that do not conform to strict dominance relationships are pruned, thus improving the quality of conditional plans. By detecting the relationship between the characteristic parameter values ​​and the pruning threshold, first values ​​that are approximately dominant are pruned. At the same time, a pruning threshold with dynamic threshold adjustment is set, that is, the pruning threshold is set to be related to the value function size of the current conditional plan set, thereby achieving a balance between computational efficiency and conditional plan quality.

[0058] In some embodiments, such as Figure 3 As shown, a maximum step size is preset, and the initial first step size is set to 1. After removing condition plans from the initial condition plan set using the two methods described above, it is checked whether the current first step size is less than the preset maximum step size. If the current first step size is less than the preset maximum step size, the current first step size is incremented by 1 to generate a new condition plan set. Then, the new first value of each new condition plan in the new condition plan set is recalculated, and low-value condition plans in the new condition plan set are removed again. The above process is repeated until the final first step size equals the preset maximum step size. The final condition plan set is the target condition plan set of this embodiment.

[0059] Specifically, a new set of conditional plans can be generated based on the following formula: ;in, For the new set of conditional plans, For the current conditional plan set, This is the set of actions for the robot.

[0060] Step 106: Respond to the control trigger command and determine the robot's target action based on the target condition plan set.

[0061] The method for determining the target action of the robot is described in detail in subsequent embodiments, and will not be repeated here to avoid repetition.

[0062] Compared with the related art, the embodiment of the application has at least the following advantages: by determining the joint action value when the robot and the human collaboratively act according to the environment state space, the value preference space, the initial condition plan set and the set of executable actions of the human, since the environment state space is used to represent a plurality of environment states in which the robot and the human can be, and the value preference space is used to represent the preference degree of the human to an event, the defined joint action value conforms to the human decision-making mode. By determining the predicted execution action of the human predicted by the robot according to the joint action value, and then determining the first value of each condition plan according to the predicted execution action, on the one hand, the influence of the robot and the human behavior on the future human decision-making is considered, the decrease in solving accuracy caused by the robot's unknown preference for the human is alleviated, and thus the accuracy of human-robot collaborative work is improved; on the other hand, by determining the predicted execution action of the human by the robot, the traversal search of the action decision of the human is avoided, and thus the action space of the human can be compressed, fast solving is achieved, and thus the control efficiency of the robot is improved. In addition, by screening a plurality of condition plans according to the first value, the solving efficiency of the target execution action can be improved, and thus the control efficiency of the robot is improved.

[0063] Please refer to Figure 4 , Figure 4 is a step flowchart of an embodiment of the robot control method for human-robot collaboration of the present application. The order of the steps in the flowchart can be changed according to different needs, and some steps can be omitted. The robot control method for human-robot collaboration can be applied to the robot control device for human-robot collaboration described above, but is not limited thereto, and the embodiments of the present application do not limit this.

[0064] The embodiment is a specific description of the foregoing embodiment, mainly explaining how to determine the target execution action of the robot according to the set of target condition plans. In this way, the accuracy of the target execution action can be ensured, and thus the accuracy of the robot control method is further improved.

[0065] The specific process of the embodiment is shown in Figure 4 , including the following steps: Step 201, determining the initial condition plan set of the robot, wherein the initial condition plan set includes a plurality of condition plans, and the condition plan is used to represent the executable action of the robot.

[0066] Step 202, determining the environment state space and the value preference space.

[0067] Step 203, determining the joint action value when the robot and the human collaboratively act according to the environment state space, the value preference space, the initial condition plan set and the set of executable actions of the human.

[0068] Step 204, determining a predicted execution action of the human predicted by the robot according to the joint action value.

[0069] Step 205, determining a first value of each conditional plan according to the predicted execution action, and screening the plurality of conditional plans according to the first value to obtain a target conditional plan set.

[0070] Steps 201 to 205 of the embodiment are similar to steps 101 to 105 of the foregoing embodiment, and are not described here again to avoid repetition.

[0071] Step 206, determining a target first value of each target conditional plan in the target conditional plan set according to the environment state information.

[0072] Specifically, the control trigger instruction is the environment state information of the current environment of the robot. After receiving the environment state information sent by the environment sensor, the robot determines the joint state s according to the environment state information, and then obtains the target first value of each target conditional plan in the target conditional plan set under the joint state s in the manner of the foregoing embodiment.

[0073] Step 207, determining a current belief state of the robot, wherein the belief state is used to represent the preference degree of the human predicted by the robot to an event.

[0074] Step 208, calculating a second value of the robot in the belief state executing the target conditional plan according to the belief state and the target first value.

[0075] Specifically, each target conditional plan corresponds to a second value.

[0076] In some embodiments, the second value is calculated according to the following formula: ; wherein, the second value is V, the belief state is B, and the target first value is V.

[0077] Step 209, determining a final conditional plan in the target conditional plan set according to the second value, and determining a target execution action of the robot according to the final conditional plan.

[0078] In some embodiments, the final conditional plan is determined according to the following formula: ; wherein, the final conditional plan is F, and the target conditional plan set is T.

[0079] It is worth noting that the embodiment also updates the belief state of the robot according to the target execution action of the robot and the actual execution action of the human, thereby improving the accuracy of the robot control method.

[0080] Specifically, the belief state is updated according to the following formula: ; wherein, is the new belief state. That is, for each joint state s, if the current actual execution action of the human does not maximize the corresponding joint action value Q, the weight b(s) is reduced, that is, the possibility of the human being in the joint state s corresponding to the preference is reduced.

[0081] In some embodiments, a second step size and a maximum interaction step number are also set, and the initial value of the second step size is set to 1. After updating the belief state of the robot, it is also detected whether the size of the second step size is greater than or equal to the maximum interaction step number. If the second step size is less than the maximum interaction step number, the second step size is increased by 1, and new environment state information is obtained again to determine the new action of the human and the robot, and the belief state is updated again based on the new action. The above process is repeated until the second step size is equal to the maximum interaction step number.

[0082] Compared with the related art, the embodiment of the present application has at least the following advantages: by determining the joint action value of the collaborative action of the robot and the human according to the environment state space, the value preference space, the initial condition plan set and the executable action set of the human, since the environment state space is used to represent the various environment states in which the robot and the human can be in, and the value preference space is used to represent the preference degree of the human to the event, the defined joint action value conforms to the human decision-making mode. By determining the predicted execution action of the human predicted by the robot according to the joint action value, and then determining the first value of each condition plan according to the predicted execution action, on the one hand, the influence of the behavior of the robot and the human on future human decision-making is considered, the decrease in solving accuracy caused by the robot's unknown preference for the human is reduced, thereby improving the accuracy of human-robot collaborative work; on the other hand, by determining the predicted execution action of the human by the robot, the traversal search of the action decision of the human is avoided, thereby the action space of the human can be compressed, fast solving is realized, and the control efficiency of the robot is improved. In addition, by screening the multiple condition plans according to the first value, the solving efficiency of the target execution action can be improved, thereby the control efficiency of the robot is improved.

[0083] Based on the same idea as the robot control method of human-robot collaboration in the above embodiment, the application also provides a robot control device for human-robot collaboration, which can be used to execute the robot control method of human-robot collaboration. For the convenience of description, only the parts related to the embodiments of the application are shown in the structural schematic diagram of the robot control device for human-robot collaboration, and it can be understood by those skilled in the art that the diagram does not constitute a limitation on the device, and it can include more or less components than the diagram, or combine some components, or arrange different components.

[0084] As shown in Figure 5 , the robot control device for human-robot collaboration 50 includes a first determining module 501, a second determining module 502, a third determining module 503, a predicting module 504, a screening module 505, and a responding module 506. In some embodiments, the above modules can be programmable software instructions stored in a memory and executable by a processor. It can be understood that in other embodiments, the above modules can also be program instructions or firmware fixed in the processor.

[0085] The first determining module 501 is configured to determine an initial condition plan set of a robot, wherein the initial condition plan set includes a plurality of condition plans, and each condition plan is used to represent an executable action of the robot; The second determining module 502 is configured to determine an environment state space and a value preference space, wherein the environment state space is used to represent a plurality of environment states in which the robot and a human can be, and the value preference space is used to represent a preference degree of the human to an event; The third determining module 503 is configured to determine a joint action value when the robot and the human collaboratively act according to the environment state space, the value preference space, the initial condition plan set, and a preset executable action set of the human; The predicting module 504 is configured to determine a predicted execution action of the human predicted by the robot according to the joint action value; The screening module 505 is configured to determine a first value of each condition plan according to the predicted execution action, and screen a plurality of condition plans according to the first value to obtain a target condition plan set; The responding module 506 is configured to respond to a control trigger instruction, and determine a target execution action of the robot according to the target condition plan set.

[0086] Please refer to Figure 6 , Figure 6 for the schematic diagram of an embodiment of the electronic device of the application.

[0087] The electronic device 100 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. The processor 30 implements the steps in the above-described human-robot collaborative robot control method embodiments when executing the computer program 40, for example Figure 1 The steps 101-106 are shown.

[0088] By way of example, the computer program 40 can also be divided into one or more modules / units, one or more of which are stored in the memory 20 and executed by the processor 30. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 40 in the electronic device 100. For example, the first determining module 501, the second determining module 502, the third determining module 503, the predicting module 504, the screening module 505, and the responding module 506 shown can be divided.

[0089] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 100 and does not constitute a limitation on the electronic device 100, which can include more or fewer components than the diagram, or combine certain components, or different components, for example, the electronic device 100 can also include an input / output device, a network access device, a bus, etc.

[0090] The processor 30 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip computer, or the processor 30 can also be any conventional processor, etc.

[0091] The memory 20 can be used to store a computer program 40 and / or modules / units, and the processor 30 realizes various functions of the electronic device 100 by running or executing the computer program and / or modules / units stored in the memory 20 and calling data stored in the memory 20. The memory 20 can mainly include a program storage area and a data storage area, where the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device.

[0092] The modules / units integrated in the electronic device 100, if implemented in the form of software function units and sold or used as independent products, can be stored in a storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be implemented by a computer program instructing related hardware to complete, and the computer program can be stored in a storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The storage medium can include any entity or device capable of carrying computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the storage medium can be appropriately increased or decreased according to the requirements of patent practice. For example, according to patent practice, the storage medium does not include electrical carrier signals and telecommunication signals.

[0093] The above provides a human-computer collaborative robot control method, device, electronic device and storage medium, and the principle and implementation manner of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description of the specification should not be understood as limiting the present application.

Claims

1. A human-robot collaboration method of controlling a robot, characterized by, The method comprises the following steps: determining an initial conditional plan set of a robot, wherein the initial conditional plan set comprises a plurality of conditional plans for representing executable actions of the robot; determining an environment state space and a value preference space, wherein the environment state space is used for representing a plurality of environment states in which the robot and a human being can be, and the value preference space is used for representing a preference degree of the human being to an event; determining a joint action value of the robot and the human being when they act together according to the environment state space, the value preference space, the initial conditional plan set and a preset executable action set of the human being; determining a predicted execution action of the human being predicted by the robot according to the joint action value; determining a first value of each conditional plan according to the predicted execution action, and screening a plurality of conditional plans according to the first value to obtain a target conditional plan set; determining a target execution action of the robot according to the target conditional plan set in response to a control trigger instruction.

2. The human-robot collaboration method of claim 1, wherein, The step of determining the joint action value of the robot and the human being when they act together according to the environment state space, the value preference space, the initial conditional plan set and the preset executable action set of the human being comprises: calculating a Cartesian product of the environment state space and the value preference space as a joint state space; setting the human being to make an action decision in the executable action set of the human being according to the joint action value, and calculating the joint action value according to the following formula: ； wherein, is the joint action value, is one element of the environment state space, is one human decision action of the set of executable actions of the human, is the conditional plan, is one robot decision action of the set of executable actions of the robot, is the mapping of human next actions to next conditional plans, is one element of the value preference space, is one joint state of the joint state space, is the state is the next joint state to which the robot and human transition after performing the actions.

3. The human-robot collaboration method of claim 2, wherein The step of determining the predicted execution action of the human being predicted by the robot according to the joint action value comprises calculating the predicted execution action according to the following formula: ; wherein, is the predicted execution action; The step of determining the first value of each conditional plan according to the predicted execution action comprises calculating the first value according to the following formula: ;in, Plan for the conditions in the joint state The first value below, for The reward function, This is the discount factor.

4. The human-robot collaboration control method according to claim 3, wherein The conditional plan has different first values in different joint states; The step of screening a plurality of conditional plans according to the first value comprises: for the first value of each conditional plan, detecting whether there is a dominated conditional plan in a plurality of conditional plans, wherein the dominated first value of the dominated conditional plan is not less than the first value in the same joint state, and the dominated first value is greater than the first value in at least one joint state; in the case where the dominated conditional plan is detected, the conditional plan is retained; otherwise, the conditional plan is deleted.

5. The human-robot collaboration control method according to claim 4, wherein The method further comprises: calculating a value function size of the initial conditional plan set according to the first value, and determining a pruning threshold according to the value function size; selecting a first conditional plan and a second conditional plan from the initial conditional plan set at random, and calculating a feature parameter value according to the following formula: M = 1 ; wherein M is the characteristic parameter value, is a first value of the first condition plan, is a Euclidean norm of the first value of the first condition plan, is a first value of the second condition plan, is a Euclidean norm of the first value of the second condition plan; Detect a size relationship between the feature parameter value and the pruning threshold value, and in a case where it is detected that the feature parameter value is less than the pruning threshold value, retain the first conditional plan and the second conditional plan; in a case where it is detected that the feature parameter value is greater than or equal to the pruning threshold value, delete the second conditional plan.

6. The human-robot collaboration method of claim 1, wherein, The control trigger instruction is environment state information of a current environment of the robot; The target execution action of the robot is determined according to the target conditional plan set, including: According to the environment state information, a target first value of each target conditional plan in the target conditional plan set is determined; A current belief state of the robot is determined, wherein the belief state is used to represent a predicted preference degree of the human to an event; According to the belief state and the target first value, a second value of the robot executing the target conditional plan in the belief state is calculated, wherein each target conditional plan corresponds to a second value; According to the second value, a final conditional plan is determined in the target conditional plan set, and a target execution action of the robot is determined according to the final conditional plan.

7. The human-robot collaboration control method according to claim 6, wherein According to the belief state and the target first value, the second value of the robot executing the target conditional plan in the belief state is calculated, including: The second value is calculated according to the following formula: ; wherein, is the second value, is the belief state, is the target first value; According to the second value, the final conditional plan is determined in the target conditional plan set, including: ; wherein, is the final condition plan, is the set of target condition plans.

8. A human-robot collaborative robot control device, characterized by, including: A first determination module, a second determination module, a third determination module, a prediction module, a screening module, and a response module; The first determination module is used to determine an initial conditional plan set of a robot, wherein the initial conditional plan set includes a plurality of conditional plans, and the conditional plan is used to represent an executable action of the robot; The second determination module is used to determine an environment state space and a value preference space, wherein the environment state space is used to represent a plurality of environment states in which the robot and the human can be, and the value preference space is used to represent a preference degree of the human to an event; The third determination module is used to determine a joint action value when the robot and the human cooperatively act according to the environment state space, the value preference space, the initial conditional plan set, and a preset executable action set of the human; The prediction module is used to determine a predicted execution action of the human predicted by the robot according to the joint action value; The screening module is used to determine a first value of each conditional plan according to the predicted execution action, and to screen a plurality of conditional plans according to the first value to obtain a target conditional plan set; The response module is used to respond to a control trigger instruction, and to determine a target execution action of the robot according to the target conditional plan set.

9. An electronic device comprising a processor and a memory, wherein The memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device executes the human-robot collaborative robot control method according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium stores computer instructions, and when the computer instructions run on the electronic device, the electronic device executes the human-robot collaborative robot control method according to any one of claims 1 to 7.

Images