Man-machine cooperative robot control method and device, electronic equipment and storage medium
By defining the environmental state space and value preference space, and combining belief state to calculate the value of joint actions, the problem of value alignment caused by the fixed reward function is solved by using incomplete rationality modeling and particle filtering algorithm, thus improving the accuracy and applicability of human-machine collaboration.
Patent Information
- Application Number
- CN202511538667.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-27
- Publication Date
- 2026-02-27
AI Technical Summary
In existing technologies, the difficulty in value alignment caused by fixed reward functions is particularly difficult to achieve quickly in human-machine collaboration scenarios. Furthermore, algorithms based on fully rational modeling suffer from significant accuracy loss, while non-fully rational modeling has considerable limitations and cannot effectively guide robot learning.
By defining the environmental state space, value preference space, first belief state, and second belief state, the joint action value of robot and human collaborative actions is calculated. Incomplete rationality modeling is adopted to model humans as individuals who make decisions according to the probability of joint action value, reflecting the randomness and instructiveness of human decision-making. The robot's belief distribution is approximated by a particle filtering algorithm to dynamically estimate human value preferences.
It improves the accuracy of human-machine collaborative work, is applicable to any human-machine collaborative scenario, reduces the impact of reduced solution accuracy caused by human modeling, and improves the accuracy of robot control.
Smart Images

Figure CN121578875A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of reinforcement learning and value alignment, and more particularly to a human-machine collaborative robot control method, device, electronic device, and storage medium. Background Technology
[0002] With the development of artificial intelligence technology, the demand for intelligent robotic assistants is growing. People urgently hope that robots can understand human intentions to better achieve human-machine collaboration. In the field of reinforcement learning, a fixed reward function is typically pre-designed, and the robot continuously interacts with the environment to obtain rewards based on this function. Through feedback from these rewards, the robot adjusts its behavior and eventually learns the corresponding strategy. However, reward functions designed by humans based on experience are unreliable. Generally, it is necessary to wait until the training is stable, judge the rationality of the reward function based on the robot's behavior, and then adjust the reward function. This method consumes a lot of time and computing power and cannot achieve rapid value alignment in real-time scenarios such as human-machine collaboration.
[0003] To address the difficulty of value alignment caused by fixed reward functions, existing methods introduce value preference parameters and map the reward function to a parameterized function of value preference. The robot dynamically estimates human value preferences through interaction, adjusts its own reward function accordingly, and ultimately learns a strategy that aligns with human value preferences. Based on this, there are generally two methods to solve the robot's strategy. The first is to model the human-robot collaboration problem as a distributed Markov decision process and use a multi-agent inverse reinforcement learning algorithm to solve the strategy. The second is to model the human, transforming the two-player game into a special single-agent partially observable Markov decision process problem. Depending on the degree of human modeling, it can be divided into fully rational modeling and imperfectly rational modeling. Fully rational modeling portrays the human as a self-interested individual that executes actions that maximize its own gains, without considering the robot's learning behavior. Imperfectly rational modeling assumes that the human can perceive the current value alignment level of the robot and may execute some actions that do not maximize gains, but these actions can guide the robot's learning, thereby improving the robot's alignment level.
[0004] However, single-agent problem-solving algorithms based on perfectly rational modeling suffer from significant accuracy loss: The main idea of these algorithms is to model the human role in a two-player game, assuming it makes decisions according to certain fixed rules. The robot predicts the human's actions based on its current estimated human value preferences, thus compressing the human's action space and achieving rapid solution. However, because perfectly rational modeling ignores the randomness and instructiveness of human decision-making, it deviates from actual human behavior. This method may incorrectly predict human actions to some extent, leading to inaccurate estimations of current human value preferences and ultimately affecting the value alignment effect. Single-agent problem-solving algorithms based on imperfectly rational modeling have significant limitations: These algorithms allow humans to perceive the robot's current learning level and selectively choose actions to guide the robot's learning. However, this method assumes that humans can directly observe or calculate the robot's belief distribution regarding human values, i.e., the degree of robot alignment. This assumption simplifies the solution process, but in reality, the degree of robot alignment is often not directly observable, thus limiting its effectiveness. Summary of the Invention
[0005] In view of this, this application provides a human-machine collaborative robot control method, device, electronic device and storage medium, which can improve the accuracy of human-machine collaborative work and is applicable to any human-machine collaborative work scenario.
[0006] A first aspect of this application provides a human-machine collaborative robot control method, comprising: determining an environmental state space and a value preference space, wherein the environmental state space is used to characterize various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of human preference for an event; determining a current first belief state of the robot, wherein the first belief state is used to characterize the degree of human preference for the event predicted by the robot; setting a current second belief state of the human as the first belief state, wherein the second belief state is used to characterize the human's estimate of the belief state of the robot; calculating a joint action value when the robot and the human cooperate in action based on the first belief state, the second belief state, the environmental state space, and the value preference space; determining a target decision model of the robot based on the joint action value; acquiring environmental state information of the current environment; and determining a first target execution action of the robot based on the environmental state information and the target decision model.
[0007] In one possible implementation, before calculating the joint action value of the robot and the human when they cooperate based on the first belief state, the second belief state, the environmental state space, and the value preference space, the method further includes: determining a first set of executable actions for the robot and a second set of executable actions for the human; the step of calculating the joint action value of the robot and the human when they cooperate based on the first belief state, the second belief state, the environmental state space, and the value preference space includes: calculating the joint action value of the robot and the human when they cooperate based on the first set of executable actions, the second set of executable actions, the first belief state, the second belief state, the environmental state space, and the value preference space.
[0008] In one possible implementation, calculating the joint action value of the robot and the human during collaborative actions based on the first set of executable actions, the second set of executable actions, the first belief state, the second belief state, the environmental state space, and the value preference space includes: defining the human's action decision model according to the following formula: ; in, For the strategies of the aforementioned humans, Let be an element of the environmental state space. This is a human decision action within the second set of executable actions. This is a robot decision action from the first set of executable actions. For an element of the value preference space, This is the second belief state. This represents the first belief state. The correlation coefficient is... The value of the combined action is calculated according to the following formula: ,in, This is a parameterized reward function.
[0009] In one possible implementation, determining the robot's target decision model based on the joint action value includes: determining the target decision model according to the following formula: ;in, This is the target decision model.
[0010] In one possible implementation, after determining the robot's first target action based on the environmental state information and the target decision model, the method further includes: showing the first target action to the human and receiving a second target action performed by the human; determining a joint action performed by the human and the robot based on the first target action and the second target action; determining environmental state update information of the current environment after the human and the robot perform the joint action; updating the first belief state based on the second target action and the environmental state update information; and updating the second belief state based on the first target action and the environmental state update information.
[0011] In one possible implementation, updating the first belief state based on the second target action and the environmental state update information includes updating the first belief state according to the following formula: ; in, The updated first belief state, For the strategies of the aforementioned humans, Let be an element of the environmental state space. This is a human decision action within the second set of executable actions. This is a robot decision action from the first set of executable actions. For an element of the value preference space, This is the second belief state. This represents the first belief state.
[0012] In one possible implementation, updating the second belief state based on the first target action and the environmental state update information includes: collecting a set of belief particles from the prior distribution of the robot's belief states; calculating the weight of each belief particle in the set of belief particles based on the first target action and the environmental state update information; resampling the set of belief particles based on the weights to obtain a target set of belief particles; and updating the second belief state based on the target set of belief particles.
[0013] Secondly, embodiments of this application also provide a human-machine collaborative robot control device, comprising: a first determining module, a second determining module, a setting module, a calculation module, a third determining module, an acquisition module, and a fourth determining module; the first determining module is used to determine an environmental state space and a value preference space, wherein the environmental state space is used to characterize multiple environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of human preference for events; the second determining module is used to determine the robot's current first belief state, wherein the first belief state is used to characterize the degree of human preference for events predicted by the robot; the setting module is used to set the human's current second belief state as the first belief state, wherein the second belief state is used to characterize the human's estimate of the robot's belief state; the calculation module is used to calculate the joint action value when the robot and the human cooperate in action based on the first belief state, the second belief state, the environmental state space, and the value preference space; the third determining module is used to determine the robot's target decision model based on the joint action value; the acquisition module is used to acquire environmental state information of the current environment; and the fourth determining module is used to determine the robot's first target execution action based on the environmental state information and the target decision model.
[0014] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the human-machine collaborative robot control method as described in the first aspect.
[0015] Fourthly, embodiments of this application also provide a storage medium that stores computer instructions, which, when executed on an electronic device, cause the electronic device to perform the human-machine collaborative robot control method as described in the first aspect.
[0016] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the joint action value of robot and human collaborative actions based on the environmental state space, value preference space, first belief state, and second belief state, since the environmental state space is used to characterize the various environmental states that the robot and human may be in, the value preference space is used to characterize the degree of human preference for events, the first belief state is used to characterize the degree of human preference for events predicted by the robot, and the second belief state is used to characterize the human's estimate of the robot's belief state, the defined joint action value conforms to the human decision-making pattern. Modeling humans as individuals making decisions according to the probability of joint action value reflects the randomness of human decision-making, and modeling the human's estimate of the robot's belief state reflects the instructive nature of human decision-making. Because the modeling of the human's belief estimate of the robot solves the problem that humans cannot directly observe the robot's belief state, the human-machine collaborative robot control method of this application can be applied to any human-machine collaborative work scenario. Furthermore, setting the human's current second belief state as the first belief state, i.e., modeling humans as not entirely rational, is more in line with real human behavior, mitigating to some extent the impact of reduced solution accuracy caused by human modeling, and improving the accuracy of human-machine collaborative work.
[0017] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description
[0018] Figure 1 A schematic diagram of the implementation environment of a robot control system provided in an embodiment of this application; Figure 2 A flowchart illustrating the steps of a human-machine collaborative robot control method provided in an embodiment of this application; Figure 3 Another flowchart of the human-machine collaborative robot control method provided in an embodiment of this application; Figure 4 This is a functional block diagram of a robot control system provided in one embodiment of this application; Figure 5 A functional block diagram of a human-machine collaborative robot control device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0019] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0020] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.
[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.
[0022] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0023] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.
[0024] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0025] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.
[0026] Environmental state space: refers to the set of all possible states in an environment, used to describe the dynamic characteristics and evolution of the environment at a specific moment.
[0027] Please refer to Figure 1 , Figure 1This is a schematic diagram of the implementation environment of the human-machine collaborative robot control system provided in the embodiments of this application.
[0028] Those skilled in the art will understand that Figure 1 The schematic diagram of the implementation environment of the human-machine collaborative robot system shown is merely an example. The implementation environment described in this application embodiment is for the purpose of more clearly illustrating the technical solution of this application embodiment and does not constitute a limitation on the technical solution provided by this application embodiment. As those skilled in the art will know, with the emergence of new business scenarios, the technical solution provided by this application embodiment is also applicable to similar technical problems.
[0029] exist Figure 1 In this application, the human-machine collaborative robot system provided in the embodiments may include a server 110, a human 120, and a robot 130. The server 110 may integrate the human-machine collaborative robot control device provided in the embodiments of this application.
[0030] Server 110 can communicate with human 120 and robot 130. For example, server 110 and robot 130 can be connected via a wired communication link or a wireless communication link, etc. This application embodiment does not limit this. In addition, server 110 provides an interface for interacting with human 120, and server 110 exchanges information with human 120 through the interface.
[0031] It is understood that server 110 can be a standalone server, or a server network or server cluster. For example, server 110 described in the embodiments of this application includes, but is not limited to, computers, network hosts, single network servers, multiple sets of network servers, or cloud servers composed of multiple servers. Among them, cloud servers are composed of a large number of computers or network servers based on cloud computing.
[0032] Server 110 can execute the human-machine collaborative robot control method provided in the embodiments of this application to determine the target action to be performed by robot 130.
[0033] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps of an embodiment of the human-machine collaborative robot control method of the present application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted. The human-machine collaborative robot control method of this application can be applied to human-machine collaborative robot control devices, but is not limited thereto, and the embodiments of this application do not limit this application.
[0034] The specific process of this embodiment is as follows: Figure 2 As shown, it includes the following steps: S201, Determine the environmental state space and value preference space.
[0035] Specifically, the environmental state space is used to characterize the various environmental states that robots and humans may be in, while the value preference space is used to characterize the degree of human preference for events.
[0036] S202, determine the robot's current first belief state, where the first belief state is used to characterize the degree of human preference for the event as predicted by the robot.
[0037] S203, set the human's current second belief state as the first belief state, whereby the second belief state is used to characterize the human's estimate of the robot's belief state.
[0038] The reason for setting the second belief state as the first belief state will be explained in detail in subsequent steps, and will not be repeated here to avoid repetition.
[0039] S204. Calculate the value of the joint action when the robot and the human cooperate, based on the first belief state, the second belief state, the environmental state space, and the value preference space.
[0040] In some embodiments, before calculating the joint action value when the robot and human cooperate in action based on the first belief state, the second belief state, the environmental state space, and the value preference space, the method further includes: determining a first set of executable actions for the robot and a second set of executable actions for the human; calculating the joint action value when the robot and human cooperate in action based on the first belief state, the second belief state, the environmental state space, and the value preference space, including: calculating the joint action value when the robot and human cooperate in action based on the first set of executable actions, the second set of executable actions, the first belief state, the second belief state, the environmental state space, and the value preference space.
[0041] Specifically, the human action decision-making model is defined according to the following formula: ; in, For the strategies of the aforementioned humans, Let be an element of the environmental state space. This is a human decision action within the second set of executable actions. This is a robot decision action from the first set of executable actions. For an element of the value preference space, This is the second belief state. This represents the first belief state. The correlation coefficient is... The value of the combined action is calculated according to the following formula: ,in, This is a parameterized reward function.
[0042] Understandable It is known to humankind Execute in the case of these four elements The probability of.
[0043] S205, Determine the robot's target decision model based on the value of joint actions.
[0044] Specifically, the target decision model is determined according to the following formula: ;in, This is the target decision model.
[0045] To facilitate understanding, the following is a detailed explanation of how the robot's target decision model is obtained in this embodiment: 1. In this embodiment, the robot is modeled as a perfectly rational decision-maker whose sole objective is to align with human values to obtain higher rewards. In this example, the robot makes a decision first, and the human makes a decision after observing the robot's decision-making action. Therefore, the robot needs to consider the impact of the human's subsequent decisions. Thus, the robot's strategy is represented as: .
[0046] 2. Since the human action decision-making model is unknown, this embodiment considers the possibility of non-perfectly rational modeling of human behavior. A correlation coefficient is introduced. Combining the Boltzmann model and Bayesian theory of mind, the human action decision-making model is as follows: .
[0047] 3. In the above formula, the value of the combined action is defined as follows: ; Due to human belief in robots, it is estimated that... This is unknown to the robot, considering the eventual convergence. = You can directly use the first belief state. Replace the second belief state If it eventually converges, then use... and The calculated results are exactly the same, therefore the robot's target decision model is as follows: .
[0048] S206, Obtain the current environmental status information.
[0049] In some embodiments, environmental state information of the current environment is acquired through sensors.
[0050] S207, based on environmental state information and the target decision model, determines the robot's first target action to execute.
[0051] In some embodiments, environmental state information is input into the target decision model, and the target decision model outputs the first target execution action.
[0052] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the joint action value of robot and human collaborative actions based on the environmental state space, value preference space, first belief state, and second belief state, since the environmental state space is used to characterize the various environmental states that the robot and human may be in, the value preference space is used to characterize the degree of human preference for events, the first belief state is used to characterize the degree of human preference for events predicted by the robot, and the second belief state is used to characterize the human's estimate of the robot's belief state, the defined joint action value conforms to the human decision-making pattern. Modeling humans as individuals making decisions according to the probability of joint action value reflects the randomness of human decision-making, and modeling the human's estimate of the robot's belief state reflects the instructive nature of human decision-making. Because the modeling of the human's belief estimate of the robot solves the problem that humans cannot directly observe the robot's belief state, the human-machine collaborative robot control method of this application can be applied to any human-machine collaborative work scenario. Furthermore, setting the human's current second belief state as the first belief state, i.e., modeling humans as not entirely rational, is more in line with real human behavior, mitigating to some extent the impact of reduced solution accuracy caused by human modeling, and improving the accuracy of human-machine collaborative work.
[0053] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the steps of an embodiment of the human-machine collaborative robot control method proposed by the applicant. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted. This human-machine collaborative robot control method can be applied to the aforementioned human-machine collaborative robot control device, but is not limited thereto, and the embodiments of this application do not limit it in this regard.
[0054] This embodiment is a further improvement upon the aforementioned embodiment. The main improvement lies in the following: In this embodiment, a first target execution action is also shown to the human, and a second target execution action performed by the human is received. A joint execution action is determined based on the first and second target execution actions. Then, environmental state update information after the human and robot perform the joint execution action is obtained. Finally, the first and second belief states are updated based on the environmental state update information. This approach ensures the accuracy of the target execution action, thereby further improving the precision of the robot control method.
[0055] The specific process of this embodiment is as follows: Figure 3 As shown, it includes the following steps: S301, Determine the environmental state space and value preference space.
[0056] S301, Determine the robot's current first belief state, where the first belief state is used to characterize the degree of human preference for the event as predicted by the robot.
[0057] S303, set the human's current second belief state as the first belief state, whereby the second belief state is used to characterize the human's estimate of the robot's belief state.
[0058] S304. Calculate the value of the joint action when the robot and the human cooperate, based on the first belief state, the second belief state, the environmental state space, and the value preference space.
[0059] S305, Determine the robot's target decision model based on the value of joint actions.
[0060] S306, Obtain the current environmental status information.
[0061] S307 determines the robot's first target action based on environmental state information and the target decision model.
[0062] S301 to S307 of this embodiment are similar to S201 to S207 of the previous embodiment. To avoid repetition, they will not be described again here.
[0063] S308 demonstrates the first objective execution action to the human and receives the second objective execution action performed by the human.
[0064] S309, determine the joint execution action of the human and the robot based on the first target execution action and the second target execution action.
[0065] S310, determine the environmental state update information after the human and robot perform a joint action.
[0066] S311, Update the first belief state based on the action performed for the second objective and the environmental status update information.
[0067] In some embodiments, the first belief state is updated according to the following formula: ; in, The updated first belief state, For human strategies, As an element of the environmental state space, This is a human decision action within the second set of executable actions. For a robot decision action in the first set of executable actions, As an element of the value preference space, This is the second belief state. This is the first belief state.
[0068] Considering directly using the first belief state Replace the second belief state The update formula for the first belief state is obtained as follows: .
[0069] S312, Update the second belief state based on the action performed for the first objective and the environmental status update information.
[0070] In some embodiments, updating the second belief state based on the first target action and environmental state update information includes: collecting a set of belief particles from the prior distribution of the robot's belief states; calculating the weight of each belief particle in the set of belief particles based on the first target action and environmental state update information; resampling the set of belief particles based on the weights to obtain a target set of belief particles; and updating the second belief state based on the target set of belief particles.
[0071] To facilitate understanding, the following will be combined with... Figure 4 The human-machine collaborative robot control method of this embodiment will be described in detail below: Please refer to Figure 4 This is a schematic diagram of the functional modules of the human-machine collaborative robot control system provided in the embodiments of this application.
[0072] The human-machine collaborative robot control system includes: an environmental perception unit, a human decision-making unit, a robot decision-making unit, a joint action execution unit, a human value update unit, a robot value update unit, and a reward receiving unit.
[0073] The environmental perception unit monitors the current environmental status information; the human decision-making unit acquires human decision-making actions; the robot decision-making unit generates the best robot decision-making actions based on the currently estimated human value; the joint action execution unit acquires and executes the decision-making actions submitted by humans and robots; the human value update unit updates the estimated human value based on the human's decision-making actions in the current round; the robot value update unit updates the estimated robot value based on the robot's decision-making actions in the current round; and the reward receiving unit receives environmental feedback after the joint action is executed and rewards humans and robots accordingly.
[0074] Specifically, the human-machine collaborative robot control method includes the following steps: 1. The human decision-making unit and the robot decision-making unit initialize their own belief states, that is, the human decision-making unit initializes the second belief state, and the robot decision-making unit initializes the first belief state. 2. The environmental perception unit acquires current environmental status information through sensors and sends it to the human decision-making unit and the robot decision-making unit; 3. The robot decision-making unit receives environmental state information and submits the first target action to the joint action execution unit; the method by which the robot decision-making unit determines the first target action based on the environmental state information has been described in detail in the foregoing embodiments and will not be repeated here.
[0075] 4. The joint action execution unit sends the robot's first target action to the human decision-making unit; 5. The human decision-making unit shows the robot the first objective action to be performed in this round and the estimated belief state of the robot, that is, the second belief state; 6. Humans select the second objective to execute and submit it to the human decision-making unit, which then submits the second objective to the joint action execution unit. 7. The joint action execution unit receives the first target execution action and the second target execution action submitted by the human decision-making unit and the robot decision-making unit, executes the joint action, and sends the decision information to the human value update unit and the robot value update unit. 8. The reward receiving unit receives feedback from the environment after the joint action is performed and sends the reward to the human value update unit and the robot value update unit. 9. The human value update unit receives human decision-making information and reward information, updates the robot's belief state, i.e. the first belief state, and sends it to the robot decision-making unit. 10. The robot value update unit updates the human belief state estimate of the robot, that is, updates the second belief state; Specifically, the robot value update unit is used to update human estimates of robot beliefs. , represented as: ; By marginalizing the belief distribution, historical interaction trajectories are collected. It can handle human needs Uncertainty: This embodiment approximates the belief distribution of R using particle filtering. First, from the prior distribution... Medium sampling A belief particle Then for each time step According to historical actions and Predicted particles are generated using the robot's belief update formula: Based on the observed robot actions, calculate the particle weights: ; After the weights are updated, weight degradation may occur, meaning that the weights of most particles approach 0, with only a few particles dominating the estimation. Therefore, it is necessary to resample the particle set to avoid weight concentration on a few particles while keeping the total number of particles unchanged. First, calculate the cumulative distribution function: ; Since resampling can lead to a loss of particle diversity, with many particles becoming duplicates of high-weight particles, resulting in a failure to cover key regions of the belief space and causing sample impoverishment, we consider using systematic resampling for uniformly spaced sampling. This forces a proportional allocation of replication counts based on weights, maintaining coverage. Sampling from a uniform distribution... And construct a sequence of sampling points: ; For each Find satisfaction Particle index ,copy To get a new set of particles in the new set. .final, The particle set approximation is as follows: ;in, It is the Dirac function.
[0076] 11. Determine if the maximum number of interaction steps has been reached. If yes, end the interaction; otherwise, return to step 2.
[0077] In some embodiments, a first step length and a maximum number of interaction steps are also set, with the initial value of the first step length set to 1. After updating the robot's belief state, it is also checked whether the size of the first step length is greater than or equal to the maximum number of interaction steps. If the first step length is less than the maximum number of interaction steps, the first step length is incremented by 1, and new environmental state information is reacquired to determine the new actions of the human and the robot. The belief state is then updated again based on the new actions, and the above process is repeated until the first step length equals the maximum number of interaction steps.
[0078] 12. The robot decision-making unit calculates the maximum value preference estimate and the optimal strategy.
[0079] In some embodiments, the specific implementation includes: First, The largest value in the middle corresponds to Calculated as a reflection of genuine human value preferences, it is expressed as: Then, the optimal strategy for the robot is derived through the combined action value: .
[0080] It is worth noting that this embodiment proposes a human-machine collaborative framework based on cooperative inverse reinforcement learning and incomplete irrationality modeling, and a value alignment method adapted to this framework. The key idea of the human-machine collaborative framework is to enable the robot to dynamically estimate human value preferences through continuous interaction between humans and robots, thereby adjusting its own reward function. The key idea of the value alignment method adapted to the human-machine collaborative framework is to model humans as individuals making decisions based on the probabilistic value of joint actions, reflecting the randomness of human decision-making, and to model human estimations of the agent's belief state, reflecting the edifying nature of human decision-making. To address the problem that humans cannot directly observe the agent's belief state, a particle filter algorithm is used to approximate the robot's belief distribution, and a corresponding update method is designed. The framework defines dynamic value preferences, thus avoiding the value alignment problem caused by a fixed reward function; the use of human modeling avoids the low solution efficiency problem in multi-agent inverse reinforcement learning algorithms; the incomplete irrationality modeling of humans more closely reflects real-world human behavior, mitigating the impact of reduced solution accuracy caused by human modeling to some extent; and the modeling of human belief estimations of the robot and the design of corresponding belief update methods solve the problem that humans cannot directly observe the robot's belief state.
[0081] Compared with related technologies, the embodiments of this application have at least the following advantages: By determining the joint action value of robot and human collaborative actions based on the environmental state space, value preference space, first belief state, and second belief state, since the environmental state space is used to characterize the various environmental states that the robot and human may be in, the value preference space is used to characterize the degree of human preference for events, the first belief state is used to characterize the degree of human preference for events predicted by the robot, and the second belief state is used to characterize the human's estimate of the robot's belief state, the defined joint action value conforms to the human decision-making pattern. Modeling humans as individuals making decisions according to the probability of joint action value reflects the randomness of human decision-making, and modeling the human's estimate of the robot's belief state reflects the instructive nature of human decision-making. Because the modeling of the human's belief estimate of the robot solves the problem that humans cannot directly observe the robot's belief state, the human-machine collaborative robot control method of this application can be applied to any human-machine collaborative work scenario. Furthermore, setting the human's current second belief state as the first belief state, i.e., modeling humans as not entirely rational, is more in line with real human behavior, mitigating to some extent the impact of reduced solution accuracy caused by human modeling, and improving the accuracy of human-machine collaborative work.
[0082] Based on the same idea as the human-machine collaborative robot control method in the above embodiments, this application also provides a human-machine collaborative robot control device, which can be used to execute the above-described human-machine collaborative robot control method. For ease of explanation, the structural schematic diagram of the human-machine collaborative robot control device embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0083] like Figure 5 As shown, the human-machine collaborative robot control device 50 includes: a first determining module 501, a second determining module 502, a setting module 503, a calculation module 504, a third determining module 505, an acquisition module 506, and a fourth determining module 507. In some embodiments, the above modules can be programmable software instructions stored in a memory and executable by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware embedded in a processor.
[0084] The first determining module 501 is used to determine the environmental state space and the value preference space, wherein the environmental state space is used to characterize the various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of human preference for events. The second determining module 502 is used to determine the robot's current first belief state, wherein the first belief state is used to characterize the degree of human preference for the event predicted by the robot. The setting module 503 is used to set the human's current second belief state to the first belief state, wherein the second belief state is used to characterize the human's estimate of the belief state of the robot; The calculation module 504 is used to calculate the value of the joint action when the robot and the human cooperate, based on the first belief state, the second belief state, the environmental state space, and the value preference space. The third determining module 505 is used to determine the target decision model of the robot based on the value of the joint action; The acquisition module 506 is used to acquire the environmental status information of the current environment; The fourth determining module 507 is used to determine the robot's first target action based on the environmental state information and the target decision model.
[0085] The human-machine collaborative robot control device 50 provided in the above embodiments can realize the technical solutions described in the above resource collaborative scheduling method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above resource collaborative scheduling method embodiments, and will not be repeated here.
[0086] Please refer to Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the electronic device of this application.
[0087] In some embodiments, processor 601 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 602 or process data, such as the resource collaborative scheduling method of the present invention.
[0088] In some embodiments, processor 601 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 601 may be local or remote. In some embodiments, processor 601 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.
[0089] In some embodiments, memory 602 may be an internal storage unit of electronic device 600, such as a hard disk or memory of electronic device 600. In other embodiments, memory 602 may also be an external storage device of electronic device 600, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 600.
[0090] Furthermore, the memory 602 may include both internal storage units of the electronic device 600 and external storage devices. The memory 602 is used to store application software and various types of data installed on the electronic device 600.
[0091] In some embodiments, display 603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 603 is used to display information from electronic device 600 and to display visual user applications. Components 601-603 of electronic device 600 communicate with each other via a system bus.
[0092] In one embodiment, when the processor 601 executes the human-machine collaborative robot control program stored in the memory 602, the following steps can be implemented: Define an environmental state space and a value preference space, wherein the environmental state space is used to characterize the various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of preference of the human for events; Determine the robot's current first belief state, wherein the first belief state is used to characterize the degree of human preference for the event as predicted by the robot; The human's current second belief state is set as the first belief state, wherein the second belief state is used to characterize the human's estimate of the robot's belief state; Based on the first belief state, the second belief state, the environmental state space, and the value preference space, calculate the value of the joint action when the robot and the human cooperate. The target decision model of the robot is determined based on the value of the combined action; Obtain the current environmental status information; Based on the environmental state information and the target decision model, the robot determines its first target action.
[0093] It should be understood that when the processor 601 executes the human-machine collaborative robot control program in the memory 602, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.
[0094] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 600 mentioned. Electronic device 600 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 600 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).
[0095] Accordingly, this application also provides a storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the human-machine collaborative robot control methods provided in the above-described method embodiments.
[0096] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0097] The above provides a detailed description of the human-machine collaborative robot control method, device, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A human-machine collaborative robot control method, characterized in that, include: Define an environmental state space and a value preference space, wherein the environmental state space is used to characterize the various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of preference of the human for events; Determine the robot's current first belief state, wherein the first belief state is used to characterize the degree of human preference for the event as predicted by the robot; The human's current second belief state is set as the first belief state, wherein the second belief state is used to characterize the human's estimate of the robot's belief state; Based on the first belief state, the second belief state, the environmental state space, and the value preference space, calculate the value of the joint action when the robot and the human cooperate. The target decision model of the robot is determined based on the value of the combined action; Obtain the current environmental status information; Based on the environmental state information and the target decision model, the robot determines its first target action.
2. The human-machine collaborative robot control method according to claim 1, characterized in that, Before calculating the value of the joint action when the robot and the human cooperate based on the first belief state, the second belief state, the environmental state space, and the value preference space, the method further includes: Determine the first set of executable actions for the robot and the second set of executable actions for the human. The step of calculating the value of the joint action when the robot and the human cooperate, based on the first belief state, the second belief state, the environmental state space, and the value preference space, includes: The combined action value of the robot and the human is calculated based on the first set of executable actions, the second set of executable actions, the first belief state, the second belief state, the environmental state space, and the value preference space.
3. The human-machine collaborative robot control method according to claim 2, characterized in that, The step of calculating the joint action value of the robot and the human during collaborative actions based on the first set of executable actions, the second set of executable actions, the first belief state, the second belief state, the environmental state space, and the value preference space includes: The human action decision-making model is defined according to the following formula: ; in, For the strategies of the aforementioned humans, Let be an element of the environmental state space. This is a human decision action within the second set of executable actions. This is a robot decision action from the first set of executable actions. For an element of the value preference space, This is the second belief state. This represents the first belief state. The correlation coefficient is... The value of the joint action; The value of the combined action is calculated using the following formula: ,in, This is a parameterized reward function.
4. The human-machine collaborative robot control method according to claim 3, characterized in that, The step of determining the robot's target decision model based on the value of the combined action includes: The target decision model is determined according to the following formula: ; in, This is the target decision model.
5. The human-machine collaborative robot control method according to claim 1, characterized in that, After determining the robot's first target action based on the environmental state information and the target decision model, the method further includes: The system displays the first target action to the human and receives the second target action performed by the human. The combined action of the human and the robot is determined based on the action of the first target and the action of the second target; Determine the environmental state update information of the current environment after the human and the robot perform the joint action; The first belief state is updated based on the action performed for the second objective and the environmental state update information. The second belief state is updated based on the action performed on the first target and the environmental state update information.
6. The human-machine collaborative robot control method according to claim 5, characterized in that, The step of updating the first belief state based on the action performed according to the second target and the environmental state update information includes: updating the first belief state according to the following formula: ; in, The updated first belief state, For the strategies of the aforementioned humans, Let be an element of the environmental state space. This is a human decision action within the second set of executable actions. This is a robot decision action from the first set of executable actions. For an element of the value preference space, This is the second belief state. This represents the first belief state.
7. The human-machine collaborative robot control method according to claim 5, characterized in that, The step of updating the second belief state based on the action performed on the first target and the environmental state update information includes: Collect a set of belief particles from the prior distribution of the robot's belief states; The weight of each belief particle in the belief particle set is calculated based on the action performed by the first target and the environmental state update information. The set of belief particles is resampled according to the weights to obtain the target set of belief particles. Update the second belief state based on the target belief particle set.
8. A human-machine collaborative robot control device, characterized in that, include: The module comprises a first determination module, a second determination module, a setting module, a calculation module, a third determination module, an acquisition module, and a fourth determination module. The first determining module is used to determine the environmental state space and the value preference space, wherein the environmental state space is used to characterize the various environmental states that the robot and the human may be in, and the value preference space is used to characterize the degree of human preference for events; The second determining module is used to determine the robot's current first belief state, wherein the first belief state is used to characterize the degree of human preference for the event predicted by the robot; The setting module is used to set the human's current second belief state to the first belief state, wherein the second belief state is used to characterize the human's estimate of the robot's belief state; The calculation module is used to calculate the value of the joint action when the robot and the human cooperate, based on the first belief state, the second belief state, the environmental state space, and the value preference space. The third determining module is used to determine the robot's target decision model based on the value of the joint action; The acquisition module is used to acquire environmental status information of the current environment; The fourth determining module is used to determine the robot's first target action based on the environmental state information and the target decision model.
9. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the human-machine collaborative robot control method as described in any one of claims 1 to 7.
10. A storage medium, characterized in that, The storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the human-machine collaborative robot control method as described in any one of claims 1 to 7.
