Human operation intention prediction and robot active cooperation method based on hand postures

Abstract

The invention is suitable for the technical field of robot man-machine interaction, and provides a human operation intention prediction and robot active cooperation method based on hand postures, and the method comprises the steps: firstly building a behavior mapping graph related to human identities, objects and cooperation strategies in an off-line manner through a large language model, and enabling the behavior mapping graph to serve as a prior knowledge base; then, the hand 3D posture and the object state are sensed in real time through a relative coordinate system with the hand as the center, the human operation target is directly predicted, and the limitation of track prediction is avoided; and finally, locking a stable intention by adopting a time window verification mechanism, and retrieving from a knowledge base to generate a safe cooperative action sequence. The method effectively solves the problems that in the prior art, explicit instructions are relied on, intention recognition is not accurate, and large model delay is high, and smooth, safe and efficient robot active cooperation in the dynamic unstructured environment is achieved.

Description

Technical Field

[0001] This invention belongs to the field of robot human-computer interaction technology, and particularly relates to a method for predicting human operation intentions based on hand posture and for active robot collaboration. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, robots are considered key intelligent agents for solving labor shortages and supporting human activities, and are gradually entering unstructured environments such as homes, hospitals, and public spaces. However, achieving efficient and natural human-robot collaboration in such highly dynamic environments with uncertain task objectives faces significant challenges. Currently, human-robot collaboration technologies are mainly divided into the following categories: The first category is traditional methods based on explicit instructions or strict task allocation. These methods typically rely on predefined workflows or depend entirely on explicit commands such as voice and gestures issued by humans to execute tasks. However, these methods have significant limitations: on the one hand, they impose a huge cognitive and time burden on humans, with the cost of formulating and issuing commands often exceeding the cost of manual execution, severely reducing collaboration efficiency; on the other hand, in dynamic environments, task objectives are often unpredictable, and predefined allocation methods cannot adapt to changes.

[0003] The second category is intent recognition methods based on trajectory prediction. These methods attempt to infer intent by predicting the future movement trajectory of the hand. However, existing methods have significant drawbacks: the trajectory only captures the surface movement patterns (i.e., "how the hand moves"), and cannot reveal the underlying operational intent driving these movements (i.e., "what one wants to do"). In complex collaborative scenarios, the same intent may correspond to multiple trajectories, and trajectory prediction alone is insufficient to accurately understand the deeper motivations.

[0004] The third category is planning methods based on large language models. These methods utilize the reasoning capabilities of large models for task planning. While this improves flexibility, large models typically have slow reasoning speeds, making it difficult to meet the demands for millisecond-level real-time responses in close human-computer collaboration, resulting in poor interaction fluency.

[0005] In summary, the main problems facing current human-machine collaboration technologies include: heavy reliance on commands leading to low efficiency, inability to accurately understand intent through trajectory prediction alone, and high inference latency in large models hindering real-time response. Therefore, there is an urgent need for a collaboration method that can directly infer intent from hand gestures and scene context, and provide real-time, proactive context-aware assistance. Summary of the Invention

[0006] The purpose of this invention is to provide a method for predicting human operational intentions and enabling proactive robot collaboration based on hand posture, thereby addressing the problems mentioned in the background art.

[0007] The present invention is implemented as follows: a method for predicting human operational intentions based on hand posture and for active robot collaboration includes the following steps: Step 1: Construct a collaborative prior knowledge base and use a large language model (LLM) to generate a behavior mapping graph containing the correspondence between human intentions and robot collaborative strategies based on human identity and information about operable objects in the scene. Step 2: Real-time perception of human operation intentions, acquisition of hand pose sequence in relative coordinate system centered on the hand and 6D pose of objects in the scene, and prediction of the probability score of each object becoming the operation target through perception model; Step 3: Intent locking and collaborative action generation. The predicted score output in Step 2 is subjected to stability verification based on a time window. After locking the operation target, the corresponding collaborative strategy is retrieved from the behavior map and an execution sequence is generated.

[0008] A further technical solution, wherein step 1 includes the following steps: Step 1.1: Information Input and Senior Management Guidance; When a robot enters a collaborative scenario, it first perceives and extracts information about operable objects in the current environment and the identity characteristics of human collaborators. Then, it constructs this information into structured input prompts and inputs them into the LLM. Through the reasoning capabilities of the LLM, it provides high-level guidance, enabling the robot to pre-plan how to assist humans based on inferred intentions. Step 1.2: Interactive reasoning, strategy generation, and knowledge base construction; By employing a one-time reasoning set of human hand-object interactions, corresponding robot collaboration strategies are inferred for these interactions. The inferred robot actions must satisfy physical feasibility and human benefit, and ensure compliance with safety and ethical standards. Finally, this set of robot behaviors is compiled into a behavior map as prior knowledge to guide the robot's strategy selection in real-time collaborative decision-making.

[0009] A further technical solution, step 2 includes the following steps: Step 2.1: Representation of motion and scene information; The input information is divided into hand motion information and object state information, and the hand-object relationship is defined as a 3D relative position vector from the hand to each object. Thus, these poses and object relationships are represented in the relative coordinate system of the hand center. Step 2.2: Encoding and prediction of scene state features; The encoded scene state features are fed as tokens into the Transformer encoder to capture the overall collaborative scene state between the hand and the object. The encoded state features are then fed into the POP predictor, which ultimately outputs a POP score vector, where each value indicates the likelihood that the corresponding object will become a target of human manipulation. Step 2.3: Training objective and loss function; During the training phase, the standard cross-entropy loss function is used for optimization; loss function Defined as: ; in, This represents the total number of operable objects in the current scene. For the model to predict the first Each object is a probability score of the operation target; For the first The actual label of an object, with 1 for the target object and 0 for non-target objects.

[0010] In a further technical solution, in step 2.1, the hand motion information is represented as a hand pose sequence, each frame containing the 3D position of the palm, the direction of the palm, and the bending angle of the 15 finger joints, all directions and angles are represented using Euler angles; the object state information includes the 6D pose of each object, namely the 3D position and the 3D Euler angle direction.

[0011] In a further technical solution, step 3 includes the following specific steps: Step 3.1: Intent stability verification and target locking; Employing a time-window-based intent locking mechanism, the system locks each frame... The prediction results are then subjected to continuity verification, and the specific calculation process is as follows: First, from the output of step 2 The index of the object with the highest confidence in the current frame is identified. : ; in, A function that determines the index position of a function or array to obtain its maximum value; Then, determine the highest confidence level of the object. Does it exceed the preset confidence threshold? If satisfied If the object with the highest confidence in the current frame is the same as the locked object in the previous frame, then the stability counter is incremented. Otherwise, reset the counter; the determination condition for intent to lock is defined as: ; in, It was ultimately confirmed as the target object for human manipulation after undergoing time stability verification. The threshold for the number of consecutive frames used to filter transient noise; Step 3.2: Strategy retrieval and collaborative action sequence unfolding; once target If locked, the most suitable collaboration plan is retrieved from the behavior mapping graph constructed in step 1 by combining the object's attributes and human identity; then, a primitive action is generated first, and then the primitive action is expanded into a complete sequence of collaborative actions by analyzing the required spatial relationships between humans, robots and objects defined in the selected collaboration plan. Step 3.3: Optimization of action execution and real-time response; The robot executes a pre-planned sequence of collaborative actions. When the robot enters the collaborative scene, the set of manipulable objects and their feasible primitive actions are analyzed in advance. During real-time collaboration, the robot only needs to select and refine actions based on the current collaboration strategy and the 6D pose of the objects. After the actions are completed, the locked state and counters will be reset.

[0012] The method for predicting human operational intentions and promoting active robot collaboration based on hand posture provided in this invention has the following beneficial effects: (1) It solves the contradiction between large model reasoning latency and real-time response: Through the decoupled architecture of "prior knowledge base construction" and "real-time perception decision", it not only utilizes the semantic reasoning capability of LLM, but also achieves millisecond-level real-time response.

[0013] (2) Improved accuracy and robustness of intent recognition: The relative coordinate system centered on the hand is used to represent the hand-object relationship, which effectively eliminates environmental noise and improves the prediction accuracy in dynamic scenes.

[0014] (3) Enhanced safety and standardization of collaborative behavior: Human identity recognition is introduced into the knowledge base construction, enabling robots to generate strategies that comply with safety ethics (such as preventing children from accessing dangerous products).

[0015] (4) Effectively prevents accidental triggering of actions: The introduction of an intent locking mechanism based on time windows filters out instantaneous noise and unconscious actions, ensuring that the system only triggers collaboration when the intent is clear and stable, thereby improving the system's stability and security. Attached Figure Description

[0016] Figure 1 This is a framework diagram of a method for predicting human operational intentions and promoting active robot collaboration based on hand posture, provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0019] like Figure 1 As shown, an embodiment of the present invention provides a method for predicting human operational intentions based on hand posture and for active robot collaboration, which includes the following steps: Step 1: Construct a collaborative prior knowledge base based on high-level reasoning; This step aims to address the difficulty of predefining collaborative models and task rules in unstructured environments. By leveraging the extensive human knowledge and reasoning capabilities of large language models, a prior knowledge base, or behavior mapping graph, is built for the robot, thereby avoiding the inherent latency of LLM from affecting real-time human-computer interaction.

[0020] Step 1.1: Information Input and Senior Management Guidance; When a robot enters a collaborative scenario, it first perceives and extracts information about manipulable objects in the current environment and the identity characteristics of human collaborators (e.g., child or adult male). This information is then used to construct structured input cues, which are fed into the LLM (Local Management Module) to leverage its reasoning capabilities to provide high-level guidance, enabling the robot to pre-plan how to assist humans based on inferred intentions.

[0021] Step 1.2: Interactive reasoning, strategy generation, and knowledge base construction; To address the inference latency issue of LLM, a one-time inference set of human hand-object interactions (e.g., "picking up a can") is employed. For these interactions, the model infers corresponding robot collaborative strategies (e.g., "delivering a can"). The inferred robot actions must satisfy physical feasibility and human benefit, and ensure compliance with safety and ethical standards; for example, when a child attempts to touch a dangerous object, the robot should infer and execute an intervention or prevention strategy. Finally, this set of robot behaviors is compiled into a behavior map. This map serves as prior knowledge, guiding the robot's strategy selection in real-time collaborative decision-making, thereby maintaining adaptive responsiveness while effectively mitigating the high latency problem of LLM.

[0022] Step 2: Real-time perception of human operational intentions and target objects; This step designs a core perception model designed to infer in real-time the operational goals of humans in unstructured scenes—that is, "why" they move, rather than simply predicting "how" they move their hand trajectories. This model uses hand movements as the primary cue and integrates contextual scene information for target prediction.

[0023] Step 2.1: Representation of motion and scene information; To perform real-time intent inference, the input information is divided into hand motion information and object state information. Hand motion information is represented as a sequence of hand poses, with each frame containing the 3D position of the palm, the orientation of the palm, and the bending angles of the 15 finger joints (45 dimensions in total), all orientations and angles represented using Euler angles. Object state information includes the 6D pose (3D position and 3D Euler angle orientation) of each object. Furthermore, the hand-object relationship is explicitly defined as a 3D relative position vector from the hand to each object. Representing these poses and object relationships in a relative coordinate system centered on the hand significantly improves the model's intent perception accuracy.

[0024] Step 2.2: Encoding and prediction of scene state features; The encoded scene state features are fed as tokens into the Transformer encoder to capture the overall collaborative scene state between the hand and the object. Subsequently, the encoded state features are fed into the object predictor, ultimately outputting a manipulation intent score vector, where each value indicates the likelihood that the corresponding object will be a target of human manipulation.

[0025] Step 2.3: Training objective and loss function; During the training phase, the hand gesture intention perception module, as a trainable component within the framework, is optimized using the standard cross-entropy loss function. Loss function Defined as: ; in, This represents the total number of operable objects in the current scene. For the model to predict the first Each object represents a probability score of the operation target. For the first The true labels for each object are assigned (1 for target objects and 0 for non-target objects). The loss function aims to maximize the predicted operational intent score. With ground real target labels Consistency between models encourages them to assign higher confidence to real manipulation targets, thereby improving prediction accuracy.

[0026] Step 3: Intent locking and context-aware collaborative action generation; This step integrates the prediction results from step 2 with the prior knowledge from step 1 to ensure the safety and social norms of robot behavior.

[0027] Step 3.1: Intent stability verification and target locking; To prevent prediction noise from causing erroneous robot actions, an intent-locking mechanism based on a time window is employed. The system performs intent locking on each frame... The prediction results are then subjected to continuity verification, and the specific calculation process is as follows: First, from the output of step 2 The index of the object with the highest confidence in the current frame is identified. : ; in, An index position function that allows a function or array to reach its maximum value.

[0028] Then, determine the highest confidence level of the object. Does it exceed the preset confidence threshold? If satisfied If the object with the highest confidence in the current frame is the same as the locked object in the previous frame, then the stability counter is incremented. Otherwise, reset the counter. The determination condition for intent locking is defined as follows: ; in, It was ultimately confirmed as the target object for human manipulation after undergoing time stability verification. This is the threshold for the number of consecutive frames used to filter transient noise.

[0029] Step 3.2: Strategy retrieval and collaborative action sequence unfolding; once target Once locked, this step combines the object's attributes and the human's identity to retrieve the most suitable collaboration plan from the behavior map constructed in Step 1. The system then first generates a primitive action (e.g., "grab"). Next, by analyzing the required spatial relationships between the human, robot, and object defined in the selected collaboration plan, the collaboration execution module expands this primitive action into a complete sequence of collaborative actions. For example, in a delivery task, the sequence guides the robot to hand the object to the human's hand, while in a relocation task, the sequence guides the robot to transport the object to a location away from the human.

[0030] Step 3.3: Optimization of action execution and real-time response; The robot executes a pre-planned sequence of collaborative actions. To further improve response speed, the system pre-analyzes the set of manipulable objects and their feasible primitive actions when the robot enters the collaborative scenario. During real-time collaboration, the robot only needs to select and refine appropriate actions based on the current collaboration strategy and the 6D poses of the objects. After the actions are executed, the locked state and counters are reset to prepare for the next round of collaboration.

[0031] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting human operational intentions and promoting active robot collaboration based on hand posture, characterized in that, Includes the following steps: Step 1: Construct a collaborative prior knowledge base. Using LLM, based on human identity and information about operable objects in the scene, generate a behavior mapping graph containing the correspondence between human intentions and robot collaborative strategies in one go. Step 2: Real-time perception of human operation intentions, acquisition of hand pose sequence in relative coordinate system centered on the hand and 6D pose of objects in the scene, and prediction of the probability score of each object becoming the operation target through perception model; Step 3: Intent locking and collaborative action generation. The predicted score output in Step 2 is subjected to stability verification based on a time window. After locking the operation target, the corresponding collaborative strategy is retrieved from the behavior map and an execution sequence is generated.

2. The method for predicting human operational intentions and promoting active robot collaboration based on hand posture according to claim 1, characterized in that, Step 1 includes the following steps: Step 1.1: Information Input and Senior Management Guidance; When a robot enters a collaborative scenario, it first perceives and extracts information about operable objects in the current environment and the identity characteristics of human collaborators. Then, it constructs this information into structured input prompts and inputs them into the LLM. Through the reasoning capabilities of the LLM, it provides high-level guidance, enabling the robot to pre-plan how to assist humans based on inferred intentions. Step 1.2: Interactive reasoning, strategy generation, and knowledge base construction; By employing a one-time reasoning set of human hand-object interactions, corresponding robot collaboration strategies are inferred for these interactions. The inferred robot actions must satisfy physical feasibility and human benefit, and ensure compliance with safety and ethical standards. Finally, this set of robot behaviors is compiled into a behavior map as prior knowledge to guide the robot's strategy selection in real-time collaborative decision-making.

3. The method for predicting human operational intentions and promoting active robot collaboration based on hand posture according to claim 2, characterized in that, Step 2 includes the following steps: Step 2.1: Representation of motion and scene information; The input information is divided into hand motion information and object state information, and the hand-object relationship is defined as a 3D relative position vector from the hand to each object. Thus, these poses and object relationships are represented in the relative coordinate system of the hand center. Step 2.2: Encoding and prediction of scene state features; The encoded scene state features are input as tokens into the Transformer encoder to capture the overall collaborative scene state between the hand and the object; then, the encoded state features are input into the POP, and finally the operation intention score vector is output, where each value in the vector indicates the likelihood that the corresponding object will become a target of human manipulation. Step 2.3: Training objective and loss function; During the training phase, the standard cross-entropy loss function is used for optimization; loss function Defined as: ； in, This represents the total number of operable objects in the current scene. For the model to predict the first Each object is a probability fraction of the operation target; For the first The actual label of an object, with 1 for the target object and 0 for non-target objects.

4. The method for predicting human operational intentions and promoting active robot collaboration based on hand posture according to claim 3, characterized in that, In step 2.1, the hand motion information is represented as a hand pose sequence. Each frame contains the 3D position of the palm, the direction of the palm, and the bending angle of the 15 finger joints. All directions and angles are represented using Euler angles. The object state information includes the 6D pose of each object, namely the 3D position and the 3D Euler angle direction.

5. The method for predicting human operational intentions and promoting active robot collaboration based on hand posture according to claim 3, characterized in that, Step 3 includes the following specific steps: Step 3.1: Intent stability verification and target locking; Employing a time-window-based intent locking mechanism, the system locks each frame... The prediction results are then subjected to continuity verification. The specific calculation process is as follows: First, from the output of step 2 The index of the object with the highest confidence in the current frame is identified. : ； in, A function that determines the index position of a function or array to obtain its maximum value; Then, determine the highest confidence level of the object. Does it exceed the preset confidence threshold? If satisfied If the object with the highest confidence in the current frame is the same as the locked object in the previous frame, then the stability counter is incremented. Otherwise, reset the counter; the determination condition for intent to lock is defined as: ； in, It was ultimately confirmed as the target object for human manipulation after undergoing time stability verification. The threshold for the number of consecutive frames used to filter transient noise; Step 3.2: Strategy retrieval and collaborative action sequence unfolding; once target If locked, the most suitable collaboration plan is retrieved from the behavior mapping graph constructed in step 1 by combining the object's attributes and human identity; then, a primitive action is generated first, and then the primitive action is expanded into a complete sequence of collaborative actions by analyzing the required spatial relationships between humans, robots and objects defined in the selected collaboration plan. Step 3.3: Optimization of action execution and real-time response; The robot executes a pre-planned sequence of collaborative actions. When the robot enters the collaborative scene, the set of manipulable objects and their feasible primitive actions are analyzed in advance. During real-time collaboration, the robot only needs to select and refine actions based on the current collaboration strategy and the 6D pose of the objects. After the actions are completed, the locked state and counters will be reset.

Images