A human-machine action joint prediction method based on human-machine-object interaction

Abstract

The application discloses a human-machine action joint prediction method based on human-machine-object interaction, and relates to the technical field of robots, comprising: acquiring time sequence motion data of a human body, a robot and a target object, and constructing unified human-machine-object time sequence motion representation; performing spatial feature extraction and time sequence feature coding on the three, to generate respective motion coding features; inferring the human-machine-object interaction state at the current moment based on the coding features, and constructing human-machine-object interaction behavior representation accordingly; based on the interaction behavior representation and the coding features of the three, jointly predicting future continuous multiple frames of human body action and robot action, and synchronously outputting future human body action prediction sequence and robot action prediction sequence. Through the joint prediction mechanism, the application overcomes the defect that the robot lacks active cooperation ability in the traditional "human body action prediction + robot passive response" mode, and significantly improves the active cooperation ability and interaction safety of the robot in a complex human-machine cooperation scene.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically to a method for joint prediction of human-machine actions based on human-machine-object interaction. Background Technology

[0002] Currently, with the development of industrial intelligence and collaborative robot technology, human-robot collaboration (HRC) has gradually become an important research direction in the field of intelligent manufacturing and service robots. In HRC scenarios, robots need to understand human intentions in real time and actively generate coordinated actions based on the environment and task status, thereby achieving safe, efficient, and natural human-robot collaboration.

[0003] Existing methods for predicting human motion mainly include probabilistic models, convolutional neural networks, recurrent neural networks, and Transformer-based methods. These methods typically only model human motion sequences and use historical human motion information to predict future human motion. In recent years, some research has begun to consider the impact of robots on human motion, predicting human motion before planning robot motion to achieve robot response in collaborative tasks.

[0004] However, most current methods adopt a two-stage structure of "human motion prediction + robot passive response". The generation of robot motion depends on the prediction results of the human body and lacks the ability to actively cooperate. Existing methods usually only perform single kinematic modeling of human motion and fail to explicitly model the interaction behavior between the human body, robot and target object. It is difficult to accurately describe the dynamic changes of human-machine-object interaction behavior during the cooperation process, resulting in the prediction results lacking interaction consistency and physical rationality.

[0005] Therefore, there is an urgent need to propose a method that can explicitly model the human-machine-object interaction relationship in the human-computer interaction process and jointly predict human actions and robot active response actions. Summary of the Invention

[0006] In view of the above problems, the present invention proposes a human-machine action joint prediction method based on human-machine-object interaction to overcome or at least partially solve the above problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention provides a method for joint prediction of human-machine actions based on human-machine-object interaction, comprising the following steps: S1. Acquire the temporal motion data of the human body, robot, and target object, and synchronize the human body temporal motion data, robot temporal motion data, and target object temporal motion data in time and coordinate unify them to construct a unified human-machine-object temporal motion representation. S2. Spatial feature extraction and temporal feature encoding are performed on the human body temporal motion data, robot temporal motion data and target object temporal motion data respectively to generate human body motion coding features, robot motion coding features and target object motion coding features. Based on the human motion coding features, robot motion coding features, and target object motion coding features, the current human-machine-object interaction state is inferred, and a human-machine-object interaction behavior representation is constructed based on the human-machine-object interaction state. The process of inferring the current human-machine-object interaction state based on the human motion coding features, robot motion coding features, and target object motion coding features includes: Based on the human body temporal motion data, robot temporal motion data and target object temporal motion data within a preset time window, the human body motion trend, robot motion trend and target object motion trend are calculated respectively. Based on the human motion trend, robot motion trend and target object motion trend, a human-machine-object interaction factor is calculated. The human-machine-object interaction factor is used to characterize the smoothness of the interaction between the human body and the target object, as well as between the robot and the target object. The human motion coding features, robot motion coding features, and target object motion coding features are averaged and pooled in the time dimension and then concatenated. The concatenation is input into the classification network, and the probability distribution of human-machine-object interaction states is output. The human-machine-object interaction states include no interaction state, human-object interaction state, robot-object interaction state, and human-robot-object collaborative interaction state. The process of constructing a human-machine-object interaction behavior representation includes: Based on the human motion trend, robot motion trend and target object motion trend, human-object interaction attractors, robot-object interaction attractors and target object interaction attractors are constructed respectively. The attractor is used to characterize the spatial constraint features of the future interaction position. Based on the probability distribution of the human-machine-object interaction state, the human-object interaction attractor, robot-object interaction attractor, target object interaction attractor and the robot position at the current moment are weighted and fused to generate the human-machine-object interaction position at the current moment. The expression for the human-object interaction attractor is:

[0009] in, for Momentary human-object interaction attractor , , This represents the magnification factor of the movement trend of the human wrist joint. For a moment The wrist position, which is closest to the robot on the human body. Position of the robot's end effector. For the trend of human body movement; The expression for the robot-object interaction attractor is:

[0010] in, For robot-object interaction attractors , The magnification factor for the robot's motion trend. The center position of the target object. Trends in robot movements; The expression for the target object interaction attractor is:

[0011] in, for At any given moment, the target object interaction attractor, The tendency of an object's movement; The expression for the human-machine-object interaction position is:

[0012] in, These are the weighting coefficients; The locations of human-machine-object interactions are mapped to a latent feature space and expanded along the time dimension to form a sequence of human-machine-object interaction behaviors. Based on the attention mechanism, the robot motion coding features are used as the query vector, and the human motion coding features, robot motion coding features, target object motion coding features and the human-machine-object interaction behavior embedding sequence are used as the key vector and value vector. Multi-head attention calculation is performed to generate the human-machine-object interaction behavior representation. The expression representing the human-machine-object interaction behavior is as follows:

[0013] in, , These are learnable mapping parameters.

[0014] S3. Based on the human-machine-object interaction behavior representation and the human motion coding features, robot motion coding features and target object motion coding features, jointly predict future continuous multi-frame human actions and future continuous multi-frame robot actions, and output future human action prediction sequence and future robot action prediction sequence. Specifically, it includes: The human-machine-object interaction behavior representation is concatenated with the human motion coding feature and the robot motion coding feature, respectively, and used as the input of the human decoder and the robot decoder. The human body action at the last observation moment in the unified human-machine-object temporal motion representation is used as the human body query sequence, and the robot action at the last observation moment is used as the robot query sequence. The human decoder is input into the first Transformer decoder along with the human query sequence to generate human decoding features; the robot decoder is input into the second Transformer decoder along with the robot query sequence to generate robot decoding features. The future human motion prediction sequence is predicted based on the human body decoding features, and the future robot motion prediction sequence is predicted based on the robot decoding features.

[0015] Furthermore, the human movement trend is characterized by the average displacement change of the key joint of the human body closest to the target object within the preset time window. The robot's motion trend is characterized by the average displacement change of the robot's end effector within the preset time window. The target object's motion trend is characterized by the average displacement change of the target object's center position within the preset time window.

[0016] Furthermore, the human-object interaction attractor is constructed based on the position of the human wrist joint, the historical relative displacement between the human wrist joint and the target object, and the trend of human movement. The robot-object interaction attractor is constructed based on the position of the robot end effector, the historical relative displacement between the robot end effector and the target object, and the robot's motion trend; the target object interaction attractor is constructed based on the center position of the target object and the target object's motion trend.

[0017] Furthermore, when performing weighted fusion of the human-object interaction attractor, robot-object interaction attractor, target object interaction attractor, and the robot position at the current moment, the weights corresponding to each attractor are generated by the probability distribution of the human-machine-object interaction state through a learnable nonlinear mapping. The weight allocation of the human-object interaction attractor and the target object interaction attractor is dynamically adjusted based on the human-machine-object interaction factor.

[0018] Furthermore, the first Transformer decoder and the second Transformer decoder adopt a non-autoregressive prediction structure, using the action at the last observation moment as the initial decoding input, to simultaneously generate action sequences for multiple consecutive future frames.

[0019] Furthermore, in step S3, before inputting the human decoding features and the robot decoding features into the prediction network respectively, a multi-head attention mechanism is used to enhance the self-attention of the human decoding features and the robot decoding features respectively, so as to improve the temporal continuity of the prediction results.

[0020] Furthermore, in step S3, a joint loss function is used to constrain the prediction results. The joint loss function includes human motion prediction loss, robot motion prediction loss, and human-machine-object interaction consistency loss. The human-machine-object interaction consistency loss is constructed based on the error between the predicted human-machine-object interaction position and the actual interaction position.

[0021] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a human-computer action joint prediction method based on human-computer-object interaction, which has the following beneficial effects: 1. This invention constructs a Transformer-based joint action prediction method for human-machine-object interaction behavior by jointly modeling human-machine-object interaction behavior. This overcomes the problem in existing technologies that only model human actions and are difficult to describe complex interaction relationships. By introducing an interaction state reasoning mechanism, an interaction attractor mechanism, and a human-machine-object interaction attention mechanism, a unified modeling of the interaction relationship between the human body, the robot, and the target object is achieved, improving the interaction consistency, physical rationality, and temporal continuity of the action prediction results. By jointly predicting the future human body action and the robot's active response action, this invention overcomes the problem of the robot's lack of active collaboration ability in the traditional "human action prediction + robot passive response" method, and improves the robot's active collaboration ability in complex human-machine collaboration scenarios.

[0022] 2. This invention further employs a non-autoregressive prediction structure and an interactive consistency constraint method, effectively reducing the error accumulation problem in the long-term prediction process and improving the temporal continuity and prediction stability of the action prediction results. This invention is applicable to scenarios such as industrial collaborative assembly, human-robot collaborative handling, service robot interaction, and embodied intelligent collaboration. It can effectively enhance the robot's ability to understand human action intentions and its proactive collaboration capabilities, thereby improving the safety and collaboration efficiency in complex human-robot collaborative tasks. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 This is a flowchart of the human-machine action joint prediction method provided in the embodiments of the present invention; Figure 2 This is an overall architecture diagram of the human-machine-object interaction action prediction method provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the graph convolutional network structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the implementation prediction provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention discloses a joint prediction method for human-machine actions based on human-machine-object interaction, such as... Figure 1 As shown, it includes the following steps: S1. Acquire the temporal motion data of the human body, robot, and target object, and synchronize the human body temporal motion data, robot temporal motion data, and target object temporal motion data in time and coordinate unify them to construct a unified human-machine-object temporal motion representation. S2. Spatial feature extraction and temporal feature encoding are performed on the human body temporal motion data, robot temporal motion data and target object temporal motion data respectively to generate human body motion coding features, robot motion coding features and target object motion coding features. Based on the human motion coding features, robot motion coding features, and target object motion coding features, the current human-machine-object interaction state is inferred, and a human-machine-object interaction behavior representation is constructed based on the human-machine-object interaction state. S3. Based on the human-machine-object interaction behavior representation and the human motion coding features, robot motion coding features and target object motion coding features, perform joint prediction of future continuous multi-frame human actions and future continuous multi-frame robot actions, and output future human action prediction sequence and future robot action prediction sequence.

[0027] This invention overcomes the problem of robots lacking proactive collaboration ability in the traditional "human motion prediction + robot passive response" method by jointly predicting future human actions and robot proactive response actions, thereby improving the robot's proactive collaboration ability in complex human-robot collaboration scenarios.

[0028] The following is a detailed description of each of the above steps; In step S1, motion observation and data acquisition are performed; First, the temporal motion data of the human body, robot, and target object during the collaborative process are acquired. Then, the temporal motion data of the human body, robot, and target object are synchronized in time and their coordinates are unified to construct a unified human-machine-object temporal motion representation. The specific implementation method of this step is as follows: 1. Acquisition of human motion sequences; Human motion sequences are constructed by acquiring three-dimensional coordinate information of human skeletal joints using human pose estimation systems, motion capture systems, or RGB-D cameras. (1) in, , J Indicates the number of joints in the human skeleton. k Indicates the length of the action observation sequence. H It refers to the human body.

[0029] Human skeletal joints can be represented using joint formats defined by human pose estimation standards such as COCO, MPII, or OpenPose. In this embodiment, a pose estimation format containing 21 human skeletal key points is preferred, including major joints such as the head, shoulders, elbows, wrists, hips, knees, and ankles. The three-dimensional coordinate information of each joint can be obtained through a multi-view RGB-D camera system or a monocular camera combined with a depth estimation algorithm.

[0030] It should be noted that the number of human skeletal joints J can be adjusted according to the actual application scenario and accuracy requirements. For example, in industrial collaborative assembly scenarios, the acquisition density of hand and finger joints can be increased to capture upper limb movements more precisely; while in human-machine collaborative handling scenarios, the focus can be on acquiring the main joints of the torso and lower limbs.

[0031] 2. Robot motion sequence acquisition; Obtain robot end effector pose information through robot controller or robot state feedback interface, and construct robot motion sequence: (2) in, R It refers to a robot.

[0032] The pose information of the robot's end effector can be read in real time through the state feedback interface inside the robot controller. The sampling frequency is preferably 30Hz-120Hz to ensure that it matches the time resolution of human motion data. The robot can be a collaborative robot, an industrial robot, or a service robot, and its end effector can be a gripper, a suction cup, a welding torch, or a special tool, etc.

[0033] 3. Obtaining the action sequence of the target object; By acquiring the spatial pose information of the target object through visual sensors, depth cameras, or object detection systems, a sequence of actions of the target object can be constructed. (3) In this context, O represents an object.

[0034] The target object refers to the object manipulated by the human body and / or robot during human-robot collaboration, such as parts to be assembled in industrial assembly scenarios, boxes or workpieces in collaborative handling scenarios, and items to be transferred in service robot scenarios. The pose information of the target object is preferably referenced to the world coordinate system to ensure coordinate consistency with human motion data and robot motion data.

[0035] 4. Time synchronization and coordinate unification; Time synchronization and coordinate unification are performed on human motion sequences, robot motion sequences, and target object motion sequences to construct a unified human-machine-object temporal motion representation. Time synchronization employs a timestamp-based alignment method, mapping data from different sensors or sampling frequencies to a unified time reference. In cases of inconsistent timestamps, linear interpolation is used to fill in missing states. Coordinate unification utilizes a homogeneous coordinate transformation-based spatial alignment method. Through a pre-calibrated extrinsic parameter matrix, the coordinates of key points on the human skeleton, the pose of the robot's end effector, and the spatial pose of the target object are uniformly mapped to the same world coordinate system, thus achieving alignment of motion sequence data under a unified spatiotemporal reference.

[0036] 5. Construct an observation window; Construct an action observation window based on a unified human-machine-object motion representation: (4) in, To represent future human movement sequences, To represent the future sequence of robot actions.

[0037] In step S2, interaction behavior is modeled; like Figure 2 As shown, Figure 2 The left side is the data input layer, which inputs the human action sequence, robot action sequence and target object action sequence constructed in the previous steps into the human graph convolutional network (GCN_H), robot graph convolutional network (GCN_R) and target object graph convolutional network (GCN_O) respectively to extract their respective spatial topological features; Figure 2 The middle part is the Transformer encoder layer, which receives the outputs of three graph convolutional networks, performs temporal feature encoding, and outputs human motion encoding features, robot motion encoding features, and target object motion encoding features. Figure 2 The interaction state reasoning branch in the middle performs average pooling and concatenation on the three encoded features and inputs them into the classification network, outputting the probability distribution of the interaction state; Figure 2 The interactive position inference network on the right calculates the human-machine-object interaction position at the current moment based on the probability distribution of the interaction state and formulas (17)-(19), and obtains the interaction behavior embedding sequence after mapping and broadcasting. Figure 2 The bottom is a multi-head interactive attention network. The robot motion coding features are used as the query vector, and the human motion coding features, robot motion coding features, target object motion coding features and interactive behavior embedding sequences are used as the key vector and value vector. After multi-head attention calculation by formula (22), the human-machine-object interactive behavior representation is output for subsequent steps.

[0038] Spatial features are extracted and temporal features are encoded from human temporal motion data, robot temporal motion data, and target object temporal motion data using encoders, generating human motion encoding features, robot motion encoding features, and target object motion encoding features. Based on these encoded features, the current human-machine-object interaction state is inferred, and a human-machine-object interaction behavior representation is constructed based on this state. The specific implementation method of this step is as follows: 1. Spatial feature extraction and temporal feature encoding; Construct convolutional networks for human motion graphs, robot motion graphs, and target object motion graphs, such as... Figure 3 As shown, Figure 3 The left side is the input layer: the input is the historical temporal motion data obtained in step S1, that is, the node feature matrix corresponding to the human action sequence. The middle part is a three-layer graph convolution stacked structure, and the right side is the output layer.

[0039] Spatial features were extracted from human motion sequences, robot motion sequences, and target object motion sequences, respectively. The calculation of the graph convolutional layer is as follows: (5) in, Indicates the number of convolutional layers. For activation function, For the first The adjacency matrix of a layer represents the relationship between joint poses; This represents the output of a certain layer; Indicates the first Layer weight parameters.

[0040] In this embodiment, the human motion graph convolutional network, the robot motion graph convolutional network, and the target object motion graph convolutional network all adopt a 3-layer graph convolutional structure, with output dimensions of 64, 128, and 256 for each layer, respectively. That is, the first graph convolutional layer maps the input feature dimensions to 64 dimensions, the second layer maps 64 dimensions to 128 dimensions, and the third layer maps 128 dimensions to 256 dimensions. Each graph convolutional layer is followed by a batch normalization layer and a ReLU activation function to accelerate training convergence and prevent gradient vanishing.

[0041] Spatial features are fused with positional codes and input into a Transformer encoder to output human motion coding features, robot motion coding features, and target object motion coding features. Position coding can be sine-cosine position coding or learnable position coding. In this embodiment, sine-cosine position coding is preferred to preserve the relative time position information between frames in the sequence.

[0042] It should be noted that the number of network layers, attention heads, and hidden layer dimensions mentioned above are merely illustrative examples, and those skilled in the art can adjust them according to the actual amount of training data and computing resources. For example, for large-scale training data, a deeper network structure can be used; for scenarios with limited computing resources, the number of layers and hidden layer dimensions can be appropriately reduced.

[0043] 2. Calculation of movement trends; To analyze human-machine-object interaction, the pose changes of key human joints, robot end effectors, and the center position of the target object within a continuous time window are used to calculate human motion trends, robot motion trends, and target object motion trends. These trends reflect the proximity relationships and interaction state changes between different entities. The human motion trend is preferably represented by the motion changes of the wrist joint, which is closest to the target object, to reflect the main interaction intentions of the human in collaborative tasks.

[0044] Specifically, the trend of human body movement is represented as: (6) The robot's movement trend is represented as follows: (7) The trend of an object's movement is represented as: (8) in, as well as Let i represent the position of the human wrist closest to the robot, the position of the robot's end effector, and the center position of the target object at time i, respectively, and m represent the length of the time window.

[0045] The time window length m in the above formulas (6)-(8) can be adjusted according to the interaction speed. For fast interaction scenarios, the value of m can be appropriately reduced to enhance the sensitivity of trend response; for slow and delicate operation scenarios, the value of m can be appropriately increased to smooth noise interference.

[0046] 3. Calculation of human-machine-object interaction factors; To represent the characteristics of human-machine-object interaction, the distance between the human wrist and the target object is calculated based on the spatial positional relationship between the human body, the robot, and the target object. and the distance between the robot's end effector and the target object. : (9) Calculate the mean and variance of the human-object distance and the robot-object distance within a time window of length m; (10) (11) From this, the human-machine-object interaction factor can be calculated. .

[0047] (12) in, , As weight. The larger the value, the smoother the interaction between the person and the object. Before the robot interacts with the object, The impact will be suppressed.

[0048] 4. Infer the human-machine-object interaction state; Based on human motion embedding sequences, robot motion embedding sequences, target object motion embedding sequences, and human-machine-object interaction features, the human-machine-object interaction state is inferred.

[0049] Specifically, the features of the embedded subsequences are averaged and pooled. The action embedding sequences of the three entities are then concatenated through a linear layer and input into a Softmax layer to obtain the probability distribution of the interaction states. : (13) in, , The concatenated feature dimensions These are the weight matrices, , , The subsequence after average pooling. This indicates a concatenation operation, and Softmax() is the Softmax function. The resulting probability distribution... This represents the probability distribution of four interaction states: no interaction state, human-object interaction state, human-machine-object interaction state, and robot-object interaction state.

[0050] 5. Construct interaction attractors and infer interaction locations; Based on human motion trends, robot motion trends, and target object motion trends, human-object interaction attractors, robot-object interaction attractors, and target object interaction attractors are constructed as spatial constraint features to describe the proximity positions of future collaborative interactions, thereby characterizing different human-machine-object interaction states. The human-object interaction attractor is represented as: (14) in, The magnification factor indicating the tendency of human wrist joint movement. It reflects the continuity of the interaction between the human body and objects.

[0051] To ensure the feasibility of robot movement, a robot-object interaction attractor is introduced: (15) in, This indicates the magnification factor representing the robot's motion trend. It reflects the continuity of the interaction between the robot and the object, and is used to ensure that the robot moves gradually to the target object, avoiding sudden large movements.

[0052] When the interaction process is stable, such as when a person is steadily moving the target object or a robot is steadily moving in contact with the object, the target object becomes the primary reference, and the target object attractor is introduced as follows: (16) in, The magnification factor that indicates the tendency of an object to move.

[0053] Therefore, based on the constructed attractor parameters, the human-machine-object interaction position is represented as: (17) in, The weighting parameter can be calculated using the following formula: (18) in, and For learnable weight matrix and bias vector, This represents the probability distribution of the interaction states.

[0054] To better reflect the changes in the human-machine-object interaction process, the weights have been adjusted as follows: (19) This enables reasoning about the interaction positions between humans, machines, and objects.

[0055] 6. Construct human-machine-object interaction behavior representations; By mapping future interaction locations to a latent feature space using a learnable nonlinear mapping function, a representation of human-machine-object interaction states is constructed. (20) in, and These represent the learnable mapping parameters.

[0056] 7. Time dimension expansion; Extend the human-machine-object interaction location along the time dimension to embed the human-machine-object interaction behavior sequence: (twenty one) 8. Joint attention calculation; Based on the human-machine-object interaction attention mechanism, the human-machine-object interaction behavior embedding sequence is jointly attention-calculated with human action coding features, robot action coding features, and target object action coding features to generate a human-machine-object interaction behavior representation: (twenty two) The robot motion encoding sequence serves as the query vector, representing that the robot motion generation process is jointly constrained by human motion, robot motion, target object motion, and interaction state representation. The key vector and value vector are composed of human motion encoding features, robot motion encoding features, target object motion encoding features, and interaction state embedding sequence, thereby realizing the joint modeling of the interaction relationship between the human body, robot, and target object, and generating a human-machine-object interaction behavior representation.

[0057] In one embodiment, all the learnable parameters mentioned above, such as W and b, are initialized using a uniform Xavier distribution and optimized during the training phase using the backpropagation algorithm and the Adam optimizer, with the learning rate set to 0.001.

[0058] In step S3, future human and robot movements are jointly predicted; Based on the aforementioned human-machine-object interaction behavior representation, this step constructs a joint action prediction model to jointly predict future human actions and robot active response actions, optimizes the temporal consistency of the prediction results, and finally outputs a future human action prediction sequence and a future robot action prediction sequence.

[0059] Specifically, human motion encoding features, robot motion encoding features, target object motion encoding features, and human-machine-object interaction behavior representations are jointly input into the Transformer decoder. An attention mechanism is used to establish the correlation between historical and future actions. Furthermore, a non-autoregressive prediction structure is used to simultaneously generate future human action sequences and future robot action sequences, and the prediction results are optimized for temporal consistency based on a temporal attention mechanism. This includes the following sub-steps: 1. Construct the decoder query sequence; The human and robot actions at the last observation time are used as the human query sequence and robot query sequence, respectively, and input into the Transformer decoder: (twenty three) in, and These represent the human and robot actions at the last observation moment, respectively. 2. Transformer decoding; Encoding human motion features Robot motion coding features Target object motion coding features and human-machine-object interaction behavior representation Common input Transformer decoder decoding: (twenty four) Where C represents H or O.

[0060] 3. Increased self-attention; Before inputting the human and robot decoding features into the prediction network, a multi-head attention mechanism is used to enhance the self-attention of the human and robot decoding features respectively, in order to improve the temporal continuity of the prediction results. (25) 4. Non-autoregressive parallel prediction; A non-autoregressive prediction structure is adopted to simultaneously generate future human action sequences and future robot action sequences, thereby reducing the error accumulation problem in traditional autoregressive prediction structures. The results of future human motion prediction are expressed as follows: (26) The prediction results of future robot actions are expressed as follows: (27) in, and These represent human motion prediction networks and robot motion prediction networks based on MLP, respectively.

[0061] 5. Joint loss function; A joint loss function is constructed based on human motion prediction loss, robot motion prediction loss, and human-machine-object interaction consistency loss. The loss for human motion prediction is expressed as: (28) The robot action prediction loss is expressed as: (29) The human-machine-object interaction consistency loss is expressed as: (30) in, Indicates the predicted interaction location, Indicates the actual interaction location.

[0062] The joint loss function is expressed as: (31) in, and λ1 and λ2 represent the weights of the robot action prediction loss and the human-machine-object interaction consistency loss, respectively. Preferably, λ1=1 and λ2=0.5.

[0063] This invention achieves joint prediction of future human actions and proactive robot responses in complex human-machine collaboration scenarios by modeling human-machine-object interaction behavior and designing a joint action prediction model based on an attention mechanism. By introducing an interactive state reasoning mechanism, an interactive attractor mechanism, and a human-machine-object interaction attention mechanism, the invention improves the interactive consistency, physical plausibility, and temporal continuity of the prediction results.

[0064] This invention is applicable to scenarios such as industrial collaborative assembly, human-machine collaborative handling, service robot interaction, and embodied intelligent collaboration. It can effectively improve the robot's ability to understand human intentions and its proactive collaboration capabilities, thereby improving safety and collaboration efficiency in complex human-machine collaborative tasks.

[0065] like Figure 4 The illustrations show implementation examples of this invention in three typical human-machine-object interaction tasks. The object transfer interaction task describes the handover and transfer process of a target object between a human and a robot; the object retrieval interaction task describes the grasping and picking up process of a target object by a human or robot; and the post-retrieval collaborative interaction task describes the further collaborative operations performed by the human and robot around the target object after it has been acquired, including operations such as handling, adjustment, or assembly. This invention can effectively model the dynamic interaction relationships between the human, robot, and target object in all three types of tasks. By introducing an interaction state reasoning mechanism and an interaction attractor mechanism, it accurately captures key interaction positions and motion trends during collaboration. Furthermore, by combining a human-machine-object interaction attention mechanism, it enhances joint modeling capabilities, thereby significantly improving the interaction consistency, spatial rationality, and temporal continuity of action prediction in complex collaborative scenarios.

[0066] The above description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A joint prediction method for human-computer actions based on human-computer-object interaction, characterized in that, Includes the following steps: S1. Acquire the temporal motion data of the human body, robot, and target object, and synchronize the human body temporal motion data, robot temporal motion data, and target object temporal motion data in time and coordinate unify them to construct a unified human-machine-object temporal motion representation. S2. Spatial feature extraction and temporal feature encoding are performed on the human body temporal motion data, robot temporal motion data and target object temporal motion data respectively to generate human body motion coding features, robot motion coding features and target object motion coding features. Based on the human motion coding features, robot motion coding features, and target object motion coding features, the current human-machine-object interaction state is inferred, and a human-machine-object interaction behavior representation is constructed based on the human-machine-object interaction state. The process of inferring the current human-machine-object interaction state based on the human motion coding features, robot motion coding features, and target object motion coding features includes: Based on the human body temporal motion data, robot temporal motion data and target object temporal motion data within a preset time window, the human body motion trend, robot motion trend and target object motion trend are calculated respectively. Based on the human motion trend, robot motion trend and target object motion trend, a human-machine-object interaction factor is calculated. The human-machine-object interaction factor is used to characterize the smoothness of the interaction between the human body and the target object, as well as between the robot and the target object. The human motion coding features, robot motion coding features, and target object motion coding features are averaged and pooled in the time dimension and then concatenated. The concatenation is input into the classification network, and the probability distribution of human-machine-object interaction states is output. The human-machine-object interaction states include no interaction state, human-object interaction state, robot-object interaction state, and human-robot-object collaborative interaction state. The process of constructing a human-machine-object interaction behavior representation includes: Based on the human motion trend, robot motion trend and target object motion trend, human-object interaction attractors, robot-object interaction attractors and target object interaction attractors are constructed respectively. The attractor is used to characterize the spatial constraint features of the future interaction position. Based on the probability distribution of the human-machine-object interaction state, the human-object interaction attractor, robot-object interaction attractor, target object interaction attractor and the robot position at the current moment are weighted and fused to generate the human-machine-object interaction position at the current moment. The expression for the human-object interaction attractor is: in, for Human-object interaction attractor at all times Let k be the position and orientation of the human skeletal joints. The time window length, , This represents the magnification factor of the movement trend of the human wrist joint. For a moment The wrist position, which is closest to the robot on the human body. Position of the robot's end effector. For the trend of human body movement; The expression for the robot-object interaction attractor is: in, For robot-object interaction attractors, Let k be the pose of the robot's end effector. , The magnification factor for the robot's motion trend. The center position of the target object. Trends in robot movements; The expression for the target object interaction attractor is: in, for At any given moment, the target object interaction attractor, Let k be the spatial pose of the target object at time k. The tendency of an object's movement; The expression for the human-machine-object interaction position is: in, These are the weighting coefficients; The locations of human-machine-object interactions are mapped to a latent feature space and expanded along the time dimension to form a sequence of human-machine-object interaction behaviors. Based on the attention mechanism, the robot motion coding features are used as the query vector, and the human motion coding features, robot motion coding features, target object motion coding features and the human-machine-object interaction behavior embedding sequence are used as the key vector and value vector. Multi-head attention calculation is performed to generate the human-machine-object interaction behavior representation. The expression representing the human-machine-object interaction behavior is as follows: in, , These are learnable mapping parameters; S3. Based on the human-machine-object interaction behavior representation and the human motion coding features, robot motion coding features and target object motion coding features, jointly predict future continuous multi-frame human actions and future continuous multi-frame robot actions, and output future human action prediction sequence and future robot action prediction sequence. Specifically, it includes: The human-machine-object interaction behavior representation is concatenated with the human motion coding feature and the robot motion coding feature, respectively, and used as the input of the human decoder and the robot decoder. The human body action at the last observation moment in the unified human-machine-object temporal motion representation is used as the human body query sequence, and the robot action at the last observation moment is used as the robot query sequence. The human decoder is input into the first Transformer decoder along with the human query sequence to generate human decoding features; the robot decoder is input into the second Transformer decoder along with the robot query sequence to generate robot decoding features. The future human motion prediction sequence is predicted based on the human body decoding features, and the future robot motion prediction sequence is predicted based on the robot decoding features.

2. The human-computer action joint prediction method based on human-computer-object interaction as described in claim 1, characterized in that, The human movement trend is characterized by the average displacement change of the key joint of the human body closest to the target object within the preset time window. The robot's motion trend is characterized by the average displacement change of the robot's end effector within the preset time window. The target object's motion trend is characterized by the average displacement change of the target object's center position within the preset time window.

3. The human-computer action joint prediction method based on human-computer-object interaction as described in claim 1, characterized in that, The human-object interaction attractor is constructed based on the position of the human wrist joint, the historical relative displacement between the human wrist joint and the target object, and the human movement trend. The robot-object interaction attractor is constructed based on the position of the robot end effector, the historical relative displacement between the robot end effector and the target object, and the robot's motion trend; the target object interaction attractor is constructed based on the center position of the target object and the target object's motion trend.

4. The human-computer action joint prediction method based on human-computer-object interaction as described in claim 1, characterized in that, When performing weighted fusion of the human-object interaction attractor, robot-object interaction attractor, target object interaction attractor, and the robot position at the current moment, the weights corresponding to each attractor are generated by the probability distribution of the human-machine-object interaction state through a learnable nonlinear mapping. The weight allocation of the human-object interaction attractor and the target object interaction attractor is dynamically adjusted based on the human-machine-object interaction factor.

5. The human-computer action joint prediction method based on human-computer-object interaction as described in claim 1, characterized in that, The first Transformer decoder and the second Transformer decoder adopt a non-autoregressive prediction structure, using the action at the last observation time as the initial decoding input, and simultaneously generate action sequences for multiple consecutive future frames.

6. The human-computer action joint prediction method based on human-computer-object interaction as described in claim 1, characterized in that, In step S3, before inputting the human decoding features and the robot decoding features into the prediction network respectively, a multi-head attention mechanism is used to enhance the self-attention of the human decoding features and the robot decoding features respectively, so as to improve the temporal continuity of the prediction results.

7. The human-computer action joint prediction method based on human-computer-object interaction as described in claim 1, characterized in that, In step S3, a joint loss function is used to constrain the prediction results. The joint loss function includes human motion prediction loss, robot motion prediction loss, and human-machine-object interaction consistency loss. The human-machine-object interaction consistency loss is constructed based on the error between the predicted human-machine-object interaction position and the actual interaction position.

Images