Man-machine physical twin system based on multi-modal video analysis and adaptive mapping and control method

Abstract

The invention belongs to the technical field of man-machine interaction and robot control, and particularly relates to a man-machine physical twin system based on multi-modal video analysis and adaptive mapping and a control method, and the system comprises a multi-modal collection unit which is used for collecting original video streams of human body actions, expressions and voices, and compensating and repairing dynamic shielding; the semantic analysis unit comprises an action analysis module used for extracting a human skeleton key point set object interaction track from the original video stream, and an expression / voice analysis module used for outputting a facial action unit coefficient and a voice text with an emotion label; the self-adaptive mapping unit is used for establishing human body-robot joint kinematics mapping, distributing control bandwidth in real time according to task types and degrading non-key modes based on network states to guarantee action continuity, and non-contact action capture is achieved through a pure vision scheme by eliminating dependence on a wearable sensor; and the problems of failure and the like of a traditional visual scheme under shielding and illumination variation are solved.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and robot control technology, specifically a human-computer physical twin system and control method based on multimodal video analysis and adaptive mapping. Background Technology

[0002] Traditional motion capture requires wearable sensors (such as IMUs), which are costly and restrict user freedom; pure vision solutions (such as OpenPose) are greatly affected by occlusion and lighting, and cannot handle fine operations (such as finger grasping); cross-modal synchronization (such as voice + facial expression + motion) lacks a unified scheduling mechanism, resulting in robot response lag or distortion. Summary of the Invention

[0003] To address the problems existing in the background technology, this invention proposes a human-machine physical twin system and control method based on multimodal video parsing and adaptive mapping. This invention eliminates the dependence on wearable sensors and achieves contactless motion capture through a pure vision solution, overcoming the failure problems of traditional vision solutions under occlusion and lighting changes. It supports complex scenes and solves the problem of robot response distortion caused by the delay in the transmission of action, expression, and voice data.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of this invention provides a human-computer physical twin system based on multimodal video parsing and adaptive mapping, comprising: The multimodal acquisition unit is used to acquire raw video streams of human movements, expressions, and speech, and to compensate for and repair dynamic occlusion. The semantic parsing unit includes an action parsing module for extracting human skeleton key point set and object interaction trajectory from the original video stream, and an expression / speech parsing module for outputting facial action unit coefficients and emotionally labeled speech text. The adaptive mapping unit is used to establish human-robot joint kinematic mapping, allocate control bandwidth in real time according to task type, and degrade non-critical modes based on network state to ensure action continuity.

[0005] As one technical solution of the present invention, the action parsing module includes: In space, a skeletal topology adaptive matrix is ​​introduced to dynamically adjust joint connection weights through learnable parameters; In terms of time, cascaded dilated temporal convolutional layers are used to enhance the perception of temporal sequence; Object interaction trajectory extraction: The operation target in the video is segmented by the Grounded-SAM model. The hand key point coordinates and the 3D point cloud of the target object are combined to generate a multi-degree-of-freedom operation trajectory, including contact force estimation.

[0006] As a technical solution of the present invention, in the case of occlusion, the feature maps of different cameras are stitched together along the channel dimension through a multi-view feature fusion layer, and occlusion robustness key points are generated by 3D convolution.

[0007] As one technical solution of the present invention, the establishment of human-robot joint kinematic mapping includes virtual joint compensation, as detailed below: Identify the degree-of-freedom gap in robot movements relative to human movements; By inserting software-simulated virtual joints into the robot's kinematic chain, the redundant degrees of freedom of human motion are mapped to the elastic control parameters of the robot's end effector. The mapping relationship is expressed as follows: ; In the formula, Ke represents the stiffness matrix of the robot's end effector. Describe the pseudo-inverse of the Jacobian matrix of the robot. This indicates the amount of change in the angle of redundant joints in the human body.

[0008] As a technical solution of the present invention, the method of real-time allocation of control bandwidth according to task type and degradation of non-critical modes based on network status includes: The bandwidth allocation strategy sets task bandwidth thresholds A and B, which correspond to bandwidth allocation for fine-tuning tasks and bandwidth allocation for fast-movement tasks, respectively, as well as joint angle error and trajectory position error, respectively.

[0009] The network adaptive mechanism shuts down the transmission of facial expression parameters when the network round-trip delay exceeds a preset threshold; key instructions are transmitted using dual-channel redundancy, including emergency stop instructions and force feedback instructions.

[0010] As a technical solution of the present invention, the system implements cross-modal synchronization in the following way: A precise clock protocol is used to timestamp action, facial expression, and voice data uniformly. Set up a circular buffer and discard the corresponding facial expression and voice data packets when the delay of motion data exceeds a threshold.

[0011] The human-machine physical twin control method based on multimodal video parsing and adaptive mapping, as proposed as a second aspect of this invention, includes: Step S100: Real-time synchronous collection of human body movement, facial expression and voice data; Step S200: Analyze the key points of the human skeleton and the object's operation trajectory; Step S300: Analyze the facial motion unit coefficients and the speech text with emotion tags; Step S400: Generate robot joint commands based on kinematic mapping and virtual joint compensation; Step S500: Dynamically allocate control bandwidth according to task type and perform cross-modal data synchronization.

[0012] As a technical solution of the present invention, in step S200, the analysis of key points of the human skeleton includes: Collect two-dimensional key points and solve for three-dimensional joint coordinates; For occluded joints, a spatiotemporal attention mechanism is used to weight the visibility scores of adjacent joints, and a long short-term memory network is used to predict the motion trajectory.

[0013] As a technical solution of the present invention, in step S400, the virtual joint compensation includes: For bipedal robots, the movement of human legs is mapped to a zero-torque point trajectory, and ankle joint torque commands are generated through a proportional-derivative controller. For robotic arms, the lateral movement of the human wrist is mapped to the stiffness coefficient adjustment of the end effector.

[0014] As a technical solution of the present invention, in step S500, the cross-modal data synchronization includes: Activate the corresponding facial motion unit coefficients based on voice emotion tags; The keyframes of gesture actions are aligned with the timing of voice commands using a dynamic time warping algorithm.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: Occlusion robustness is improved through multi-view RGB-D fusion and spatiotemporal attention mechanisms, significantly enhancing key point detection accuracy and reducing errors. Human redundant degrees of freedom are dynamically mapped to the robot's end-effector elastic parameters, reducing end-effector position errors when the 6DOF robotic arm reproduces human movements. Cross-modal precise alignment ensures accurate action-expression-voice output delay time, and the Dynamic Time Warping (DTW) algorithm matches gestures and voice commands, minimizing synchronization errors. Furthermore, by eliminating reliance on wearable sensors, a pure vision-based approach achieves contactless motion capture, overcoming the failures of traditional vision solutions under occlusion and lighting changes. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the control flow of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] refer to Figure 1 The first aspect of the present invention provides a human-machine physical twin system based on multimodal video parsing and adaptive mapping, comprising: a multimodal acquisition unit for acquiring raw video streams of human actions, expressions, and speech, and compensating for and repairing dynamic occlusion; a semantic parsing unit, including an action parsing module for extracting human skeletal key point sets and object interaction trajectories from the raw video stream, and an expression / speech parsing module for outputting facial action unit coefficients and speech text with emotion tags; and an adaptive mapping unit for establishing human-robot joint kinematic mapping, allocating control bandwidth in real time according to task type, and degrading non-critical modalities based on network state to ensure action continuity.

[0019] It should be noted that the acquisition unit uses a multi-view RGB-D camera array with infrared compensation and supports dynamic occlusion repair. The semantic parsing unit consists of two parts: an action parsing sub-part, an expression / speech parsing sub-part, and a mapping sub-part. The action parsing sub-part fuses human skeleton key points and object interaction trajectories based on an improved spatiotemporal graph convolutional network, such as the estimation of contact force when grasping a cup. The expression / speech parsing sub-part uses a multi-task learning model to synchronously output facial expression parameters and speech text from a single video stream. The adaptive mapping sub-part utilizes a robot kinematic chain optimization algorithm to establish a human-robot joint kinematic mapping through Lie groups and Lie algebras, solving the problem of heterogeneous degrees of freedom matching (e.g., human 7DOF arm → robot 6DOF robotic arm). According to a dynamic priority scheduler, the control bandwidth allocation is adjusted in real time based on the task type (e.g., fine manipulation vs. rapid movement).

[0020] The adaptive mapping sub-section of this application will be further elaborated below. This sub-section is responsible for converting the parsed human action, expression, and speech data into instructions that can be executed by the robot, solving the heterogeneity problem between the human body and the robot, and ensuring the real-time performance and coordination of multimodal output.

[0021] The kinematic chain optimization algorithm establishes a kinematic mapping model based on Lie groups and Lie algebras to describe the spatial motion relationship between human and robot joints. This addresses the issue of heterogeneous degree-of-freedom matching, such as mismatches between human joint degrees of freedom (e.g., 7DOF in an arm) and robot degrees of freedom (e.g., 6DOF in a robotic arm).

[0022] Virtual joint compensation is used to insert virtual joints into the robot's kinematic chain to fill differences in degrees of freedom. For example, redundant rotational degrees of freedom of the human arm can be mapped to elastic deformation parameters of the robot's end effector. Inverse kinematics optimization employs damped least squares (DLS) to solve the inverse kinematics, avoiding singular configurations and ensuring smooth motion. Joint weight coefficients are dynamically adjusted in real time according to task requirements (such as fine manipulation or rapid movement), prioritizing critical degrees of freedom (such as the precision of the robotic arm's end effector).

[0023] Specifically, the establishment of the human-robot joint kinematic mapping includes virtual joint compensation, as detailed below: Identify the degree-of-freedom gap in robot movements relative to human movements; By inserting software-simulated virtual joints into the robot's kinematic chain, the redundant degrees of freedom of human motion are mapped to the elastic control parameters of the robot's end effector. The mapping relationship is expressed as follows: ; In the formula, Ke represents the stiffness matrix of the robot's end effector. Describe the pseudo-inverse of the Jacobian matrix of the robot. This indicates the amount of change in the angle of redundant joints in the human body.

[0024] It is further necessary to clarify that the real-time allocation of control bandwidth based on task type and the degradation of non-critical modes based on network status includes: The bandwidth allocation strategy sets task bandwidth thresholds A and B, which correspond to bandwidth allocation for fine-tuning tasks and bandwidth allocation for fast-movement tasks, respectively, as well as joint angle error and trajectory position error, respectively.

[0025] The network adaptive mechanism shuts down the transmission of facial expression parameters when the network round-trip delay exceeds a preset threshold; key instructions are transmitted using dual-channel redundancy, including emergency stop instructions and force feedback instructions.

[0026] The system implements cross-modal synchronization in the following way: A precise clock protocol is used to timestamp action, facial expression, and voice data uniformly. Set up a circular buffer and discard the corresponding facial expression and voice data packets when the delay of motion data exceeds a threshold.

[0027] The following are merely examples and do not constitute any limitation on the present invention.

[0028] Task Classification and Bandwidth Allocation: Fine-grained operations (such as grasping, tightening screws): allocate high control bandwidth (≥100Hz) to prioritize motion accuracy; Fast movements (such as waving): reduce bandwidth to 50Hz to allow for small trajectory errors in order to improve speed; Network Adaptation: Monitor 5G / Wi-Fi signal strength. If the latency is greater than 100ms, automatically downgrade non-critical modalities (such as turning off facial expression mapping and retaining only actions and voice) and adopt a data packet redundancy transmission mechanism to ensure the reliability of critical commands (such as emergency stop signals).

[0029] Cross-modal synchronization protocol: Timestamp alignment: During the semantic parsing stage, a unified timestamp (accuracy ±1ms) is applied to action, expression, and speech data; Buffer management: A circular buffer is set up to dynamically adjust the waiting window of each modality data (e.g., fusion is triggered when the delay of action data is ≤20ms).

[0030] In one embodiment, the motion parsing module includes: spatially, introducing a skeletal topology adaptive matrix and dynamically adjusting joint connection weights through learnable parameters; temporally, employing cascaded dilated temporal convolutional layers to enhance temporal perception; and object interaction trajectory extraction, which involves segmenting the operation target in the video using a Grounded-SAM model, combining the coordinates of key hand points with the 3D point cloud of the target object, and generating a multi-degree-of-freedom operation trajectory, including contact force estimation.

[0031] In occlusion scenarios, feature maps from different cameras are stitched together along the channel dimension through a multi-view feature fusion layer, and occlusion robustness key points are generated through 3D convolution.

[0032] The second aspect of the present invention proposes a human-computer physical twin control method based on multimodal video parsing and adaptive mapping, comprising: step S100, real-time synchronous acquisition of human body movement, facial expression and voice data.

[0033] The analysis of key points of the human skeleton includes: collecting two-dimensional key points and solving three-dimensional joint coordinates; for occluded joints, a spatiotemporal attention mechanism is used to weight the visibility scores of adjacent joints, and a long short-term memory network is used to predict the motion trajectory.

[0034] Step S200: Analyze the key points of the human skeleton and the object's operation trajectory.

[0035] Step S300: Analyze the facial motion unit coefficients and the speech text with emotion tags.

[0036] Step S400: Generate robot joint commands based on kinematic mapping and virtual joint compensation.

[0037] The virtual joint compensation includes: for bipedal robots, mapping the human leg movement to a zero-torque point trajectory and generating ankle joint torque commands through a proportional-derivative controller; for robotic arms, mapping the lateral movement of the human wrist to the stiffness coefficient adjustment of the end effector.

[0038] Step S500: Dynamically allocate control bandwidth according to task type and perform cross-modal data synchronization.

[0039] The cross-modal data synchronization includes: activating the corresponding facial motion unit coefficients based on voice emotion tags; and aligning the keyframes of gesture actions with the timing points of voice commands using a dynamic time warping algorithm.

[0040] The problem of how to make robots with different configurations (such as bipedal humanoids, wheeled chassis, and multi-axis robotic arms) reproduce human movements, especially when the degrees of freedom and range of motion are inconsistent, remains unsolved.

[0041] A more detailed explanation of the above plan: Virtual joint insertion involves analyzing the robot's actual joint configuration, identifying gaps in degrees of freedom (such as the lack of clavicle rotational degree of freedom in the human shoulder), inserting virtual joints into the robot's kinematic chain, and simulating the missing degrees of freedom through software.

[0042] As an example, in this embodiment, for a wheeled robot, the movement of the human leg is mapped to a virtual "tilting joint," which is then converted into chassis speed commands. For a robotic arm, the lateral movement of the human wrist is mapped to compliant control parameters for the end effector. Through dynamic constraints, the range of motion of the virtual joints must match the physical limits of the robot (e.g., to prevent the robotic arm from exceeding its limits).

[0043] The invention will be further explained below through specific examples: a. Use OpenPose or BlazePose to extract human key points (such as joints and hands) from ordinary camera videos, and map human movements to robot joint angles (such as robotic arms and humanoid robots) through inverse kinematics (IK). This can be applied to educational robots (such as dancing robots), home service robots (such as companion robots that follow movements), etc.

[0044] b. Wearing low-cost IMU sensors (such as Perception Neuron, Xsens DOT) to collect human motion data and transmitting the data via Bluetooth / Wi-Fi to drive robot joint movement can be applied to remotely controlled robots (such as industrial maintenance, surgical assistance), VR / AR interactive robots, etc.

[0045] c. Use an RGB-D camera (such as Intel RealSense) to record video of human operations, and use a GPT-4V+ visual segmentation model (such as Grounded-SAM) to identify objects and motion trajectories. The robot automatically generates motion strategies (such as grasping and placing). This can be applied to logistics sorting robots (such as mimicking human packing) and kitchen robots (such as mimicking sprinkling salt and pouring water).

[0046] d. Input a video of human operation, and AI will generate a 3D spatial trajectory (including depth information). The robot will then directly execute the generated trajectory (without requiring real machine training data). This can be applied to industrial automation (such as assembly and welding), medical robots (such as mimicking a doctor's surgical movements), etc.

[0047] e. Using AR / VR devices (such as Apple Vision Pro) to overlay virtual robot movements onto real-world scenes in real time, the robot can synchronously execute the adjustments made by the operator. This can be applied to high-risk operations (such as power maintenance) and entertainment performances (such as virtual idols interacting with robots).

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A human-computer physical twin system based on multimodal video parsing and adaptive mapping, characterized in that, include: The multimodal acquisition unit is used to acquire raw video streams of human movements, expressions, and speech, and to compensate for and repair dynamic occlusion. The semantic parsing unit includes an action parsing module for extracting human skeleton key point set and object interaction trajectory from the original video stream, and an expression / speech parsing module for outputting facial action unit coefficients and emotionally labeled speech text. The adaptive mapping unit is used to establish human-robot joint kinematic mapping, allocate control bandwidth in real time according to task type, and degrade non-critical modes based on network state to ensure action continuity.

2. The human-computer physical twin system based on multimodal video parsing and adaptive mapping according to claim 1, characterized in that, The action parsing module includes: In space, a skeletal topology adaptive matrix is ​​introduced to dynamically adjust joint connection weights through learnable parameters; In terms of time, cascaded dilated temporal convolutional layers are used to enhance the perception of temporal sequence; Object interaction trajectory extraction: The operation target in the video is segmented by the Grounded-SAM model. The hand key point coordinates and the 3D point cloud of the target object are combined to generate a multi-degree-of-freedom operation trajectory, including contact force estimation.

3. The human-computer physical twin system based on multimodal video parsing and adaptive mapping according to claim 2, characterized in that, In occlusion scenarios, feature maps from different cameras are stitched together along the channel dimension through a multi-view feature fusion layer, and occlusion robustness key points are generated through 3D convolution.

4. The human-computer physical twin system based on multimodal video parsing and adaptive mapping according to claim 1, characterized in that, The establishment of the human-robot joint kinematic mapping includes virtual joint compensation, as detailed below: Identify the degree-of-freedom gap in robot movements relative to human movements; By inserting software-simulated virtual joints into the robot's kinematic chain, the redundant degrees of freedom of human motion are mapped to the elastic control parameters of the robot's end effector. The mapping relationship is expressed as follows: ； In the formula, Ke represents the stiffness matrix of the robot's end effector. Describe the pseudo-inverse of the Jacobian matrix of the robot. This indicates the amount of change in the angle of redundant joints in the human body.

5. The human-computer physical twin system based on multimodal video parsing and adaptive mapping according to claim 1, characterized in that, The method of allocating control bandwidth in real time according to task type and degrading non-critical modes based on network status includes: The bandwidth allocation strategy sets task bandwidth thresholds A / B, which correspond to bandwidth allocation for fine-tuning tasks and bandwidth allocation for fast-moving tasks, and also correspond to joint angle error and trajectory position error, respectively. The network adaptive mechanism shuts down the transmission of facial expression parameters when the network round-trip delay exceeds a preset threshold; key instructions are transmitted using dual-channel redundancy, including emergency stop instructions and force feedback instructions.

6. The human-computer physical twin system based on multimodal video parsing and adaptive mapping according to claim 1, characterized in that, The system implements cross-modal synchronization in the following way: A precise clock protocol is used to timestamp action, facial expression, and voice data uniformly. Set up a circular buffer and discard the corresponding facial expression and voice data packets when the delay of motion data exceeds a threshold.

7. A human-machine physical twin control method based on multimodal video analysis and adaptive mapping, characterized in that, include: Step S100: Real-time synchronous collection of human body movement, facial expression and voice data; Step S200: Analyze the key points of the human skeleton and the object's operation trajectory; Step S300: Analyze the facial motion unit coefficients and the speech text with emotion tags; Step S400: Generate robot joint commands based on kinematic mapping and virtual joint compensation; Step S500: Dynamically allocate control bandwidth according to task type and perform cross-modal data synchronization.

8. The human-machine physical twin control method based on multimodal video parsing and adaptive mapping according to claim 7, characterized in that, In step S200, the parsing of key points of the human skeleton includes: Collect two-dimensional key points and solve for three-dimensional joint coordinates; For occluded joints, a spatiotemporal attention mechanism is used to weight the visibility scores of adjacent joints, and a long short-term memory network is used to predict the motion trajectory.

9. The human-machine physical twin control method based on multimodal video parsing and adaptive mapping according to claim 7, characterized in that, In step S400, the virtual joint compensation includes: For bipedal robots, the movement of human legs is mapped to a zero-torque point trajectory, and ankle joint torque commands are generated through a proportional-derivative controller. For robotic arms, the lateral movement of the human wrist is mapped to the stiffness coefficient adjustment of the end effector.

10. The human-machine physical twin control method based on multimodal video parsing and adaptive mapping according to claim 7, characterized in that, In step S500, the cross-modal data synchronization includes: Activate the corresponding facial motion unit coefficients based on voice emotion tags; The keyframes of gesture actions are aligned with the timing of voice commands using a dynamic time warping algorithm.

Images