Remote humanoid mapping control method based on virtual reality motion capture mechanical arm

Abstract

The invention discloses a remote humanoid mapping control method based on a virtual reality motion capture mechanical arm. A motion data acquisition module transmits a matrix of a head coordinate system relative to a world coordinate system and a homogeneous transformation matrix to a coordinate system conversion module, establishes a base coordinate system and a tail end coordinate system, and establishes a kinematic chain conversion model; extreme position data of forearm extension of a user is collected, a proportionality coefficient is calculated, and a motion space boundary of the mechanical arm is output; pose normalization processing is carried out to form a discretized mechanical arm tail end pose sequence, and then the spatial relation between the mechanical arm sphere model and the obstacle is analyzed; generating a collision-free track by adopting a parallelization gradient optimization algorithm; the trajectory optimization module realizes trajectory smoothness and time optimization by constructing a multi-objective optimization function; and a mechanical arm motion simulation system is constructed, the tail end track precision is verified, and the controller drives the mechanical arm to achieve high-fidelity reproduction of user actions according to the track. According to the invention, high-precision and smooth-path motion migration of the mechanical arm is realized.

Description

Technical Field

[0001] The present invention relates to robot teleoperation, virtual reality (VR) control, and robotic arms in the field of intelligent manufacturing, and in particular to a remote humanoid mapping control method for a robotic arm based on virtual reality motion capture. Background Art

[0002] Teleoperation of robotic arms is a key technology in robotic control, enabling close interaction between the human operator and the robot through human-robot collaboration. Teleoperation plays a crucial role in data collection for imitation learning, providing not only precise operational demonstrations but also generating natural and smooth trajectories, enabling the generalization of learned policies to new environmental configurations and tasks.

[0003] Currently, the main methods for achieving robotic arm teleoperation include visual motion capture, inertial motion capture, mechanical motion capture, and hybrid motion capture. Visual motion capture uses a camera to capture the operator's movements, combines computer vision algorithms to extract key points of the human body, and maps them to the robotic arm control. It does not require wearable equipment, is convenient to operate, and is low-cost. However, it relies on ambient lighting conditions and is easily affected by factors such as occlusion and background clutter, resulting in image quality degradation and affecting the accuracy of teleoperation. Inertial motion capture uses an inertial measurement unit (IMU) worn by the operator, such as a somatosensory glove equipped with an accelerometer and gyroscope, to estimate the human posture and transmit it to the robotic arm. This motion capture method has low latency and is not restricted by environmental conditions, but it has cumulative errors, and prolonged wearing of the device can lead to operator fatigue. Mechanical motion capture uses a mechanical device connected to the robotic arm, such as an operator's hand or another robotic arm, to directly map joint angles to the robotic arm for control. It has high real-time performance and almost no latency, but is costly and has a limited range of motion. Hybrid motion capture combines multiple motion capture technologies to complement each other, offering high robustness and accuracy, and can adapt to complex environments. However, it is complex, costly, and difficult to maintain. Summary of the Invention

[0004] Purpose of the invention: In response to the shortcomings of the existing technology, the present invention proposes a remote humanoid mapping control method for a robotic arm based on virtual reality motion capture. Through VR glasses, visual feedback and intelligent algorithms, high-precision and smooth path motion migration of the robotic arm and low-latency and high-precision robotic arm control are achieved; the problems of accuracy, adaptability and comfort existing in traditional remote operation methods are overcome.

[0005] Technical solution: The present invention is based on a virtual reality motion capture robotic arm remote humanoid mapping control system, which includes a robotic arm, a controller, a host computer, a workbench, an object, a user, VR glasses, a terminal processor and a camera array.

[0006] The terminal processor includes a motion data acquisition module, a coordinate system conversion module, a workspace scale scaling module, a motion redirection module, a three-dimensional reconstruction module, a collision detection module, an inverse kinematics solution module, a trajectory optimization module, and a simulation environment generation module.

[0007] The motion data acquisition module realizes high-precision motion capture. By capturing the user's arm movements through VR glasses, it calculates in real time the wrist-to-head coordinate system transformation matrix of the user's wrist joint relative to the head coordinate system and the head-to-world coordinate system rotation matrix of the head coordinate system relative to the world coordinate system, and transmits these two matrices to the coordinate system conversion module.

[0008] The coordinate system conversion module converts the coordinate system of the posture data, receives the wrist-head coordinate system transformation matrix and the head-world coordinate system rotation matrix output by the motion data acquisition module, and uses the quaternion attitude reprojection algorithm to convert the data into a coordinate system according to the orientation of the robot arm end link coordinate system and the base coordinate system. It converts the natural movement of the user's arm recognized by the VR glasses into standardized posture data recognized by the robot arm, and outputs the obtained standardized posture data to the motion redirection module.

[0009] The workspace scaling module is used to scale the position data within the standardized pose data of the robotic arm. It receives the user's input of the robotic arm model or custom working radius, analyzes the user's forearm extension limit data based on the user's arm motion output from the VR glasses, and dynamically adjusts the operating space scale coefficient using a nonlinear scaling algorithm to output the actual movable space of the robotic arm. The workspace scaling module is equipped with an industrial robot database containing characteristic values for the working radius and joint range of motion of various robotic arm models, automatically matching the DH parameters and kinematic constraints of different robotic arm models. The workspace scaling module monitors the user's arm motion in real time. If the user's arm motion is detected outside the range of motion, a progressive scaling strategy is triggered to prevent the robotic arm's end link position from exceeding the workspace.

[0010] The motion redirection module performs posture normalization on the standardized posture data output by the received coordinate system conversion module and the actual movable space output by the workspace scale scaling module. On the premise of ensuring that the motion trajectory of the end effector meets the expectations of human operation, the standardized posture data is converted and calculated in the actual motion space and a discrete posture sequence of the robotic arm end is output.

[0011] The 3D reconstruction module reconstructs the motion environment of the robotic arm in 3D, receives multi-view video data recorded by the camera array, uses the 3D Gaussian splatter (3DGS) algorithm to fuse multi-source point cloud data in real time, and generates a textured millimeter-level precision motion environment model through adaptive point cloud density optimization.

[0012] The collision detection module detects self-collisions and collisions with the environment during the robot's motion. It receives the robot's motion environment model generated by the 3D reconstruction module and constructs a spherical collision model for the robot. Based on the end-point pose sequence output by the motion redirection module, the collision detection module calculates the spatial relationship between the robot's spherical collision model and obstacles in real time. When the joint sphere is detected approaching an obstacle, the speed suppression function is automatically triggered, slowing the robot's movement in the danger zone to ensure safe movement.

[0013] The inverse kinematics solver module calculates the manipulator's pose data, receives the end-point pose sequence output by the motion redirection module, and combines the sphere collision model constructed by the collision detection module with the end-point pose sequence output by the motion redirection module to output a collision-free trajectory for the manipulator. The inverse kinematics solver module uses a gradient-optimized inverse kinematics algorithm and completes the solution by launching parallel threads through shared memory, ensuring end-point positioning accuracy while improving solution speed.

[0014] The trajectory optimization module optimizes the collision-free trajectory output by the inverse kinematics solver module, achieving trajectory smoothness and time optimization while ensuring collision avoidance, and ultimately outputs high-precision joint control instructions after trajectory smoothing and time optimization of the robotic arm.

[0015] The simulation environment generation module establishes the motion simulation system of the robotic arm and verifies the high-precision joint control instructions of the robotic arm output by the trajectory optimization module; the simulation environment generation module constructs a virtual-reality integrated robotic arm motion simulation system based on the motion environment model output by the 3D reconstruction module, receives the joint control instructions of the robotic arm output by the trajectory optimization module, and outputs the joint control instructions of the robotic arm to the robotic arm host computer if the motion trajectory of the robotic arm meets the requirements; if the trajectory does not meet the requirements, it returns to the collision detection module to reprocess the data to form a closed-loop verification mechanism.

[0016] The present invention provides a remote humanoid mapping control method for a virtual reality motion capture robotic arm, comprising the following steps:

[0017] 1) Arrange a camera array around the robotic arm and workbench, allowing the cameras to capture the work area from multiple angles. Place the object on the workbench, connect the robotic arm to the controller, and connect the controller to the host computer.

[0018] 2) The user wears VR glasses, and the motion capture system establishes a low-latency data transmission channel with the terminal processor through a wireless network. It uses an encrypted transmission protocol based on TLS1.3, implements two-way authentication through a self-signed certificate, establishes a secure communication link at the application layer, and achieves end-to-end latency. Where B is the channel bandwidth and SNR is the signal-to-noise ratio, meeting the transmission requirements for real-time motion capture. Once activated, the motion capture system continuously captures the user's arm movements, ensuring high-fidelity transmission of motion data.

[0019] 3) After the user wears VR glasses, the motion data acquisition module establishes a coordinate system H = (x h ,y h ,z h ) and W=(x w ,y w ,z w ), the movement of the head coordinate system H relative to the world coordinate system G and the movement of the wrist coordinate system W relative to the head coordinate system H are identified in real time through the built-in gyroscope and accelerometer of the VR glasses.

[0020] 4) The rotation matrix of the head coordinate system H relative to the world coordinate system G obtained by the motion data acquisition module Calculated by gyroscope data, the rotation matrix is updated by real-time angular velocity integration in, is the skew-symmetric matrix of the angular velocity at the current moment; the obtained posture of the wrist coordinate system W relative to the head coordinate system H is obtained by the homogeneous transformation matrix Represents, where the rotation matrix Obtained from gyroscope data through quaternion conversion: Translation vector Obtained by integrating the accelerometer data: The motion data acquisition module will matrix and Transmitted to the coordinate system conversion module in real time.

[0021] 5) The coordinate system conversion module establishes a base coordinate system B = (x b ,y b ,z b ), establish the end coordinate system E=(x e ,y e ,z e ), based on the head-world coordinate system rotation matrix output by the motion data acquisition module and wrist-head coordinate system transformation matrix Combined with the preset coordinate system transformation relationship and Establish the kinematic chain transformation model. Through matrix operation Get the posture matrix of the end effector of the robotic arm and convert it into standardized posture data using the quaternion reprojection algorithm The quaternion components are passed through This conversion process achieves accurate mapping from user movements to standardized pose data of the robotic arm, converting the user's arm movements recognized by VR glasses into standardized pose data that can be recognized by the robotic arm.

[0022] 6) The workspace scale scaling module collects the user's forearm extension extreme position data in real time according to the user's arm movement process Working radius R based on the robotic arm model parameter database robot and joint range of motion θ lim , establish a spatial mapping relationship. Use nonlinear scaling algorithm to calculate the scale coefficient: Where k and c are smoothing coefficients, is the current hand position. When the progressive scaling strategy is triggered: The actual motion space boundary of the output robot arm is:

[0023] 7) The motion redirection module receives the standardized posture data output by the coordinate system conversion module and the actual movable space output by the workspace scale scaling module Perform pose normalization: X robot =(P robot ,q w ,q x ,q y ,q z ), where P robot =(x robot ,y robot ,z robot ), forming a discrete sequence of manipulator end poses At this point, the data mapping from the user action space to the robotic arm operation space is completed, and data processing of the mapped robotic arm end pose sequence begins.

[0024] 8) A multi-view camera array is installed around the robotic arm and workbench, with all lens optical axes aligned with the center of the work area. This ensures that data from the robotic arm's end-effector workspace, the workpiece manipulation area, and the safety monitoring area are captured. The multi-view video data recorded by the camera array is transmitted to the terminal processor via wireless communication.

[0025] 9) The collision detection module represents each link of the robotic arm as a sphere set S according to the size of the robotic arm link i ={s j |s j =(x j ,y j )}, where x j ∈R 3 is the center position of the sphere, rj is the radius of the sphere, and the overall collision model is At the same time, the collision detection module receives the robot arm motion environment model generated by the 3D reconstruction module, and analyzes the spatial relationship between the robot arm sphere model and obstacles in real time according to the end posture sequence output by the motion redirection module. It uses a continuous collision detection algorithm: when checking the current sphere s t The symbol d=SD(s t )-r, if d≤0, it is determined to be a collision; if not, scan the buffer distance η in the reverse direction of the movement and calculate the scanning point s'=s t +k(s t-1 -s t ), where k = 1-(η+r) / ||s t -s t-1 ||, detect potential collision risks; the collision penalty term uses a piecewise smooth function: When the joint sphere is detected approaching an obstacle, d c When ≤η, the speed suppression function is triggered, and the speed weighting item Reduce the movement speed of the robotic arm to ensure safety.

[0026] 10) The inverse kinematics solution module uses a parallelized gradient optimization algorithm to generate a collision-free trajectory based on the spherical collision model of the robotic arm provided by the collision detection module and the end pose sequence output by the motion redirection module.

[0027] 11) The trajectory optimization module receives the collision-free trajectory generated by the inverse kinematics solution module and further optimizes it. The trajectory optimization module achieves trajectory smoothness and time optimization by constructing a multi-objective optimization function: by minimizing the high-order derivatives of joint motion Achieve smoothness optimization; by dynamically adjusting the time step Achieve time optimization. Finally, output the robot arm's trajectory smoothing optimization and time-optimized high-precision joint control instructions.

[0028] 12) The simulation environment generation module receives the motion environment model provided by the 3D reconstruction module, aligns the motion environment model with the robot model in coordinate system and space, and builds a virtual-real fusion robot motion simulation system. In the motion simulation system, the high-precision joint control instructions of the robot arm output by the trajectory optimization module are executed. Verify the end trajectory accuracy, detect whether the movement of each joint exceeds the physical limit, and whether the robot arm 1 collides with the environment. If the end position error ||Δx||>5mm and the joint speed exceeds the limit, Or if the robot arm collides with the environment, return to step 10) and run again.

[0029] 13) After the simulation test passes, the terminal processor sends the optimized control instructions for each joint of the robotic arm to the robotic arm host computer, which then transmits them to the controller.

[0030] 14) Finally, the controller drives the robotic arm to complete the predetermined operation task along the predetermined trajectory, achieving high-fidelity reproduction of the user's actions.

[0031] In step 8), the 3D reconstruction module receives the multi-view video data output by the camera array, uses the 3D Gaussian splatter (3DGS) algorithm to fuse the multi-source point cloud data in real time, generates a textured millimeter-level precision motion environment model through adaptive point cloud density optimization, and transmits it to the collision detection module and the simulation environment generation module.

[0032] In step 10), the inverse kinematics solution module minimizes the composite objective function m including the end pose error and collision penalty. θ in(α||X g -K e (θ)|| 2 +βC coll (K s (θ))) to solve the joint configuration of the robot arm, where X g is the target pose, K e (θ) is the forward kinematics, C coll is the collision penalty term, C coll (K s (θ))=β2·speed(S t-1,t,t+1 )·smooth(sweep(S t-1,t,t+1 )), where sweep() is the maximum penetration distance output by the continuous collision detection algorithm, and smooth() is the maximum penetration distance of d c Perform piecewise smoothing, and speed() is the speed weighting item.

[0033] To improve computing efficiency, the inverse kinematics solution module adopts the CUDA parallel architecture and starts 16 dedicated threads to work together, including 4 matrix transformation calculation threads and 12 gradient backpropagation threads. It processes the accumulated transformation matrix of the kinematic chain through a shared memory mechanism and uses warp-level parallel reduction to quickly screen the optimal solution, ensuring the end positioning accuracy while improving the solution speed.

[0034] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0035] (1) This invention achieves motion redirection of human motion to the end of a robotic arm through high-precision motion capture and multi-level coordinate mapping using VR glasses. The motion capture system uses a 3D Gaussian splatter algorithm to reconstruct a millimeter-level environmental model in real time. Combined with progressive spatial scaling and collision detection mechanisms, it achieves a dynamic balance between safe obstacle avoidance and intuitive operation in a synchronized virtual and real simulation environment.

[0036] (2) The present invention generates a smooth trajectory under the premise of meeting energy consumption constraints and joint limits through multimodal inverse kinematics solution and adaptive trajectory optimization, and integrates visual perception, motion planning and virtual-reality synchronization functions through a modular architecture. It is compatible with the industrial robot database and supports parameter customization, thereby improving the remote control accuracy and safety in complex scenarios, and providing an efficient and intelligent solution for industrial automation.

[0037] (3) Compared with the traditional hybrid motion capture method, the method of the present invention does not rely on professional motion capture equipment. The operator only needs to wear VR glasses to achieve low-latency, high-precision robotic arm control, which significantly reduces costs; it has stability and anti-interference capabilities in complex environments; at the same time, it is intuitive to operate and easy to maintain, providing an efficient and reliable solution for remote operation of industrial robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the structure of a remote humanoid mapping control system for a robotic arm in an embodiment of the present invention;

[0039] Figure 2 A front view of a schematic diagram of environmental data collection according to an embodiment of the present invention;

[0040] Figure 3 A top view of a schematic diagram of environmental data collection according to an embodiment of the present invention;

[0041] Figure 4 Flowchart of data processing according to an embodiment of the present invention;

[0042] FIG5 is a schematic diagram of an implementation structure of an embodiment of the present invention;

[0043] 5( a ) is a front view of a schematic diagram of an implementation structure of an embodiment of the present invention;

[0044] FIG5( b ) is a top view of a schematic diagram of an implementation structure of an embodiment of the present invention;

[0045] FIG6 is a diagram illustrating an implementation process of an embodiment of the present invention;

[0046] 6( a ) is a schematic diagram of motion data collection according to an embodiment of the present invention;

[0047] FIG6( b ) is a schematic diagram of motion data processing according to an embodiment of the present invention;

[0048] FIG6( c ) is a top view of a schematic diagram of multi-view video data acquisition according to an embodiment of the present invention;

[0049] FIG6( d ) is a front view of a schematic diagram of multi-view video data acquisition according to an embodiment of the present invention;

[0050] FIG7 is a diagram illustrating a data processing implementation process according to an embodiment of the present invention;

[0051] FIG7( a ) is a schematic diagram of an information flow for collecting motion data through VR glasses according to an embodiment of the present invention;

[0052] FIG7( b ) is a schematic diagram of an information flow for coordinate conversion of motion data according to an embodiment of the present invention;

[0053] FIG7( c ) is a schematic diagram of an information flow of motion data redirection according to an embodiment of the present invention;

[0054] FIG7( d ) is a schematic diagram of an information flow for performing data processing based on a 3D reconstructed model according to an embodiment of the present invention;

[0055] FIG7( e ) is a schematic diagram of an information flow for solving redirected motion data according to an embodiment of the present invention;

[0056] FIG7( f ) is a schematic diagram of an information flow for optimizing the motion trajectory of a robotic arm according to an embodiment of the present invention;

[0057] FIG7( g ) is a schematic diagram of the information flow of robot arm trajectory verification and transmission according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] like Figure 1 , Figure 2 , Figure 3As shown, the remote humanoid mapping control system for a virtual reality motion capture robotic arm in an embodiment of the present invention includes a robotic arm 1, a controller 2, a host computer 3, a workbench 4, an object 5, a user 6, VR glasses 7, a terminal processor 8, and a camera array 9. In this embodiment, robotic arm 1 is a six-degree-of-freedom robotic arm with a gripping device at its end for gripping and moving other objects. Controller 2 is the control processor corresponding to robotic arm 1, monitoring the motion state of the motors in each joint of robotic arm 1 and feeding it back to host computer 3. It also receives instructions from host computer 3 to control the joints of robotic arm 1 to perform corresponding movements. The host computer 3 is a computer with data processing capabilities, connected to the controller 2, and has multiple data processing modules inside. It receives the robot arm motion data, issues the robot arm motion instructions and controls the robot arm motion; the workbench 4 is a processing table for placing the robot arm and the object, which is fixed to the ground; the object 5 is the geometric block to be clamped by the robot arm 1; the user 6 is an experimenter wearing VR glasses 7, who remotely controls the robot arm 1; the VR glasses 7 are virtual reality equipment with built-in gyroscopes and accelerometers for identifying the movements of the user 6's arm and converting the head coordinate system and wrist coordinate system of the user 6 into the coordinate system of the user 6. The posture is output to the terminal processor 8; the terminal processor 8 is a computer equipped with a variety of model training algorithm modules, which is used to receive the posture data transmitted by the VR glasses 7, and redirect the movements of the user's 6 arm to the robotic arm 1, establish a simulation environment and solve the motion data of the robotic arm 1; the camera array 9 is composed of multiple fixed-posture cameras, which are arranged around the geometric center circle of the robotic arm 1 and the workbench 4. The camera angles are all facing the robotic arm 1, the workbench 4 and the object 5. A data connection is established with the host computer 3 through wireless communication, and the multi-view video is recorded synchronously under the control of the host computer 3.

[0059] like Figure 4 As shown, the terminal processor 8 includes a motion data acquisition module 81, a coordinate system conversion module 82, a workspace scale scaling module 83, a motion redirection module 84, a three-dimensional reconstruction module 85, a collision detection module 86, an inverse kinematics solution module 87, a trajectory optimization module 88, and a simulation environment generation module 89. The motion data acquisition module 81 is used to achieve high-precision motion capture. By capturing the arm movements of the user 6 through the VR glasses 7, the wrist-to-head coordinate system transformation matrix of the user 6 wrist joint relative to the head coordinate system and the head-to-world coordinate system rotation matrix of the head coordinate system relative to the world coordinate system are calculated in real time, and these two matrices are transmitted to the coordinate system conversion module 82.

[0060] The coordinate system conversion module 82 is used to convert the coordinate system of the posture data, receive the wrist-head coordinate system transformation matrix and the head-world coordinate system rotation matrix output by the motion data acquisition module 81, compare the orientation of the end link coordinate system and the base coordinate system of the robotic arm 1, use the quaternion attitude reprojection algorithm to convert the data into a coordinate system, and finally convert the movement of the user 6 arm recognized by the VR glasses 7 into standardized posture data that can be recognized by the robotic arm 1, and output the obtained standardized posture data to the motion redirection module 84.

[0061] The workspace scaling module 83 is used to scale the position data within the standardized pose data of the robotic arm 1. It receives the model or custom working radius of the robotic arm 1 input by the user 6 and analyzes the extreme position data of the user 6's forearm extension based on the motion of the user 6's arm output by the VR glasses 7. It dynamically adjusts the operating space scale coefficient using a nonlinear scaling algorithm and outputs the actual motion space of the robotic arm 1. The workspace scaling module 83 is equipped with an industrial robot database containing characteristic values such as the working radius and joint range of motion of various robotic arm models. It automatically matches the DH parameters and kinematic constraints of different robotic arm models. The workspace scaling module 83 monitors the motion of the user 6's arm in real time. If it detects that the user 6's arm moves beyond the range of motion, it triggers a progressive scaling strategy to prevent the end link position of the robotic arm 1 from exceeding the workspace.

[0062] The motion redirection module 84 performs posture normalization processing on the standardized posture data output by the received coordinate system conversion module 82 and the actual movable space output by the workspace scale scaling module 83. On the premise of ensuring that the motion trajectory of the end effector meets the human operation expectations, the standardized posture data is converted and calculated in the actual motion space and a discrete posture sequence of the end of the robot arm 1 is output.

[0063] The three-dimensional reconstruction module 85 performs three-dimensional reconstruction of the motion environment of the robotic arm 1, receives multi-view video data recorded by the camera array 9, uses the 3D Gaussian splatter (3DGS) algorithm to fuse multi-source point cloud data in real time, and generates a textured millimeter-level precision motion environment model through adaptive point cloud density optimization.

[0064] The collision detection module 86 detects self-collisions and collisions with the environment during the motion of the robotic arm 1. It receives the motion environment model of the robotic arm 1 generated by the 3D reconstruction module 85 and constructs a spherical collision model of the robotic arm 1. Based on the end-point pose sequence output by the motion redirection module 84, the collision detection module 86 calculates the spatial relationship between the spherical collision model of the robotic arm 1 and obstacles in real time. When the joint sphere is detected approaching an obstacle, the speed suppression function is triggered, controlling the robotic arm 1 to move at a slower speed in the danger zone, ensuring the safety of the robotic arm 1's motion.

[0065] The inverse kinematics solver module 87 is used to solve the pose data of the robotic arm 1. It receives the end-point pose sequence of the robotic arm 1 output by the motion redirection module 84, combines the spherical collision model constructed by the collision detection module 86 with the end-point pose sequence output by the motion redirection module 84, and outputs a collision-free trajectory for the robotic arm 1. The inverse kinematics solver module 87 uses an inverse kinematics algorithm based on gradient optimization and completes the solution by launching 16 parallel threads through shared memory, ensuring end-point positioning accuracy while improving the solution speed.

[0066] The trajectory optimization module 88 optimizes the collision-free trajectory output by the inverse kinematics solution module 87, achieves trajectory smoothness and time optimization while ensuring collision avoidance, and finally outputs high-precision joint control instructions after trajectory smoothing optimization and time optimization of the robot arm 1.

[0067] The simulation environment generation module 89 establishes a motion simulation system for the robotic arm 1 and verifies the high-precision joint control instructions of the robotic arm 1 output by the trajectory optimization module 88; the simulation environment generation module 89 constructs a virtual-real fusion motion simulation system for the robotic arm 1 based on the motion environment model output by the three-dimensional reconstruction module 85, receives the joint control instructions of the robotic arm 1 output by the trajectory optimization module 88, and if the motion trajectory of the robotic arm 1 meets the requirements, the joint control instructions of the robotic arm 1 are output to the robotic arm host computer 3; if the trajectory does not meet the requirements, it returns to the collision detection module 86 to reprocess the data and perform closed-loop verification.

[0068] In this embodiment, the remote humanoid mapping control method of a virtual reality motion capture robot arm, as shown in FIG5 , FIG6 and FIG7 , includes the following steps:

[0069] 1) As shown in Figures 5(a) and 5(b), a camera array 9 is arranged around the robotic arm 1 and workbench 4, allowing the cameras to capture the work area from multiple angles. Object 5 is placed on workbench 4, and the robotic arm 1 is connected to the controller 2, which is then connected to the host computer 3.

[0070] 2) As shown in Figure 6(a), user 6 wears VR glasses 7, and the motion capture system establishes a low-latency data transmission channel with the terminal processor 8 via a wireless network. It uses an encrypted transmission protocol based on TLS1.3, implements two-way authentication through a self-signed certificate, establishes a secure communication link at the application layer, and achieves end-to-end latency. Where B is the channel bandwidth and SNR is the signal-to-noise ratio, meeting the transmission requirements for real-time motion capture. Once activated, the motion capture system continuously captures the movements of the user's six arms, ensuring high-fidelity transmission of motion data.

[0071] 3) As shown in Figure 6(a) and Figure 7(a), after the user 6 wears the VR glasses 7, the motion data acquisition module 81 establishes a coordinate system H=(x h ,yh ,z h ) and W=(x w ,y w ,z w ), the movement of the head coordinate system H relative to the world coordinate system G and the movement of the wrist coordinate system W relative to the head coordinate system H are identified in real time through the built-in gyroscope and accelerometer of the VR glasses 7.

[0072] 4) As shown in FIG7(a), the rotation matrix of the head coordinate system H relative to the world coordinate system G obtained by the motion data acquisition module 81 is Calculated by gyroscope data, the rotation matrix is updated by real-time angular velocity integration in, is the skew-symmetric matrix of the angular velocity at the current moment; the obtained posture of the wrist coordinate system W relative to the head coordinate system H is obtained by the homogeneous transformation matrix Represents, where the rotation matrix Also calculated from gyroscope data: Translation vector Obtained by integrating the accelerometer data: The motion data acquisition module 81 converts these two matrices and Transmitted to the coordinate system conversion module 82 in real time.

[0073] 5) As shown in FIG6(b) and FIG7(b), the coordinate system conversion module 82 establishes a base coordinate system B=(x b ,y b ,z b ), establish the end coordinate system E=(x e ,y e ,z e ), based on the head-world coordinate system rotation matrix output by the motion data acquisition module 81 and wrist-head coordinate system transformation matrix Combined with the preset coordinate system transformation relationship and Represents the rotation matrix of the world coordinate system relative to the robot base coordinate system, Represents the rotation matrix of the robot end coordinate system relative to the wrist coordinate system, Represents the rotation matrix of the wrist coordinate system relative to the head coordinate system, Represents the rotation matrix of the robot end coordinate system relative to the robot base coordinate system, and establishes a complete kinematic chain transformation model. Get the rotation matrix of the end effector of the robotic arm and convert it into standardized pose data using the quaternion reprojection algorithm The quaternion components are passed through Calculated; R ij Represents the element in row i and column j of the rotation matrix, (q w ,q x ,q y ,q z ) is the quaternion of the manipulator end coordinate system, which represents the rotation of the manipulator end coordinate system relative to the base coordinate system, where q w The cosine component of the rotation, which is related to the rotation angle; q x ,q y ,q z Represents the direction and sine component of the rotation axis.

[0074] This conversion process achieves accurate mapping from the user's natural movements to the standardized posture data of the robotic arm, converting the movements of the user's 6 arm recognized by the VR glasses 7 into standardized posture data that can be recognized by the robotic arm 1.

[0075] 6) As shown in FIG6(b) and FIG7(c), the workspace scale scaling module 83 collects the extreme position data of the forearm extension of the user 6 in real time according to the movement process of the user 6's arm Based on the working radius R in the parameter database of robot arm 1 model robot and joint motion angle θ lim , establish a spatial mapping relationship. Use a nonlinear scaling algorithm to calculate the scaling coefficient of the manipulator end coordinate system relative to the base coordinate system position information: Where k and c are smoothing coefficients, is the current hand position. When , the progressive scaling strategy is triggered, and the adaptive scale coefficient update strategy is triggered. The motion capture system dynamically calculates the scale coefficient λ according to the current position of the human hand, and then obtains the position of the robot arm end coordinate system relative to the base coordinate system: P robot Represents the position information of the robot end coordinate system relative to the base coordinate system, P robot =(x robot ,y robot ,z robot ). The actual motion space boundary of the output robot arm 1 is:

[0076] 7) As shown in FIG7(c), the motion redirection module 84 receives the standardized posture data output by the coordinate system conversion module 82 and the actual motion space output by the workspace scale scaling module 83 Perform pose normalization: X robot =(P robot ,q w ,q x ,q y ,qz ), where P robot =(x robot ,y robot ,z robot ), forming a discrete end pose sequence of the robot arm 1 At this point, the data mapping from the user 6 action space to the robot arm 1 operation space is completed, and data processing of the mapped robot arm end pose sequence begins.

[0077] 8) As shown in Figure 6(c) and Figure 6(d), a multi-view camera array 9 is set up around the robot arm 1 and the workbench 4, and the optical axes of all lenses are aligned with the center point of the working area to ensure that all data of the robot arm end effector workspace, workpiece operation area and safety monitoring area are collected.

[0078] As shown in Figure 7(d), the multi-view video data recorded by the camera array 9 is transmitted to the terminal processor 8 via wireless communication. The 3D reconstruction module 85 receives the multi-view video data output by the camera array 9 and uses the 3D Gaussian splatter (3DGS) algorithm to fuse the multi-source point cloud data in real time. It then generates a textured millimeter-level precision motion environment model through adaptive point cloud density optimization and transmits the model to the collision detection module 86 and the simulation environment generation module 89.

[0079] 9) As shown in FIG7(d), the collision detection module 86 represents each link of the robot arm as a sphere set S according to the size of the link of the robot arm 1. i ={s j |s j =(x j ,y j )}, where x j ∈R 3 is the center position of the sphere, r j is the radius of the sphere, and the overall collision model is At the same time, the collision detection module 86 receives the motion environment model of the manipulator 1 generated by the 3D reconstruction module 85, and analyzes the spatial relationship between the manipulator sphere model and the obstacle in real time according to the end posture sequence output by the motion redirection module 84. The continuous collision detection algorithm is used: when checking the current sphere s t The symbol d=SD(s t )-r, if d≤0, it is determined to be a collision; if not, scan the buffer distance η in the reverse direction of the movement and calculate the scanning point s'=s t +k(s t-1 -s t ), where k = 1-(η+r) / ||s t -s t-1 ||, detect potential collision risks; the collision penalty term uses a piecewise smooth function: When the joint sphere is detected approaching an obstacle, d cWhen ≤η, the speed suppression function is triggered, and the speed weighting item Reduce the movement speed of the robotic arm to ensure safety.

[0080] 10) As shown in Figure 7(e), the inverse kinematics solver module 87 generates a collision-free trajectory using a parallelized gradient optimization algorithm based on the spherical collision model of the manipulator 1 provided by the collision detection module 86 and the end-position sequence output by the motion redirection module 84. This module minimizes the composite objective function that includes the end-position error and the collision penalty. To solve the joint configuration of robot arm 1, where X g is the target pose, K e (θ) is the forward kinematics, C coll is the collision penalty term, C coll (K s (θ))=β2·speed(S t-1,t,t+1 )·smooth(sweep(S t-1,t,t+1 )), where sweep() is the maximum penetration distance output by the continuous collision detection algorithm, and smooth() is the maximum penetration distance of d c Segmented smoothing is performed, with speed() being a speed-weighted term. To improve computational efficiency, the inverse kinematics solution module utilizes the CUDA parallel architecture, enabling 16 threads to work together, including four matrix transformation calculation threads and 12 gradient backpropagation threads. This module processes the accumulated transformation matrices of the kinematic chain through a shared memory mechanism and utilizes warp-level parallel reduction to quickly select the optimal solution, ensuring end-point positioning accuracy while improving solution speed.

[0081] 11) As shown in Figure 7(f), the trajectory optimization module 88 receives the collision-free trajectory generated by the inverse kinematics solution module 87 and performs further optimization. The trajectory optimization module achieves trajectory smoothness and time optimization by constructing a multi-objective optimization function: by minimizing the high-order derivatives of the joint motion Achieve smoothness optimization; by dynamically adjusting the time step Achieve time optimization. Finally, output the trajectory smoothing optimization of robot arm 1 and high-precision joint control instructions after time optimization.

[0082] 12) As shown in FIG7(g), the simulation environment generation module 89 receives the motion environment model provided by the 3D reconstruction module 85, performs coordinate system and spatial registration on the motion environment model and the dynamic model of the manipulator 1, and constructs a virtual-real fusion motion simulation system for the manipulator 1; in the motion simulation system, the high-precision joint control instructions of the manipulator 1 output by the trajectory optimization module 88 are executed. Verify the end trajectory accuracy, detect whether the movement of each joint exceeds the physical limit and whether the robot 1 collides with the environment. If the end position error of the robot 1's motion trajectory ||Δx||>5mm, the joint speed exceeds the limit Or if the robot arm 1 collides with the environment, return to step 10) and run again.

[0083] 13) After the simulation test is passed, the terminal processor 8 sends the optimized control instructions of each joint of the robot arm 1 to the robot arm host computer 3, which then transmits them to the controller 2.

[0084] 14) Finally, the controller 2 drives the robotic arm 1 to complete the predetermined operation task along the predetermined trajectory, thereby achieving high-fidelity reproduction of the user 6's actions.

Claims

1. A remote humanoid mapping control method for a virtual reality motion capture robot arm, characterized by: The following steps are involved: 1) Arrange a camera array around the robotic arm and the workbench; 2) The user wears VR glasses, and the motion capture system captures the user's arm movements; 3) The motion data acquisition module establishes the user's head coordinate system H = (x h ,y h ,z h ) and wrist coordinate system W=(x w ,y w ,z w ); 4) Calculate the rotation matrix of the head coordinate system H relative to the world coordinate system G using the gyroscope data in the VR glasses Update the rotation matrix by integrating the angular velocity in real time in is the skew-symmetric matrix of the angular velocity at the current moment; the position of the wrist coordinate system W relative to the head coordinate system H is obtained by the homogeneous transformation matrix indicates; among them is the rotation matrix, is the translation vector; 5) The coordinate system conversion module establishes the base coordinate system B = (x b ,y b ,z b ), establish the end coordinate system E=(x e ,y e ,z e ),based on and Combine the world coordinate system to rotate the robot base coordinate system And the rotation matrix of the robot end coordinate system to the wrist coordinate system Establish a kinematic chain transformation model; through the matrix Convert the obtained rotation matrix of the end effector of the robotic arm into standardized pose data in, Convert the user's arm movements recognized by VR glasses into standardized posture data recognized by the robotic arm; 6) The workspace scale scaling module collects the user's forearm extension extreme position data based on the user's arm movement Based on the working radius R of the robot arm robot and joint motion angle θ lim , establish a spatial mapping relationship; use a nonlinear scaling algorithm to calculate the scale coefficient: Where k and c are smoothing coefficients, is the current hand position; when detected When , we get: The output motion space boundary of the robot arm is: 7) The motion redirection module receives the standardized posture data output by the coordinate system conversion module and the actual motion space output by the workspace scale scaling module Perform pose normalization: X robot =(P robot ,q w ,q x ,q y ,q z ), where P robot =(x robot ,y robot ,z robot ), forming the end pose sequence of the robotic arm 8) The 3D reconstruction module receives the multi-view video data output by the camera array, generates a textured millimeter-level motion environment model, and transmits it to the collision detection module and the simulation environment generation module; 9) The collision detection module represents the robot arm link as a sphere set S i ={s j |s j =(x j ,y j )}, where x j ∈R 3 is the center position of the sphere, r j is the radius of the sphere, and the overall collision model is At the same time, the collision detection module receives the robot arm motion environment model generated by the 3D reconstruction module; checks the current sphere s t The symbol d=SD(s t )-r, if d≤0, it is determined to be a collision; 10) The inverse kinematics solver uses a parallelized gradient optimization algorithm to generate collision-free trajectories based on the robot's spherical collision model and the end-position sequence output by the motion redirection module; 11) The trajectory optimization module constructs a multi-objective optimization function to smooth and optimize the collision-free trajectory generated by the inverse kinematics solver module; it outputs the joint control instructions after trajectory smoothing and time optimization. 12) The simulation environment generation module aligns the motion environment model with the dynamic model in coordinate system and space to build a robotic arm motion simulation system; it executes the joint control instructions output by the trajectory optimization module Verify the end trajectory accuracy, detect whether the movement of each joint exceeds the physical limit and whether the robotic arm collides with the environment, otherwise return to step 10); 13) The terminal processor transmits the optimized control instructions of each joint of the robotic arm to the controller; 14) The controller drives the robotic arm to reproduce the user's action according to the predetermined trajectory.

2. The method for remote humanoid mapping control of a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 4), the rotation matrix The gyroscope data is converted into quaternion:

3. The method for remote humanoid mapping control of a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 4), the translation vector Obtained by integrating the accelerometer data:

4. The method for remote humanoid mapping control of a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 8), the 3D reconstruction module receives the multi-view video data output by the camera array, fuses the multi-source point cloud data using the 3D Gaussian splash algorithm, and transmits the textured motion environment model generated by adaptive point cloud density optimization to the collision detection module and the simulation environment generation module.

5. The remote humanoid mapping control method for a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 9), if there is no collision, scan the buffer distance η in the reverse direction of the movement and calculate the scanning point s'=s t +k(s t-1 -s t ), where k = 1-(η+r) / ||s t -s t-1 ||, detect collision risk.

6. The method for remote humanoid mapping control of a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 9), the collision penalty term uses a piecewise smooth function: When d c ≤η, through the velocity weighting term Reduce the robot arm movement speed.

7. The method for remote humanoid mapping control of a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 10), the inverse kinematics solution module minimizes the composite objective function To solve the joint configuration of the robot arm, where X g is the target pose, K e (θ) is the forward kinematics, C coll is the collision penalty term, C coll (K s (θ))=β2·speed(S t-1,t,t+1 )·smooth(sweep(S t-1,t,t+1 )), where sweep() is the maximum penetration distance output by the continuous collision detection algorithm, and smooth() is the maximum penetration distance of d c Perform piecewise smoothing, and speed() is the speed weighting item.

8. The method for remote humanoid mapping control of a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 11), by minimizing the high-order derivatives of the joint motion Perform smooth optimization.

9. The remote humanoid mapping control method for a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 11), by adjusting the time step Perform time optimization.

10. The remote humanoid mapping control method for a virtual reality motion capture robot arm according to claim 1, characterized in that: In step 14), if the end position error of the motion trajectory of the robot arm (1) ||Δx||>5mm, the joint speed exceeds the limit Or the robot arm collides with the environment, then return to step 10).

Images