Robotic arm immersive tele-manipulation system based on mixed reality force-teleportation mapping

Abstract

This application provides an immersive telemanipulation system for a robotic arm based on mixed reality force mapping, including master-end control, slave-end execution, communication transmission, spatiotemporal synchronization, heterogeneous kinematic mapping, bidirectional force control, edge trajectory planning, and mixed reality presentation modules. The master end collects joint motion information, while the slave end collects interactive forces, tactile deformation, and environmental images. The spatiotemporal synchronization module achieves synchronization between visual and force feedback through predictive motion compensation. The heterogeneous mapping module utilizes an adaptive scaling matrix and null-space projection to achieve high-precision master-slave mapping. Bidirectional force control estimates environmental stiffness online through a perturbation observer and adaptively switches between force and position hybrid control. Edge planning employs sparse sampling and temporal elastic band optimization to achieve lightweight real-time obstacle avoidance. The mixed reality presentation registers a virtual robotic arm and overlays force vectors and safety boundaries, adaptively switching display modes based on lighting conditions. This invention significantly improves the sense of presence, operational accuracy, and safety of telemanipulation.

Description

Technical Field

[0001] This specification relates to the field of robotics, and in particular to an immersive remote control system for a robotic arm based on mixed reality force mapping. Background Technology

[0002] Robot teleoperation technology refers to the real-time control of a remote robot by an operator using local control devices, enabling the robot to perform tasks in dangerous, hazardous, or inaccessible environments, replacing humans. With the rapid development of mixed reality, force feedback, and collaborative robot technologies, immersive teleoperation systems are showing broad application prospects in fields such as live-line work, bomb disposal, and remote precision assembly.

[0003] Currently, there are various robot teleoperation technology solutions in the industry. One common solution is to use virtual reality or augmented reality technology to achieve visual teleoperation, where the operator observes the remote scene through a head-mounted display, achieving basic visual immersion control. However, this type of solution mainly addresses the visual immersion issue and lacks feedback capability for the operator's force perception channel. The operator cannot perceive physical characteristics such as contact forces and object stiffness in the remote environment.

[0004] Another approach is master-slave control based on force feedback handles or exoskeleton gloves, achieving bidirectional control on the master device through force-position hybrid mapping. However, existing solutions of this kind generally suffer from excessive delays in force rendering and a lack of synchronization between force feedback and visual feedback. This can cause dizziness due to tactile lag, affecting the efficiency and safety of long-term operations.

[0005] Another approach integrates visual and tactile sensors onto the slave robotic arm, sensing contact force through the deformation of the soft adhesive layer. However, this approach is limited by the sensor's field of view, making it difficult to achieve force perception coverage across the entire workspace. Furthermore, while trajectory planning algorithms based on model predictive control can achieve obstacle avoidance, the prediction latency introduced by the algorithm further reduces the sense of force presence, and it also demands significant computing resources from the graphics processor, making it difficult to run in real time on edge computing devices.

[0006] In summary, the main limitations of existing technologies are as follows: there is a spatiotemporal mismatch between force feedback and visual feedback, resulting in a lack of synchronization in the multimodal information presented to the operator; there are significant differences in the kinematic structure between the master-end human arm and the slave-end robotic arm, leading to insufficient accuracy when directly mapping positions, and a tendency for sudden speed changes and jitter when approaching singularities; there is a lack of real-time estimation capabilities for environmental stiffness and adaptive impedance adjustment methods, resulting in insufficient adaptability to objects with drastic stiffness changes in extreme environments such as live-line work and bomb disposal; and the computational complexity and power consumption of high-performance trajectory planning algorithms are too high, making it difficult to meet real-time requirements on edge devices that are not connected to the cloud.

[0007] Therefore, there is an urgent need for an immersive teleoperation processing system that can achieve multimodal feedback spatiotemporal synchronization, high-precision mapping of master-slave heterogeneous kinematics, environmental adaptive bidirectional force control, and real-time lightweight planning at the edge. Summary of the Invention

[0008] In view of this, embodiments of this specification provide an immersive remote control processing system for a robotic arm based on mixed reality force mapping to address the technical deficiencies existing in the prior art.

[0009] According to a first aspect of the embodiments of this specification, an immersive remote control processing system for a robot arm based on mixed reality force mapping is provided, including: a master-end control module, a slave-end execution module, a communication transmission module, a spatiotemporal synchronization module, a heterogeneous kinematics mapping module, a bidirectional force control module, an edge trajectory planning module, and a mixed reality presentation module; The main control module collects joint motion and torque information of the operator's arm and sends it through the communication transmission module; The end-effector module collects interaction force data, tactile deformation data, and environmental image data between the robotic arm's end effector and the environment, and transmits them back via the communication transmission module. The communication transmission module provides bidirectional data transmission between the master control module and the slave execution module; The spatiotemporal synchronization module establishes master-slave clock synchronization, constructs visual feedback pipeline and force feedback pipeline respectively, generates predicted motion trajectory based on joint motion information, and generates force feedback compensation command in advance based on the predicted motion trajectory, so that the visual feedback pipeline and force feedback pipeline achieve synchronization at the perception level. The heterogeneous kinematics mapping module receives joint motion information, generates a Jacobian matrix based on the master-end kinematics model and the slave-end kinematics model, constructs an adaptive scaling matrix to map the master-end task space velocity to the slave-end task space desired velocity, and obtains the slave-end robotic arm joint velocity command through closed-loop inverse kinematics solution and null space projection optimization. The bidirectional force control module generates tactile channel feedback and force channel feedback based on interactive force data and tactile deformation data, runs an adaptive impedance control law, and estimates the environmental stiffness online through a disturbance observer. When the environmental stiffness undergoes a step change, it automatically switches to the force-position hybrid control mode. The edge trajectory planning module generates a trajectory sample set based on obstacle information extracted from environmental image data using a sparse sampling strategy, generates an initial collision-free path using the artificial potential field method, and obtains a real-time obstacle avoidance trajectory through time elastic band optimization. The mixed reality presentation module registers the virtual robotic arm model with the real environment based on the operator's head position and pose, overlays force vector information and safety boundary information, and adaptively switches the display mode according to the ambient light intensity.

[0010] In one possible implementation, the spatiotemporal synchronization module performs the following processing: Run a precise time protocol stack to synchronize the master clock and slave clock to a preset precision through link delay measurement and clock skew compensation; The visual feedback pipeline is configured to sequentially include binocular camera synchronous exposure acquisition, image cropping, hardware accelerated encoding, transmission via communication module, decoding, and virtual-real fusion rendering, and the visual feedback pipeline latency is controlled within a preset visual latency range. The force feedback pipeline is configured to include synchronous sampling of interactive force data and tactile deformation data, online estimation of environmental stiffness, force rendering model solving, and generation and driving of exoskeleton arm joint torque commands in sequence, and the force feedback pipeline delay is controlled within a preset force feedback delay range that is lower than the visual feedback pipeline delay. Using the joint angle sequence of multiple consecutive frames from the master control module as input, a pre-trained long short-term memory network predictor is run to generate the pose prediction value of the operator's hand in the future time period. Based on the pose prediction value, a force feedback compensation instruction is generated for a predetermined duration and placed in the exoskeleton drive buffer to compensate for the delay difference between the visual feedback pipeline and the force feedback pipeline.

[0011] In one possible implementation, the heterogeneous kinematics mapping module performs the following processing: Improved DH parameter kinematic models were established for the master-end exoskeleton arm and the slave-end collaborative robotic arm, respectively. Solve for the master Jacobian matrix of the master exoskeleton arm and the slave Jacobian matrix of the slave collaborative robot arm. The master Jacobian matrix maps the exoskeleton joint angular velocities to the operation task space velocities, and the slave Jacobian matrix maps the robot arm joint angular velocities to the end-effector task space velocities. An adaptive scaling matrix is ​​constructed as a diagonal matrix. The diagonal elements of the adaptive scaling matrix are dynamically adjusted according to the proximity of the current position of the end of the collaborative robotic arm to the boundary of the workspace. The scaling factor in the corresponding direction is attenuated when the end approaches the constraint limit. The desired velocity in the task space of the slave arm is input into the closed-loop inverse kinematics solver. The slave arm's joint velocity command is obtained by optimizing the joint limit offset penalty, singularity avoidance penalty, and operability maximization term through the null space projection matrix using the slave redundant degrees of freedom.

[0012] In one possible implementation, the bidirectional force sensing control module performs the following processing: The tactile channel uses the fingertip tactile sensor of the slave execution module to collect images of gel body contact deformation, uses photometric stereo vision algorithm to reconstruct the three-dimensional force distribution vector field of the contact area, and converts the three-dimensional force distribution vector field into a resultant force vector sequence, which is then mapped to drive the exoskeleton arm finger sleeve actuator of the master control module to generate fingertip pressure feedback. The force sensing channel, after gravity compensation and inertia compensation, maps the data collected by the six-dimensional torque sensor in the wrist to the torque commands of the elbow and wrist joints of the exoskeleton arm in the main control module through coordinate transformation, so as to present the reaction force and torque perceived by the operator.

[0013] In one possible implementation, the bidirectional force sensing control module also performs adaptive impedance control, which adjusts the target dynamic response relationship at the end of the slave-end collaborative robot arm in real time based on the desired inertia parameter, desired damping parameter, and desired stiffness parameter, according to the force deviation and position deviation.

[0014] In one possible implementation, the online estimation of environmental stiffness is performed by a disturbance observer. The disturbance observer takes the wrist torque sensor signal and the small displacement change at the end as input, outputs an estimated value of the external contact force, and combines it with the end displacement to obtain a real-time estimate of the environmental contact stiffness. When the real-time estimate increases by a step beyond a preset threshold, it is determined that the slave-end collaborative robot arm has contacted the rigid obstacle. The damping gain in the impedance control is automatically reduced and the force control weight is increased to suppress contact force overshoot.

[0015] In one possible implementation, the edge trajectory planning module performs the following processing: On the edge computing platform, 3D point clouds are extracted from environmental image data and obstacle occupancy grids are generated. A sparse sampling strategy is adopted, which increases the trajectory sampling density only in the spatial region where the distance to the obstacle surface is less than a specified threshold as the distance decreases, and sparsely samples in the open area far away from the obstacle to generate a trajectory sample set; An initial collision-free path from the current end pose to the target pose is generated using the artificial potential field method, where the gravitational field points to the target point and the repulsive field originates from the obstacle boundary. Using an initial collision-free path as the initial value, an optimization problem is constructed that includes path smoothness, dynamic feasibility, obstacle avoidance distance constraints, and initial path similarity constraints. The real-time obstacle avoidance trajectory is obtained by iteratively solving the problem through a graph optimization solver, and the delay of a single solution meets the preset real-time requirements.

[0016] In one possible implementation, the mixed reality rendering module performs the following processing: The operator's head six-degree-of-freedom pose is acquired by the head-mounted display of the main control module through the in-out tracking sensor. The virtual robotic arm model is rendered in real time based on the head pose, and the registration error between the virtual robotic arm model and the real environment meets the preset accuracy requirements. Force vector visualization elements are overlaid on the virtual robotic arm model, with color coding for the magnitude of the force, 3D arrows indicating the direction of the force, and red semi-transparent bounding boxes indicating the boundaries of dangerous areas that are off-limits.

[0017] In one possible implementation, the mixed reality presentation module also automatically switches the display mode according to the ambient light intensity. When the ambient light exceeds a preset threshold, it switches to video perspective mode, using the camera to block external strong light and merge virtual images. Under other conditions, it switches back to optical perspective mode.

[0018] In one possible implementation, the master control module includes a force feedback exoskeleton arm with multiple joint degrees of freedom, a mixed reality head-mounted display with integrated inside-out tracking sensors, and an ARM-based master controller running a real-time operating system; the slave execution module includes a multi-axis collaborative robotic arm, a binocular stereo vision camera, a six-dimensional torque sensor mounted on a wrist flange, and a fingertip tactile sensor array based on gel optics; the communication transmission module adopts a hybrid link of mobile communication private network and fiber optic, and configures a quality of service policy to ensure that bidirectional data transmission meets preset bandwidth and low latency requirements.

[0019] This specification provides an immersive telemanipulation system for a robotic arm based on mixed reality force mapping, including master-end control, slave-end execution, communication transmission, spatiotemporal synchronization, heterogeneous kinematic mapping, bidirectional force control, edge trajectory planning, and mixed reality presentation modules. The master end collects joint motion information, while the slave end collects interactive forces, tactile deformation, and environmental images. The spatiotemporal synchronization module achieves synchronization between visual and force feedback through predictive motion compensation. The heterogeneous mapping module utilizes an adaptive scaling matrix and null-space projection to achieve high-precision master-slave mapping. Bidirectional force control estimates environmental stiffness online using a perturbation observer and adaptively switches between force and position hybrid control. Edge planning employs sparse sampling and temporal elastic band optimization to achieve lightweight real-time obstacle avoidance. The mixed reality presentation registers a virtual robotic arm and overlays force vectors and safety boundaries, adaptively switching display modes based on lighting conditions. This invention significantly improves the sense of presence, operational accuracy, and safety of telemanipulation. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an immersive remote control system for a robotic arm based on mixed reality force mapping, provided in one embodiment of this specification. Detailed Implementation

[0021] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0022] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0024] This specification provides an immersive telecontrol system for robotic arms based on mixed reality force mapping, which will be described in detail in the following embodiments.

[0025] See Figure 1 , Figure 1This diagram illustrates a system schematic of an immersive remote control system for a robotic arm based on mixed reality force mapping, according to an embodiment of this specification. Specifically, it includes a master control module, a slave execution module, a communication transmission module, a spatiotemporal synchronization module, a heterogeneous kinematics mapping module, a bidirectional force control module, an edge trajectory planning module, and a mixed reality presentation module. The master control module collects joint motion and torque information of the operator's arm and transmits it through the communication transmission module. The slave execution module collects interaction force data, tactile deformation data, and environmental image data between the robotic arm's end effector and the environment, and transmits them back through the communication transmission module. The communication transmission module provides bidirectional data transmission between the master control module and the slave execution module. The spatiotemporal synchronization module establishes master-slave clock synchronization, constructs visual feedback pipelines and force feedback pipelines respectively, generates predicted motion trajectories based on joint motion information, and generates force feedback compensation commands in advance based on the predicted motion trajectories, thereby synchronizing the visual feedback pipeline and the force feedback pipeline at the perception level. The heterogeneous kinematics mapping module receives joint motion information, generates a Jacobian matrix based on the master-end kinematic model and the slave-end kinematic model, constructs an adaptive scaling matrix to map the master-end task space velocity to the slave-end task space desired velocity, and obtains the slave-end robotic arm joint velocity command through closed-loop inverse kinematics solution and null space projection optimization. The bidirectional force control module generates tactile and force channel feedback based on interactive force data and tactile deformation data, runs an adaptive impedance control law, and estimates environmental stiffness online through a disturbance observer, automatically switching the force-potential hybrid control mode when environmental stiffness undergoes a step change. The edge trajectory planning module generates a trajectory sample set based on obstacle information extracted from environmental image data using a sparse sampling strategy, generates an initial collision-free path using the artificial potential field method, and obtains a real-time obstacle avoidance trajectory through time elastic band optimization. The mixed reality presentation module registers the virtual robotic arm model with the real environment based on the operator's head pose, overlays force vector information and safety boundary information, and adaptively switches the display mode according to the ambient light intensity.

[0026] The master control module can refer to a set of devices located on the operator's side, used to collect the operator's movements and apply force feedback. Examples include a multi-jointed exoskeleton worn on the operator's arm, a head-mounted display for showing mixed reality images, and a processing host running real-time control software. It can convert the operator's arm movement intentions into digital signals and send them to a remote location. The slave execution module can refer to a set of robotic devices located in a remote work environment, used to perform operational tasks and collect environmental information. Examples include a multi-axis collaborative robotic arm, a stereo vision camera, wrist force sensors, and fingertip tactile sensors. It can execute actions based on received commands and collect and transmit force and visual information from the work site. The communication transmission module can refer to a network device connecting the master and slave ends, providing a bidirectional data transmission channel. Examples include a hybrid link using mobile communication networks and wired fiber optics, employing a data distribution service protocol, and configuring quality of service policies to ensure reliable transmission of control commands and sensor data. The spatiotemporal synchronization module can refer to a functional unit used to unify the time base of the master and slave ends and coordinate the timing of visual and force feedback. For example, it synchronizes the system clocks at both ends through a precise clock protocol, maintains the visual processing pipeline and the force processing pipeline separately, and generates forward-looking force feedback commands based on a motion prediction mechanism to compensate for the delay differences between the two pipelines. The heterogeneous kinematic mapping module can refer to a solution unit used to convert operator arm movements into robotic arm joint movement commands. For example, it establishes parameterized kinematic models for the master and slave ends respectively, constructs a task space mapping using differential kinematic relationships, and optimizes using redundant degrees of freedom during velocity-level solution. The bidirectional force control module can refer to a control unit used to process slave-end sensor data and convert it into operator-perceptible force feedback. For example, it constructs two feedback channels to process fingertip tactile sensation and wrist force sensation respectively, runs an impedance control law, and adjusts the control mode based on online estimated environmental stiffness parameters. The edge trajectory planning module can refer to an algorithm unit running on a local computing platform used to generate obstacle avoidance paths for the robotic arm. For example, it generates candidate trajectories using sparse sampling based on obstacle distribution and generates smooth and feasible motion paths through optimization methods. The mixed reality rendering module can refer to a rendering unit used to overlay virtual information onto the operator's field of vision. For example, it uses the spatial positioning function of a head-mounted display to align the virtual robotic arm with the real environment and overlays auxiliary information such as force direction, force magnitude, and safety boundaries onto the screen.

[0027] The present invention will be further described below through a detailed embodiment: In one implementation, the operator wears a force feedback exoskeleton arm with seven degrees of freedom and a mixed reality head-mounted display. Each joint of the exoskeleton arm integrates a torque sensor and an angle encoder, enabling real-time acquisition of angle and torque values ​​at the operator's shoulder, elbow, and wrist joints. The head-mounted display uses built-in inside-out tracking sensors to acquire the operator's six-degree-of-freedom head pose in space in real time. This joint motion information of the operator's arm and head pose information are aggregated by a master controller and transmitted to a remote location via a communication transmission module. The communication transmission module employs a hybrid network consisting of a dedicated mobile communication network and a fiber optic link, runs a data distribution service communication protocol, and is configured with a low-latency and high-reliability quality of service strategy to ensure stable bidirectional transmission of command and sensor data.

[0028] In the remote work site, the seven-DOF collaborative robotic arm of the slave execution module receives control commands from the master end and executes corresponding actions. A six-dimensional torque sensor is installed at the wrist flange of the robotic arm's end effector, capable of simultaneously detecting forces in three orthogonal directions and torque components around three axes. The end effector is also equipped with a fingertip tactile sensor array based on gel optics principles. This sensor contains a transparent gel and a miniature camera; when the gel deforms upon contact with an object, the miniature camera captures a deformation image, thus obtaining tactile deformation data of the contact area. Furthermore, a binocular stereo vision camera mounted above the robotic arm synchronously acquires left and right views of the work scene at a set frame rate, serving as environmental image data. The aforementioned interactive force data, tactile deformation data, and environmental image data are transmitted back to the master end via a communication transmission module.

[0029] The spatiotemporal synchronization module runs a precise time protocol stack on both the master and slave controllers. By measuring link transmission delay and calculating clock offset, it aligns the system clocks at both ends to microsecond-level consistency. This module constructs two processing pipelines: a visual feedback pipeline sequentially executes binocular camera synchronous exposure acquisition, image region of interest cropping, hardware-accelerated encoding, transmission to the master via the communication module, decoding, and virtual-real fusion rendering for the head-mounted display; and a force feedback pipeline sequentially executes torque and tactile sensor synchronous sampling, environmental stiffness estimation, force rendering calculation, exoskeleton joint torque command generation, and drive execution. Since the processing latency of the force feedback pipeline is typically lower than that of the visual feedback pipeline, this module runs a motion predictor based on a long short-term memory network. Using a sequence of joint angles from multiple consecutive frames of the operator's exoskeleton arm as input, it predicts the operator's hand pose in a short future timeframe and generates force feedback compensation commands in advance based on the predicted pose, placing them in the exoskeleton's drive buffer. This eliminates the time difference between force perception and vision at the operator's perception level.

[0030] The heterogeneous kinematics mapping module runs on the master controller. This module first establishes improved DH parameter kinematic models for both the master exoskeleton arm and the slave collaborative manipulator, describing the geometric relationships between link lengths, link torsion angles, link offsets, and joint angles. Then, it solves for the Jacobian matrices corresponding to each model. The master-end Jacobian matrix maps the exoskeleton joint angular velocities to linear and angular velocities in the task space, while the slave-end Jacobian matrix maps the manipulator joint angular velocities to linear and angular velocities in the end-effector task space. Next, it constructs a diagonal adaptive scaling matrix, a 6x6 diagonal matrix. The six elements on the main diagonal correspond to the translational scaling factors along the three coordinate axes and the rotational scaling factors about the three coordinate axes in the task space. The current value of each diagonal element is dynamically determined based on the proximity of the slave manipulator's end-effector's current position to the corresponding workspace boundary. Specifically, for each degree of freedom direction, the distance between the current end-effector position and the nearest workspace boundary in that direction is calculated and compared to a preset safety threshold. When the distance value is greater than the safety threshold, the corresponding scaling factor remains at the set maximum scaling value; when the distance value is less than the safety threshold, the scaling factor decreases proportionally according to the ratio of the distance value to the safety threshold; when the distance value approaches zero, i.e., the end effector is close to the boundary, the scaling factor also approaches zero, thereby suppressing the generation of commands to continue moving towards the boundary in that direction. The components of the master-end operation task space velocity vector are multiplied by the corresponding diagonal elements in the adaptive scaling matrix to obtain the desired velocity in the slave-end task space. This desired velocity is fed as the target input into the closed-loop inverse kinematics solver. The closed-loop inverse kinematics solver adopts an iterative solution method based on the pseudo-inverse of the Jacobian matrix. In each iteration, based on the pose deviation between the current end effector pose and the desired pose of the slave-end robot arm, the pseudo-inverse of the slave-end Jacobian matrix is ​​used to map this pose deviation into a correction amount in the joint space, which is then superimposed with the joint velocity command of the previous iteration step to obtain the joint velocity command of the current iteration step. The iteration terminates when the norm of the pose deviation is less than the preset convergence tolerance or the number of iterations reaches the preset maximum number of iterations. In each iteration, the solver also utilizes the redundant degrees of freedom of the slave manipulator to find self-motion components in joint space that do not affect the tracking accuracy in the task space. The null space projection matrix is ​​constructed by subtracting the product of the pseudo-inverse of the Jacobian matrix and the Jacobian matrix from the identity matrix. This matrix can project any joint space vector into the null space of the Jacobian matrix. Through the null space projection matrix, the optimized vector, which is a weighted synthesis of the joint limit offset penalty term, the singularity avoidance penalty term, and the operability maximization term, is projected into the null space and superimposed with the joint velocity command of the main task, driving the joint angle away from the limit and singular configuration.The joint limit offset penalty term is calculated based on the deviation between the current angle of each joint and the midpoint of the corresponding joint's range of motion; the larger the deviation, the larger the penalty term value. The singularity avoidance penalty term is implemented by monitoring the minimum singular value of the Jacobian matrix; when the minimum singular value is lower than a preset threshold, a repulsion effect is activated. The operability maximization term uses the determinant or condition number of the product of the Jacobian matrix and its transpose as the optimization index to guide the robotic arm to maintain a posture with balanced motion capabilities in all directions. The final output is the speed command of each joint of the slave robotic arm, which takes into account both task tracking and configuration optimization.

[0031] The bidirectional force control module receives interactive force data and tactile deformation data from the receiving end and processes them in two channels. In the tactile channel, the gel deformation image captured by the fingertip tactile sensor is analyzed using a photometric stereo vision algorithm. This algorithm is based on pre-calibrated information about multiple light source directions, including at least three known non-parallel directions, which sequentially illuminate the gel surface. For each pixel in the image, the brightness value sequence under different light source directions is obtained. A system of linear equations is established based on the Lambertian reflection model, relating the brightness value to the light source direction and the surface normal vector. This system of equations is solved using the least squares method to obtain the unit normal vector of the gel surface at that pixel location. This normal vector calculation is performed on all pixels within the contact area of ​​the entire image to obtain the surface normal distribution field of the contact area. After obtaining the surface normal distribution field, the normal distribution is transformed into a three-dimensional force distribution vector field based on the elastic properties of the gel material. The gel material was pre-calibrated using mechanical calibration experiments to establish a correlation curve between the normal deflection angle and the normal contact pressure. For each pixel position within the contact area, the corresponding contact pressure amplitude was found from the calibration curve based on the deflection angle of the normal vector at that position relative to the reference normal vector in the non-contact state. This, combined with the normal vector direction, determined the three-dimensional force vector at that point. The three-dimensional force vectors of each pixel within the contact area were then summed in space to obtain an equivalent resultant force vector. The magnitude of this resultant force vector represents the total contact pressure, and its direction represents the direction of the total contact pressure. The mapping based on spatial position correspondence is as follows: a pre-defined one-to-one correspondence exists between the physical distribution of each sensing unit in the fingertip tactile sensor array at the end of the robotic arm and the arrangement of each micro-linear actuator on the finger sleeve of the main exoskeleton arm. The resultant force vectors independently calculated by each sensing unit at the end are sent to the corresponding actuators at the main end. Each actuator converts the received resultant force vector into a pushing force amplitude, determines the pushing force's application method based on the direction of the resultant force vector, and drives the actuator to extend or retract to generate corresponding fingertip pressure feedback. In terms of the force sensing channel, the data collected by the wrist six-dimensional torque sensor, after gravity compensation and inertia compensation processing, is mapped to the driving torque commands of the operator's elbow and wrist exoskeleton joints through coordinate transformation, allowing the operator to feel the reaction force and torque generated when interacting with the manipulated object. Simultaneously, this module operates an adaptive impedance control law, adjusting the dynamic response of the robotic arm's end effector in real time based on preset desired inertia, damping, and stiffness parameters, taking into account contact force deviation and position deviation. Furthermore, the disturbance observer, using the wrist torque sensor signal and minute changes in end effector displacement as input, outputs an estimated value of the external contact force, and combines this estimate with the end effector displacement to obtain a real-time estimate of the environmental contact stiffness.The real-time estimate of environmental contact stiffness is obtained by calculating the ratio of the increment of the estimated external contact force to the corresponding increment of the end displacement. A recursive least squares method with a forgetting factor is used to fit continuous sampling points online to avoid interference from instantaneous measurement noise. When the increase in the real-time estimate exceeds a preset threshold within a preset monitoring time window, it is determined that the robotic arm has contacted a rigid obstacle. The system automatically reduces the damping gain in impedance control and correspondingly increases the force control weight, achieving a smooth transition from impedance control mode to force-position hybrid control mode to suppress contact force overshoot.

[0032] The edge trajectory planning module runs on an edge computing platform deployed at the slave end. This module first extracts 3D point clouds from environmental image data acquired by a binocular stereo vision camera and generates an obstacle distribution map represented in an occupancy grid format. Then, a sparse sampling strategy is used to generate a trajectory sample set: the sampling density increases with decreasing distance only in spatial regions where the distance to the obstacle surface is less than a set threshold; in open areas far from obstacles, the sampling density is significantly reduced. Subsequently, an initial collision-free path from the current end-effector pose to the target pose is quickly generated using an artificial potential field method. A virtual gravitational field is constructed to guide the path towards the target point, and a virtual repulsive field is constructed to push the path away from the obstacle surface. Using this initial path as the initial value for time-elastic band optimization, a multi-objective optimization problem is constructed, including path smoothness constraints, dynamic feasibility constraints, obstacle avoidance distance constraints, and initial path similarity constraints. A graph optimization solver is used for iterative solving to obtain the final smooth obstacle avoidance trajectory. The single-solution delay meets the requirements of the real-time control cycle.

[0033] The mixed reality rendering module runs on the rendering unit associated with the main head-mounted display. This module acquires the operator's six-DOF head pose using sensors tracking outwards within the head-mounted display, and renders a virtual robotic arm model in real-time in 3D space. A spatial registration algorithm aligns this virtual model with the real environment seen by the operator through the display, keeping the registration deviation within a preset accuracy range. On the virtual robotic arm model, the module overlays force vector visualization elements, using color coding to represent the magnitude of the force, 3D arrows to represent the direction of the force, and a red semi-transparent bounding box to mark the boundaries of dangerous areas where the robotic arm is prohibited. Furthermore, the module continuously monitors ambient light intensity. When the light exceeds a preset threshold, causing a decrease in the recognizability of the virtual content, it automatically switches to video perspective mode, using the head-mounted display's front camera to capture the external scene and merge it with the virtual image before presenting it to the operator. When the lighting conditions return to normal, it switches back to optical perspective mode.

[0034] The workflow of the aforementioned system is as follows: The operator wears an exoskeleton arm and a head-mounted display, controlling the remote robotic arm to perform tasks through natural arm movements. The master control module collects the motion angles and torques of each joint of the operator's arm and sends them to the slave end via the communication transmission module. The slave robotic arm executes actions according to the received joint speed commands and senses the contact state with the environment through a six-dimensional torque sensor and fingertip tactile sensor. Combined with the on-site images captured by a binocular camera, this data is transmitted back to the master end. The spatiotemporal synchronization module ensures the synchronous presentation of visual and force information, the heterogeneous kinematics mapping module ensures that the operator's movements are accurately reproduced at the end of the robotic arm, the bidirectional force control module allows the operator to perceive contact forces in real time and adaptively adjust the control mode, the edge trajectory planning module generates obstacle avoidance paths when necessary, and the mixed reality presentation module provides the operator with intuitive visual assistance information.

[0035] The beneficial effects of this embodiment in this specification include at least the following: by integrating master-end control, slave-end execution, communication transmission, spatiotemporal synchronization, heterogeneous kinematic mapping, bidirectional force control, edge trajectory planning, and mixed reality presentation into a single system, a complete immersive teleoperation processing system is constructed. This system utilizes precise clock synchronization and motion prediction mechanisms to control the time difference between visual and force feedback within the human perception threshold, eliminating the dizziness and tactile lag commonly found in traditional teleoperation. Simultaneously, through closed-loop inverse kinematic mapping and zero-space projection optimization, the problem of motion transmission distortion between master and slave heterogeneous structures is solved, enabling the operator's intentions to be precisely executed by the robotic arm.

[0036] Based on the aforementioned system, this embodiment further defines the specific processing method of the spatiotemporal synchronization module. The spatiotemporal synchronization module performs the following processing: running a precise time protocol stack, synchronizing the master clock and slave clock to a preset precision through link delay measurement and clock offset compensation. The visual feedback pipeline is configured to sequentially include binocular camera synchronous exposure acquisition, image cropping, hardware-accelerated encoding, transmission via the communication transmission module, decoding, and virtual-real fusion rendering, controlling the visual feedback pipeline delay within a preset visual delay range. The force feedback pipeline is configured to sequentially include synchronous sampling of interactive force data and tactile deformation data, online estimation of environmental stiffness, force rendering model calculation, and exoskeleton arm joint torque command generation and driving, controlling the force feedback pipeline delay within a preset force feedback delay range lower than the visual feedback pipeline delay. Using the joint angle sequence of multiple consecutive frames from the master control module as input, a pre-trained long short-term memory network predictor is run to generate the pose prediction value of the operator's hand in the future time period. Based on the pose prediction value, a force feedback compensation instruction is generated for a predetermined duration and placed in the exoskeleton drive buffer to compensate for the delay difference between the visual feedback pipeline and the force feedback pipeline.

[0037] A precise time protocol stack can refer to a software protocol stack implemented according to the precision clock synchronization standard protocol of a network measurement and control system. For example, it could be a PTP slave clock and master clock instance running on the master and slave controllers respectively, calculating link transmission delay and clock skew by exchanging timestamped network messages. Link delay measurement can refer to calculating the transmission time of the network link in both directions by sending and receiving synchronization messages and delay request messages containing timestamps of the sending and receiving timestamps. Clock skew compensation can refer to the process of aligning the frequency or phase of the local clock with the peer clock based on the measured clock skew value. Preset accuracy can refer to the allowable range of master-slave clock time skew determined during system design, such as being controlled at the microsecond level. A visual feedback pipeline can refer to the ordered sequence of processing steps from image sensor exposure to image display. A preset visual delay range can refer to the maximum allowable delay value for visual feedback determined based on the operator's visual perception characteristics. A force feedback pipeline can refer to the ordered sequence of processing steps from force / tactile sensor sampling to exoskeleton actuator torque output. The preset force feedback delay range refers to the maximum allowable delay value for force feedback determined based on the operator's tactile perception characteristics, typically required to be lower than the visual feedback delay. A Long Short-Term Memory (LSTM) predictor refers to a recurrent neural network model with memory units and gating mechanisms, which, after offline training, can predict future state values ​​based on input time-series data. The predetermined duration refers to the prediction time advance of the predictor output, corresponding to the delay difference between the visual feedback pipeline and the force feedback pipeline. The exoskeleton drive buffer refers to a temporary data storage area located in the main controller's memory, used to pre-store torque commands to be sent to the exoskeleton joint actuators.

[0038] Under the overall architecture described in the aforementioned system embodiment, the specific working process of the spatiotemporal synchronization module in this embodiment is as follows: After the system is powered on and initialized, the master controller and the slave controller each start the precise time protocol stack process. The master acts as the slave clock node, and the slave acts as the master clock node. The two exchange synchronization messages periodically through the network channel established by the communication transmission module. The master calculates the deviation between its own clock and the slave clock based on the timestamp of the received message, and fine-tunes the local clock frequency through a proportional-integral controller to gradually reduce the deviation to a preset accuracy range, thereby achieving microsecond-level synchronization between the two clocks.

[0039] Based on clock synchronization, this module constructs visual feedback pipelines and force feedback pipelines respectively. The processing flow of the visual feedback pipeline is as follows: the slave binocular stereo vision camera performs synchronous exposure according to the external trigger signal, and the left and right image frames are simultaneously acquired and transmitted to the image preprocessing unit of the slave controller; the image preprocessing unit extracts the region of interest related to the working area in the image and performs cropping to reduce the amount of data for subsequent processing; the cropped image is sent to the hardware video encoder for compression encoding, and in this example, a hardware-accelerated low-latency encoding method is used; the encoded bitstream is transmitted to the master end through the communication transmission module; after being received by the master end, it is decoded and restored into image frames by the hardware decoder; finally, the decoded image frames are sent to the rendering pipeline of the mixed reality rendering module, and after being merged with virtual elements such as the virtual robotic arm model, they are displayed on the head-mounted display. The total latency of this pipeline from camera exposure to image display is controlled within the preset visual latency range through the time budget allocation of each stage.

[0040] The force feedback pipeline's processing flow is as follows: the six-dimensional torque sensor and fingertip tactile sensor of the slave execution module synchronously sample at a unified sampling clock, and the sampled values ​​are timestamped and packaged for transmission; after preprocessing the sampled data, the slave controller runs an online environmental stiffness estimation algorithm to calculate the stiffness parameters of the currently contacting object in real time; after receiving the stiffness parameters and force / tactile sensing data, the master controller runs a force rendering model to calculate the feedback torque that should be applied to each joint based on the current posture of the master exoskeleton arm and the contact state of the slave end; finally, the joint torque command is converted from digital to analog and amplified to drive the joint motors of the exoskeleton arm. The pipeline's delay is also optimized at each stage and controlled within a preset force feedback delay range lower than that of the visual feedback pipeline.

[0041] Because the delay in the force feedback pipeline is typically shorter than that in the visual feedback pipeline, the operator may feel the force before seeing the visual change, resulting in perceptual asynchrony. To address this issue, the module runs a pre-trained Long Short-Term Memory (LSTM) network predictor. The predictor takes a sequence of joint angles from multiple consecutive frames of the operator's exoskeleton arm as input, models the arm's movement trend using internal gating and memory units, and outputs a predicted six-DOF pose of the hand after a predetermined time interval. Based on this pose prediction, the system pre-calculates the corresponding force feedback torque command and pushes it into the exoskeleton's drive buffer. When the clock reaches the corresponding moment of the predetermined time interval, the pre-prepared torque command in the buffer is retrieved and applied to the exoskeleton joints, allowing the operator to perceptually associate the force feedback with the corresponding visual change.

[0042] The beneficial effects of this embodiment include at least the following: achieving microsecond-level synchronization of master and slave clocks through a precise time protocol, providing a unified time reference for subsequent timing coordination; controlling the latency of the visual feedback pipeline and the force feedback pipeline within their respective preset ranges by constructing and optimizing them separately; and effectively compensating for the latency differences between the two pipelines by predicting the operator's motion trajectory and generating force feedback commands in advance through a long short-term memory network predictor, enabling the operator to obtain an immersive teleoperation experience with high synchronization of vision and force.

[0043] Based on the aforementioned system embodiment, this embodiment further defines the specific processing method of the heterogeneous kinematics mapping module. The heterogeneous kinematics mapping module performs the following processing: Improved DH parameter kinematic models are established for both the master exoskeleton arm and the slave collaborative manipulator. The master-end Jacobian matrix of the master exoskeleton arm and the slave-end Jacobian matrix of the slave collaborative manipulator are solved. The master-end Jacobian matrix maps the exoskeleton joint angular velocities to the operational task space velocities, and the slave-end Jacobian matrix maps the manipulator joint angular velocities to the end-effector task space velocities. An adaptive scaling matrix is ​​constructed as a diagonal matrix. The diagonal elements of the adaptive scaling matrix are dynamically adjusted according to the proximity of the current position of the slave collaborative manipulator's end-effector to the workspace boundary, and the scaling coefficient in the corresponding direction is attenuated when the end-effector approaches the constraint limit. The desired velocity in the slave task space is input into the closed-loop inverse kinematics solver. Using the slave redundant degrees of freedom, the joint limit offset penalty term, singularity avoidance penalty term, and operability maximization term are optimized through the null space projection matrix to obtain the slave manipulator joint velocity command.

[0044] The improved DH parameter kinematic model refers to a model that mathematically describes the translational and rotational relationships between the joint coordinate systems of a cascade linkage mechanism using the Denavit-Hartenberg convention. Different link parameters are set for the human joint configuration of the master exoskeleton arm and the offset wrist joint configuration of the slave robotic arm. The master-end Jacobian matrix refers to a matrix function that linearly maps the angular velocities of each joint of the exoskeleton arm to the linear and angular velocities in the task space, and its value varies with the current joint angle configuration. The slave-end Jacobian matrix refers to a matrix function that linearly maps the angular velocities of each joint of the robotic arm to the linear and angular velocities in the task space at the end flange center. The task space velocity refers to the velocity of the operator's hand or the end effector of the robotic arm described in the task coordinate system, including three translational velocity components and three rotational velocity components. The diagonal matrix refers to a square matrix where only the main diagonal elements are non-zero. The adaptive scaling matrix refers to a diagonal matrix whose main diagonal elements automatically adjust according to the system state. The workspace boundary refers to the spatial range that the end effector of a robotic arm cannot exceed due to joint limitations, link length constraints, and structural limitations. Constraint limits refer to physical limitations such as upper or lower limits of joint angles and the reachable boundaries of the end effector. A closed-loop inverse kinematics solver refers to an algorithm module that iteratively solves for joint velocity commands based on the deviation between the current pose and the desired value through closed-loop feedback. The null space projection matrix refers to a matrix constructed from Jacobian matrices that can project any joint space vector into a null space subspace that does not affect the task space velocity. The joint limit offset penalty term refers to a metric representing the deviation between the current joint angle and the joint center position; a larger value indicates that the joint is closer to its limit. The singularity avoidance penalty term refers to a metric measuring how close the current configuration of the robotic arm is to a singular configuration. The operability maximization term refers to an index characterizing the balance of the robotic arm's motion capabilities in all directions.

[0045] Under the overall architecture described in the aforementioned system embodiment, the heterogeneous kinematic mapping module in this embodiment works as follows: During the system initialization phase, improved DH parameter kinematic models are first established for the master exoskeleton arm and the slave collaborative robotic arm, respectively. The model parameters of the exoskeleton arm are calibrated based on the actual dimensions and range of motion of the shoulder, elbow, and wrist joints worn on the human body; the model parameters of the robotic arm are configured based on its link length, joint offset, and joint torsion angle according to factory specifications. Both models can calculate the position and orientation of the corresponding end effector in Cartesian space based on the input joint angle vector.

[0046] During teleoperation, the angle encoders of each joint of the master exoskeleton arm output joint angle values ​​in real time. This module performs differential kinematics solutions on the master-end kinematic model and the slave-end kinematic model respectively, obtaining the master-end Jacobian matrix and the slave-end Jacobian matrix at the current moment. The master-end Jacobian matrix maps the joint angular velocity vectors of the exoskeleton arm to linear velocity vectors and angular velocity vectors in the operation task space; the slave-end Jacobian matrix maps the joint angular velocity vectors of the robotic arm to linear velocity vectors and angular velocity vectors in the end-effector task space. Both Jacobian matrices are time-varying matrices dependent on the current joint configuration.

[0047] Subsequently, the module constructs a diagonal adaptive scaling matrix. Each diagonal element of this matrix corresponds to a degree of freedom in the task space. The current value of each diagonal element is determined by the proximity of the current position of the slave end effector to the workspace boundary in the corresponding direction. Specifically, when the end effector is far from the workspace boundary in a certain direction, the scaling factor in that direction remains at a set maximum value; as the end effector gradually approaches the constraint limit, the scaling factor in the corresponding direction smoothly decreases according to a preset decay function; when the end effector is close to the boundary, the scaling factor approaches zero, thereby suppressing the generation of further motion commands towards the boundary direction. This mechanism ensures that the slave's desired speed generated by the master end motion mapping is always within the feasible range of the robotic arm.

[0048] Multiplying each component of the velocity vector in the master-end task space by the corresponding diagonal element in the adaptive scaling matrix yields the desired velocity in the slave-end task space. This desired velocity serves as the target input and is fed into the closed-loop inverse kinematics solver. The solver iteratively solves for joint velocity commands based on the deviation between the current end-effector pose and the target pose of the slave manipulator, combined with the pseudo-inverse of the slave-end Jacobian matrix. In each iteration, the solver also utilizes the redundant degrees of freedom of the slave manipulator to search for self-motion components in the joint space that do not affect the tracking accuracy in the task space through the null projection matrix. These self-motion components are driven by an optimization direction that is a weighted synthesis of joint limit offset penalty, singularity avoidance penalty, and maneuverability maximization term: the joint limit offset penalty term encourages the joint angle to move closer to the center position away from the limit; the singularity avoidance penalty term generates a repulsive force when the manipulator approaches a singular configuration, preventing a sharp increase in velocity commands; and the maneuverability maximization term guides the manipulator to maintain a balanced posture in all directions of motion. The three optimization terms are superimposed onto the main task solution result through the null projection matrix, and finally output the slave arm joint speed command that takes into account both task tracking and configuration optimization, which is sent to the robotic arm joint servo driver for execution.

[0049] The beneficial effects of this embodiment include at least the following: by establishing kinematic models for the master and slave ends respectively and solving their respective Jacobian matrices, accurate motion transfer between heterogeneous structures is achieved; by dynamically adjusting the mapping ratio according to the workspace boundary using an adaptive scaling matrix, the problem of the mapped target exceeding the reach of the robotic arm is avoided; by combining closed-loop inverse kinematics solution with null space projection optimization, redundant degrees of freedom are driven to avoid joint constraints and singular configurations while ensuring the tracking accuracy in the task space, significantly improving the stability and accuracy of master-slave mapping.

[0050] Based on the aforementioned system embodiment, this embodiment further defines the specific processing methods of the tactile and force channels in the bidirectional force control module. The bidirectional force control module performs the following processing: The tactile channel acquires gel contact deformation images from the fingertip tactile sensor of the slave execution module, reconstructs the three-dimensional force distribution vector field of the contact area using a photometric stereo vision algorithm, and transforms the three-dimensional force distribution vector field into a resultant force vector sequence, which is then mapped to drive the exoskeleton arm finger actuator of the master control module to generate fingertip pressure feedback. The force channel, after gravity compensation and inertia compensation, maps the data acquired by the wrist six-dimensional torque sensor to the elbow and wrist joint torque commands of the exoskeleton arm of the master control module through coordinate transformation, so as to present the reaction force and torque perceived by the operator.

[0051] A gel contact deformation image refers to an image captured by a built-in miniature camera when the surface of a transparent gel inside a tactile sensor deforms under pressure from an external object. The image records the distortion and displacement information of the gel surface texture. A photometric stereo vision algorithm refers to a computational method that reconstructs the three-dimensional shape of an object's surface based on changes in brightness under different light sources. For example, multiple light sources from known directions can be used to sequentially illuminate the gel surface, and the surface normal vector can be calculated by analyzing the brightness values ​​of each pixel under different lighting conditions. A three-dimensional force distribution vector field refers to a set of vectors containing the magnitude and direction of forces at various points on the contact surface, describing the spatial distribution of contact forces. A resultant force vector sequence refers to a vector sequence representing the magnitude and direction of the total contact force obtained by combining the force vectors at all points in the entire contact area in both time and space. An exoskeleton arm finger actuator refers to a miniature linear drive element installed on the finger part of an exoskeleton arm, capable of generating a pushing force perpendicular to the fingertip direction according to commands, allowing the operator to feel the contact at the fingertip. Gravity compensation refers to the process of removing the torque component caused by the weight of the end tool and the sensor itself from the torque sensor readings. Inertia compensation refers to the process of removing the inertial torque component caused by end-effector acceleration from torque sensor readings. Coordinate transformation refers to the rotation and translation operations that convert the torque vector from the sensor coordinate system to the exoskeleton joint coordinate system or the operator's limb coordinate system.

[0052] Under the overall architecture described in the aforementioned system embodiment, the specific working process of the bidirectional force control module in this embodiment is as follows: Regarding the tactile channel, when the fingertip tactile sensor at the end of the robotic arm comes into contact with the object surface, the transparent gel inside the sensor is compressed, causing surface deformation. A built-in miniature camera continuously captures images of the gel surface under sequential illumination from multiple light sources, obtaining a sequence of deformation images under different lighting conditions. These deformation images are transmitted back to the main controller via a communication transmission module. The main controller runs a photometric stereo vision algorithm, calculating the normal vector of the gel surface at each pixel location based on the brightness change under different light source directions. Then, based on the elastic properties of the gel material, the surface normal vector distribution is transformed into a three-dimensional force distribution vector field in the contact area. Subsequently, the algorithm calculates the resultant force of the force vectors at each point within the entire contact area in space, generating a power-equivalent resultant force vector sequence. According to the temporal and spatial correspondence, the resultant force vector sequence is mapped to the driving force command of each micro linear actuator on the finger sleeve of the main exoskeleton arm. The actuator pushes the contact surface on the inside of the finger sleeve to act on the operator's fingertip with corresponding pressure and frequency, so that the operator feels the same tactile sensation as contact with a distant object.

[0053] In terms of force sensing, a six-dimensional torque sensor mounted on the wrist of the slave-end robotic arm outputs three orthogonal force components and three orthogonal torque components in real time when the robotic arm comes into contact with an external object. This raw data is first preprocessed by the slave-end controller, corrected using pre-calibrated sensor zero-drift parameters and temperature compensation coefficients. Subsequently, the slave-end controller calculates and removes the contributions of gravity and inertia to the sensor readings based on the current mass parameters, center-of-mass position parameters, and current acceleration value of the end effector, obtaining the external force and torque components purely caused by external contact. This processed data is then transmitted back to the master end via a communication module. After receiving the external force / torque data, the master controller transforms the external force / torque into the coordinate system corresponding to the elbow and wrist joints of the exoskeleton arm through rotation and translation operations, based on the preset spatial transformation relationship between the end-tool coordinate system and the master exoskeleton arm joint coordinate system. Then, combining the principle of virtual work, the Cartesian space force / torque is converted into the driving torque of the corresponding joint, generating torque commands for the elbow and wrist joints of the exoskeleton arm and applying them to the corresponding joint motors, so that the operator's arm feels the reaction force corresponding to the remote contact state.

[0054] The beneficial effects of this embodiment include at least the following: Firstly, by using photometric stereoscopic vision through the tactile channel to reconstruct a fine three-dimensional force distribution from the gel deformation image, high-fidelity perception and feedback of the contact surface texture and local force information are achieved. Secondly, by using the force channel to map the six-dimensional torque sensor data into exoskeleton joint torques after gravity and inertia compensation, accurate transmission of the magnitude and direction of macroscopic contact forces is achieved. The two channels work together to allow the operator to simultaneously obtain macroscopic force and microscopic tactile sensations, significantly enhancing the immersive force perception during teleoperation.

[0055] Building upon the aforementioned system, this embodiment further specifies that the bidirectional force sensing control module also performs adaptive impedance control. The adaptive impedance control adjusts the target dynamic response relationship at the end of the slave-end cooperative robotic arm in real time based on the desired inertia parameter, desired damping parameter, and desired stiffness parameter, taking into account force and position deviations.

[0056] The desired inertia parameter refers to the virtual mass characteristic set in the impedance control model, characterizing the response of the robotic arm's end effector to external forces. This parameter affects the acceleration response of the end effector under contact forces. The desired damping parameter refers to the virtual damping coefficient set in the impedance control model, characterizing the velocity-related drag characteristics of the robotic arm's end effector. This parameter affects the degree to which the end effector resists velocity changes during motion. The desired stiffness parameter refers to the virtual stiffness coefficient set in the impedance control model, characterizing the position-related restoring force characteristics of the robotic arm's end effector. This parameter affects the magnitude of the restoring force generated when the end effector deviates from the desired position. Force deviation refers to the difference between the actual measured external contact force and the desired contact force. Position deviation refers to the difference between the current actual position of the robotic arm's end effector and the desired target position. The target dynamic response relationship refers to the expected mass-spring-damped dynamic behavior characteristics of the robotic arm's end effector when subjected to external forces, i.e., the dynamic constraint relationship between force and motion.

[0057] In the operation of the system described above, the adaptive impedance control in this embodiment further refines the control method of the force sensing channel. When the end effector of the robotic arm moves from free space into the constrained space that contacts the work object, the bidirectional force sensing control module operates the adaptive impedance control law.

[0058] The control law establishes a virtual mass-spring-damped system model within the controller. The dynamic behavior of this model is defined by three preset parameters: the desired inertia parameter determines the virtual system's inertial response to external forces; the desired damping parameter determines the virtual system's velocity decay characteristics; and the desired stiffness parameter determines the virtual system's elastic restoring force characteristics. The control law drives this virtual model through two input quantities: force deviation and position deviation. Specifically, the force deviation is obtained by subtracting the preset desired contact force from the external contact force measured and compensated by the six-dimensional torque sensor at the wrist; the position deviation is obtained by subtracting the actual position of the robotic arm's end effector from the desired trajectory position generated through motion planning. Under the combined action of the two deviation signals, the virtual mass-spring-damped system calculates the desired acceleration that the robotic arm's end effector should exhibit in the current contact state. This desired acceleration is converted into end-effector velocity and position corrections through kinematic relationships, and superimposed on the original desired trajectory to form an impedance-adjusted end-effector target pose sequence. Subsequently, this target pose sequence is converted into joint space commands through inverse kinematics solving, driving the robotic arm to execute.

[0059] By adjusting the desired inertia, damping, and stiffness parameters, the system enables the robotic arm's end effector to exhibit different compliance characteristics when contacting environments with varying stiffness. For example, a lower desired stiffness parameter results in higher compliance with external forces, making it suitable for handling fragile items; a higher desired stiffness parameter provides stronger resistance to external forces, making it suitable for tasks requiring precise positional maintenance. The impedance parameter can be pre-configured according to the task type or adaptively adjusted during operation based on environmental stiffness estimation results.

[0060] The beneficial effects of this embodiment include at least the following: by introducing adaptive impedance control based on a virtual mass-spring-damping model, the robotic arm can exhibit adjustable compliance characteristics in contact operations, and adjust the end effector dynamic response in real time according to force deviation and position deviation, thereby achieving a smooth transition between free space motion and constrained space operations and avoiding contact force impact caused by direct position control.

[0061] Based on the system described above, this embodiment further defines the specific processing method for online environmental stiffness estimation and control mode switching. Online environmental stiffness estimation is performed by a disturbance observer. The disturbance observer takes the wrist torque sensor signal and the small displacement change at the end as inputs, outputs an estimated value of the external contact force, and combines it with the end displacement to obtain a real-time estimated value of the environmental contact stiffness. When the real-time estimated value increases by a step beyond a preset threshold, it is determined that the slave-end cooperative robotic arm has contacted a rigid obstacle. The damping gain in the impedance control is automatically reduced and the force control weight is increased to suppress contact force overshoot.

[0062] A disturbance observer can refer to a state observer capable of real-time estimation of unknown external disturbances based on a known dynamic model of the system and measurable input / output signals. Wrist torque sensor signals refer to the external force and torque components output from the six-dimensional torque sensor at the wrist of the end-effector robotic arm, after gravity and inertia compensation. End-effector micro-displacement change refers to the minute positional offset of the robotic arm end-effector due to force after contact, which can be obtained through forward kinematics calculations using a joint angle encoder. Estimated external contact force refers to the external contact force vector acting on the end-effector of the robotic arm, calculated by the disturbance observer based on the system dynamic model and measurement signals. Real-time estimated environmental contact stiffness refers to the environmental stiffness parameter calculated online using the ratio between the change in external contact force and the corresponding change in end-effector displacement. A step increase refers to a significant upward jump in the estimated environmental stiffness over a short time interval. A preset threshold refers to a stiffness change discrimination threshold pre-set based on experience. Damping gain refers to the adjustment parameter acting on the velocity-related terms in impedance control. Force control weight refers to the proportional factor that emphasizes one mode over the other in a control system, between position control and force control. Contact force overshoot refers to the amount by which the actual peak contact force exceeds the expected steady-state contact force after contact occurs.

[0063] This embodiment adds online environmental stiffness estimation and adaptive mode switching functions based on a disturbance observer. A disturbance observer is deployed internally within the bidirectional force sensing control module. The disturbance observer is designed based on a simplified dynamic model of the end effector of the collaborative robotic arm in Cartesian space with a single degree of freedom, treating the contact force applied to the end effector by the external environment as an unknown disturbance input to the system. The disturbance observer uses the contact force measurement signal provided by the wrist torque sensor and the end effector displacement signal calculated by the joint encoder via forward kinematics as inputs. The observer includes a low-pass filter to suppress high-frequency noise components in the sensor signal. Through inversion of the known dynamic model of the system and low-pass filtering of the measurement signal, the disturbance observer outputs a real-time estimate of the external contact force. This estimate has a lower noise level and stronger robustness to model uncertainties compared to directly using torque sensor readings.

[0064] Based on the estimated external contact force, this module calculates a real-time estimate of the environmental contact stiffness by incorporating minute changes in end-effector displacement. The calculation method involves taking the ratio of the increment of the estimated external contact force to the corresponding increment of the end-effector displacement, and then fitting continuous data points during the contact process using a recursive least squares method or an online parameter estimation method with a forgetting factor to obtain the equivalent linear stiffness estimate parameters of the current contact environment. This estimate is dynamically updated as the contact process progresses.

[0065] When the robotic arm moves from free space and comes into contact with an object, the real-time estimate of the environmental contact stiffness will experience an increase from near zero to a certain positive value. If the object being contacted is a normal operational target, the stiffness estimate will usually change smoothly within a certain range. However, if the robotic arm accidentally collides with a rigid obstacle, the stiffness estimate will increase dramatically in a very short time, with both the rate of increase and the absolute value far exceeding those under normal operating conditions. This module continuously monitors the changes in the stiffness estimate, and when it detects that the increase in the estimate within a preset time window exceeds a preset threshold, it determines that the robotic arm has come into contact with a rigid obstacle.

[0066] After determining rigid contact, the module automatically triggers a control mode switch. During the switch, the damping gain in impedance control is gradually reduced to minimize the interference of velocity damping force on contact force adjustment; simultaneously, the force control weight is gradually increased, transforming the control system from an impedance control mode primarily based on position tracking and secondarily on force feedback to a force-position hybrid control mode primarily based on force tracking. In the force-position hybrid control mode, the controller prioritizes responding to force deviation signals, actively unloading excessive contact force by applying a backward velocity command in the contact normal direction, thus stabilizing the actual contact force near the desired value. Through this gain smoothing transition strategy, contact force overshoot is effectively suppressed.

[0067] When the stiffness estimate is detected to return to normal levels, the system automatically reverts to impedance control mode, restoring normal compliant operation.

[0068] The beneficial effects of this embodiment include at least the following: achieving low-noise and robust estimation of external contact forces through a disturbance observer; estimating environmental contact stiffness online through the ratio of force increment to displacement increment, providing the system with real-time perception of environmental mechanical properties; and automatically identifying unexpected rigid contact and quickly switching control modes through stiffness mutation detection, effectively suppressing contact force overshoot and ensuring the safety of equipment and operated objects.

[0069] Based on the aforementioned system embodiments, this embodiment further defines the specific processing method of the edge trajectory planning module. The edge trajectory planning module performs the following processing: On the edge computing platform, a 3D point cloud is extracted from the environmental image data and an obstacle occupancy grid is generated. A sparse sampling strategy is adopted, increasing the trajectory sampling density only in spatial regions where the distance to the obstacle surface is less than a specified threshold as the distance decreases, and sparsely sampling in open areas far from the obstacle to generate a trajectory sample set. An initial collision-free path from the current end pose to the target pose is generated using the artificial potential field method, where the gravitational field points to the target point and the repulsive field originates from the obstacle boundary. Using the initial collision-free path as the initial value, an optimization problem is constructed that includes path smoothness, dynamic feasibility, obstacle avoidance distance constraints, and initial path similarity constraints. The real-time obstacle avoidance trajectory is obtained by iteratively solving the problem using a graph optimization solver, and the single solution delay meets the preset real-time requirements.

[0070] Edge computing platforms can refer to embedded processing devices with a certain computing power deployed on-site, such as compact computing modules integrating multi-core CPUs and GPUs, capable of performing perception data processing and trajectory planning calculations locally without relying on remote cloud servers. 3D point clouds can refer to a set of discrete points in 3D space obtained by back-projecting the disparity map from left and right images acquired by a binocular stereo vision camera using a stereo matching algorithm, where each point contains spatial coordinates in the camera coordinate system. Obstacle occupancy grids can refer to a voxelized map representation that divides 3D space into regular cubic units and marks each unit as occupied by an obstacle based on point cloud data. Sparse sampling strategies can refer to methods of non-uniformly distributing trajectory sample points in configuration space or task space. Specified thresholds can refer to distance thresholds used to distinguish between areas near obstacles and open areas. Trajectory sample sets can refer to a set of several alternative paths generated in a certain way between a given start and end pose. Artificial potential field method refers to a classic robot path planning method. It establishes a virtual gravitational potential field at the target point and a virtual repulsive potential field at obstacles, guiding the robot to avoid obstacles and move towards the target under the guidance of the resultant potential field force. A gravitational field can be defined as a spatial scalar field whose potential function value monotonically increases with increasing distance from the target point. A repulsive field can be defined as a spatial scalar field whose potential function value increases sharply with decreasing distance from the obstacle surface. An initial collision-free path refers to a geometric path generated by the artificial potential field method that runs from the start point to the end point without crossing obstacles and occupying any grid. Path smoothness refers to the geometric characteristic measure of a path with continuous curvature and no sharp turns. Dynamic feasibility refers to the characteristic measure that the velocity and acceleration of each point on the path do not exceed the physical limitations of the robot arm joints. Obstacle avoidance distance constraint refers to the constraint that the distance between each point on the path and the nearest obstacle surface is not less than a preset safety margin. Initial path similarity constraint refers to the constraint that the optimized path maintains similarity to the initial path in overall orientation. A graph optimization solver can refer to a numerical optimization algorithm framework that represents an optimization problem as a graph structure and solves it through iterative linearization. Preset real-time requirements refer to the upper limit imposed by the system control cycle on the computation time of a single solution by the trajectory planning module.

[0071] Under the overall architecture described in the aforementioned system embodiment, the specific working process of the edge trajectory planning module in this embodiment is as follows: This module runs on an edge computing platform deployed at the slave work site. When the robotic arm needs to perform point-to-point movement in an environment with obstacles, the module first obtains the latest environmental image data collected by the binocular stereo vision camera through the communication interface. A stereo matching algorithm is executed on the left and right views to calculate the disparity map, and the disparity map is back-projected into three-dimensional space according to the calibrated camera baseline parameters and focal length parameters to generate a three-dimensional point cloud represented in the camera coordinate system. Subsequently, the three-dimensional point cloud is transformed to the robotic arm base coordinate system, and the space is divided into equally sized cubic grid units in this coordinate system. For each grid unit, the number of point clouds falling into it is counted. When the number exceeds a preset occupancy threshold, the grid is marked as occupied by an obstacle, ultimately generating a three-dimensional obstacle-occupied grid map.

[0072] After obtaining the occupied grid map, this module employs a sparse sampling strategy to generate a trajectory sample set in the task space between the current pose and the target pose of the robotic arm's end effector. Specifically, for each sampling candidate position in the task space, the Euclidean distance between that position and the grid surface occupied by the nearest obstacle is calculated. When this distance is greater than a specified threshold, it indicates that the position is in an open area far from obstacles, and a small number of trajectory samples are generated at that location with a lower density. When the distance is less than the specified threshold, it indicates that the position is in a region near obstacles, and the sampling density increases inversely as the distance decreases, generating denser trajectory sample points at that location. This distance-based adaptive sampling mechanism concentrates computational resources on local key areas near obstacles, significantly saving computational overhead in open areas.

[0073] After generating the trajectory sample set, this module uses an artificial potential field method to quickly generate an initial collision-free path. In the task space, the target pose point is defined as a gravitational potential field source with a global minimum, whose potential function value monotonically increases with the distance from the target point; the points on the grid surface occupied by obstacles are defined as repulsive potential field sources, whose potential function value reaches its maximum on the obstacle surface and rapidly decays to zero with increasing distance. The end effector of the robotic arm is regarded as a point mass moving in the resultant potential field. Starting from the initial pose, it moves step by step along the negative gradient direction of the resultant potential field and finally reaches the target pose. A local minimum escape strategy is introduced in the potential field. When the end effector gets stuck in a local minimum point, it jumps out through random perturbation and continues the search. This process is repeated until the end effector reaches the neighborhood of the target pose, resulting in an initial collision-free path that does not pass through the grid occupied by obstacles.

[0074] Using the initial collision-free path as the initial value for optimization, this module constructs a comprehensive optimization problem containing multiple objective and constraint terms. The path smoothness objective uses the angle change between adjacent segments on the path as the cost term, requiring the path to be as straight and continuous as possible. The dynamic feasibility objective uses the estimated velocity and acceleration amplitude of each point on the path as the cost term, requiring the path to meet the motion capability limitations of the robotic arm joints. The obstacle avoidance distance constraint uses the distance between each point on the path and the nearest obstacle as the constraint condition, requiring the minimum distance to be no less than a preset safety margin. The initial path similarity constraint uses the geometric deviation between the optimized path and the initial path as the cost term, preventing the optimization result from deviating excessively from the original path. This optimization problem is encoded as a sparse graph structure with the coordinates of each vertex on the path as the optimization variables, and is solved iteratively using a graph optimization solver. In each iteration, the solver linearizes the objective function and constraint function at the current estimated point, constructs a local linear least squares subproblem, solves the incremental correction, and updates the path vertex coordinates. Iteration continues until the decrease in the objective function or the incremental correction is less than a preset convergence criterion. The final path obtained from the solution is the real-time obstacle avoidance trajectory, which is sent to the robotic arm joint controller for execution. Through the combination of sparse sampling and efficient optimization algorithms, the latency of a single solution is stably controlled within the preset real-time requirements.

[0075] The beneficial effects of this embodiment include at least the following: by completing trajectory planning locally on the edge computing platform, it eliminates the dependence on the cloud and is suitable for field and mobile operation scenarios where the network is unreliable; by using a sparse sampling strategy guided by obstacle distance, it significantly reduces the complexity of sampling and optimization calculations while ensuring obstacle avoidance accuracy; by using the artificial potential field method to quickly generate an initial collision-free path and then optimizing and refining it with a time elastic band, it balances planning speed and path quality; the overall computational latency meets the requirements of real-time control, and the power consumption has a significant advantage compared with traditional dense sampling methods.

[0076] Based on the aforementioned system embodiments, this embodiment further defines the specific processing method of the mixed reality presentation module. The mixed reality presentation module performs the following processing: It acquires the operator's six-degree-of-freedom head pose using the outward tracking sensors within the head-mounted display of the main control module; renders a virtual robotic arm model in real time based on the head pose; and ensures that the registration error between the virtual robotic arm model and the real environment meets preset accuracy requirements. Force vector visualization elements are overlaid on the virtual robotic arm model, using color coding to encode the magnitude of the force, three-dimensional arrows to indicate the direction of the force, and a red semi-transparent bounding box to mark the boundary of the prohibited danger zone.

[0077] Inside-out tracking sensors refer to a combination of sensors mounted on the head-mounted display that estimates its own pose by observing features of the external environment. Examples include a visual-inertial simultaneous localization and mapping (VIM) module containing multiple fisheye grayscale cameras and inertial measurement units. Head six-degree-of-freedom pose refers to the three-dimensional spatial coordinates of the operator's head in a preset reference coordinate system and its attitude angles around the three axes. Real-time rendering refers to the process of generating virtual content images on a graphics processor at a refresh rate of tens of frames per second or higher based on the current head pose data. A virtual robotic arm model refers to a three-dimensional digital model pre-constructed in a computer using polygonal meshes and material properties, which is consistent with the appearance and joint configuration of an actual robotic arm in the distance. Registration error refers to the spatial deviation between the image position of the virtual robotic arm model observed by the operator through the head-mounted display and its corresponding reference position in the actual environment. Preset accuracy requirements refer to the allowable upper limit set for registration deviation during system design. Force vector visualization elements refer to multi-layered overlay content that graphically presents force information in a virtual scene. Color-coded force magnitude refers to mapping the force amplitude to different hues or brightness in a color space; for example, a gradient from blue to red represents a range of force values ​​from small to large. A 3D arrow refers to a 3D spatial arrow model with a starting point, direction, and length; the arrow direction indicates the direction of the force. A red semi-transparent bounding box refers to a cuboid or complex curved surface bounding box model rendered with a red material and set to partial transparency. A prohibited hazardous area boundary refers to the spatial boundaries that are pre-calibrated or sensed in real-time within the end-user working environment, and which the robotic arm's end effector is not allowed to enter.

[0078] Under the overall architecture described in the aforementioned system embodiments, the specific working process of the mixed reality presentation module in this embodiment is as follows: The operator wears a mixed reality head-mounted display, and the inside-out tracking sensor built into the display continuously collects image features and inertial data of the external environment. Multiple cameras in the sensors synchronously capture grayscale images of the environment around the operator at a preset frame rate, and the inertial measurement unit synchronously outputs three-axis angular velocity and acceleration. The visual-inertial synchronous localization and mapping algorithm extracts and performs cross-frame matching of image features, and tightly couples and fuses them with the inertial data to estimate the six-degree-of-freedom pose of the operator's head in the environmental reference coordinate system in real time, including the three-dimensional spatial position and three attitude angles.

[0079] The rendering engine calculates the view matrix and perspective projection matrix to be drawn on the left and right displays of the virtual content based on the updated six-DOF head pose of each frame. The engine loads a pre-built virtual robotic arm model into the virtual 3D space. The joint angles of this model are updated in real-time by the actual joint angle data transmitted back from the slave robotic arm via a communication module, ensuring complete synchronization between the virtual robotic arm's movements and the real robotic arm at a distance. The engine renders the virtual robotic arm model to a position corresponding to the real environment seen by the operator through the display. To maintain spatial consistency between the virtual image and the real world, the system performs a spatial registration process during initialization. This involves aligning preset reference points in the virtual coordinate system with corresponding physical markers in the real environment within the operator's field of vision to determine the spatial transformation relationship between the virtual and real worlds. During operation, when cumulative drift occurs in the sensor positioning, visual feature loop closure detection corrects the pose estimation, ensuring that the registration error between the virtual robotic arm model and the real environment remains within the preset accuracy requirements.

[0080] Building upon the basic virtual-real registration and rendering, this module overlays force vector visualization elements. The current contact force vector data output by the bidirectional force control module is fed into the rendering pipeline. The rendering engine selects the corresponding color from a preset color palette based on the force magnitude, simultaneously generating a 3D arrow model with its origin at the contact point at the end of the robotic arm, its direction along the actual force direction, and its length proportional to the force magnitude. This arrow model updates with each frame of rendering, allowing the operator to intuitively see the magnitude and direction of the current contact force. Furthermore, based on the danger zone boundaries identified from the end-environment perception data, this module overlays a red semi-transparent bounding box model onto the virtual scene, marking the prohibited areas for the robotic arm. The operator sees a virtual robotic arm image precisely aligned with the real environment on the head-mounted display, overlaid with a force vector arrow, and danger zones marked by a red transparent bounding box.

[0081] The beneficial effects of this embodiment include at least the following: by acquiring the high refresh rate six-DOF pose of the head through an inside-out tracking sensor and driving the virtual robotic arm to render in real time, accurate registration of virtual images with the real environment is achieved; by encoding the color and length of the force vector arrows and indicating the three-dimensional direction, the contact force information existing in digital form is transformed into visual elements that the operator can intuitively perceive; and by marking dangerous areas with red semi-transparent bounding boxes, clear spatial safety constraints are provided to the operator, effectively reducing the risk of misoperation.

[0082] Based on the system described above, this embodiment further defines the adaptive switching function of the display mode of the mixed reality presentation module. The mixed reality presentation module also automatically switches the display mode according to the ambient light intensity. When the ambient light exceeds a preset threshold, it switches to video perspective mode, using the camera to block external strong light and merge virtual images. Under other conditions, it switches back to optical perspective mode.

[0083] Ambient light intensity refers to the overall brightness level of visible light in the operator's environment, which can be measured in real time by the ambient light sensor on the head-mounted display. Preset threshold refers to the critical light intensity value at which the system makes switching decisions between different display modes. Video see-through mode refers to a working mode where the head-mounted display blocks external ambient light from directly entering the operator's eyes, instead using a front-facing camera to capture the external scene and displaying the camera image and virtual content merged on an internal display screen. Optical see-through mode refers to a working mode where the head-mounted display allows external ambient light to directly penetrate a semi-transparent optical lens into the operator's eyes, with virtual content reflected and superimposed on the natural field of vision. Camera blocking of strong external light refers to the feature in video see-through mode where the head-mounted display's enclosed casing physically blocks external light from directly entering, and the brightness of the image captured by the camera can be adjusted electronically.

[0084] This embodiment adds adaptive switching logic for display modes. During operation, the mixed reality presentation module continuously measures the ambient light intensity around the operator using an ambient light sensor integrated into the head-mounted display. When the measured light intensity is at a normal or low level, i.e., below a preset threshold, the module maintains optical perspective mode. In this mode, external natural light passes through the display's optical lenses into the operator's pupils. The virtual robotic arm model and related auxiliary information are projected onto the optical lenses via a micro-projection system, reflected by the lenses, and then enter the operator's eyes, superimposed on the natural field of vision to form a mixed reality image. The advantage of optical perspective mode is that the real world seen by the operator has completely natural resolution and extremely low visual latency.

[0085] When an operator moves to a brightly lit environment, such as an outdoor scene with direct sunlight, the illuminance measured by the ambient light sensor exceeds a preset threshold. In this situation, if the optical perspective mode is used, the strong external light will severely reduce the contrast of the virtual content displayed on the micro-projection, making it difficult for the operator to identify key information such as force vector arrows and danger zone bounding boxes. The mixed reality rendering module automatically triggers a display mode switch, switching from optical perspective mode to video perspective mode. In video perspective mode, the physical light-shielding structure of the head-mounted display prevents direct external light from entering; the front-facing camera captures the external scene, and the camera's exposure parameters and gain are automatically adjusted to adapt to the bright light environment, obtaining a digital image of the realistic scene with appropriate brightness; the virtual robotic arm model and overlaid information are rendered and fused onto this digital image, and the overall image is displayed on the internal screen of the head-mounted display. This method ensures the visibility of virtual content in bright light environments.

[0086] When the operator returns to a normal lighting environment and the ambient light sensor readings fall below a preset threshold, the module automatically switches the display mode back to optical perspective mode, restoring the natural visual experience.

[0087] The beneficial effects of this embodiment include at least the following: by monitoring ambient light intensity and adaptively switching display modes, the problem of unstable mixed reality display quality under different lighting conditions is solved; in strong light environments, the video perspective mode is used to ensure clear visibility of virtual information, and in normal lighting conditions, the optical perspective mode is used to maintain a high-quality visual experience of the real world, so that the system can provide operators with a good mixed reality visual experience under various working lighting conditions.

[0088] Building upon the aforementioned system embodiments, this embodiment further defines the specific hardware configuration and communication parameter settings of the master control module, slave execution module, and communication transmission module. The master control module includes a force feedback exoskeleton arm with multiple joint degrees of freedom, a mixed reality head-mounted display integrating an inside-out tracking sensor, and an ARM-based master controller running a real-time operating system. The slave execution module includes a multi-axis collaborative robotic arm, a binocular stereo vision camera, a six-dimensional torque sensor mounted on a wrist flange, and a fingertip tactile sensor array based on gel optics. The communication transmission module employs a hybrid link of a dedicated mobile communication network and fiber optics, configured with a quality of service (QoS) policy to ensure that bidirectional data transmission meets preset bandwidth and low latency requirements.

[0089] Force feedback exoskeleton arms can refer to an external skeletal electromechanical device worn on an operator's arm, with its joint axes aligned with the human arm's joint axes. Each joint is equipped with a torque motor, angle encoder, and torque sensor, capable of measuring the operator's joint angles and applying external forces / torques to the operator. Multiple joint degrees of freedom refer to the number of independent motion axes possessed by the exoskeleton arm, corresponding to the main motion degrees of freedom of the operator's shoulder, elbow, and wrist, enabling comprehensive capture and feedback of arm spatial movements. Mixed reality head-mounted displays integrating inside-out tracking sensors refer to head-mounted displays that incorporate a sensor suite capable of estimating their own pose by observing the external environment, typically including multiple wide-angle cameras and inertial measurement units. ARM-based master controllers running real-time operating systems refer to computing devices employing ARM architecture processor chips and running embedded operating systems with deterministic task scheduling capabilities, used to perform tasks with stringent response time requirements, such as exoskeleton arm servo control, communication protocol stacks, and force rendering. Multi-axis collaborative robotic arms refer to robotic arm devices with a multi-joint serial structure designed to safely collaborate with operators in a shared space, offering flexible end-effector reach and high motion accuracy. Wrist flanges refer to the mechanical mounting interface at the output end of the robotic arm's end joint, typically a planar disc structure with standard screw holes for securing end-effector sensors and tools. Six-dimensional torque sensors refer to sensors capable of simultaneously measuring forces and torque components applied to them in three directions within a Cartesian coordinate system. Gel optics-based fingertip tactile sensor arrays refer to multiple tactile sensing units arranged at the fingertips of the robotic arm's end-effector grippers. Each unit contains a transparent elastic gel and a miniature camera pointing towards the gel, using image changes caused by gel contact deformation to sense contact information. Private mobile communication networks refer to independent mobile communication networks deployed for specific application scenarios that do not rely on the public internet. Fiber optic hybrid links refer to transmission links that simultaneously include wireless mobile communication segments and wired fiber optic segments. Quality of Service (QoS) policies refer to management mechanisms that allocate different service levels within a network based on the priority and latency requirements of different data streams. Preset bandwidth and low latency requirements refer to the minimum available data transmission rate and maximum permissible network latency indicators set during system design to ensure the real-time performance and accuracy of remote operation.

[0090] Under the overall architecture described in the aforementioned system embodiments, the hardware configuration of each module in this embodiment is as follows: The main control module is directly worn and used by the operator. The force feedback exoskeleton arm has multiple joint degrees of freedom corresponding to the shoulder, elbow, and wrist joints of the human arm. Each joint is equipped with a high-resolution angle encoder and torque sensor, which can sense the angular position and applied torque of each joint of the operator in real time. Each joint of the exoskeleton arm is also equipped with a servo motor and transmission mechanism, which can apply the torque command generated by the controller in reverse to the operator's arm to realize force perception. The operator also wears a mixed reality head-mounted display, which integrates an inside-out tracking sensor and acquires the operator's head precise six-degree-of-freedom spatial pose in real time through a multi-camera array and an inertial measurement unit. The main controller adopts an ARM architecture embedded processor platform and runs a real-time optimized embedded operating system. It is responsible for core processing tasks such as exoskeleton arm servo loop control, communication data transmission and reception, force rendering calculation, and mixed reality rendering drive, ensuring that the entire processing link from sensing acquisition to drive output has a deterministic short-latency response.

[0091] The end-effector module is deployed at the remote work site. The robotic arm adopts a multi-axis collaborative design, featuring a serial joint configuration with the same degrees of freedom as the master exoskeleton arm. Its load capacity and repeatability meet the requirements of precision operations. A binocular stereo vision camera is fixedly mounted above or to the side of the robotic arm's working area, synchronously outputting left and right views at a fixed baseline distance and set frame rate, providing a data source for environmental perception and obstacle detection. A six-dimensional torque sensor is bolted to the mounting flange of the wrist flange, connected in series between the wrist joint and the end effector, directly sensing all external forces and torque components acting on the end effector. A fingertip tactile sensor array is mounted on the inner contact surface of the end gripper. Each sensing unit contains a transparent gel with a high-friction coefficient marker texture on its bottom and outer surface. A miniature camera captures the marker pattern from the back of the gel. When the gel comes into contact with an object and deforms under pressure, the markers shift accordingly, and the change in the marker position in the camera image carries information about the contact force distribution.

[0092] The communication transmission module connects the master and slave ends. Mobile communication private network base stations and access equipment are deployed at both the master and slave ends to establish independent wireless communication coverage areas. Simultaneously, the slave end connects the base station to the core network equipment via fiber optic lines, forming a hybrid transmission link of mobile communication private network and fiber optics. The communication protocol adopts a data distribution service protocol, which supports publish-subscribe communication, allowing nodes at the master and slave ends to exchange data on a topic-by-topic basis. By configuring a quality of service policy, control commands and real-time sensor data are marked as high-priority data streams, ensuring they are sent in a low-latency, high-reliability transmission mode; non-real-time logs and monitoring data are marked as low-priority data streams and transmitted in a best-effort mode. The above link and protocol configuration ensures that the total effective bandwidth for bidirectional data transmission is not less than the preset bandwidth value, and the end-to-end average network latency is controlled within the preset low-latency requirement.

[0093] The beneficial effects of this embodiment include at least the following: the master control module, through a combination of a multi-degree-of-freedom force feedback exoskeleton arm, a mixed reality head-mounted display, and a real-time controller, achieves high-precision capture and multi-modal information presentation of the operator's arm movements; the slave execution module, through a combination of a collaborative robotic arm, a binocular stereo camera, a six-dimensional torque sensor, and a tactile sensor array, achieves multi-dimensional perception of the remote environment and precise operation capabilities; and the communication transmission module, through a hybrid link of mobile communication private network and fiber optics and configured quality of service policies, ensures high-bandwidth and low-latency reliable transmission of teleoperation data. The hardware selection and software configuration of each module are mutually matched, providing a hardware foundation guarantee for the stable and efficient operation of the entire immersive teleoperation processing system.

[0094] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0095] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0096] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A robot arm immersive teleoperation processing system based on mixed reality force mapping, characterized in that, include: The system includes a master control module, a slave execution module, a communication transmission module, a spatiotemporal synchronization module, a heterogeneous kinematics mapping module, a two-way force control module, an edge trajectory planning module, and a mixed reality presentation module. The main control module collects the joint movement and torque information of the operator's arm and sends it through the communication transmission module; The slave execution module collects the interaction force data, tactile deformation data and environmental image data between the robotic arm end and the environment, and transmits them back through the communication transmission module; The communication transmission module provides bidirectional data transmission between the master control module and the slave execution module; The spatiotemporal synchronization module establishes master-slave clock synchronization, constructs a visual feedback pipeline and a force feedback pipeline, generates a predicted motion trajectory based on the joint motion information, and generates a force feedback compensation command in advance based on the predicted motion trajectory, so that the visual feedback pipeline and the force feedback pipeline achieve synchronization at the perception level. The heterogeneous kinematics mapping module receives the joint motion information, generates a Jacobian matrix based on the master-end kinematics model and the slave-end kinematics model, constructs an adaptive scaling matrix to map the master-end task space velocity to the slave-end task space desired velocity, and obtains the slave-end robotic arm joint velocity command through closed-loop inverse kinematics solution and null space projection optimization. The bidirectional force control module generates tactile channel feedback and force channel feedback based on the interactive force data and the tactile deformation data, runs an adaptive impedance control law, and estimates the environmental stiffness online through a disturbance observer. When the environmental stiffness undergoes a step change, it automatically switches to the force-position hybrid control mode. The edge trajectory planning module generates a trajectory sample set based on obstacle information extracted from the environmental image data using a sparse sampling strategy, generates an initial collision-free path using the artificial potential field method, and obtains a real-time obstacle avoidance trajectory through time elastic band optimization. The mixed reality presentation module registers the virtual robotic arm model with the real environment based on the operator's head posture, overlays and displays force vector information and safety boundary information, and adaptively switches the display mode according to the ambient light intensity.

2. The system of claim 1, wherein, The spatiotemporal synchronization module performs the following processing: Run a precise time protocol stack to synchronize the master clock and slave clock to a preset precision through link delay measurement and clock skew compensation; The visual feedback pipeline is configured to sequentially include binocular camera synchronous exposure acquisition, image cropping, hardware accelerated encoding, transmission via the communication transmission module, decoding, and virtual-real fusion rendering, and the visual feedback pipeline delay is controlled within a preset visual delay range. The force feedback pipeline is configured to sequentially include synchronous sampling of the interactive force data and the tactile deformation data, online estimation of environmental stiffness, force rendering model calculation, and generation and driving of exoskeleton arm joint torque commands, and the force feedback pipeline delay is controlled within a preset force feedback delay range that is lower than the visual feedback pipeline delay. Using the joint angle sequence of multiple consecutive frames from the main control module as input, a pre-trained long short-term memory network predictor is run to generate a pose prediction value of the operator's hand in the future time period. Based on the pose prediction value, a force feedback compensation command is generated for a predetermined duration and placed in the exoskeleton drive buffer to compensate for the delay difference between the visual feedback pipeline and the force feedback pipeline.

3. The system of claim 2, wherein, The heterogeneous kinematics mapping module performs the following processing: Improved DH parameter kinematic models were established for the master-end exoskeleton arm and the slave-end collaborative robotic arm, respectively. Solve for the master Jacobian matrix of the master exoskeleton arm and the slave Jacobian matrix of the slave collaborative robot arm. The master Jacobian matrix maps the exoskeleton joint angular velocity to the operation task space velocity, and the slave Jacobian matrix maps the robot arm joint angular velocity to the end-effector task space velocity. Construct the adaptive scaling matrix as a diagonal matrix. The diagonal elements of the adaptive scaling matrix are dynamically adjusted according to the proximity of the current position of the end of the slave-end collaborative robot arm to the boundary of the workspace. When the end of the slave-end collaborative robot arm approaches the constraint limit, the scaling factor in the corresponding direction is attenuated. The desired velocity in the slave task space is input into the closed-loop inverse kinematics solver. The slave redundant degrees of freedom are used to optimize the joint limit offset penalty term, singularity avoidance penalty term, and operability maximization term through the null space projection matrix to obtain the joint velocity command of the slave robotic arm.

4. The system of claim 3, wherein, The bidirectional force sensing control module performs the following processing: The tactile channel acquires gel contact deformation images by the fingertip tactile sensor of the slave execution module, reconstructs the three-dimensional force distribution vector field of the contact area using a photometric stereo vision algorithm, and converts the three-dimensional force distribution vector field into a resultant force vector sequence, which is then mapped to drive the exoskeleton arm finger sleeve actuator of the master control module to generate fingertip pressure feedback. The force sensing channel, after gravity compensation and inertia compensation, maps the data collected by the six-dimensional torque sensor at the wrist into torque commands for the elbow and wrist joints of the exoskeleton arm in the main control module through coordinate transformation, so as to present the reaction force and torque perceived by the operator.

5. The system of claim 4, wherein, The bidirectional force sensing control module also performs adaptive impedance control, which adjusts the target dynamic response relationship at the end of the slave-end cooperative robotic arm in real time based on the desired inertia parameter, desired damping parameter and desired stiffness parameter, and force deviation and position deviation.

6. The system of claim 5, wherein, The online estimation of environmental stiffness is performed by a disturbance observer. The disturbance observer takes the wrist torque sensor signal and the small displacement change at the end as input, outputs the estimated value of the external contact force, and combines the end displacement to obtain the real-time estimated value of the environmental contact stiffness. When the real-time estimated value increases by a step beyond a preset threshold, it is determined that the slave-end collaborative robotic arm has contacted the rigid obstacle. The damping gain in the impedance control is automatically reduced and the force control weight is increased to suppress contact force overshoot.

7. The system according to claim 1, characterized in that, The edge trajectory planning module performs the following processing: On the edge computing platform, a 3D point cloud is extracted from the environmental image data and an obstacle occupancy grid is generated. A sparse sampling strategy is adopted to increase the trajectory sampling density only in spatial regions where the distance to the obstacle surface is less than a specified threshold, and to sparsely sample in open areas far away from the obstacle, so as to generate the trajectory sample set. The initial collision-free path from the current end pose to the target pose is generated using the artificial potential field method, wherein the gravitational field points to the target point and the repulsive field originates from the obstacle boundary; Using the initial collision-free path as the initial value, an optimization problem is constructed that includes path smoothness, dynamic feasibility, obstacle avoidance distance constraints, and initial path similarity constraints. The real-time obstacle avoidance trajectory is obtained by iteratively solving the problem using a graph optimization solver, and the delay of a single solution meets the preset real-time requirements.

8. The system according to claim 1, characterized in that, The mixed reality rendering module performs the following processing: The operator's head six-degree-of-freedom pose is acquired by the head-mounted display of the main control module through the in-out tracking sensor. The virtual robotic arm model is rendered in real time based on the head pose, and the registration error between the virtual robotic arm model and the real environment meets the preset accuracy requirements. Force vector visualization elements are overlaid on the virtual robotic arm model, with color coding for the magnitude of the force, three-dimensional arrows indicating the direction of the force, and red semi-transparent bounding boxes indicating the boundaries of dangerous areas that are off-limits.

9. The system according to claim 8, characterized in that, The mixed reality presentation module also automatically switches the display mode according to the ambient light intensity. When the ambient light exceeds a preset threshold, it switches to video perspective mode, using the camera to block external strong light and merge virtual images. Under other conditions, it switches back to optical perspective mode.

10. The system according to claim 1, characterized in that, The master control module includes a force feedback exoskeleton arm with multiple joint degrees of freedom, a mixed reality head-mounted display integrating an inside-out tracking sensor, and an ARM-based master controller running a real-time operating system; the slave execution module includes a multi-axis collaborative robotic arm, a binocular stereo vision camera, a six-dimensional torque sensor mounted on a wrist flange, and a fingertip tactile sensor array based on gel optics; the communication transmission module adopts a hybrid link of mobile communication private network and optical fiber, and configures a quality of service policy to ensure that bidirectional data transmission meets preset bandwidth and low latency requirements.

Images