A teleoperation system and method based on dense point cloud and dynamic multi-constraint CBF robotic arm

Abstract

The application discloses a teleoperation system and method based on a dense point cloud and a dynamic multi-constraint CBF mechanical arm, wherein a master end interaction module comprises a display to realize visual display of an environment three-dimensional distance field map and a mechanical arm pose, and a teleoperation unit issues a teleoperation nominal control instruction; an execution module of a slave end execution module serves as a work execution main body; a global perception module and a local perception module are used for multi-view environment perception and three-dimensional mapping, and a master control calculation unit is used for realizing multi-source point cloud fusion, map construction and active safety obstacle avoidance; the scheme adopts a global solid-state radar and local depth camera fusion architecture to eliminate a blind area at a perception end; a remote operation part and a display are utilized at an interaction end to construct a hybrid incremental mapping-based anti-jump control closed loop; a dynamic multi-constraint CBF and a quadratic programming solver are introduced at a control bottom layer, so that absolute safety and refined obstacle avoidance with high fault tolerance are realized.

Description

Technical Field

[0001] This invention belongs to the field of robot teleoperation and intelligent control, specifically relating to a teleoperation system and method for a robotic arm based on dense point cloud and dynamic multi-constraint CBF. Background Technology

[0002] With the development of robotics technology, six-degree-of-freedom robotic arm teleoperation has been widely used in high-risk or complex unstructured environments such as special operations, live-line repair, disaster relief and rescue, and precision manufacturing. Traditional teleoperation systems primarily rely on operators observing through two-dimensional video feeds and issuing commands using a keyboard or joystick. However, existing systems face the following technical bottlenecks in practical engineering applications:

[0003] 1. Perception dimensionality reduction and blind spots: Traditional teleoperation relies heavily on a single RGB camera installed at the end of a robotic arm or in a fixed position in the environment. A single viewpoint not only fails to provide accurate depth and spatial position information, but is also prone to blind spots due to the robotic arm itself or environmental blind spots, leading to misjudgments and collisions by the operator in complex spaces.

[0004] 2. Lack of underlying safety mechanisms and rigid traditional obstacle avoidance: Existing teleoperations are mostly open-loop control with low fault tolerance. Some systems that introduce obstacle control functions often rely on sparse point clouds or fixed geometry for calculation, which can easily lead to computing power collapse when faced with dense point clouds of hundreds of thousands. Furthermore, traditional CBF uses static boundary parameters, which can easily cause the robotic arm to be directly "locked up" when the operation command is directed at an obstacle, resulting in severe "stickiness" and failing to meet the operation requirements such as edge sliding and precise grasping.

[0005] 3. Sudden command changes and loss of control: When the operator releases the joystick, switches control modes, or there is a network delay, the underlying control often causes command jumps due to the deviation between the target position and the actual physical position, resulting in violent twitching of the robotic arm. In addition, the operator has difficulty intuitively perceiving the physical limits of the robotic arm's joints and kinematic singularities, which can easily lead to hardware damage.

[0006] Therefore, it is necessary to propose a higher-performance, safer, and more intuitive teleoperation system and method. Summary of the Invention

[0007] To address the shortcomings of existing technologies, such as large blind spots, rigid obstacle avoidance algorithms with high computational costs, and susceptibility to command jumps, this paper proposes a teleoperation system and method based on dense point clouds and dynamic multi-constraint CBF robotic arms. The system employs a fusion architecture of global solid-state radar and local depth cameras at the perception end to eliminate blind spots. At the interaction end, a hybrid incremental mapping-based anti-mutation control closed loop is constructed using the teleoperation component and a display. At the control layer, dynamic multi-constraint CBF and a quadratic programming solver are introduced to achieve high fault tolerance, absolute safety, and refined obstacle avoidance.

[0008] The specific technical solution for achieving the objective of this invention is as follows:

[0009] A teleoperation system for a robotic arm based on dense point cloud and dynamic multi-constraint CBF includes a master-end interaction module and a slave-end execution module.

[0010] The main interaction module includes a display and a remote control unit. The display is used to visualize the three-dimensional distance field map of the environment and the pose of the robotic arm, and the remote control unit is used to issue remote operation nominal control commands.

[0011] The slave execution module includes a master control computing unit, an execution unit, a global perception unit, a local perception unit, and a drive unit; the execution module is a six-degree-of-freedom robotic arm, which serves as the main body for task execution; the global perception module and the local perception module are set on the six-degree-of-freedom robotic arm and are used for multi-view environmental perception and three-dimensional mapping.

[0012] The main control computing unit is used to realize multi-source point cloud fusion, map building, and active safety obstacle avoidance;

[0013] The drive unit is used to receive control commands and provide power to the motors of each joint of the robotic arm, enabling it to complete the task according to a given safe path.

[0014] The master-end interaction module and the slave-end execution module communicate with each other via a communication unit.

[0015] Furthermore, the main interaction module first uses the main control computing unit to obtain a real-time 3D map and render the robot arm pose, and then streams it to the display.

[0016] Subsequently, according to the control commands from the remote control unit, the main control computing unit collects point cloud data obtained by the global perception unit and the local perception unit, and obtains the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map in real time. The solver is used to determine whether the nominal command violates the CBF safety constraint. If it does not violate the constraint, the nominal command is sent directly. If it violates the constraint, the dangerous normal component that causes the collision is eliminated. Finally, the optimal safety command is sent to the drive unit and the execution unit. The above process is repeated until the operation ends.

[0017] Furthermore, the main control computing unit includes a multi-source sensing fusion and preprocessing subunit, an ESDF range field mapping subunit, an instantaneous resynchronization integral control subunit, and a safety filtering subunit based on dynamic multi-constraint CBF-QP.

[0018] The main control computing unit performs perception fusion and preprocessing on the point cloud data acquired by the global perception unit and the local perception unit based on the multi-source perception fusion and preprocessing subunit.

[0019] Then, using the ESDF distance field mapping submodule, distance field mapping is performed based on the preprocessed point cloud data to obtain the shortest safe distance and anti-collision gradient information for any point in space;

[0020] The instantaneous resynchronization integral control subunit is used to process the jump commands in the control commands of the remote control unit.

[0021] The safety filtering subunit based on dynamic multi-constraint CBF-QP is used to enforce safety constraints on the control commands of the remote control unit.

[0022] Furthermore, the process by which the multi-source sensing fusion and preprocessing subunit performs sensing fusion and preprocessing on point cloud data is as follows:

[0023] The global perception unit and the local perception unit collect data and input it into their respective point cloud preprocessing modules; at the same time, the robotic arm joint encoder collects real-time joint angles and calculates the pose of each link through the forward kinematics calculation unit and the URDF robotic arm model unit.

[0024] After being calibrated by the extrinsic parameter calibration unit, the point cloud data is sent to the TF coordinate transformation module to be unified into the global absolute coordinate system.

[0025] Let the three-dimensional point in the depth camera coordinate system be denoted as The three-dimensional points in the lidar coordinate system are The three-dimensional points of the robotic arm base in the absolute coordinate system are ,but:

[0026]

[0027] in for The rotation matrix, for The translation vector, and Depend on Overview, For the sensor coordinate system to the base coordinate system Homogeneous transformation matrix;

[0028] Subsequently, the fused point cloud enters the body self-filtering module, which uses the geometric envelope provided by the URDF model to remove point clouds belonging to the robotic arm itself, and finally outputs a clean external environment point cloud.

[0029] Furthermore, the process by which the ESDF distance field mapping submodule obtains the shortest safe distance and anti-collision gradient information for any point in space is as follows:

[0030] Input the preprocessed environmental perception point cloud data and perform spatial rasterization;

[0031] Secondly, the truncated symbolic range field (TSDF) is constructed and updated, and voxels are updated using a weighted multi-frame fusion algorithm. Distance value at and weight :

[0032]

[0033] in, This represents the currently updated voxel space coordinates. Indicates the updated number Frame TSDF distance value, Prehistoric The cumulative TSDF distance value of the frames. Prehistoric The weight value accumulated across frames. This represents the depth distance value observed by the sensing unit in the latest frame. This represents the weight of the latest frame of sensor observation data, determined based on the sensor noise model or distance setting.

[0034] Based on this, the Euclidean symbolic distance field (ESDF) is generated to solve for arbitrary voxels in space. Nearest obstacle set shortest distance :

[0035]

[0036] in, Voxel representation The Euclidean shortest distance to the nearest obstacle surface This represents a set of point clouds or voxels that are identified as obstacles in three-dimensional space. Represents the set of obstacles Any obstacle point in the map, express Norm, i.e., Euclidean geometric distance;

[0037] Based on the above field map, the shortest safe distance to any point in space is output in real time to the safety filtering subunit of the dynamic multi-constraint CBF-QP. With collision avoidance gradient information And the distance field is incrementally maintained and updated along with the robot's movement.

[0038] Furthermore, when the instantaneous resynchronization integral control subunit receives a jump command, i.e., when the safety enable button of the remote control unit is pressed:

[0039] The initial trigger moment of the safety enable lock when the operator presses the joystick. Physical joint angle at time As the starting point for integration, in the Cartesian coordinate mapping mode, The joystick thrust of the telescopic unit at a given moment is mapped to the end-space velocity spinor. The differential inverse kinematics is solved using the damped least squares method, and the joint velocity increment is output. :

[0040]

[0041] in, This represents the nominal joint velocity increment vector output by the solution. This represents the terminal Cartesian nominal velocity spinor issued by the operator via the flight joystick. Indicates the current joint position of the robotic arm. The Jacobian matrix below is abbreviated as: , Describes the pseudo-inverse of the Jacobian matrix. Describes the transpose of the Jacobian matrix. This represents the damping coefficient used to limit the abrupt divergence of joint velocities in a robotic arm near singular points. Represents the identity matrix;

[0042] The virtual integrator generates a smooth nominal control command through closed-loop accumulation calculation:

[0043]

[0044] in, express The absolutely smooth nominal joint position command generated at every moment. express The actual physical joint positions of the robotic arm are constantly read from the hardware encoder. This represents the time integral variable.

[0045] Furthermore, the safety filtering subunit based on dynamic multi-constraint CBF-QP receives nominal control commands. And import the constraint pool;

[0046] The quadratic programming QP solver uses minimizing the degree of instruction intervention as its objective function.

[0047]

[0048] in, This represents the actual safe joint velocity command output by the QP solver after safety constraint filtering. This indicates the nominal joint speed command passed from the front end;

[0049] Four major boundary constraints are applied in parallel during the solution process:

[0050] (1) Dynamic ESDF collision avoidance: Define the collision safety margin function

[0051]

[0052] Introducing Lie derivative and dynamic speed adjustment coefficient Construct CBF constraints:

[0053]

[0054]

[0055] in, This represents the obstacle avoidance control function for the environment. Indicates the current joint angle The position of the closest point to the obstacle on the surface of the lower robotic arm's anti-collision capsule. This represents the system's preset absolute safety distance threshold. This indicates that the ESDF map is at the point. The gradient vector of the three-dimensional distance field obtained by looking up the table. Indicates the closest point from the robotic arm base to the capsule body. The local Jacobian matrix, Indicates the current speed of the robotic arm's end effector. The negatively correlated dynamic adjustment function, where Based on the gain, As a speed attenuation factor, it ensures that constraints tighten at high speeds and flexibility is maintained at low speeds;

[0056] (2) Joint physical limits: To avoid exceeding mechanical limits, damping constraints are applied:

[0057]

[0058] in, Indicates the current real-time joint position. and These represent the lower and upper physical limits of each joint of the robotic arm, respectively. This represents the constant approach velocity decay coefficient;

[0059] (3) Singularity avoidance: based on operability index

[0060]

[0061] Preventing approximation of exotic configurations:

[0062]

[0063] in, The Yoshikawa index represents the kinematic maneuverability of a robotic arm; a smaller value indicates that it is closer to a singularity. This represents the operation of finding the determinant of a matrix. This indicates the operability index for joint angles. The gradient of the partial derivative, This represents the minimum operability safety threshold allowed by the system. This represents the adjustment coefficient for singularity avoidance control;

[0064] (4) Self-collision avoidance: The distance between each link is calculated based on the simplified capsule model. By applying a similar self-collision prevention CBF, when faced with an approaching obstacle, the solver, under the premise of strictly satisfying all the above inequalities, uses the gradient direction to orthogonally filter out the normal velocity and forcibly retain the tangential velocity, giving the robotic arm a forward safe sliding capability without any stickiness.

[0065] The capsule envelope parameters of each link are automatically extracted based on the URDF model, and their spatial pose is updated in real time through forward kinematics to calculate the shortest distance between non-adjacent link pairs. Construct a self-collision control barrier function And apply CBF constraints:

[0066]

[0067] in, The minimum distance between non-adjacent links calculated for the capsule model. To preset a safety threshold, This is the adjustment coefficient; this constraint is input into the QP solver in parallel with other safety constraints to ensure that the robotic arm does not self-collision under any configuration.

[0068] This solution also provides a teleoperation method for a robotic arm based on dense point clouds and dynamic multi-constraint CBF, including the following steps:

[0069] Step 1: The global perception unit and the local perception unit respectively acquire point cloud data and transmit it to the main control computing unit. The main control computing unit performs TF coordinate transformation and timestamp alignment, and then removes the point cloud of the robotic arm body to update the ESDF distance field map. The generated 3D map and the robotic arm pose are rendered and streamed to the display in real time.

[0070] Step 2: The main control computing unit reads the input from the remote control unit and generates nominal joint commands through the real-time resynchronization integral control subunit;

[0071] Step 3: Obtain the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map. Use the safety filtering subunit based on dynamic multi-constraint CBF-QP to determine whether the safety constraints are violated. If so, eliminate the dangerous normal component; otherwise, send the nominal command and generate a safety command to the drive unit and execution unit.

[0072] Step 4: Repeat the above steps until the teleoperation operation is completed.

[0073] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0074] (1) This invention introduces an active safety filtering mechanism based on dynamic multi-constraint CBF and quadratic programming (QP) solver. By introducing dynamic parameters related to velocity and using the gradient of the distance field for orthogonal projection, not only is collision prevention and over-limit prevention achieved at the bottom layer, but also the "viscous feeling" of traditional CBF is eliminated, allowing the robotic arm to slide tangentially along the surface of the obstacle, thus realizing zero-collision teleoperation function.

[0075] (2) Realize the fusion and dimensionality reduction mapping of multi-source heterogeneous sensor data in a “global + local” manner. Use the external lidar on the base and the depth camera at the end of the robotic arm to eliminate blind spots in the field of view; by constructing the Euclidean symbolic distance field (ESDF), the collision detection for tens of thousands of dense point clouds is transformed from O(N) traversal to O(1) microsecond-level voxel lookup table, which completely solves the computing power bottleneck of high-frequency servo obstacle avoidance.

[0076] (3) A highly immersive human-computer interaction closed loop without sudden changes was constructed. The underlying "instant resynchronization" virtual integrator, combined with incremental control logic based on hybrid mapping, fundamentally eliminates the jumps and twitches in robotic arm commands caused by mode switching or network fluctuations, and significantly reduces the cognitive load on the operator.

[0077] The present invention will be further described below with reference to specific embodiments. Attached Figure Description

[0078] Figure 1 This is a block diagram of the teleoperation system of the present invention based on dense point cloud and dynamic multi-constraint CBF robotic arm.

[0079] Figure 2 This is a flowchart of the teleoperation process of the robotic arm based on dense point cloud and dynamic multi-constraint CBF according to the present invention.

[0080] Figure 3 This is a flowchart of the multi-source sensing fusion and preprocessing process of the present invention.

[0081] Figure 4 This is a flowchart of the ESDF distance field mapping process of the present invention.

[0082] Figure 5This is a flowchart of the real-time resynchronization integral control of the present invention.

[0083] Figure 6 This is a flowchart of the security filtering process based on dynamic multi-constraint CBF-QP of the present invention. Detailed Implementation

[0084] Example

[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0086] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0087] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0088] Combination Figure 1 A teleoperation system for a robotic arm based on dense point cloud and dynamic multi-constraint CBF includes a master-end interaction module and a slave-end execution module.

[0089] The main interaction module includes a display and a remote control unit. The display is used to visualize the three-dimensional distance field map of the environment and the pose of the robotic arm, and the remote control unit is used to issue remote operation nominal control commands.

[0090] The slave execution module includes a master control computing unit, an execution unit, a global perception unit, a local perception unit, and a drive unit; the execution module is a six-degree-of-freedom robotic arm, which serves as the main body for task execution; the global perception module and the local perception module are set on the six-degree-of-freedom robotic arm and are used for multi-view environmental perception and three-dimensional mapping.

[0091] The main control computing unit is used to realize multi-source point cloud fusion, map building, and active safety obstacle avoidance;

[0092] The drive unit is used to receive control commands and provide power to the motors of each joint of the robotic arm, enabling it to complete the task according to a given safe path.

[0093] The master-end interaction module and the slave-end execution module communicate with each other via a communication unit.

[0094] During operation, the global and local acquisition and perception units transmit environmental point cloud data to the main control computing unit via high-speed data cables. The main control computing unit processes the data based on an internally constructed 3D obstacle map and joint status feedback received from the communication module, generating safety control commands. These control commands are then sent to the drive unit to drive the execution unit's movement. The main terminal interaction unit and the main control computing unit are connected via a communication module unit to enable the issuance of remote control commands and the rendering feedback of the 3D scene. The power supply unit provides stable power support for the entire system.

[0095] Combination Figure 2 The main interaction module first uses the main control computing unit to obtain real-time 3D map and robot arm pose rendering and push it to the display;

[0096] Subsequently, according to the control commands from the remote control unit, the main control computing unit collects point cloud data obtained by the global perception unit and the local perception unit, and obtains the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map in real time. The solver is used to determine whether the nominal command violates the CBF safety constraint. If it does not violate the constraint, the nominal command is sent directly. If it violates the constraint, the dangerous normal component that causes the collision is eliminated. Finally, the optimal safety command is sent to the drive unit and the execution unit. The above process is repeated until the operation ends.

[0097] The main control computing unit includes a multi-source sensing fusion and preprocessing subunit, an ESDF range field mapping subunit, an instantaneous resynchronization integral control subunit, and a safety filtering subunit based on dynamic multi-constraint CBF-QP.

[0098] The main control computing unit performs perception fusion and preprocessing on the point cloud data acquired by the global perception unit and the local perception unit based on the multi-source perception fusion and preprocessing subunit.

[0099] Then, using the ESDF distance field mapping submodule, distance field mapping is performed based on the preprocessed point cloud data to obtain the shortest safe distance and anti-collision gradient information for any point in space;

[0100] The instantaneous resynchronization integral control subunit is used to process the jump commands in the control commands of the remote control unit.

[0101] The safety filtering subunit based on dynamic multi-constraint CBF-QP is used to enforce safety constraints on the control commands of the remote control unit.

[0102] Combination Figure 3 The process by which the multi-source sensing fusion and preprocessing subunit performs sensing fusion and preprocessing on point cloud data is as follows:

[0103] The global perception unit and the local perception unit collect data and input it into their respective point cloud preprocessing modules; at the same time, the robotic arm joint encoder collects real-time joint angles and calculates the pose of each link through the forward kinematics calculation unit and the URDF robotic arm model unit.

[0104] After being calibrated by the extrinsic parameter calibration unit, the point cloud data is sent to the TF coordinate transformation module to be unified into the global absolute coordinate system.

[0105] Let the three-dimensional point in the depth camera coordinate system be denoted as The three-dimensional points in the lidar coordinate system are The three-dimensional points of the robotic arm base in the absolute coordinate system are ,but:

[0106]

[0107] in for The rotation matrix, for The translation vector, and Depend on Overview, For the sensor coordinate system to the base coordinate system Homogeneous transformation matrix;

[0108] Subsequently, the fused point cloud enters the body self-filtering module, which uses the geometric envelope provided by the URDF model to remove point clouds belonging to the robotic arm itself, and finally outputs a clean external environment point cloud.

[0109] Combination Figure 4 The process by which the ESDF distance field mapping submodule obtains the shortest safe distance and anti-collision gradient information for any point in space is as follows:

[0110] Input the preprocessed environmental perception point cloud data and perform spatial rasterization;

[0111] Secondly, the truncated symbolic range field (TSDF) is constructed and updated, and voxels are updated using a weighted multi-frame fusion algorithm. Distance value at and weight :

[0112]

[0113] in, This represents the currently updated voxel space coordinates. Indicates the updated number Frame TSDF distance value, Prehistoric The cumulative TSDF distance value of the frames. Prehistoric The weight value accumulated across frames. This represents the depth distance value observed by the sensing unit in the latest frame. This represents the weight of the latest frame of sensor observation data, determined based on the sensor noise model or distance setting.

[0114] Based on this, the Euclidean symbolic distance field (ESDF) is generated to solve for arbitrary voxels in space. Nearest obstacle set shortest distance :

[0115]

[0116] in, Voxel representation The Euclidean shortest distance to the nearest obstacle surface This represents a set of point clouds or voxels that are identified as obstacles in three-dimensional space. Represents the set of obstacles Any obstacle point in the map, express Norm, i.e., Euclidean geometric distance;

[0117] Based on the aforementioned field map, the system outputs the shortest safe distance to any point in space in real time to the safety filtering subunit of the dynamic multi-constraint CBF-QP. With collision avoidance gradient information And the distance field is incrementally maintained and updated along with the robot's movement.

[0118] Combination Figure 5 When the instantaneous resynchronization integral control subunit receives a jump command, i.e., when the safety enable button of the remote control unit is pressed:

[0119] The initial trigger moment of the safety enable lock when the operator presses the joystick. Physical joint angle at time As the starting point for integration, in the Cartesian coordinate mapping mode, The joystick thrust of the telescopic unit at a given moment is mapped to the end-space velocity spinor. The differential inverse kinematics (Diff IK) is solved using the damped least squares method, and the joint velocity increments are output. :

[0120]

[0121] in, This represents the nominal joint velocity increment vector output by the solution. This represents the terminal Cartesian nominal velocity spinor issued by the operator via the flight joystick. Indicates the current joint position of the robotic arm. The Jacobian matrix below is abbreviated as: , Describes the pseudo-inverse of the Jacobian matrix. Describes the transpose of the Jacobian matrix. This represents the damping coefficient used to limit the abrupt divergence of joint velocities in a robotic arm near singular points. Represents the identity matrix;

[0122] The virtual integrator generates a smooth nominal control command through closed-loop accumulation calculation:

[0123]

[0124] in, express The absolutely smooth nominal joint position command generated at every moment. express The actual physical joint positions of the robotic arm are constantly read from the hardware encoder (as the absolute starting point for resynchronization of the integral). This represents the time integral variable.

[0125] Combination Figure 6 The safety filtering subunit based on dynamic multi-constraint CBF-QP receives the nominal control command. And import the constraint pool;

[0126] The quadratic programming QP solver uses minimizing the degree of instruction intervention as its objective function.

[0127]

[0128] in, This represents the actual safe joint velocity command output by the QP solver after safety constraint filtering. This indicates the nominal joint speed command passed from the front end;

[0129] Four major boundary constraints are applied in parallel during the solution process:

[0130] (1) Dynamic ESDF collision avoidance: Define the collision safety margin function

[0131]

[0132] Introducing Lie derivative and dynamic speed adjustment coefficient Construct CBF constraints:

[0133]

[0134]

[0135] in, This represents the obstacle avoidance control function for the environment. Indicates the current joint angle The position of the closest point to the obstacle on the surface of the lower robotic arm's anti-collision capsule. This represents the system's preset absolute safety distance threshold. This indicates that the ESDF map is at the point. The gradient vector of the three-dimensional distance field obtained by looking up the table. Indicates the closest point from the robotic arm base to the capsule body. The local Jacobian matrix, Indicates the current speed of the robotic arm's end effector. The negatively correlated dynamic adjustment function, where Based on the gain, As a speed attenuation factor, it ensures that constraints tighten at high speeds and flexibility is maintained at low speeds;

[0136] (2) Joint physical limits: To avoid exceeding mechanical limits, damping constraints are applied:

[0137]

[0138] in, Indicates the current real-time joint position. and These represent the lower and upper physical limits of each joint of the robotic arm, respectively. This represents the constant approach velocity decay coefficient;

[0139] (3) Singularity avoidance: based on operability index

[0140]

[0141] Preventing approximation of exotic configurations:

[0142]

[0143] in, The Yoshikawa index represents the kinematic maneuverability of a robotic arm; a smaller value indicates that it is closer to a singularity. This represents the operation of finding the determinant of a matrix. This indicates the operability index for joint angles. The gradient of the partial derivative, This represents the minimum operability safety threshold allowed by the system. This represents the adjustment coefficient for singularity avoidance control;

[0144] (4) Self-collision avoidance: The distance between each link is calculated based on the simplified capsule model. By applying a similar self-collision prevention CBF, when faced with approaching obstacles, the solver, under the premise of strictly satisfying all the above inequalities, uses the gradient direction to orthogonally filter out the normal velocity and forcibly retain the tangential velocity, giving the robotic arm a forward safe sliding capability without any stickiness:

[0145] The capsule envelope parameters of each link are automatically extracted based on the URDF model, and their spatial pose is updated in real time through forward kinematics to calculate the shortest distance between non-adjacent link pairs. Construct a self-collision control barrier function And apply CBF constraints:

[0146]

[0147] in, The minimum distance between non-adjacent links calculated for the capsule model. To preset a safety threshold, This is the adjustment coefficient; this constraint is input into the QP solver in parallel with other safety constraints to ensure that the robotic arm does not self-collision under any configuration.

[0148] The present invention employs a fusion architecture of global solid-state radar and local depth camera at the sensing end to eliminate blind spots; at the interaction end, it utilizes the remote control part and the display to construct a mutation-resistant control closed loop based on hybrid incremental mapping; and at the control layer, it introduces dynamic multi-constraint CBF and quadratic programming solver to achieve high fault tolerance, absolute safety, and refined obstacle avoidance.

[0149] This solution also provides a teleoperation method for a robotic arm based on dense point clouds and dynamic multi-constraint CBF, including the following steps:

[0150] Step 1: The global perception unit and the local perception unit respectively acquire point cloud data and transmit it to the main control computing unit. The main control computing unit performs TF coordinate transformation and timestamp alignment, and then removes the point cloud of the robotic arm body to update the ESDF distance field map. The generated 3D map and the robotic arm pose are rendered and streamed to the display in real time.

[0151] Step 2: The main control computing unit reads the input from the remote control unit and generates nominal joint commands through the real-time resynchronization integral control subunit;

[0152] Step 3: Obtain the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map. Use the safety filtering subunit based on dynamic multi-constraint CBF-QP to determine whether the safety constraints are violated. If so, eliminate the dangerous normal component; otherwise, send the nominal command and generate a safety command to the drive unit and execution unit.

[0153] Step 4: Repeat the above steps until the teleoperation operation is completed.

[0154] This solution also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0155] Step 1: The global perception unit and the local perception unit respectively acquire point cloud data and transmit it to the main control computing unit. The main control computing unit performs TF coordinate transformation and timestamp alignment, and then removes the point cloud of the robotic arm body to update the ESDF distance field map. The generated 3D map and the robotic arm pose are rendered and streamed to the display in real time.

[0156] Step 2: The main control computing unit reads the input from the remote control unit and generates nominal joint commands through the real-time resynchronization integral control subunit;

[0157] Step 3: Obtain the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map. Use the safety filtering subunit based on dynamic multi-constraint CBF-QP to determine whether the safety constraints are violated. If so, eliminate the dangerous normal component; otherwise, send the nominal command and generate a safety command to the drive unit and execution unit.

[0158] Step 4: Repeat the above steps until the teleoperation operation is completed.

[0159] This solution also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the following steps:

[0160] Step 1: The global perception unit and the local perception unit respectively acquire point cloud data and transmit it to the main control computing unit. The main control computing unit performs TF coordinate transformation and timestamp alignment, and then removes the point cloud of the robotic arm body to update the ESDF distance field map. The generated 3D map and the robotic arm pose are rendered and streamed to the display in real time.

[0161] Step 2: The main control computing unit reads the input from the remote control unit and generates nominal joint commands through the real-time resynchronization integral control subunit;

[0162] Step 3: Obtain the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map. Use the safety filtering subunit based on dynamic multi-constraint CBF-QP to determine whether the safety constraints are violated. If so, eliminate the dangerous normal component; otherwise, send the nominal command and generate a safety command to the drive unit and execution unit.

[0163] Step 4: Repeat the above steps until the teleoperation operation is completed.

[0164] The embodiments described above are merely one implementation method of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A teleoperation system for a robotic arm based on dense point clouds and dynamic multi-constraint CBF, characterized in that, It includes a master-side interaction module and a slave-side execution module; The main interaction module includes a display and a remote control unit. The display is used to visualize the three-dimensional distance field map of the environment and the pose of the robotic arm, and the remote control unit is used to issue remote operation nominal control commands. The slave execution module includes a master control computing unit, an execution unit, a global perception unit, a local perception unit, and a drive unit; the execution module is a six-degree-of-freedom robotic arm, which serves as the main body for task execution; the global perception module and the local perception module are set on the six-degree-of-freedom robotic arm and are used for multi-view environmental perception and three-dimensional mapping. The main control computing unit is used to realize multi-source point cloud fusion, map building, and active safety obstacle avoidance; The drive unit is used to receive control commands and provide power to the motors of each joint of the robotic arm, enabling it to complete the task according to a given safe path. The master-end interaction module and the slave-end execution module communicate with each other via a communication unit.

2. The teleoperation system for a robotic arm based on dense point clouds and dynamic multi-constraint CBF as described in claim 1, characterized in that, The main interactive module first uses the main control computing unit to obtain real-time 3D map and robotic arm pose rendering and push it to the display. Subsequently, according to the control commands from the remote control unit, the main control computing unit collects point cloud data obtained by the global perception unit and the local perception unit, and obtains the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map in real time. The solver is used to determine whether the nominal command violates the CBF safety constraint. If it does not violate the constraint, the nominal command is sent directly. If it violates the constraint, the dangerous normal component that causes the collision is eliminated. Finally, the optimal safety command is sent to the drive unit and the execution unit. The above process is repeated until the operation ends.

3. The teleoperation system for a robotic arm based on dense point clouds and dynamic multi-constraint CBF as described in claim 2, characterized in that, The main control computing unit includes a multi-source sensing fusion and preprocessing subunit, an ESDF range field mapping subunit, an instantaneous resynchronization integral control subunit, and a safety filtering subunit based on dynamic multi-constraint CBF-QP. The main control computing unit performs perception fusion and preprocessing on the point cloud data acquired by the global perception unit and the local perception unit based on the multi-source perception fusion and preprocessing subunit. Then, using the ESDF distance field mapping submodule, distance field mapping is performed based on the preprocessed point cloud data to obtain the shortest safe distance and anti-collision gradient information for any point in space; The instantaneous resynchronization integral control subunit is used to process the jump commands in the control commands of the remote control unit. The safety filtering subunit based on dynamic multi-constraint CBF-QP is used to enforce safety constraints on the control commands of the remote control unit.

4. The teleoperation system for a robotic arm based on dense point clouds and dynamic multi-constraint CBF as described in claim 3, characterized in that, The process by which the multi-source sensing fusion and preprocessing subunit performs sensing fusion and preprocessing on point cloud data is as follows: The global perception unit and the local perception unit collect data and input it into their respective point cloud preprocessing modules; at the same time, the robotic arm joint encoder collects real-time joint angles and calculates the pose of each link through the forward kinematics calculation unit and the URDF robotic arm model unit. After being calibrated by the extrinsic parameter calibration unit, the point cloud data is sent to the TF coordinate transformation module to be unified into the global absolute coordinate system. Let the three-dimensional point in the depth camera coordinate system be denoted as The three-dimensional points in the lidar coordinate system are The three-dimensional points of the robotic arm base in the absolute coordinate system are ,but: ； in for The rotation matrix, for The translation vector, and Depend on Overview, For the sensor coordinate system to the base coordinate system Homogeneous transformation matrix; Subsequently, the fused point cloud enters the body self-filtering module, which uses the geometric envelope provided by the URDF model to remove point clouds belonging to the robotic arm itself, and finally outputs a clean external environment point cloud.

5. The teleoperation system for a robotic arm based on dense point clouds and dynamic multi-constraint CBF as described in claim 3, characterized in that, The process by which the ESDF distance field mapping submodule obtains the shortest safe distance and anti-collision gradient information for any point in space is as follows: Input the preprocessed environmental perception point cloud data and perform spatial rasterization; Secondly, the truncated symbolic range field (TSDF) is constructed and updated, and voxels are updated using a weighted multi-frame fusion algorithm. Distance value at and weight : ； in, This represents the currently updated voxel space coordinates. Indicates the updated number Frame TSDF distance value, Prehistoric The cumulative TSDF distance value of the frames. Prehistoric The weight value accumulated across frames. This represents the depth distance value observed by the sensing unit in the latest frame. This represents the weight of the latest frame of sensor observation data, determined based on the sensor noise model or distance setting. Based on this, the Euclidean symbolic distance field (ESDF) is generated to solve for arbitrary voxels in space. Nearest obstacle set shortest distance : ； in, Voxel representation The Euclidean shortest distance to the nearest obstacle surface This represents a set of point clouds or voxels that are identified as obstacles in three-dimensional space. Represents the set of obstacles Any obstacle point in the map, express Norm, i.e., Euclidean geometric distance; Based on the above field map, the shortest safe distance to any point in space is output in real time to the safety filtering subunit of the dynamic multi-constraint CBF-QP. With collision avoidance gradient information And the distance field is incrementally maintained and updated along with the robot's movement.

6. The teleoperation system for a robotic arm based on dense point clouds and dynamic multi-constraint CBF as described in claim 3, characterized in that, When the instantaneous resynchronization integral control subunit receives a jump command, i.e., when the safety enable button of the remote control unit is pressed: The initial trigger moment of the safety enable lock when the operator presses the joystick. Physical joint angle at time As the starting point for integration, in the Cartesian coordinate mapping mode, The joystick thrust of the telescopic unit at a given moment is mapped to the end-space velocity spinor. The differential inverse kinematics is solved using the damped least squares method, and the joint velocity increment is output. : ； in, This represents the nominal joint velocity increment vector output by the solution. This represents the terminal Cartesian nominal velocity spinor issued by the operator via the flight joystick. Indicates the current joint position of the robotic arm. The Jacobian matrix below is abbreviated as: , Describes the pseudo-inverse of the Jacobian matrix. Describes the transpose of the Jacobian matrix. This represents the damping coefficient used to limit the abrupt divergence of joint velocities in a robotic arm near singular points. Represents the identity matrix; The virtual integrator generates a smooth nominal control command through closed-loop accumulation calculation: ； in, express The absolutely smooth nominal joint position command generated at every moment. express The actual physical joint positions of the robotic arm are constantly read from the hardware encoder. This represents the time integral variable.

7. The teleoperation system for a robotic arm based on dense point clouds and dynamic multi-constraint CBF as described in claim 3, characterized in that, The safety filter subunit based on dynamic multi-constraint CBF-QP receives the nominal control command. And import the constraint pool; The quadratic programming QP solver uses minimizing the degree of instruction intervention as its objective function. ； in, This represents the actual safe joint velocity command output by the QP solver after safety constraint filtering. This indicates the nominal joint speed command passed from the front end; Four major boundary constraints are applied in parallel during the solution process: (1) Dynamic ESDF collision avoidance: Define the collision safety margin function ； Introducing Lie derivative and dynamic speed adjustment coefficient Construct CBF constraints: ； ； in, This represents the obstacle avoidance control function for the environment. Indicates the current joint angle The position of the closest point to the obstacle on the surface of the lower robotic arm's anti-collision capsule. This represents the system's preset absolute safety distance threshold. This indicates that the ESDF map is at the point. The gradient vector of the three-dimensional distance field obtained by looking up the table. Indicates the closest point from the robotic arm base to the capsule body. The local Jacobian matrix, Indicates the current speed of the robotic arm's end effector. The negatively correlated dynamic adjustment function, where Based on the gain, As a speed attenuation factor, it ensures that constraints tighten at high speeds and flexibility is maintained at low speeds; (2) Joint physical limits: To avoid exceeding mechanical limits, damping constraints are applied: ； in, Indicates the current real-time joint position. and These represent the lower and upper physical limits of each joint of the robotic arm, respectively. This represents the constant approach velocity decay coefficient; (3) Singularity avoidance: based on operability index ； Preventing approximation of exotic configurations: ； in, The Yoshikawa index represents the kinematic maneuverability of a robotic arm; a smaller value indicates that it is closer to a singularity. This represents the operation of finding the determinant of a matrix. This indicates the operability index for joint angles. The gradient of the partial derivative, This represents the minimum operability safety threshold allowed by the system. This represents the adjustment coefficient for singularity avoidance control; (4) Self-collision avoidance: The distance between each link is calculated based on the simplified capsule model. By applying a similar self-collision prevention CBF, when faced with approaching obstacles, the solver, under the premise of strictly satisfying all the above inequalities, uses the gradient direction to orthogonally filter out the normal velocity and forcibly retain the tangential velocity, giving the robotic arm a forward safe sliding capability without any stickiness: The capsule envelope parameters of each link are automatically extracted based on the URDF model, and their spatial pose is updated in real time through forward kinematics to calculate the shortest distance between non-adjacent link pairs. Construct a self-collision control barrier function And apply CBF constraints: ； in, The minimum distance between non-adjacent links calculated for the capsule model. To preset a safety threshold, This is the adjustment coefficient; this constraint is input into the QP solver in parallel with other safety constraints to ensure that the robotic arm does not self-collision under any configuration.

8. A teleoperation method for a robotic arm based on dense point clouds and dynamic multi-constraint CBF, characterized in that, Includes the following steps: Step 1: The global perception unit and the local perception unit respectively acquire point cloud data and transmit it to the main control computing unit. The main control computing unit performs TF coordinate transformation and timestamp alignment, and then removes the point cloud of the robotic arm body to update the ESDF distance field map. The generated 3D map and the robotic arm pose are rendered and streamed to the display in real time. Step 2: The main control computing unit reads the input from the remote control unit and generates nominal joint commands through the real-time resynchronization integral control subunit; Step 3: Obtain the shortest distance and gradient from each simplified capsule of the robotic arm to the ESDF map. Use the safety filtering subunit based on dynamic multi-constraint CBF-QP to determine whether the safety constraints are violated. If so, eliminate the dangerous normal component; otherwise, send the nominal command and generate a safety command to the drive unit and execution unit. Step 4: Repeat the above steps until the teleoperation operation is completed.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 8.

10. A computer-storable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8.

Images