A mobile robot arm-based signal homing operation method and system

Abstract

The application discloses a signal resetting operation method and system based on a mobile manipulator, wherein a visual perception module collects image and depth data of a signal resetting button, realizes resetting button recognition and positioning; an inertial sensor senses the angular velocity and acceleration of the end of the manipulator and feeds back to a control module to actively inhibit the end jitter; a force sensor senses the pressure of the end of the mobile manipulator on the signal resetting button; a coordinate transformation module converts pixel coordinates and depth information into target pose under the base coordinate system of the manipulator; a planning module generates approaching, pressing and withdrawing trajectories according to the target pose; the control module controls the manipulator to move along the generated trajectories to press the signal resetting button, while the acceleration and angular velocity information sensed by the inertial sensor is used to inhibit the end jitter of the manipulator. The application effectively inhibits the end jitter of the mobile manipulator and can effectively replace manual work to complete dangerous or repetitive signal resetting tasks.

Description

Technical Field

[0001] This invention relates to the field of robot operation and intelligent control technology, and in particular to a signal recovery operation method and system based on a mobile robotic arm. Background Technology

[0002] In industrial automation, power line inspection, and scientific research scenarios, equipment panels are generally equipped with various signal indicators and alarm buttons. During operation, when a momentary alarm occurs or manual confirmation is required, the operator needs to press the corresponding reset button to clear the alarm and restore the system to normal operation. Traditional methods rely on close-range manual operation, which is prone to fatigue and misoperation due to the high frequency and repetitive nature of the work. Furthermore, some sites are located in dangerous environments such as high pressure, high temperature, toxic gases, or radiation, posing a high risk to personal safety.

[0003] Existing automated systems utilize fixed robotic arms to perform preset actions, but their recognition capabilities and environmental adaptability are limited, making it difficult to locate and manipulate target components in unknown or complex environments. With the development of visual perception and multi-sensor fusion technologies, combining robotic arms with visual and force information can achieve automatic recognition and flexibly pressing of target buttons.

[0004] Currently, depth vision cameras can provide control modules with high-precision image information and depth data in real time for spatial point cloud reconstruction and target localization; six-degree-of-freedom robotic arms can achieve motion control of complex trajectories in three-dimensional space. By integrating force sensors at the end of the robotic arm, the magnitude of contact force during the pressing process can be sensed, and protective measures can be taken in time when the contact force exceeds a set threshold, thereby improving the safety and compliance of the operation process.

[0005] However, relying solely on visual perception and force sensors still has certain limitations: force sensors mainly reflect changes in contact force in the pressing direction, making it difficult to timely characterize end-effector vibrations and jitters caused by uneven ground, base movement, the flexibility of the robotic arm structure, load changes, and improper trajectory planning; and the joint speed feedback from the motor encoder is also difficult to accurately reflect the vibration state of the end effector in space, often resulting in a situation where "the motor speed is stable but the end-effector speed still wobbles significantly." Sometimes, even when the robotic arm is stopped, the end effector still jitters, causing the visual perception module installed at the end effector to be unable to obtain a clear target image, resulting in the target being unidentifiable or having a large position calculation error.

[0006] Therefore, there is an urgent need for a method and system for end-effector jitter suppression and signal recovery that tightly integrates visual recognition, inertial sensors, and force sensing feedback. This method involves installing inertial sensors at the end of a robotic arm to sense the acceleration and angular velocity data of the moving end-effector and incorporating this data into the servo control of each joint motor. Through the speed and current loops of each joint motor, end-effector jitter is actively suppressed, reducing jitter amplitude and duration, and improving the perception effect of the visual sensors at the end of the robotic arm. This achieves high-precision and highly reliable automated operation for signal reset in complex environments while ensuring safety. Summary of the Invention

[0007] Purpose of the invention: In view of the problems pointed out in the background art, the present invention discloses a signal recovery operation method and system based on a mobile robotic arm. By integrating visual perception, inertial sensors, force feedback, coordinate mapping, motion planning and robotic arm control, it realizes the suppression of end effector jitter and automatic identification, target positioning, stable approach and reliable pressing of the recovery button.

[0008] Technical solution: This invention discloses a signal recovery operating system based on a mobile robotic arm, including a six-degree-of-freedom mobile robotic arm, a vision perception module, an inertial sensor, a force sensor, a coordinate transformation module, a planning module, and a control module; the vision perception module, the inertial sensor, and the force sensor are all located at the end of the six-degree-of-freedom mobile robotic arm;

[0009] The six-degree-of-freedom mobile robotic arm is used to perform the physical operations of approaching, pressing, and retracting the signal reset button;

[0010] The visual perception module is used to acquire color and depth images containing the reset button, and to obtain the button pixel position and depth information through a target recognition algorithm.

[0011] The inertial sensor is used to acquire the linear acceleration and angular velocity of the robotic arm end in real time, and to evaluate the end motion state and jitter.

[0012] The force sensor is used to detect the normal contact force and torque during the pressing process;

[0013] The coordinate transformation module is used to convert the button pixel coordinates and depth information output by the visual perception module into spatial pose in the robot arm base coordinate system, thereby realizing the mapping relationship between the visual coordinate system and the robot arm coordinate system.

[0014] The planning module is used to generate segmented motion trajectories of a six-degree-of-freedom mobile robotic arm from its current pose to its approach pose, pressing pose, and retraction pose, based on the target pose, and to constrain the continuity of joint angles, velocity, acceleration, and jerk.

[0015] The control module is used to perform closed-loop control of the robotic arm actuator based on the received trajectory information and in combination with the real-time feedback from the force sensor module and the inertial sensor module, so as to eliminate end-effector jitter and complete the signal reset operation.

[0016] Furthermore, the visual perception module operates as follows:

[0017] A depth camera is used to acquire color images and depth information. The restoration button is identified through image preprocessing, feature extraction and template matching, and its center pixel coordinates and depth value are located.

[0018] The recognition results are then processed by multi-frame fusion and adaptive thresholding and output to the coordinate transformation module.

[0019] A pinhole imaging model is used to perform 3D back projection on the button pixel coordinates. The pixel coordinates of the restored button and the corresponding depth value are taken as input, the pixel coordinates are normalized, the 3D coordinates of the button in the camera coordinate system are obtained, and the depth of the restored button in multiple consecutive frames is fused.

[0020] Select a local depth point cloud around the return button, and obtain the plane normal vector from the depth gradient through a plane fitting model. Then, normalize the normal vector to obtain the unit normal vector of the return button in the camera coordinate system.

[0021] Furthermore, the coordinate transformation module first converts the pixel coordinates and depth information of the reset button into three-dimensional coordinates in the camera coordinate system. Then, using the extrinsic parameter relationship between the camera and the six-degree-of-freedom moving robotic arm, the three-dimensional coordinates are transformed into the target position and posture in the robotic arm base coordinate system, providing a spatial basis for trajectory planning. The unit normal vector in the camera coordinate system is converted into a normal vector in the robotic arm base coordinate system through a rotation matrix, which is used to construct the pressing target posture.

[0022] Furthermore, the six-dimensional torque vector of the force sensor can be used to obtain the true force vector after calibration matrix and zero bias compensation. Combined with the normal vector in the base coordinate system of the button robotic arm, the normal contact force along the pressing direction is calculated. The normal contact force is used to determine whether a stable contact has been established and whether the pressing is completed. On the other hand, it is compared with the safety threshold to trigger the protection action.

[0023] Furthermore, the control module monitors the rate of change of the normal contact force over time. To determine whether a typical press pulse occurs, when the condition is met... When a valid pressure pulse has been generated, the signal reset action is considered complete; simultaneously, a safety force threshold is set. ,when At that time, the protection strategy is immediately triggered, controlling the six-degree-of-freedom mobile robotic arm to retreat along the retraction trajectory.

[0024] Furthermore, the inertial sensor uploads the collected end-effector motion data to the control module after filtering and zero-point compensation. The specific operation is as follows:

[0025] Multimodal fusion of visual measurement, robotic arm encoder feedback, and inertial sensor data is performed to construct an end-effector state vector, including the end-effector's position in the robotic arm's base coordinate system, end-effector velocity, end-effector attitude parameters, and end-effector angular velocity.

[0026] An extended Kalman filter (EKF) is used to predict and update the state.

[0027] Jitter metrics are constructed using linear acceleration and angular velocity measured by inertial sensors.

[0028] ;

[0029] in, The end-line acceleration vector, is the end-effector angular velocity vector, and and are non-negative coefficients for adjusting the weights of different components; the jitter index is calculated as the root mean square value or average value within a given time window and is used as a quantitative indicator to evaluate the intensity of end-effector vibration.

[0030] During trajectory execution, the planned end velocity is adaptively corrected based on the jitter index. The corrected velocity is:

[0031] ;

[0032] in, The original velocity obtained from trajectory planning, The damping coefficient is... The actual execution speed is determined by the fact that when the jitter index increases, the exponential term decreases, thereby automatically reducing the end-effector speed.

[0033] Furthermore, based on the coordinate transformation results, the planning module plans the motion path and action sequence of the six-degree-of-freedom mobile robotic arm, generating segmented trajectories for three stages: approach, pressing, and retraction. It also imposes constraints on joint angle continuity, velocity, acceleration, and jerk, as detailed below:

[0034] Approaching Phase Trajectory: Starting from the current end-effector pose and ending at the target pressing pose, three trajectories are generated: long-distance approach, mid-distance deceleration, and close-distance fine-tuning. The fine-tuning phase uses visual servoing for end-effector fine-tuning. During visual servoing, the pixel coordinates of the return button are periodically collected and the difference between them and the desired position is calculated to form the pixel error. This is further combined with depth updates to convert the image error into the end-effector pose correction amount, and the joint velocities are solved using the Jacobian matrix of the robotic arm.

[0035] Pressing Phase Trajectory: The end effector slowly presses down along the button's normal direction, continuously monitoring changes in contact force and torque via a force sensor; when the force sensor detects a pressing pulse that matches the button's characteristics, the reset is considered successful; if the force value abnormally increases or exceeds the safety threshold during the pressing process, the system immediately stops pressing and performs a retraction; if no successful pressing signal is detected within the set maximum displacement range, the pressing is considered a failure, and a readjustment and retry process begins.

[0036] Trajectory during the retraction phase: The visual perception module re-acquires images of the target area and determines whether the button has changed from "not reset" to "reset" by comparing the button features before and after pressing. If the verification is successful, the robotic arm retracts to a safe posture according to the planned trajectory. If the expected change is not detected, the system believes that the retraction may not have been fully executed. The planning module will re-plan the fine-tuned trajectory based on the current end pose and repeat the approach and pressing operations.

[0037] Furthermore, the control module specifically performs the following:

[0038] 1) During the task completion process, based on feedback from the inertial sensor, the active suppression algorithm is used to control the motor current of each joint and adjust the movement speed and acceleration;

[0039] 2) During the pressing phase, based on the feedback from the force sensor, the normal contact force is controlled in a closed loop, and the velocity loop and current loop are compensated by combining the end angular velocity and acceleration information.

[0040] 3) During the retraction phase, residual vibration is eliminated by limiting acceleration and angular velocity, in conjunction with end-effector inertial feedback.

[0041] This invention also discloses a method for suppressing jitter and restoring signals at the end effector of a mobile robotic arm, comprising the following steps:

[0042] Step S1: After starting the system, perform visual calibration, external parameter calibration of the robotic arm and camera, and zero-point calibration of the force sensor and inertial sensor;

[0043] Step S2: Establish the transformation relationship between the camera coordinate system, the robot arm base coordinate system, and the end-effector coordinate system;

[0044] Step S3: The visual perception module identifies and determines the pixel coordinates of the reset button position;

[0045] Step S4: Calculate the 3D position of the return button in the camera coordinate system;

[0046] Step S5: The coordinate transformation module maps the 3D position of the button in the camera coordinate system to the robot arm base coordinate system;

[0047] Step S6: Determine the target pose of the robotic arm end effector based on the normal direction of the reset button and the pressing depth;

[0048] Step S7: Based on the performance constraints of each joint of the robotic arm, plan and complete the approach trajectory, pressing trajectory, and retraction trajectory;

[0049] Step S8: Control the robotic arm to move along the planned trajectory, while suppressing the shaking at the end of the robotic arm;

[0050] Step S9: Complete the signal recovery operation and verify it.

[0051] Compared with the prior art, the significant advantages of this invention are:

[0052] (1) The present invention provides the three-dimensional spatial position of the button through the visual perception module to achieve millimeter-level high-precision positioning; the inertial sensor directly measures the end jitter state, and after feeding it back into the servo control, the jitter amplitude and convergence time can be significantly reduced, and the visual recognition and positioning efficiency can be improved; the force sensor monitors the contact force in real time, and completes the reliable return operation under the premise of ensuring the safety of the equipment through pulse signal and threshold judgment logic.

[0053] (2) Based on the traditional position, speed and current multi-loop servo control, this invention introduces end inertial sensor feedback and force sensor feedback to form a composite closed-loop structure of "visual positioning + active jitter suppression + force protection", which improves the system's anti-interference capability and safety.

[0054] (3) High degree of automation. This invention realizes full-process automation from "perception - identification - positioning - trajectory planning - recursive execution - result verification - parameter self-adjustment", which greatly reduces manual intervention and improves work efficiency.

[0055] In summary, this invention integrates visual perception, inertial sensors, force feedback, coordinate mapping, motion planning, and robotic arm control to achieve end-effector jitter suppression and automatic identification, target localization, stable approach, and reliable pressing of the reset button. Furthermore, to improve end-effector estimation stability in vibration and noise environments, pixel back-projection, multi-frame deep fusion, vision-inertial-encoder multimodal fusion (EKF), quaternion attitude update, and jitter index-driven adaptive speed control are introduced to maintain high robustness under mobile conditions. Attached Figure Description

[0056] Figure 1 This is a block diagram of the overall structure of the present invention.

[0057] Figure 2 This is a flowchart of the steps of the present invention.

[0058] Figure 3This is a schematic diagram of the coordinate mapping based on visual information of the present invention, illustrating the transformation process of pixel coordinates → camera coordinates → robotic arm base coordinates;

[0059] Figure 4 This is a physical diagram of the signal recovery operating system based on a mobile robotic arm according to the present invention;

[0060] Figure 5 This is a physical image of the six-degree-of-freedom mobile robotic arm of the present invention implementing signal recovery operation. Detailed Implementation

[0061] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention is not limited to the following embodiments.

[0062] This invention proposes a signal recovery operation method and system based on a mobile robotic arm, such as... Figure 4 As shown, the signal recovery operating system based on the mobile robotic arm mainly consists of six parts: a six-degree-of-freedom mobile robotic arm, a vision perception module, an inertial sensor, a force sensor, a coordinate transformation module, a planning module, and a control module. The modules interact and coordinate control through a communication interface, forming a closed-loop system of "perception-computation-execution-feedback".

[0063] A six-DOF robotic arm is mounted on a mobile platform, with a two-finger gripper as its end effector. The vision perception module uses a depth vision camera, mounted between the robotic arm's end effector and the force sensor, ensuring its field of view covers the target panel area. The force sensor is a six-dimensional force / torque sensor, connected in series between the robotic arm's end effector and the two-finger gripper flange. The inertial sensor is an inertial measurement unit integrating a three-axis gyroscope and a three-axis accelerometer, integrated within the vision perception module. This unit is used to sense the motion state of the robotic arm's end effector in real time, assisting in the assessment of whether vibration, impact, or rapid disturbances exist at the end effector.

[0064] During the initialization phase, the system calibrates the intrinsic parameters of the camera and the extrinsic parameters between the camera and the robotic arm, establishing a consistent mapping relationship between pixel coordinates, camera coordinates, and robotic arm base coordinates.

[0065] When performing the signal recovery task, the visual perception module acquires color and depth images including the operation panel. To improve recognition stability, this invention further introduces multi-frame depth fusion and noise suppression strategies on the basis of existing edge detection, color segmentation, and template matching. It performs confidence screening and time-weighted fusion on consecutive multi-frame depth images, thereby reducing errors caused by depth jumps.

[0066] After recognizing the pressed area, the visual perception module marks the button area with a rectangle and calculates the pixel coordinates of the rectangle's center and the fusion depth as the button's initial position. Regarding the button's surface pose, this invention, based on existing image edge fitting, further incorporates a local plane estimation method using local depth point clouds to make the button's normal direction more stable, thereby ensuring the correct pressing direction.

[0067] The coordinate transformation module first converts the button's pixel coordinates into three-dimensional coordinates in the camera coordinate system based on the camera's intrinsic parameters. Then, it uses an extrinsic parameter matrix to transform the three-dimensional coordinates back to the robotic arm's base coordinate system, thereby obtaining the button's spatial position in the base coordinate system. This visual recognition and coordinate transformation strategy maintains high recognition accuracy under conditions of changing lighting, reflective noise, or faded buttons.

[0068] During the button approach phase, relying solely on visual input is affected by refresh rate, occlusion, and noise, while relying solely on the robotic arm encoder cannot reflect the actual small vibrations of the end effector. Therefore, this invention utilizes the linear acceleration and angular velocity output from inertial sensors to monitor the end effector's motion state in real time and fuses it with visual position recognition and robotic arm encoder feedback. The inertial sensor can capture rapid displacement, sudden stops, and vibration trends of the end effector, thus compensating for the slow visual update speed; the robotic arm encoder provides stable joint angle information, making the overall fusion result more continuous and smooth. Through this multimodal information interaction method, the system can maintain high attitude estimation stability even when the mobile platform shakes, the robotic arm operates at high speed, or the end effector is subjected to slight collision interference. This fusion strategy improves the stability of the end effector position and attitude, reduces random jumps after the depth image is disturbed, and thus significantly improves the overall robustness of visual servoing.

[0069] The planning module starts from the current end-effector pose and generates three trajectories—long-distance approach, mid-distance deceleration, and close-distance fine-tuning—with the target pressing pose as the endpoint. In the long-distance phase, the end-effector moves towards the button area at a relatively high speed; in the mid-distance phase, it gradually decelerates to avoid collisions. These speeds can be set according to actual needs and are not elaborated here. In the fine-tuning phase, visual servoing is used for end-effector fine-tuning. During visual servoing, the system periodically collects the pixel coordinates of the button and calculates the difference between these coordinates and the desired position to form the pixel error. Further, combined with depth updates, the image error is converted into an end-effector pose correction amount, and the joint velocities are solved using the robotic arm's Jacobian matrix. To suppress end-effector jitter, this invention incorporates inertial sensors to monitor the end-effector's angular velocity and acceleration during visual servoing: when increased end-effector vibration, sudden acceleration changes, or external disturbance signals are detected, the control module automatically reduces the end-effector velocity command, allowing the robotic arm to approach the button more smoothly, thereby avoiding visual recognition errors or pressing deviations caused by vibration. When approaching a singular position, a damped least squares method is used to keep the joint velocity stable, avoid algorithm divergence, and improve the stability and safety of the approach button phase.

[0070] When the end effector has stably approached the button and the visual and depth errors are below preset thresholds, the control module switches to press mode. The end effector slowly presses down along the button's normal direction, continuously monitoring changes in contact force and torque via a force sensor. When the force sensor detects a press pulse that matches the button's characteristics, the reset is considered successful. If the force value abnormally increases or exceeds the safety threshold during pressing, the system immediately stops pressing and performs a retraction. If no successful press signal is detected within the set maximum displacement range, the press is considered a failure, and a readjustment and retry process begins to prevent damage to the equipment.

[0071] After the pressing action ends, the visual perception module re-captures an image of the target area. By comparing the button's characteristics before and after pressing (color change, position change, structural change, etc.), it determines whether the button has changed from "not reset" to "reset." Figure 5 As shown. If the verification is successful, the robotic arm retracts to a safe posture according to the planned trajectory; if no expected change is detected, the system assumes that the return may not have been fully executed, and the planning module will replan the fine-tuned trajectory based on the current end-effector pose and repeat the approach, press, and jitter suppression process.

[0072] The specific operations of this embodiment of the invention are as follows:

[0073] A six-degree-of-freedom robotic arm, serving as the execution body of this invention, has six independent motion joints, enabling precise operation in any posture within three-dimensional space. The end effector gripper of the robotic arm can perform approach, pressing, and retraction operations on the signal reset button according to task requirements.

[0074] II. The visual perception module, installed at the end effector of the robotic arm, uses a depth camera to acquire color images and depth information. Through image preprocessing, feature extraction, and template matching algorithms, it identifies signal buttons and locates their center pixel coordinates and depth values. The recognition results are then processed through multi-frame fusion and adaptive thresholding before being output to the coordinate transformation module for spatial position calculation and attitude estimation.

[0075] To achieve accurate mapping from image coordinates to 3D spatial coordinates, a pinhole imaging model is used in the visual perception module to perform 3D back projection of the button pixel coordinates, using the button pixel coordinates... and corresponding depth values For input, the camera intrinsic parameter matrix is ​​denoted as:

[0076] ,

[0077] in, This refers to the equivalent focal length of the camera in both the horizontal and vertical directions. Let be the coordinates of the optical center on the image plane. Normalizing the pixel coordinates yields:

[0078] ,

[0079] The button's three-dimensional coordinates in the camera coordinate system are represented as follows:

[0080] .

[0081] To reduce the impact of single-frame depth noise on positioning accuracy, the button depths from multiple consecutive frames are fused. Let the depth measured in the i-th frame be... The noise variance is The fusion depth is then expressed as a weighted average.

[0082]

[0083] in This represents the effective depth after fusion. Utilizing... Instead of single-frame depth, a more stable 3D button position can be obtained. .

[0084] To further estimate the normal direction of the button surface, a local depth point cloud is selected around the button, and a planar fitting model is used:

[0085]

[0086] The plane normal vector (A, B, C) is obtained, and after normalization, the unit normal vector of the button in the camera coordinate system is obtained:

[0087] .

[0088] Third, the coordinate transformation module enables precise mapping between visual coordinates and the robotic arm's base coordinates. After the visual perception module outputs the pixel coordinates and depth information of the reset button, the coordinate transformation module first converts them into three-dimensional coordinates in the camera coordinate system. Then, using the extrinsic parameter relationship between the camera and the robotic arm, the module transforms these three-dimensional coordinates into the target position and orientation in the robotic arm's base coordinate system, providing a spatial basis for trajectory planning.

[0089] The results of extrinsic parameter calibration can usually be expressed as a homogeneous transformation matrix from the camera coordinate system to the robot arm's base coordinate system:

[0090] ,

[0091] Transform the button's coordinates to the robot arm's base coordinate system. Position of the button in the robot arm's base coordinate system. for

[0092] ,

[0093] in, It is a 3×3 rotation matrix used to describe the pose of the camera coordinate system relative to the robot arm's base coordinate system. It is a 3×1 translation vector used to describe the translation offset between the two coordinate systems. Let be the three-dimensional position vector of the button in the base coordinate system of the robotic arm.

[0094] Furthermore, to estimate the normal direction of the button panel, this invention calculates a planar fitting model within a local depth region around the button, and obtains the normal vector in the camera coordinate system from the depth gradient. Then, it is converted into a normal vector in the robot arm's base coordinate system using a rotation matrix. It is used to construct the pressing target posture.

[0095] IV. Force sensor, installed on the end effector of the robotic arm, is used to detect the contact force during the pressing process in real time. During the signal reset operation, the force sensor module continuously monitors the change in pressing force and compares it with a preset safe pressure threshold: when a pressure pulse signal is detected, the signal reset operation is considered complete; when the detected value exceeds the safe pressure threshold, the control module immediately controls the robotic arm to return along the retraction trajectory.

[0096] The force sensor output is denoted as a six-dimensional torque vector:

[0097] ,

[0098] The true force vector can be obtained after calibration matrix and zero bias compensation:

[0099]

[0100] in, To calibrate the gain matrix, This is the zero-point bias vector.

[0101] Combining button normal vector It can calculate the normal contact force along the pressing direction. This normal force is used to determine whether a stable contact has been established and whether the pressing is complete, and it is also compared with a safety threshold to trigger a protective action.

[0102] This invention monitors the rate of change of normal force over time. To determine if a typical press pulse occurs. When the condition is met... When a valid pressure pulse has been generated, the signal reset action is considered complete; simultaneously, a safety force threshold is set. ,when If the situation occurs, the protection strategy will be triggered immediately, controlling the robotic arm to retreat along the retraction trajectory to avoid damage to the button and surrounding equipment.

[0103] 5. An inertial sensor, integrated within the vision sensor, measures the linear acceleration and angular velocity of the end effector in the x, y, and z directions, as well as its angular velocity around these three axes. The inertial sensor module uploads the collected end effector motion data to the control module after filtering and zero-point compensation, reflecting the actual operating status and jitter of the robotic arm's end effector. Compared to control systems relying solely on motor encoders and visual information, the addition of an inertial sensor allows for faster detection of changes in end effector acceleration and angular velocity, enabling real-time perception and suppression of end effector jitter.

[0104] In this invention, the acceleration output of the inertial sensor can be represented using a basic measurement model as follows:

[0105] ,

[0106] in, The acceleration vector measured by the accelerometer. This represents the actual acceleration at the terminal end. For the zero bias term of the accelerometer, To measure noise. Similarly, the angular velocity measurement model is expressed as:

[0107] ,

[0108] in, This is the angular velocity vector measured by the gyroscope. This represents the actual angular velocity at the end point. For gyroscope bias For measuring noise.

[0109] The above model can be used to construct end-effector jitter indices and velocity correction strategies, enabling the control module to automatically reduce the trajectory execution speed or adjust control parameters when it detects an abnormal increase in end-effector acceleration or angular velocity, thereby suppressing end-effector jitter, as detailed below:

[0110] Multimodal fusion is performed using visual measurements, robotic arm encoder feedback, and inertial sensor data to construct the end-effector state vector.

[0111] ,

[0112] in, The position of the end effector in the robot arm's base coordinate system. For the terminal velocity, These are the end-effector attitude parameters. This represents the terminal angular velocity.

[0113] The Extended Kalman Filter (EKF) is used to predict and update the state. Its discrete-time state prediction equation is as follows:

[0114] ,

[0115] The observation update equation is:

[0116] ,

[0117] in, To control the input, For visual or inertial observation, For the system motion model, For observation models, For process noise, This represents the filter gain. The attitude component can be represented in quaternion form, where attitude quaternions are... The update can be represented as:

[0118] ,

[0119] in, This represents quaternion multiplication. The sampling period is Let be the angular velocity measurement vector at time k. Through the above multimodal fusion, a more stable end-effector pose estimation can be obtained under the presence of vibration disturbances.

[0120] To address the jitter at the end effector of a mobile robotic arm during movement, this invention utilizes linear acceleration and angular velocity measured by inertial sensors to construct a jitter index:

[0121] ,

[0122] in, The end-line acceleration vector, Let be the end-effector angular velocity vector, and and are non-negative coefficients for adjusting the weights of different components. This jitter index can calculate the root mean square value or average value within a given time window and is used as a quantitative indicator to assess the intensity of end-effector vibration.

[0123] During trajectory execution, this invention adaptively corrects the planned end velocity based on jitter indicators. The corrected velocity is:

[0124]

[0125] in, The original velocity obtained from trajectory planning, The damping coefficient is... This represents the actual execution speed. When the jitter index increases, the exponential term decreases, thereby automatically reducing the end effector speed.

[0126] VI. The planning module plans the motion path and action sequence of the robotic arm based on the coordinate transformation results, generating segmented trajectories for three stages: approach, pressing, and retraction. It also imposes constraints on joint angle continuity, velocity, acceleration, and jerk. During the planning process, the module automatically adjusts the joint motion sequence according to the velocity and acceleration limit parameters of the robotic arm joints, ensuring that the trajectory execution process meets dynamic performance requirements while avoiding significant impacts and vibrations.

[0127] To ensure the smoothness and dynamic controllability of the robotic arm's movements during the approach, pressing, and retraction phases, this invention employs a fifth-order polynomial trajectory model in joint space, specifying the trajectory for each joint angle. The plan is as follows:

[0128]

[0129] in, The trajectory coefficients are obtained by solving based on the initial and final positions, velocities, and acceleration boundary conditions of the joints. Let be the angle of the joint at time t. From this, we can further derive the expressions for the joint velocity and acceleration:

[0130] ,

[0131] .

[0132] Through the and Applying amplitude constraints ensures that joint movements meet the maximum speed and maximum acceleration limits, making the trajectory continuous, smooth, and physically feasible during the three phases of connection approach, pressing, and retraction.

[0133] VII. The control module is responsible for controlling the rotation of each joint motor, causing the robotic arm's end effector to move along a planned trajectory. This module receives trajectory information generated by the planning module and target pose provided by the coordinate transformation module, while also receiving output data from the force sensor and inertial sensor. Through velocity loop, current loop, and end effector jitter active suppression algorithm, it drives each joint of the robotic arm to perform the pressing task. Specific execution includes:

[0134] 1. During the task completion process, based on feedback from the end inertial sensor, the active suppression algorithm is used to control the motor current of each joint, adjust the movement speed and acceleration, reduce the shaking amplitude, and shorten the shaking time.

[0135] 2. During the pressing phase, based on the feedback from the force sensor, closed-loop control of the contact force is achieved, and the velocity loop and current loop are compensated by combining the end angular velocity and acceleration information, so that the pressing process is both smooth and stable.

[0136] 3. During the retraction phase, residual vibration is eliminated by limiting acceleration and angular velocity, in conjunction with end-effector inertial feedback.

[0137] For the aforementioned system, a method for suppressing jitter and restoring signals at the end effector of a mobile robotic arm based on the fusion of vision, inertial, and force sensors specifically includes:

[0138] Step S1: After starting the system, perform visual calibration, external parameter calibration of the robotic arm and camera, and zero-point calibration of the force sensor and inertial sensor.

[0139] Step S2: Establish the transformation relationship between the camera coordinate system, the robot arm base coordinate system, and the end-effector coordinate system.

[0140] Step S3: The visual perception module identifies and determines the pixel coordinates of the reset button position.

[0141] Step S4: Calculate the three-dimensional position of the return button in the camera coordinate system.

[0142] Step S5: The coordinate transformation module maps the three-dimensional position of the button in the camera coordinate system to the robot arm base coordinate system.

[0143] Step S6: Determine the target pose of the robotic arm end effector based on the normal direction of the reset button and the pressing depth.

[0144] Step S7: Based on the performance constraints of each joint of the robotic arm, plan the approach trajectory, pressing trajectory, and return trajectory.

[0145] Step S71: The planning module generates the approach, pressing, and retraction trajectories of the robotic arm from the current pose to the target pose based on the target pose.

[0146] Step S72: Based on the velocity, acceleration, and jerk constraints of each joint, smooth the planned trajectory in the joint space or Cartesian space.

[0147] Step S8: Control the robotic arm to move along the planned trajectory, while suppressing the shaking at the end of the robotic arm.

[0148] Step S81: The control module controls the mobile robotic arm to move along the planned trajectory.

[0149] Step S82: Based on the data output by the inertial sensor, the control module uses the end effector jitter active suppression algorithm to control the motor current and rotation speed of each joint, thereby reducing the jitter amplitude and shortening the jitter time at the end of the robotic arm.

[0150] Step S9: Complete the signal recovery operation and verify it.

[0151] Step S91: After the end contact button is pressed, the signal reset operation is determined to be completed and safe based on the pressure data output by the force sensor. When the force sensor measures a pulse signal in the direction of movement of the end of the moving robotic arm, the signal reset operation is considered to be completed. During this process, if the output of the force sensor exceeds the set safety threshold, the control module will immediately control the robotic arm to return along the retraction trajectory.

[0152] Step S92: After the press is completed, the visual perception module acquires the image of the target area again and checks whether the button state has switched to the "reset" state by comparing it with the image before the press.

[0153] Step S93: If visual verification or force sensor feedback indicates that the return has failed, then the process starting from step S2 is re-executed.

[0154] In summary, the mobile robotic arm end effector jitter suppression and signal restoration method and system proposed in this invention integrates visual recognition, inertial sensor feedback, and force detection to form a high-precision, highly robust automated signal restoration scheme. This method and system offer advantages such as compact structure, rapid response, high precision, and high reliability. They can replace manual labor in complex signal reset tasks in environments such as power systems, industrial control consoles, and unmanned laboratories, providing reliable technical support for intelligent inspection and remote maintenance.

Claims

1. A signal recovery operating system based on a mobile robotic arm, characterized in that, It includes a six-degree-of-freedom mobile robotic arm, a vision perception module, an inertial sensor, a force sensor, a coordinate transformation module, a planning module, and a control module; the vision perception module, inertial sensor, and force sensor are all located at the end of the six-degree-of-freedom mobile robotic arm; The six-degree-of-freedom mobile robotic arm is used to perform the physical operations of approaching, pressing, and retracting the signal reset button; The visual perception module is used to acquire color and depth images containing the reset button, and to obtain the button pixel position and depth information through a target recognition algorithm. The inertial sensor is used to acquire the linear acceleration and angular velocity of the robotic arm end in real time, and to evaluate the end motion state and jitter. The force sensor is used to detect the normal contact force and torque during the pressing process; The coordinate transformation module is used to convert the button pixel coordinates and depth information output by the visual perception module into spatial pose in the robot arm base coordinate system, thereby realizing the mapping relationship between the visual coordinate system and the robot arm coordinate system. The planning module is used to generate segmented motion trajectories of a six-degree-of-freedom mobile robotic arm from its current pose to its approach pose, pressing pose, and retraction pose, based on the target pose, and to constrain the continuity of joint angles, velocity, acceleration, and jerk. The control module is used to perform closed-loop control of the robotic arm actuator based on the received trajectory information and in combination with the real-time feedback from the force sensor module and the inertial sensor module, so as to eliminate end-effector jitter and complete the signal reset operation. The inertial sensor collects end-effector motion data, filters and performs zero-point compensation, and then uploads it to the control module. The specific operation is as follows: Multimodal fusion of visual measurement, robotic arm encoder feedback, and inertial sensor data is performed to construct an end-effector state vector, including the end-effector's position in the robotic arm's base coordinate system, end-effector velocity, end-effector attitude parameters, and end-effector angular velocity. An extended Kalman filter (EKF) is used to predict and update the state. Jitter metrics are constructed using linear acceleration and angular velocity measured by inertial sensors. ； wherein, is the end line acceleration vector, is the end angular velocity vector, and a, b are non-negative coefficients for adjusting the weights of different components; the jitter index is calculated as the root mean square value or the average value within a given time window, and is used as a quantitative index for evaluating the end vibration intensity; During trajectory execution, the planned end velocity is adaptively corrected based on the jitter index. The corrected velocity is: ； wherein, is the original velocity obtained by trajectory planning, is the vibration suppression coefficient, is the actual execution velocity, when the jitter index increases, the exponential term decreases, thereby automatically reducing the end motion velocity.

2. The mobile manipulator-based signal homing system of claim 1, wherein, The visual perception module operates as follows: A depth camera is used to acquire color images and depth information. The restoration button is identified through image preprocessing, feature extraction and template matching, and its center pixel coordinates and depth value are located. The recognition results are then processed by multi-frame fusion and adaptive thresholding and output to the coordinate transformation module. A pinhole imaging model is used to perform 3D back projection on the button pixel coordinates. The pixel coordinates of the restored button and the corresponding depth value are taken as input, the pixel coordinates are normalized, the 3D coordinates of the button in the camera coordinate system are obtained, and the depth of the restored button in multiple consecutive frames is fused. Select a local depth point cloud around the return button, and obtain the plane normal vector from the depth gradient through a plane fitting model. Then, normalize the normal vector to obtain the unit normal vector of the return button in the camera coordinate system.

3. The mobile manipulator-based signal homing system of claim 1, wherein, The coordinate transformation module first converts the pixel coordinates and depth information of the reset button into three-dimensional coordinates in the camera coordinate system. Then, using the extrinsic parameter relationship between the camera and the six-degree-of-freedom moving robotic arm, it transforms the three-dimensional coordinates into the target position and attitude in the robotic arm base coordinate system, providing a spatial basis for trajectory planning. The unit normal vector in the camera coordinate system is converted into a normal vector in the robotic arm base coordinate system through a rotation matrix, which is used to construct the pressing target attitude.

4. The mobile manipulator-based signal homing system of claim 3, wherein, The six-dimensional torque vector of the force sensor can be used to obtain the true force vector after calibration matrix and zero bias compensation. Combined with the normal vector in the base coordinate system of the button robotic arm, the normal contact force along the pressing direction is calculated. The normal contact force is used to determine whether a stable contact has been established and whether the pressing is completed. On the other hand, it is compared with the safety threshold to trigger the protection action.

5. The mobile manipulator-based signal homing system of claim 4, wherein, The control module monitors the rate of change of the normal contact force over time. To determine whether a typical press pulse occurs, when the condition is met... When a valid pressure pulse has been generated, the signal reset action is considered complete; simultaneously, a safety force threshold is set. ,when At that time, the protection strategy is immediately triggered, controlling the six-degree-of-freedom mobile robotic arm to retreat along the retraction trajectory.

6. The mobile manipulator-based signal homing system of claim 1, wherein, The planning module plans the motion path and action sequence of the six-degree-of-freedom mobile robotic arm based on the coordinate transformation results, generating segmented trajectories for three stages: approach, pressing, and retraction, and constraining the continuity of joint angles, velocity, acceleration, and jerk; specifically: Approaching Phase Trajectory: Starting from the current end-effector pose and ending at the target pressing pose, three trajectories are generated: long-distance approach, mid-distance deceleration, and close-distance fine-tuning. The fine-tuning phase uses visual servoing for end-effector fine-tuning. During visual servoing, the pixel coordinates of the return button are periodically collected and the difference between them and the desired position is calculated to form the pixel error. This is further combined with depth updates to convert the image error into the end-effector pose correction amount, and the joint velocities are solved using the Jacobian matrix of the robotic arm. Pressing phase trajectory: The end effector slowly presses down along the button normal direction, and the force sensor continuously monitors the changes in contact force and torque; when the force sensor detects a pressing pulse that matches the button characteristics, the reset is considered successful. If the force value increases abnormally or exceeds the safety threshold during the pressing process, the system will immediately stop pressing and perform a retraction; if no successful pressing signal is detected within the set maximum displacement range, the pressing is considered to have failed and a readjustment and retry process will begin. Trajectory during the retraction phase: The visual perception module re-acquires images of the target area and determines whether the button has changed from "not reset" to "reset" by comparing the button features before and after pressing. If the verification is successful, the robotic arm retracts to a safe posture according to the planned trajectory. If the expected change is not detected, the system believes that the retraction may not have been fully executed. The planning module will re-plan the fine-tuning trajectory based on the current end pose and repeat the approach and pressing operations.

7. The mobile manipulator-based signal homing system of claim 1, wherein, The control module specifically performs the following: 1) During the task completion process, based on feedback from the inertial sensor, the active suppression algorithm is used to control the motor current of each joint and adjust the movement speed and acceleration; 2) During the pressing phase, based on the feedback from the force sensor, the normal contact force is controlled in a closed loop, and the velocity loop and current loop are compensated by combining the end angular velocity and acceleration information. 3) During the retraction phase, residual vibration is eliminated by limiting acceleration and angular velocity, in conjunction with end-effector inertial feedback.

8. A method for suppressing end-shake and signal restoration of a mobile robot arm according to any one of claims 1 to 7, characterized by, Includes the following steps: Step S1: After starting the system, perform visual calibration, external parameter calibration of the robotic arm and camera, and zero-point calibration of the force sensor and inertial sensor; Step S2: Establish the transformation relationship between the camera coordinate system, the robot arm base coordinate system, and the end-effector coordinate system; Step S3: The visual perception module identifies and determines the pixel coordinates of the reset button position; Step S4: Calculate the 3D position of the return button in the camera coordinate system; Step S5: The coordinate transformation module maps the 3D position of the button in the camera coordinate system to the robot arm base coordinate system; Step S6: Determine the target pose of the robotic arm end effector based on the normal direction of the reset button and the pressing depth; Step S7: Based on the performance constraints of each joint of the robotic arm, plan and complete the approach trajectory, pressing trajectory, and retraction trajectory; Step S8: Control the robotic arm to move along the planned trajectory, while suppressing the shaking at the end of the robotic arm; Step S9: Complete the signal recovery operation and verify it.

Images