Force sense control method fusing robot visual information

Abstract

The invention relates to the technical field of robot control, in particular to a force sense control method fusing robot visual information, and solves the technical problems of operator intention distortion and man-machine control right conflict. The method comprises the following steps: identifying an interaction object through scene semantic segmentation, and establishing a virtual force model to generate a predictive guide data stream; receiving an operating force of an operator through a force sensor, executing online dynamic compensation to obtain a pure intention force, and analyzing the pure intention force by applying an algorithm to obtain an operating instruction vector; applying a dynamic projection fusion algorithm, taking the modulus length of the guide data stream as a dynamic weight, performing non-conflict fusion on the guide data stream and the manipulation instruction vector, and calculating a shared motion track; and finally, outputting a trajectory tracking signal by adopting a position tracking controller, and providing force sense guidance for an operator through the active movement of the robot. According to the invention, efficient, safe and smooth man-machine cooperative control is realized.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a force control method that integrates robot visual information. Background Technology

[0002] Existing robot remote control operating systems, especially master-slave control systems applied in complex or unstructured environments, have significant shortcomings in terms of intuitiveness, safety, and human-machine collaboration efficiency.

[0003] Traditional remote control operations mostly rely on direct control or transparent remote control modes. This mode rigidly maps the operator's movement to the robot's movement, forcing the operator to bear all perception, planning, and obstacle avoidance tasks, resulting in an extremely high cognitive load. When the robot senses a danger in the operator's blind spot through its sensors, such systems lack a mechanism to effectively feed this environmental perception back to the operator to proactively guide avoidance; furthermore, existing controllers suffer from severe distortion when interpreting operator intentions. Traditional control systems typically use the raw force / torque signal applied by the operator to the main controller directly as input, but this signal is mixed with interference signals. This signal aliasing causes the robot's motion trajectory to jitter and become uneven, making high-precision fine-tuning impossible. When existing systems attempt to introduce automated assistance, such as virtual grippers or obstacle avoidance guidance, human-robot control conflicts often arise, resulting in a stiff control feel for the operator, reducing work efficiency and safety.

[0004] To address the technical problem of distorted intent in operator input signals and conflicting control rights between human-machine guidance signals, a force control method integrating robot visual information is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a force control method that integrates robot visual information. By accurately analyzing the operator's pure intention and non-conflictingly integrating it with the robot's environmental perception, it provides a smooth, safe, and predictive guided robot collaborative control experience.

[0006] The specific solution is a force control method that integrates robot visual information, the method comprising: The robot receives and processes scene cognition information from its vision sensors, identifies key interactive objects including targets and obstacles through scene semantic segmentation, and establishes a multi-layer virtual force model based on the scene cognition information. The multi-layer virtual force model includes at least the attraction of the work object, the repulsion of the obstacle, and the constraint force of the image field of view, and generates a predictive guidance data stream. The actual joint position and applied manipulation force of the operator are received in real time via the program controller; the time-domain features of the manipulation force are extracted in real time, including torque amplitude, direction and rate of change; and a manipulation command recognition algorithm based on dual-stream parallelism is applied to analyze the operator's real-time interaction expectations to obtain the manipulation command vector. The predictive guidance data stream and the manipulation command vector are fused non-conflictingly; a shared motion trajectory is calculated; and a position tracking controller using a proportional-derivative feedforward algorithm combines the desired position, desired velocity, and desired acceleration of the shared motion trajectory with the actual joint position and actual joint velocity to output a trajectory tracking signal.

[0007] Preferably, the step of identifying key interactive objects, including targets and obstacles, through scene semantic segmentation specifically includes: The robot receives and processes scene cognition information from its vision sensors. This scene cognition information includes at least visual feedback data, 3D point cloud data of the environment, and dynamic optical flow data. The scene semantic segmentation is performed using MASKR-CNN to fuse the visual feedback data and the 3D point cloud data, perform pixel-level classification and boundary detection, and apply a Bayesian algorithm for confidence verification. The verified semantic labels and location coordinates are output. Based on the semantic labels and location coordinates, the robot distinguishes between the task target and obstacles and generates the target task path.

[0008] Preferably, the specific steps for generating the predictive guidance data stream include: Using the 3D coordinate data of the target operation path, a continuous reference path is constructed, and a gravity model based on virtual target points is established. The composite vector used for path adsorption and path forward movement is calculated to generate the gravity of the operation object. After filtering out the point cloud identified as the operation target from the 3D point cloud data, a clustering algorithm is applied to determine the obstacle clusters, and a repulsive artificial potential field function is established. Based on the closest distance between the end effector and the obstacle cluster, the repulsive vector that is inversely proportional to the distance within a safety threshold is calculated to generate the obstacle repulsive force. Using the dynamic optical flow data and the operation target offset recorded by the sensor, the disturbance compensation potential field and the field of view maintenance potential field are calculated respectively, merged into a composite potential field, and the gradient is calculated to generate the image field of view constraint force. By combining the gravitational force of the work object, the repulsive force of the obstacle, and the constraint force of the image field of view, a total virtual potential field function is constructed. By calculating the negative gradient of the function at the position of the end effector, the expected resultant force vector for guiding motion is obtained. The expected resultant force vector and the first derivative are output as a predictive guidance data stream.

[0009] Preferably, the step of receiving the operator's actual joint position and applied manipulation force in real time, and extracting temporal features, specifically includes: Using a six-axis force and torque sensor and a position encoder installed on the controller, the operator's manipulation input data is collected synchronously. The manipulation input data includes at least the six-degree-of-freedom joint position coordinates and the corresponding original force and torque vectors. Noise suppression processing is performed through a low-pass filter to obtain the actual joint positions and a smoothed manipulation force sequence. A time-domain feature extraction algorithm based on vector operation and numerical difference is applied to analyze the smoothed manipulation force sequence, calculate the torque amplitude, direction, and rate of change, and obtain the time-domain features. The physical dynamics of the controller itself, including at least gravity, inertial force, and friction, are calculated in real time to obtain the equipment compensation force vector. Online dynamic compensation is performed to subtract the equipment compensation force vector from the smoothed manipulation force sequence to obtain the pure intention force vector, which represents the operator's true interactive intention.

[0010] Preferably, the specific steps for obtaining the manipulation command vector include: The manipulation command recognition algorithm includes: a command type classification stream, which identifies the interaction expectation as a motion, fine-tuning, or position-holding command type by analyzing the transient patterns of the time-domain features; and a command parameter calculation stream, which extracts smooth raw motion parameters by fusing the actual joint position and the pure intention force vector and applying a Kalman filter to filter out the operator's physiological hand tremors and high-frequency noise. The dynamic calibration fusion module, based on the instruction type label, performs gating and nonlinear calibration processing on the smoothed raw motion parameters to generate a manipulation instruction vector; it dynamically adjusts the calibration parameters of the nonlinear calibration processing, specifically by dynamically and inversely adjusting the gain of the calibration parameters according to the magnitude of the obstacle repulsive force; and then dynamically deflects the response axis of the calibration parameters according to the direction of the gravitational force of the working object.

[0011] Preferably, the step of calculating the shared motion trajectory specifically includes: The non-conflict fusion is achieved through adaptive weight allocation. Using the desired resultant force vector, a virtual dynamics solver is used to calculate the desired velocity vector in real time. Combined with the manipulation command vector, a dynamic projection fusion algorithm is applied, using the magnitude of the desired resultant force vector as a dynamic weight to flexibly weight conflicting motion components, resulting in the fused target shared velocity. Using the fused target shared velocity and the actual joint position, multi-step prediction is performed to generate future motion path points in real time. Using the future motion path points as control points, a Bézier curve is constructed as the shared motion trajectory. After applying the minimization of abrupt change criterion to perform optimal time parameterization, the desired position, desired velocity, and desired acceleration of the trajectory are calculated.

[0012] Preferably, the specific steps for outputting the trajectory tracking signal include: Based on the desired position and the actual joint position, the position error is calculated in real time; the position error is multiplied by a proportional gain to obtain a proportional compensation term; based on the desired velocity and the real-time detected actual joint velocity, the velocity error is calculated in real time; the velocity error is multiplied by a differential gain to obtain a differential compensation term; the desired acceleration of the shared motion trajectory is multiplied by a feedforward gain to obtain a feedforward compensation term; the proportional compensation term, the differential compensation term, and the feedforward compensation term are weighted and summed to compensate for the position error and velocity error, and a trajectory tracking signal is generated; the magnitude of the trajectory tracking signal is continuously monitored, and a soft limit constraint is applied when it exceeds a preset force threshold.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By integrating visual feedback data and 3D point cloud data, this invention enables the robot to actively identify obstacles in the operator's blind spot and generate obstacle repulsion force by constructing a repulsive artificial potential field, which significantly improves the safety of remote control operation.

[0014] 2. By filtering out the physical dynamics of the main controller through online dynamic compensation and applying a Kalman filter to filter out physiological tremors, this invention can extract the operator's pure intention force vector and smooth the original motion parameters, making the source of control commands more accurate and stable, and greatly reducing misoperation.

[0015] 3. By converting the control commands into Bézier curves and applying the minimum jerk criterion to perform optimal time parameterization, the final executed shared motion trajectory is ensured to be continuous, smooth, and shock-free, eliminating the jitter commonly found in traditional control.

[0016] 4. This invention achieves a high degree of human-machine collaboration. It is no longer solely dominated by humans or machines, but rather deeply integrates AI's environmental perception with the operator's intent through dynamic projection and adaptive calibration. The algorithm optimization results in a 20% increase in frame processing speed and a reduction in response latency of less than 50ms. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of a force control method that integrates robot visual information according to the present invention. Figure 2 This is a schematic diagram of the predictive guidance data flow of the present invention; Figure 3 This is a flowchart illustrating the manipulation command vector of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figures 1 to 3 This invention provides a force control method that integrates robot visual information. The technical solution is as follows: A force control method that integrates robot visual information, referring to... Figure 1 The flowchart of the present invention includes: The robot receives and processes scene cognition information from its vision sensors, identifies key interactive objects including targets and obstacles through scene semantic segmentation, and establishes a multi-layer virtual force model based on the scene cognition information. The multi-layer virtual force model includes at least the attraction of the work object, the repulsion of the obstacle, and the constraint force of the image field of view, and generates a predictive guidance data stream. The actual joint position and applied manipulation force of the operator are received in real time via the program controller; the time-domain features of the manipulation force are extracted in real time, including torque amplitude, direction and rate of change; and a manipulation command recognition algorithm based on dual-stream parallelism is applied to analyze the operator's real-time interaction expectations to obtain the manipulation command vector. The predictive guidance data stream and the manipulation command vector are fused non-conflictingly; a shared motion trajectory is calculated; and a position tracking controller using a proportional-derivative feedforward algorithm combines the desired position, desired velocity, and desired acceleration of the shared motion trajectory with the actual joint position and actual joint velocity to output a trajectory tracking signal.

[0020] Example 1 This embodiment discloses a force control method that integrates robot visual information to achieve precise force control of the robot on target workpieces in complex environments.

[0021] First, scene perception information from the robot's vision sensor is received and processed. Specifically, the scene perception information includes at least: visual feedback data, 3D point cloud data of the environment, and dynamic optical flow data. For example, in this embodiment, an Intel RealSense D435i depth camera installed at the robot's end effector is used as the vision sensor. The visual feedback data is an RGB image stream with a resolution of 1280x720 pixels and a frame rate of 30fps. The 3D point cloud data is acquired by the depth camera and contains the (X,Y,Z) coordinates and RGB information of each pixel, with a point cloud density of approximately 300,000 points per frame. The dynamic optical flow data is calculated using the Lucas-Kanade sparse optical flow algorithm in the OpenCV library, tracking the motion vectors of 200 feature points in the image.

[0022] Key interactive objects, including targets and obstacles, are identified through scene semantic segmentation. Specifically, the scene semantic segmentation uses MASK R-CNN to fuse the visual feedback data and the 3D point cloud data, performing pixel-level classification and boundary detection, and applying a Bayesian algorithm for confidence verification, outputting verified semantic labels and location coordinates. Based on the semantic labels and location coordinates, the task target and obstacles are distinguished, and a target task path is generated.

[0023] Specifically, this embodiment employs a pre-trained MASK R-CNN model (based on a ResNet-101 backbone network). This model was initially trained on the COCO dataset and then transferred and fine-tuned on a custom dataset containing artifacts specific to this field and typical obstacles. The custom dataset contains 5000 labeled images, covering different lighting conditions, angles, and occlusion scenarios. During training, the learning rate was set to 0.001, the batch size to 2, and the training period to 50 epochs.

[0024] Key processes include: Data preprocessing: First, the input 3D point cloud and RGB data are fused and converted into a 3D voxel mesh with features; 3D Feature Extraction: Use a sparse 3D convolutional network (such as Sparse ResNet) to extract 3D features directly on this sparse voxel grid.

[0025] 3D target proposal: The 3D RPN (Region Proposal Network) slides on the 3D feature map to propose a series of 3D candidate bounding boxes (i.e. cuboids).

[0026] 3D Feature Pooling: 3D RoIAlign extracts these proposed cuboid regions from the 3D feature map and pools them into 3D feature blocks of a fixed size.

[0027] 3D Parallel Prediction: The prediction head performs three tasks simultaneously on these feature blocks: object classification, 3D bounding box regression (fine-tuning position and size), and generating voxel-level 3D masks.

[0028] Specifically, the scene semantic segmentation uses MASK R-CNN to fuse the visual feedback data and the 3D point cloud data, perform pixel-level classification and boundary detection, and apply a Bayesian algorithm for confidence verification. The verified semantic labels and location coordinates are output, and the target and obstacles are distinguished based on the semantic labels and location coordinates, and the target operation path is generated.

[0029] Furthermore, this embodiment employs a multi-task Mask R-CNN model, which extracts features from both RGB images and depth maps. The model is trained on a custom dataset containing assembly parts of various shapes (job targets) and common factory environment objects (obstacles). The dataset consists of 8000 pairs of RGB-D images with pixel-level annotations. Training parameters are set as follows: learning rate 0.0005, batch size 4, and training epochs of 80.

[0030] The visual feedback data (RGB image and depth map) are input in parallel into the Mask R-CNN dual-stream encoder to extract 2D visual features and 3D geometric features, respectively. The feature maps are fused in the FPN (Feature Pyramid Network) and then fed into the RPN and ROI Align layers.

[0031] For example, for a "circular pad" to be picked up, Mask R-CNN outputs its precise pixel-level mask on the RGB image and depth map, as well as its 3D bounding box, with an initial confidence level of 0.90.

[0032] The confidence verification process of the Bayesian algorithm is as follows: A discrete Bayesian filter is established, whose state space includes "target A exists", "target B exists", "obstacle C exists", etc. Whenever Mask R-CNN outputs a new recognition result, it is used as the likelihood function, and combined with the posterior probability of the previous time step to update the posterior probability of the current time step.

[0033] For example, if 10 consecutive frames of images identify "circular pad" with a confidence level higher than 0.8, and its three-dimensional position changes within ±2mm, the Bayesian filter updates the posterior probability of the existence of "circular pad" to 0.98, and outputs the verified semantic label "circular pad" and the precise three-dimensional position coordinates.

[0034] Based on the verified semantic tags, this embodiment identifies a "circular gasket" as the target of the operation. A "screwdriver," a "wrench," and a "cable harness" are identified as obstacles.

[0035] The target task path is generated using a sampling-based RRT* algorithm. This algorithm plans an optimal path from the robot arm's current pose to the "circular pad" pickup pose, taking into account the robot arm's kinematic constraints, self-collision, and environmental obstacles, ensuring the path length is minimized and the path is smooth. The path planner considers the robot arm's seven degrees of freedom and outputs a series of path points including joint angles and end-position poses.

[0036] The visual feedback data (RGB image) is used as input to MASK R-CNN to generate preliminary pixel-level semantic segmentation masks and bounding boxes. Simultaneously, the 3D point cloud data is projected onto the 2D image plane and fused with the output of MASK R-CNN at the feature level. The fusion strategy employs a late-fusion approach, where the 2D semantic segmentation result output by MASK R-CNN is correlated with the 3D point cloud data, utilizing the depth information of the point cloud to perform a precise 3D spatial mapping of the 2D segmentation result.

[0037] For example, for a identified bolt (the target of the task), MASK R-CNN outputs its pixel-level mask and bounding box in the image with a confidence level of 0.92. Simultaneously, using point cloud data, the bolt's 3D position coordinates and pose information in the robot coordinate system can be accurately obtained. For an obstacle identified as a "toolbox," the confidence level is 0.88, and its 3D position and size information are obtained.

[0038] The Bayesian algorithm described is used to verify the confidence level of the MASK R-CNN output. Specifically, the Bayesian classifier uses the recognition result of the previous frame as the prior probability and combines it with the recognition probability of the current frame (output by MASK R-CNN) to calculate the posterior probability. For example, if the same target is recognized in 5 consecutive frames, and the confidence level of each recognition is higher than 0.8, the final verified semantic label and location coordinates are considered valid, and the confidence level can be further improved to above 0.95.

[0039] Based on validated semantic tags, such as "bolt" being labeled as the target, and "toolbox" and "wire" being labeled as obstacles, the RRT algorithm is used to plan a collision-free path from the robot's end effector to the target object, according to the target type and the robot's current position. For example, a robot movement path is planned that avoids the toolbox and wire, with path points spaced 5mm apart.

[0040] By fusing visual image data and 3D point cloud geometric data, the problem of single sensor failure in harsh environments is overcome, providing a stable and accurate data source of obstacles and targets for subsequent virtual force models.

[0041] Furthermore, the specific steps for generating the predictive guidance data stream include: constructing a continuous reference path using the three-dimensional coordinate data of the target operation path, establishing a gravity model based on virtual target points, calculating the composite vector used for path adsorption and path forward movement, and generating the gravity of the operation object.

[0042] Specifically, in this embodiment, the target operation path (a series of three-dimensional coordinate points) is interpolated into a smooth B-spline curve as a continuous reference path. The gravity model is configured to place a virtual target point on the path, which leads the robot's end effector at a preset speed along the continuous reference path. The robot's end effector is subjected to an attractive force pointing towards the virtual target point. For example, the gravity gain is set to 50 N / m; the virtual target point leads the robot's end effector by 0.1 meters along the path. The composite vector combines the requirements of path attraction and path forward movement, ultimately generating a gravitational force on the target object, with a magnitude ranging from 0 to 10 N.

[0043] After filtering out point clouds identified as work targets from the 3D point cloud data, a clustering algorithm is applied to determine obstacle clusters. A repulsive artificial potential field function is established, and a repulsive force vector, inversely proportional to the distance within a safety threshold, is calculated based on the closest distance between the end effector and the obstacle cluster, generating obstacle repulsion force. Specifically, in this embodiment, point cloud data identified as work targets (e.g., bolts) are first removed from the acquired 3D point cloud data. The remaining point cloud data is then clustered into different obstacle clusters (e.g., toolboxes, wires) using the DBSCAN clustering algorithm with parameters set to eps=0.05 meters and min_samples=10. For each obstacle cluster, a repulsive artificial potential field function is established. When the distance between the end effector and the obstacle is less than the safety threshold, a repulsive force is generated. For example, the repulsive force gain is set to... The safety threshold is set to 0.1 meters. If the end effector is 0.08 meters away from the toolbox, a repulsive force will be generated pointing away from the toolbox, the magnitude of which is inversely proportional to the distance. The magnitude of the obstacle repulsive force ranges from 0 to 8 N.

[0044] Using dynamic optical flow data and target offset recorded by sensors, the disturbance compensation potential field and the field-of-view maintenance potential field are calculated separately, merged into a composite potential field, and the gradient is calculated to generate the image field-of-view constraint force.

[0045] Specifically, the disturbance compensation potential field utilizes the acquired dynamic optical flow data to detect minute movements of the target or environmental disturbances in real time. For example, if the optical flow data shows that the target image is moving to the left in the X direction at a rate of 5 pixels / frame, a potential field is calculated to drive the end effector to move to the right to compensate for this offset. The field-of-view maintaining potential field maintains the target in the center region of the image field of view based on the offset of the target in the image recorded by the sensor (depth camera). For example, if the target deviates from the center of the image by 50 pixels, a potential field is calculated to drive the end effector to adjust its pose and bring the target back to the center region. The disturbance compensation potential field and the field-of-view maintaining potential field are combined into a composite potential field. The image field-of-view constraint force is obtained by calculating the negative gradient of the composite potential field. For example, the magnitude of the image field-of-view constraint force ranges from 0 to 5N.

[0046] By combining the gravitational force of the work object, the repulsive force of the obstacle, and the constraint force of the image field of view, a total virtual potential field function is constructed. The desired resultant force vector for guiding motion is obtained by calculating the negative gradient of this function at the position of the end effector. This desired resultant force vector and its first derivative are then output as a predictive guidance data stream. Specifically, the total virtual potential field function is composed of the potential field functions corresponding to the gravitational force of the work object, the repulsive force of the obstacle, and the constraint force of the image field of view. The desired resultant force vector for guiding motion is obtained by calculating the negative gradient of the total virtual potential field function at the position of the end effector. For example, at a specific moment, the calculated gravitational force of the work object is (3.0, 1.0, 0.5) N, the repulsive force of the obstacle is (-0.8, -0.2, 0.0) N, and the constraint force of the image field of view is (0.1, -0.3, 0.0) N. Then the desired resultant force vector is (2.3, 0.5, 0.5) N.

[0047] Simultaneously, the first derivative of the desired resultant force vector (i.e., the desired torque) is calculated as part of the predictive guidance data stream; for example, the desired torque is (0.1, 0.05, 0.02) Nm. The desired resultant force vector and its first derivative are output at a frequency of 100 Hz as part of the predictive guidance data stream and transmitted to the robot's force controller. The force controller receives the predictive guidance data stream and outputs the robot joint torque through an impedance control or admittance control algorithm. For example, this embodiment uses an impedance control algorithm. The desired impedance model is set as follows: the difference between the desired force and the actual force is related to the acceleration, velocity, and position under the desired mass, desired damping, and desired stiffness. Specifically, the desired mass matrix is ​​set as a diagonal matrix diag(5,5,5) kg; the desired damping matrix is ​​set as a diagonal matrix diag(100,100,100) Ns / m; and the desired stiffness matrix is ​​set as a diagonal matrix diag(1000,1000,1000) N / m.

[0048] The force controller calculates the required joint torque based on the desired resultant force vector, the actual force, and the current robot state. This joint torque is then sent to the robot's joint servo system to drive the robot's movement. For example, the calculated joint torque might be (10, 8, 12, 5, 3, 2) Nm. The robot moves according to this joint torque and, during movement, interacts with the target object to achieve force-controlled operation. For instance, under force control, the robot's end effector moves above a bolt and then slowly descends. When the end effector force sensor detects that the contact force with the bolt reaches a preset threshold (e.g., 2 N), tightening begins. During tightening, the torque sensor monitors the tightening torque in real time. When the torque reaches a set value (e.g., 5 Nm), the operation stops, and the bolt tightening is complete.

[0049] Specifically, in this embodiment, the target operation path is generated into a continuous reference path through cubic spline interpolation.

[0050] The gravity model employs a variable stiffness gravitational field: the gravitational stiffness is low when the end effector is far from the virtual target point, and increases when it is close to the virtual target point.

[0051] For example, the maximum gravitational gain is set to 80 N / m, and the virtual target point is 0.05 meters ahead of the robotic arm's end effector on the path. The gravitational force of the working object ranges from 0 to 15 N.

[0052] After filtering out point clouds identified as operational targets from 3D point cloud data, a clustering algorithm is applied to determine obstacle clusters, a repulsive artificial potential field function is established, and a repulsive force vector that is inversely proportional to the distance within a safe threshold is calculated based on the closest distance between the end effector and the obstacle cluster, thereby generating obstacle repulsive force.

[0053] Specifically, in this embodiment, the K-means clustering algorithm is used to cluster the 3D point cloud data after removing the target point cloud. The number of clusters K is dynamically adjusted according to the complexity of the scene.

[0054] The artificial potential field function for repulsion has a specific form, in which the repulsive force increases rapidly as it approaches a safety threshold. For example, the repulsive force gain is set to 15 N*m^2, and the safety threshold is set to 0.12 meters. The magnitude of the obstacle repulsive force ranges from 0 to 12 N.

[0055] Using dynamic optical flow data and target offset recorded by sensors, the disturbance compensation potential field and the field-of-view maintenance potential field are calculated separately, merged into a composite potential field, and the gradient is calculated to generate the image field-of-view constraint force.

[0056] Furthermore, the perturbation compensation potential field: using dynamic optical flow data, if a global displacement of the target is detected in the image, for example, if the average optical flow vector shows the target moving at a speed of (3 pixels / frame, -2 pixels / frame), then a compensation force opposite to this direction of motion is calculated, the magnitude of which is proportional to the optical flow velocity. For example, the gain is set to 0.1 N / (pixel / frame).

[0057] Field-of-view preservation potential: Define a central region of the image (e.g., 100x100 pixels). If the center pixel coordinates of the target object extend beyond this region, a potential field is calculated. The image field-of-view constraint is obtained by calculating the negative gradient of the composite potential field. For example, the magnitude of the image field-of-view constraint ranges from 0 to 7N.

[0058] By combining the gravitational force of the work object, the repulsive force of the obstacle, and the constraint force of the image field of view, a total virtual potential field function is constructed. By calculating the negative gradient of the function at the position of the end effector, the expected resultant force vector for guiding motion is obtained. The expected resultant force vector and the first derivative are output as a predictive guidance data stream.

[0059] Specifically, the overall virtual potential field function is composed of the potential field functions corresponding to the attraction of the task object, the repulsion of obstacles, and the constraint force of the image field of view. The weight coefficients can be dynamically adjusted according to the task priority. For example, the weight of the attraction of the task object increases when approaching the target, and the weight of the repulsion of the obstacle increases when near the obstacle.

[0060] The desired resultant force vector is obtained by calculating the negative gradient of the overall virtual potential field function at the location of the end effector. The desired torque is calculated using the desired resultant force vector and the Jacobian matrix of the robotic arm.

[0061] The output frequency of the predictive guidance data stream is 200Hz.

[0062] Transforming complex visual scene analysis into an intuitive force field, and providing predictive data for downstream controllers by innovatively outputting the desired resultant force vector and its first derivative (rate of change), is a prerequisite for achieving high-precision tracking and shock-free motion.

[0063] The steps for receiving the operator's actual joint position and applied manipulation force in real time and extracting temporal features are further described in detail.

[0064] Further, the steps specifically include: synchronously acquiring the operator's manipulation input data using a six-axis force / torque sensor and position encoder mounted on the controller. The manipulation input data includes at least the six-degree-of-freedom joint position coordinates and the corresponding raw force / torque vectors. Noise suppression processing is performed using a low-pass filter to obtain the actual joint positions and a smooth manipulation force sequence. Specifically, this embodiment uses a KUKALBRiiwa14R820 lightweight robot as the robot. Its controller integrates an ATII Industrial Automation Gamma series six-axis force / torque sensor, which is mounted at the robot's end effector. This sensor can measure the force in the XYZ axes and the torque in the three directions around the XYZ axes in real time. Simultaneously, each joint of the robot is equipped with a high-precision position encoder to acquire the six-degree-of-freedom joint position coordinates generated in real time when the operator drags the robot via a teach pendant or directly.

[0065] The sensor and encoder synchronously acquire data at a frequency of 1000Hz. For example, at a certain moment, the sensor acquires the raw force-torque vector, while the encoder simultaneously obtains the six-degree-of-freedom joint position coordinates. The raw force-torque vector is processed for noise suppression using a second-order Butterworth low-pass filter with a cutoff frequency of 20Hz. After filtering, a smoothed manipulation force sequence is obtained, and the actual joint position is the six-degree-of-freedom joint position coordinate directly read by the encoder.

[0066] A time-domain feature extraction algorithm based on vector operations and numerical differences is applied to analyze the smoothed control force sequence, calculating the torque amplitude, direction, and rate of change to obtain the time-domain features. Specifically, real-time analysis of the obtained smoothed control force sequence includes: The torque amplitude is obtained by calculating the Euclidean norm of the torque vector; the direction is determined by calculating the unit vector of the torque vector. For example, a six-dimensional unit vector is calculated to represent the direction of the force applied by the operator; the rate of change is calculated by performing a first-order numerical difference on a smoothed manipulation force sequence to calculate the rate of change of the torque vector (i.e., the derivative of the force).

[0067] The time-domain characteristics, serving as a preliminary representation of the operator's intent, are output at a frequency of 50Hz. For example, the output includes the current total torque amplitude, the torque direction unit vector, and the torque rate of change vector. The controller's own physical dynamics, including at least gravity, inertial forces, and friction, are calculated in real time to obtain the device compensation force vector.

[0068] Specifically, the controller in this embodiment stores a complete dynamic model of the robot, which includes the mass, center of mass position, inertia tensor, and joint friction parameters of each link of the robot.

[0069] The real-time solution process includes: Gravity is calculated based on the robot's current joint position coordinates, determining the joint torque generated by each link under the influence of gravity. For example, if the robot is currently in an extended position, the gravitational force on its end effector will be relatively large. Inertial force is calculated based on the robot's current joint acceleration (obtained by numerical difference between joint position and velocity), and the joint torque generated by the robot's link inertia is calculated. Friction force is calculated based on the robot's current joint velocity using a pre-calibrated friction model (e.g., Coulomb friction and viscous friction models). These calculated joint moments are superimposed and transformed into the end effector coordinate system to obtain the device compensation force vector. For example, when the robot is stationary in the air, the device compensation force vector mainly reflects the equivalent effect of gravity on the end effector.

[0070] Furthermore, online dynamics compensation is performed by subtracting the device compensation force vector from the smoothed manipulation force sequence to obtain the pure intention force vector, which represents the operator's true interactive intention. Specifically, this embodiment performs frame-by-frame online dynamics compensation. The obtained smoothed manipulation force sequence is subtracted from the obtained device compensation force vector to obtain the pure intention force vector. For example, if the operator is dragging the robot upwards, the smoothed manipulation force sequence includes the force applied by the operator and the force resisting gravity. After subtracting the device compensation force vector, which mainly contains the component resisting gravity, the obtained pure intention force vector will more accurately reflect the operator's true intention to move the robot upwards. This pure intention force vector is output at a frequency of 50Hz and, as the operator's true interactive intention, is transmitted to the subsequent intention recognition module or control system.

[0071] By using online dynamic compensation, the operator's pure intention force vector is extracted from the original force signal, solving the core problem of intention being contaminated by the main controller's own physical dynamics (gravity, inertia), and greatly improving the fidelity of control.

[0072] Furthermore, the step of obtaining the manipulation command vector specifically includes: Specifically, in this embodiment, the command type classification flow uses a classifier based on support vector machine (SVM) to perform real-time analysis on the extracted time-domain features (torque amplitude, direction, rate of change). This classifier is pre-trained and can identify different transient patterns.

[0073] For example, when a sustained large torque amplitude is detected and the rate of torque change is stable, it is classified as a "motion" instruction type. For example, an operator pushes the robot with a sustained force of 5N.

[0074] When a small torque amplitude, high-frequency vibration pattern, or periodic minute torque change is detected, it is categorized under the "fine-tuning" instruction type label. For example, the operator makes a fine adjustment with a force of less than 1N.

[0075] When the detected torque amplitude is close to zero and the rate of torque change is very small, it is classified as a "position hold" instruction type. For example, the operator simply holds the robot without applying any significant force.

[0076] The instruction type label is output at a frequency of 50Hz.

[0077] In this embodiment, the command parameter calculation stream integrates the actual joint positions and the pure intention force vector, and is processed through an Extended Kalman Filter (EKF). The state vector of the EKF includes the position and velocity of the robot's end effector. The observation vector consists of the actual joint positions (converted to end effector position via forward kinematics) and the pure intention force vector.

[0078] For example, when an operator applies a force, the pure intention force vector represents the operator's desired motion trend. EKF uses the robot's dynamic model as the state transition equation and combines it with observation data to estimate the actual motion state of the end effector in real time.

[0079] The Kalman filter can effectively filter out physiological high-frequency tremors (e.g., small force fluctuations with a frequency higher than 10 Hz) generated by the operator's hand when applying force, as well as high-frequency noise introduced by sensors and the environment.

[0080] For example, after filtering, smoothed raw motion parameters, including the desired linear and angular velocities of the end effector, are extracted from the operator's intended force. These smoothed raw motion parameters are output at a frequency of 50 Hz.

[0081] Furthermore, when analyzing the operator's immediate interaction expectations, an operator uncertainty detection mechanism based on pure intention force vector frequency domain analysis is introduced. When the operator hesitates or is uncertain, the force applied to the controller is not completely static, but rather manifests as a "high-frequency, low-amplitude stiffness," which is physically different from the completely relaxed "position holding" and also different from the physiological random "hand tremor."

[0082] Specifically, the implementation steps of this detection mechanism include: after obtaining the pure intention force vector through high-frequency energy analysis, the system does not immediately use it for motion calculation, but instead applies time-frequency analysis, such as short-time Fourier transform or wavelet transform, to it in parallel.

[0083] The uncertainty feature extraction algorithm specifically monitors the energy level of the pure intention force in a specific high-frequency band (i.e., the hesitation frequency). When the system detects that the energy in this high-frequency band is consistently higher than a preset threshold, while the low-frequency motion mean of the force vector (i.e., the overall motion trend) is close to zero, the system no longer incorrectly identifies it as a position-holding command. Instead, the system identifies this state as a completely new and independent command state: intention uncertainty.

[0084] Once the "intention uncertainty" state is triggered, this invention will automatically correct the logic for generating the shared motion trajectory. At this time, the system will neither maintain position nor perform motion, but will automatically and significantly increase the fusion weight of the attraction of the work object and the repulsion of obstacles in the predictive guidance data stream. In this way, the problem of intention distortion caused by misidentifying "hesitation" as "holding" is solved, and moreover, at the moment when the operator is most uncertain and needs help the most, control is seamlessly handed over to the robot guide, realizing a higher level of human-machine intelligent collaboration.

[0085] The dynamic calibration fusion module, based on the instruction type label, performs gating and nonlinear calibration processing on the smoothed raw motion parameters to generate a manipulation instruction vector. Specifically, in this embodiment, the dynamic calibration fusion module performs different processing on the smoothed raw motion parameters according to the instruction type label. When the instruction type label is "motion," larger raw motion parameters are allowed to pass, and linear or slightly nonlinear calibration is performed to achieve rapid robot movement. When the instruction type label is "fine-tuning," the raw motion parameters are gating, allowing only extremely small motion parameters to pass, and high-gain nonlinear calibration is applied to achieve fine, slow robot movement. For example, the original desired speed is limited to within 5 mm / s. When the instruction type label is "position hold," the raw motion parameters are set to zero, and the robot maintains its current position.

[0086] The nonlinear calibration process can adjust the gain of the output parameters according to the magnitude of the input parameters. For example, when the input speed is low, the output speed is further reduced to improve accuracy; when the input speed is high, the output speed can be proportionally amplified to improve response speed. Through this process, a six-degree-of-freedom manipulation command vector is generated, containing the desired linear and angular velocities of the end effector.

[0087] The calibration parameters of the nonlinear calibration process are dynamically adjusted, specifically by dynamically and inversely adjusting the gain of the calibration parameters according to the magnitude of the obstacle's repulsive force; and by dynamically deflecting the response axis of the calibration parameters according to the direction of the object's gravitational force. Specifically, this embodiment dynamically adjusts the parameters of the nonlinear calibration process in real time, dynamically and inversely adjusting the gain of the calibration parameters according to the magnitude of the obstacle's repulsive force: when the obstacle's repulsive force is greater (i.e., the closer the robot is to the obstacle), the gain of the nonlinear calibration process will dynamically decrease. For example, if the obstacle's repulsive force increases from 1N to 5N, the calibration gain used for speed output will decrease from 1.0 to 0.5. This means that when approaching an obstacle, even if the operator applies the same intentional force, the robot's actual movement speed will slow down, thereby increasing the operator's reaction time and collision avoidance time.

[0088] The response axis of the calibration parameters is dynamically deflected based on the direction of the object's gravitational pull: when the object's gravitational pull has a definite direction, the response axis of the nonlinear calibration process will dynamically deflect. For example, if the object's gravitational pull is mainly downward along the Z-axis (perpendicular to the worktable surface), the system will slightly enhance the motion response in the Z-axis direction or slightly suppress excessive motion in the XY plane when the operator makes fine adjustments in the XY plane. This deflection makes the robot's motion response more inclined to follow the direction guided by the target, thereby assisting the operator to more accurately approach and manipulate the target.

[0089] The manipulation command vector is output at a frequency of 50Hz and is transmitted to the subsequent force control module as the robot's desired motion command.

[0090] Robust filtering of physiological tremors is achieved through a dual-flow architecture, and the operator's control gain and response axis are dynamically adjusted through AI's environmental perception (repulsion and attraction), realizing efficient human-machine collaboration between virtual damping field and virtual guide slot.

[0091] Further, the specific steps for calculating the shared motion trajectory and output trajectory tracking signal are described. The non-conflict fusion is achieved through adaptive weight allocation. Using the desired resultant force vector, a virtual dynamics solver is used to calculate the desired velocity vector in real time. Combined with the manipulation command vector, a dynamic projection fusion algorithm is applied, using the magnitude of the desired resultant force vector as a dynamic weight to flexibly weight the conflicting motion components, thus obtaining the fused target shared velocity.

[0092] Specifically, the following operations are performed in this embodiment: Virtual Dynamics Solver: The desired resultant force vector is input into a virtual dynamics solver. This solver simulates a mechanical system with virtual mass and damping, and calculates its desired linear velocity and angular velocity in the end effector coordinate system, i.e., the desired velocity vector, based on the input force.

[0093] Furthermore, the dynamic projection fusion algorithm fuses the manipulation command vector (operator's intention) with the aforementioned desired velocity vector (system guidance). The fusion algorithm achieves adaptive weight allocation through the following method: The magnitude of the desired resultant force vector is used as a dynamic weight, and an adaptive weight allocation mechanism based on "guidance urgency" is introduced. Specifically, this magnitude itself is the optimal physical quantification of the system's guidance urgency. The system maps "urgency" to a minimum. When the system only exerts a weak gravitational force on the work object, the magnitude is small, indicating low urgency, and the system should prioritize respecting the operator's intentions. When the system generates a strong repulsive force from an obstacle, the magnitude is large, indicating extremely high urgency, and the system must prioritize executing the AI's obstacle avoidance commands. The system presets a "maximum guiding force threshold" and a "minimum sensing force threshold." Using the magnitude of the expected resultant force vector, a nonlinear mapping function, such as the sigmoid function or a piecewise saturation function, is used to calibrate and calculate a dimensionless system guiding weight between 0 and 1. The operator's weight is dynamically set to the complement of this system guiding weight. In this way, the control dominance between the human and the machine achieves a dynamic and smooth transition, automatically and smoothly ignoring the operator's manipulation commands, and forcibly ensuring the robot's safety without causing control jitter.

[0094] For conflicting motion components in the maneuver command vector and the desired velocity vector (e.g., the operator intends to move left, while the system guides them to move right), a flexible weighting is applied. Conflict detection is performed by calculating the dot product of the two velocity vectors; if the dot product is negative, a conflict is considered to exist. In the case of conflict, the fused target shared velocity is used as a preset upper and lower limit for weighting, which is then used for normalization. In this way, when the system guidance force (desired resultant force) is strong, the weight of the system guidance will increase accordingly, and vice versa. For example, the weight of the system guidance will increase during obstacle avoidance, and the weight of the operator's intention will be higher during free space operations.

[0095] The final target shared velocity is obtained after fusion, and its frequency is the same as that of the manipulation command vector and the desired velocity vector, for example, 50Hz.

[0096] Furthermore, using the fused target shared velocity and the actual joint position, multi-step prediction is performed to generate future motion path points in real time. Specifically, this embodiment employs a predictive control-based algorithm. The fused target shared velocity is used as input, combined with the actual joint position (converted to the end effector position through inverse kinematics).

[0097] Furthermore, the multi-step prediction process is as follows: Starting from the current end effector pose and using the shared velocity as the velocity command, the robot end effector pose is predicted several steps ahead (e.g., 50 steps, with a step time interval of 10ms). Each prediction step considers the robot's kinematic and dynamic constraints, generating a series of discrete future motion path points. For example, a series of three-dimensional coordinates and pose points that the robot end effector will pass through within the next 0.5 seconds are predicted. These future motion path points are updated at a frequency of 50Hz.

[0098] Using the future motion path points as control points, a Bézier curve is constructed as a shared motion trajectory. After applying the minimum abrupt change criterion to perform optimal time parameterization, the expected position, expected velocity, and expected acceleration of the trajectory are calculated.

[0099] Specifically, this embodiment uses future path points as control points for the Bézier curve. For example, five predicted path points are used as control points for a fourth-order Bézier curve. A smooth, continuous Bézier curve is constructed using these control points as a shared motion trajectory. The Bézier curve is time-parameterized to minimize the jerk (derivative of acceleration) as the robot moves along the trajectory. This helps ensure the smoothness of the robot's motion and reduces shocks and vibrations. For example, an optimization algorithm is used to adjust the time allocation of each trajectory segment to meet the minimum jerk constraint. Finally, based on the time-parameterized Bézier curve, the desired position, desired velocity, and desired acceleration of the trajectory are calculated at each time point. For example, the desired position, desired velocity, and desired acceleration are calculated every 1 ms within the next 50 ms, and these desired values ​​are output at a frequency of 100 Hz.

[0100] The conflict between human and machine control was resolved by using a dynamic projection fusion algorithm in a flexible weighted manner. By applying the minimization maneuver criterion, the final shared motion trajectory was ensured to be mathematically continuous, smooth, and shock-free.

[0101] To address the extreme situation of human-machine "control conflict," namely the continuous conflict and stalemate caused by the operator's insistence on resisting AI guidance, this invention adds a human-machine conflict arbitration and escalation mechanism after the fusion module.

[0102] Specifically, the implementation steps of this arbitration mechanism include: the arbitration layer monitors in real time the magnitude of the conflict components calculated and discarded by the dynamic projection fusion algorithm, i.e., the adversarial force between the operator's intention and the AI ​​guidance. The system integrates this adversarial force over time to generate a conflict accumulation index. When the conflict accumulation index exceeds a preset arbitration threshold (i.e., the system determines that the flexible fusion has failed and the human and machine are in a continuous conflict), the arbitration layer immediately executes the conflict escalation protocol.

[0103] The conflict escalation protocol includes: immediately gating the output to force the desired velocity of the shared motion trajectory to zero, bringing the robot to a safe stop; immediately commanding kinematic navigation to stop outputting smooth resistance feedback and instead output a high-frequency, high-amplitude "warning vibration"; the system remains in a frozen and warning state until the operator's pure intention force vector is detected to return to zero, i.e., after the operator releases the device and confirms the warning, the arbitration layer resets the conflict accumulation index and returns control to the flexible fusion mode. This resolves the persistent deadlock of "control conflict," provides a fail-safe mechanism, and avoids the system remaining in a high-stress state under high-force human-robot confrontation. Instead, it arbitrates the conflict, forcing the system into a "zero-motion, high-alert" safety mode, compelling the operator to reassess the situation, and ensuring the ultimate safety of the system.

[0104] Furthermore, the specific steps for outputting the trajectory tracking signal include: calculating the position error in real time based on the desired position and the actual joint position; and multiplying the position error by the proportional gain to obtain the proportional compensation term.

[0105] Specifically, this embodiment will determine the desired trajectory position. Compared with the real-time acquired actual joint position (the actual end position obtained through forward kinematics solution) By comparing the results, the position error can be obtained. For example, if The value is (0.5, 0.2, 0.3) m. If the value is (0.49, 0.21, 0.3) m, then the position error is... m; then the position error Multiply by the preset proportional gain Obtain proportional compensation item .For example, If set to 500 N / m, then N.

[0106] Based on the desired velocity and the actual joint velocity detected in real time, the velocity error is calculated in real time; the velocity error is multiplied by the differential gain to obtain the differential compensation term. Specifically, this embodiment calculates the desired trajectory velocity. The actual joint speed of the robot is detected in real time (the actual end-effector speed obtained through differential joint encoder or speed sensor). By comparing the speeds, the speed error can be obtained. For example, if The value is (0.01, 0.005, 0.002) m / s. If the velocity is (0.012, 0.004, 0.002) m / s, then the velocity error is... =(-0.002, 0.001, 0) m / s. Then, the speed error... Multiply by the preset differential gain The differential compensation term is obtained. .For example, If set to 50 Ns / m, then =(-0.1,0.05,0)N.

[0107] The expected acceleration of the shared motion trajectory is multiplied by the feedforward gain to obtain the feedforward compensation term. Specifically, in this embodiment, the expected acceleration of the shared motion trajectory is used... Multiply by the preset feedforward gain The feedforward compensation term is obtained. For example, if The result is (0.1, 0.05, 0.01). , If set to 10kg, then =(1,0.5,0.1)N.

[0108] The proportional compensation term, the differential compensation term, and the feedforward compensation term are weighted and summed to compensate for the position error and velocity error, generating a trajectory tracking signal. The magnitude of the trajectory tracking signal is continuously monitored, and a soft limit constraint is applied when it exceeds a preset force threshold. Specifically, in this embodiment, the three compensation terms obtained above are weighted and summed to generate the final trajectory tracking signal. .For example, =(5-0.1+1,-5+0.05+0.5N,0+0+0.1)N=(5.9,-4.45,0.1)N.

[0109] The trajectory tracking signal F_track is output to the robot's underlying torque controller at a frequency of 100Hz.

[0110] Simultaneously, the magnitude of the trajectory tracking signal is continuously monitored. If the magnitude exceeds a preset force threshold (set to 10N) at any given moment, a soft limit constraint is applied. Specifically, when the signal exceeds the threshold, the original signal is no longer directly output; instead, it is limited to a maximum allowable value (e.g., 15N) or proportionally reduced to prevent the robot from generating excessive force or speed, thus protecting the safety of both the robot and the operator. For example, if the calculated magnitude of the trajectory tracking signal is 12N, exceeding the 10N threshold, the actual signal magnitude output to the controller will be limited to 10N.

[0111] The proportional-derivative feedforward controller used can accurately and stably track the planned shared motion trajectory, while the soft limit constraint also ensures that the output torque is always within the safe threshold, which complies with the robot's safety specifications.

[0112] This embodiment details a trajectory generation and tracking method for robot shared control. The core of this method lies in using a non-conflicting fusion algorithm to adaptively weight and fuse the operator's manipulation commands with the desired velocity calculated by the system based on the expected resultant force, generating a fused target shared velocity. Then, through multi-step prediction, a series of future motion path points are generated in real time. These path points are used as control points for Bézier curves, constructing a smooth shared motion trajectory. Optimal time parameterization is achieved through minimizing abrupt changes, ultimately calculating the desired position, velocity, and acceleration of the trajectory. In the trajectory tracking stage, the system compares the desired trajectory parameters with the actual robot state, calculates the position error, velocity error, and feedforward compensation term, and sums them with weights to generate the final trajectory tracking signal. This signal is continuously monitored for magnitude, and soft limit constraints are applied when it exceeds a preset force threshold to ensure the safe and stable operation of the robot. The entire process achieves smooth, safe, and efficient robot motion control under human-machine collaboration, making it particularly suitable for scenarios requiring precise operation and intelligent obstacle avoidance.

[0113] Example 2

[0114] The collaborative robot in this embodiment needs to work with human operators to complete precise assembly tasks, such as inserting an irregularly shaped part into a narrow slot while avoiding other obstacles and tools in the work area, and predicting and responding to the operator's potential intentions; this scenario places higher demands on the adaptability and intelligence of the shared control system.

[0115] Firstly, multi-sensor fusion is no longer limited to force sensors and joint encoders, but integrates visual sensors (RGB-D cameras), LiDAR, and tactile sensors; using SLAM technology to combine visual and LiDAR data, a high-precision 3D environment map is built in real time, and obstacles are dynamically identified and updated, including static obstacles, dynamic obstacles such as moving vehicles and human operators themselves.

[0116] Deep learning models are introduced for human posture recognition and prediction. Visual sensors are used to identify the operator's skeletal key points and posture in real time, and predict the operator's movement trend and contact area in the next 5 seconds, which are then added to the planning as a higher-level dynamic obstacle avoidance area. Furthermore, multimodal manipulation command fusion and intent prediction are introduced: Voice command input involves adding a natural language processing module, allowing operators to issue high-level commands via voice, such as "grab this screw," "move to the left a little," and "stop." These commands will be parsed and converted into corresponding manipulation command vectors or target points. Gesture recognition uses a visual sensor to identify specific gestures of the operator (e.g., pointing in a certain direction, clenching a fist to indicate pause) and uses them as a supplement to or replacement for manipulation commands.

[0117] Furthermore, based on deep learning intention prediction, a deep learning model LSTM is trained using historical operation data and multimodal information such as current environmental state, operator posture, and voice / gesture commands to predict the operator's next operation intention in real time, such as preparing to insert a part, adjusting the angle, or taking away a tool.

[0118] When there is a conflict between different modalities of manipulation commands or predicted intentions, such as a voice command saying "left" but a gesture pointing "forward", a hierarchical conflict resolution mechanism is introduced; for example, explicit voice commands have the highest priority, followed by predicted recent intentions, and finally lower-level manipulation command vectors.

[0119] The robot's end effector carries parts of varying masses and geometries. The system needs to estimate the mass and inertia of the current load in real time and dynamically adjust the parameters of the virtual dynamics solver to maintain consistent handling feel and response characteristics. When the robot comes into contact with the environment or parts, it accurately determines the type of contact, such as sliding, rolling, or pressing, using data from tactile and force sensors. Based on the contact state, it adaptively switches force control strategies: position control is the primary method in free space, while force-based impedance control or hybrid force-position control is used during contact assembly to achieve a compliant assembly process.

[0120] To address minor vibrations and jamming during assembly, more advanced vibration suppression algorithms and compliance adjustment mechanisms are introduced. For example, by adjusting virtual damping and stiffness parameters, the robot becomes more "soft" and "adaptive" during assembly.

[0121] Furthermore, task-level shared trajectory planning and optimization: Based on intent prediction and environmental understanding, the system can automatically plan longer shared motion trajectories containing multiple sub-tasks from discrete action instructions such as "grab", "move", and "insert".

[0122] Furthermore, when generating Bézier curves, instead of simply minimizing abrupt movements, multi-objective optimization is considered. Specifically, this includes safety (ensuring a safe distance from the operator to avoid collisions), efficiency (optimizing trajectory length and motion time), comfort (maintaining operator perceived comfort and avoiding abrupt movements), accuracy (achieving higher positioning and attitude accuracy at key assembly points), and learning-based trajectory optimization (introducing reinforcement learning methods to continuously try and learn in simulation environments or actual operations to optimize the generation strategy of shared motion trajectories, making them more in line with human operating habits and able to effectively cope with unknown situations).

[0123] Furthermore, intelligent soft limit and safety strategies are added: Specifically, the regional safety strategy combines dynamic environmental maps and human posture prediction to achieve regional force / velocity limits; for example, in areas close to the operator's body, the force and velocity thresholds are significantly reduced, even activating emergency braking to bring the entire system to a complete stop; while in open areas far from the operator, higher forces and velocities can be allowed. The safety thresholds related to the task status can be dynamically adjusted at different stages of assembly; for example, in the critical stage of part insertion, there will be stricter force feedback and speed limits in order to ensure accuracy and avoid damage. Fault diagnosis and fault tolerance are enhanced by real-time diagnostic capabilities for sensor faults, actuator faults, and network communication faults. Once a fault is detected, the system can automatically switch to a safe mode, such as emergency stop, retraction to a safe position, or reduction of movement speed, and issue an alarm to the operator.

[0124] With these enhanced features, collaborative robot assembly systems will be able to achieve deeper, smarter, and safer collaboration with human operators in complex industrial scenarios that are highly dynamic, uncertain, and require precise operation.

[0125] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A force control method integrating robot visual information, characterized in that, The method includes: The robot receives and processes scene cognition information from its vision sensors, identifies key interactive objects including targets and obstacles through scene semantic segmentation, and establishes a multi-layer virtual force model based on the scene cognition information. The multi-layer virtual force model includes at least the attraction of the work object, the repulsion of the obstacle, and the constraint force of the image field of view, and generates a predictive guidance data stream. The actual joint position, actual joint speed, and applied manipulation force of the operator are received in real time via the program controller; the time-domain features of the manipulation force are extracted, including torque amplitude, direction, and rate of change; and a manipulation command recognition algorithm based on dual-stream parallelism is applied to analyze the operator's real-time interaction expectations to obtain the manipulation command vector. The predictive guidance data stream and the manipulation command vector are non-conflictingly fused to calculate a shared motion trajectory; and a position tracking controller using a proportional-derivative feedforward algorithm is used to output a trajectory tracking signal by combining the expected position, expected velocity, and expected acceleration of the shared motion trajectory with the actual joint position and actual joint velocity.

2. The force control method integrating robot visual information according to claim 1, characterized in that, The steps for identifying key interactive objects, including targets and obstacles, through scene semantic segmentation specifically include: The robot receives and processes scene perception information from its vision sensors. This scene perception information includes at least visual feedback data, 3D point cloud data of the environment, and dynamic optical flow data. The scene semantic segmentation is performed using MASK R-CNN to fuse the visual feedback data and the 3D point cloud data, perform pixel-level classification and boundary detection, and apply a Bayesian algorithm for confidence verification. The verified semantic labels and location coordinates are output. Based on the semantic labels and location coordinates, the robot distinguishes between the task target and obstacles and generates the target task path.

3. The force control method integrating robot visual information according to claim 1, characterized in that, The specific steps for generating the predictive guided data stream include: Using the 3D coordinate data of the target operation path, a continuous reference path is constructed, and a gravity model based on virtual target points is established. The composite vector used for path adsorption and path forward movement is calculated to generate the gravity of the operation object. After filtering out the point cloud identified as the operation target from the 3D point cloud data, a clustering algorithm is applied to determine the obstacle clusters, and a repulsive artificial potential field function is established. Based on the closest distance between the end effector and the obstacle cluster, the repulsive vector that is inversely proportional to the distance within a safety threshold is calculated to generate the obstacle repulsive force. Using the dynamic optical flow data and the operation target offset recorded by the sensor, the disturbance compensation potential field and the field of view maintenance potential field are calculated respectively, merged into a composite potential field, and the gradient is calculated to generate the image field of view constraint force. By combining the gravitational force of the work object, the repulsive force of the obstacle, and the constraint force of the image field of view, a total virtual potential field function is constructed. By calculating the negative gradient of the function at the position of the end effector, the expected resultant force vector for guiding motion is obtained. The expected resultant force vector and the first derivative are output as a predictive guidance data stream.

4. The force control method integrating robot visual information according to claim 1, characterized in that, The step of receiving the operator's actual joint position and applied manipulation force in real time, and extracting temporal features, specifically includes: Using a six-axis force and torque sensor and a position encoder installed on the controller, the operator's manipulation input data is collected synchronously. The manipulation input data includes at least the six-degree-of-freedom joint position coordinates and the corresponding original force and torque vectors. Noise suppression processing is performed through a low-pass filter to obtain the actual joint positions and a smoothed manipulation force sequence. A time-domain feature extraction algorithm based on vector operation and numerical difference is applied to analyze the smoothed manipulation force sequence, calculate the torque amplitude, direction, and rate of change, and obtain the time-domain features. The physical dynamics of the controller itself, including at least gravity, inertial force, and friction, are calculated in real time to obtain the equipment compensation force vector. Online dynamic compensation is performed to subtract the equipment compensation force vector from the smoothed manipulation force sequence to obtain the pure intention force vector, which represents the operator's true interactive intention.

5. The force control method integrating robot visual information according to claim 1, characterized in that, The specific steps to obtain the manipulation command vector include: The manipulation command recognition algorithm includes: a command type classification stream, which identifies the interaction expectation as a motion, fine-tuning, or position-holding command type by analyzing the transient patterns of the time-domain features; and a command parameter calculation stream, which extracts smooth raw motion parameters by fusing the actual joint position and the pure intention force vector and applying a Kalman filter to filter out the operator's physiological hand tremors and high-frequency noise. The dynamic calibration fusion module, based on the instruction type label, performs gating and nonlinear calibration processing on the smoothed raw motion parameters to generate a manipulation instruction vector; it dynamically adjusts the calibration parameters of the nonlinear calibration processing, specifically by dynamically and inversely adjusting the gain of the calibration parameters according to the magnitude of the obstacle repulsive force; and then dynamically deflects the response axis of the calibration parameters according to the direction of the gravitational force of the working object.

6. The force control method integrating robot visual information according to claim 1, characterized in that, The steps for calculating the shared motion trajectory specifically include: The non-conflict fusion is achieved through adaptive weight allocation. Using the desired resultant force vector, a virtual dynamics solver is used to calculate the desired velocity vector in real time. Combined with the manipulation command vector, a dynamic projection fusion algorithm is applied, using the magnitude of the desired resultant force vector as a dynamic weight to flexibly weight conflicting motion components, resulting in the fused target shared velocity. Using the fused target shared velocity and the actual joint position, multi-step prediction is performed to generate future motion path points in real time. Using the future motion path points as control points, a Bézier curve is constructed as the shared motion trajectory. After applying the minimization of abrupt change criterion to perform optimal time parameterization, the desired position, desired velocity, and desired acceleration of the trajectory are calculated.

7. The force control method integrating robot visual information according to claim 1, characterized in that, The specific steps for outputting the trajectory tracking signal include: Based on the desired position and the actual joint position, the position error is calculated in real time; the position error is multiplied by a proportional gain to obtain a proportional compensation term; based on the desired velocity and the real-time detected actual joint velocity, the velocity error is calculated in real time; the velocity error is multiplied by a differential gain to obtain a differential compensation term; the desired acceleration of the shared motion trajectory is multiplied by a feedforward gain to obtain a feedforward compensation term; the proportional compensation term, the differential compensation term, and the feedforward compensation term are weighted and summed to compensate for the position error and velocity error, and a trajectory tracking signal is generated; the magnitude of the trajectory tracking signal is continuously monitored, and a soft limit constraint is applied when it exceeds a preset force threshold.

Images