Robot vision-guided grabbing control method and system

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a robot vision-guided grabbing control method and system, and the method comprises the steps: executing global approaching coarse positioning; after a preset distance threshold is entered, the frame rate is increased, a high-frequency visual servo mode is switched, geometric vectors of feature points are extracted, and pixel residual errors are calculated; kalman filtering is introduced to smooth a residual error, and an observation noise covariance matrix is dynamically adjusted according to a real-time image signal-to-noise ratio; mapping the smooth residual error into a six-degree-of-freedom speed compensation amount by using an image jacobian matrix; and the compensation amount and the original motion instruction vector are synthesized, and after S-shaped curve weighting and normalized amplitude limiting, a joint actuator is driven until the residual error converges. The system comprises a visual acquisition module, a feature extraction module, a residual calculation module, a visual servo controller and a motion execution module. By means of the scheme, high-precision and high-robustness dynamic grabbing is achieved, and dependence on camera calibration and three-dimensional reconstruction is remarkably reduced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a robot vision-guided grasping control method and system. Background Technology

[0002] With the continuous improvement of industrial automation, vision-guided grasping technology directly determines the operational accuracy and environmental adaptability of robots. In typical scenarios such as intelligent manufacturing, logistics sorting, and flexible assembly, robots need to perform high-precision positioning and grasping of target objects in unstructured or semi-structured environments, which places extremely high demands on the coordination capabilities of visual perception and motion control.

[0003] Among these, visual servoing-based grasping control methods, due to their closed-loop feedback characteristics, have become a key technical path for improving system robustness. These methods typically construct control signals based on the deviation between real-time image features and the desired target, driving the robot's end effector to approach the target pose. However, existing visual servoing solutions still face multiple technical bottlenecks in practical deployments: First, most systems rely on high-precision camera-robot hand-eye calibration parameters, and calibration errors or extrinsic parameter drift can directly lead to inaccurate 3D pose calculations, resulting in grasping failure; second, traditional solutions generally employ open-loop coarse positioning superimposed with static image processing, which cannot respond promptly to minute dynamic displacements of the target, leading to closed-loop failure; third, existing visual servoing strategies are sensitive to image noise, easily causing feature jumps under complex conditions such as sudden changes in illumination, reflections, or partial occlusion, resulting in severe oscillations in control commands; finally, mainstream methods lack smooth transition mechanisms during mode switching (such as from global approximation to fine servoing), often causing mechanical jitter due to sudden changes in speed commands, affecting system stability and lifespan. These problems collectively restrict the large-scale reliable application of vision-guided grasping systems in real industrial settings.

[0004] To address the core shortcomings of existing technologies, such as strong calibration dependence, weak dynamic response capability, lack of noise suppression mechanism, and discontinuous control commands, there is an urgent need for a novel grasping control scheme that integrates adaptive filtering, pixel-level closed-loop mapping, and dynamic trajectory synthesis to achieve a balance between low calibration dependence, high disturbance rejection, and smooth response. Summary of the Invention

[0005] The purpose of this invention is to provide a robot vision-guided grasping control method and system, which can effectively solve the problems in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A robot vision-guided grasping control method includes the following specific steps: Step (1) Execute the global approach stage: acquire the initial image of the target object through the vision acquisition device and perform coarse positioning, generate the original motion command to move towards the target area, so that the end effector approaches the target to within the preset distance threshold range; Step (2) Extracting image feature vectors: After entering the preset distance threshold, the visual acquisition frame rate is increased to the preset high frequency frame rate, and the high frequency visual servo mode is switched to acquire continuous image streams in real time and extract geometric feature vectors of no less than 4 feature points from each frame image. Step (3) Calculate pixel residuals: Compare the currently extracted geometric feature vector with the pre-stored expected feature vector, calculate the pixel-level deviation between the two in the image plane, and form a multi-dimensional pixel residual vector; Step (4) Mapping the velocity control vector: Kalman filtering is introduced to smooth the multidimensional pixel residual vector. The observation noise covariance matrix of the Kalman filter is dynamically adjusted according to the real-time image signal-to-noise ratio, and the observation noise covariance matrix is ​​inversely proportional to the image signal-to-noise ratio. Then, using the image Jacobian matrix and its pseudo-inverse, the smoothed pixel residual is mapped in real time to the six-degree-of-freedom velocity compensation of the robot end effector in Cartesian space, including three linear velocity components and three angular velocity components. Step (5) Perform dynamic trajectory correction: Combine the velocity compensation amount with the original motion command into a vector, and perform normalization and amplitude limiting processing on the angular velocities of each joint after synthesis to generate composite control commands to drive the robot joint actuators to adjust the motion trajectory until the pixel residual is less than the set convergence threshold.

[0007] Preferably, in step (5), the vector synthesis process uses an S-curve function to weight the speed compensation amount. During a predetermined time period after switching to the high-frequency visual servo mode, the weight coefficient increases linearly from 0 to 1, and the output speed is the sum of the original motion command and the weighted speed compensation amount.

[0008] Preferably, in step (4), when calculating the six-degree-of-freedom velocity compensation, the EPnP algorithm is used to solve the pose to obtain the feature point depth information, and the depth value of the previous control cycle is used as the initial value of the current iteration and input into the solver; when the coplanarity of the feature points causes the reprojection error to be greater than the preset pixel threshold, the center reference depth is obtained by the laser displacement sensor integrated on the end effector, and the depth of each point is corrected by using the cosine value of the angle between the center reference depth and each feature point relative to the center.

[0009] Preferably, in step (2), if the number of feature points successfully matched in the current frame is less than 3, the end effector is controlled to perform a spiral search trajectory with a predetermined step length centered on the current coordinates, while the vision module performs template matching at a preset high-frequency frame rate; when the matching score exceeds the preset confidence threshold, the target center is locked and the state covariance matrix of the Kalman filter is reset.

[0010] Preferably, in step (4), when mapping the six-degree-of-freedom velocity compensation amount, the gain coefficient is set within a preset range; when the Euclidean norm of the pixel residual is greater than the first preset pixel threshold, the gain coefficient is taken as the first preset value; when the Euclidean norm of the pixel residual is less than or equal to the first preset pixel threshold, the gain coefficient is taken as the second preset value.

[0011] Preferably, in step (4), the observation noise covariance matrix is ​​a diagonal matrix, and its diagonal elements are all equal to a preset constant plus the negative multiple of the image signal-to-noise ratio of the natural exponential function; when the local gray variance is lower than the preset gray variance threshold, the image signal-to-noise ratio is determined to be in a low signal-to-noise ratio state, and the value of the observation noise covariance matrix is ​​increased to the preset upper limit value.

[0012] Preferably, the specific process of normalization limiting in step (5) is as follows: determine whether the synthesized joint angular velocity exceeds the rated maximum value; if the maximum absolute value of each joint angular velocity exceeds the rated maximum value, then reduce all joint velocities proportionally, with the scaling factor being the ratio of the rated maximum value to the maximum absolute value of each joint angular velocity.

[0013] Preferably, the vision acquisition device is mounted on the robot end effector to form a hand-eye system, and the end effector also integrates a laser displacement sensor for assisting depth correction. This sensor is used to provide a reliable depth reference when the coplanarity of feature points causes pose solving to fail.

[0014] Preferably, the visual acquisition frame rate in the high-frequency visual servo mode is not lower than a preset high-frequency threshold, ensuring that the control system can respond promptly to the minute dynamic displacement of the target object, thereby maintaining the stability and accuracy of the closed-loop control.

[0015] Preferably, the preset distance threshold is detected in real time by a laser displacement sensor integrated on the end effector and triggers mode switching to ensure that the system has completed coarse positioning before entering the fine servo stage.

[0016] On the other hand, this application proposes a robot vision-guided grasping control system, comprising: The visual acquisition module is used to acquire the initial image of the target object at the first frame rate during the global approach phase, and to acquire a continuous image stream at the second frame rate after entering the preset distance threshold. The feature extraction module is used to extract geometric feature vectors of no less than 4 feature points from each frame of image; the residual calculation module is used to compare the current geometric feature vector with the pre-stored expected feature vector to generate a multi-dimensional pixel residual vector. A visual servo controller is used to perform Kalman filtering smoothing on the multidimensional pixel residual vector, wherein the observation noise covariance matrix is ​​dynamically adjusted according to the real-time image signal-to-noise ratio and is inversely proportional to the image signal-to-noise ratio, and the smoothed pixel residual is mapped to a six-degree-of-freedom velocity compensation amount using the image Jacobian matrix and its pseudo-inverse. The motion execution module is used to synthesize the speed compensation amount with the original motion command vector, and perform normalization and amplitude limiting processing on the synthesized joint angular velocities to generate composite control commands to drive the robot joint actuators to adjust the motion trajectory until the pixel residual is less than the set convergence threshold.

[0017] Preferably, the vision acquisition module and the laser displacement sensor are integrated together on the robot's end effector to form a hand-eye system; the laser displacement sensor is used to provide a center reference depth when the pose solution fails due to the coplanarity of feature points, so as to correct the depth information of each feature point.

[0018] Preferably, the visual servo controller uses a first gain coefficient when the Euclidean norm of the pixel residual is greater than a first preset pixel threshold, and uses a second gain coefficient when the Euclidean norm of the pixel residual is less than or equal to the first preset pixel threshold; the motion execution module uses an S-curve function to weight the speed compensation amount within a predetermined time period after mode switching, with the weighting coefficient increasing linearly from 0 to 1.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. High-precision closed-loop control capability: This invention completely avoids the camera calibration error and extrinsic parameter drift problems introduced by three-dimensional coordinate calculation in traditional methods by directly constructing a pixel residual closed loop in the image plane. The system can achieve pixel-level grasping and positioning accuracy without relying on high-precision hand-eye calibration parameters, significantly improving the positioning reliability under long-term operation in industrial fields.

[0020] 2. Strong adaptability to dynamic environments: This invention abandons the traditional open-loop coarse positioning and static image processing approach, and adopts a closed-loop dynamic servo strategy throughout the entire process. The system continuously senses the target's state as the robot moves towards it and generates real-time speed compensation for dynamic correction, effectively addressing minute object displacements caused by conveyor belt vibrations, airflow disturbances, etc., significantly improving the success rate of grasping in unstructured dynamic environments.

[0021] 3. Low calibration dependence and high robustness: By omitting the cumbersome 3D reconstruction steps, this invention fundamentally reduces the dependence on camera calibration accuracy. Simultaneously, the introduction of an adaptive Kalman filter mechanism based on dynamically adjusting the observation noise covariance matrix according to the real-time image signal-to-noise ratio effectively suppresses feature jumps caused by sudden changes in illumination, reflections, or partial occlusion, ensuring the smoothness of control commands and the overall robustness of the system.

[0022] 4. Efficient and smooth mode switching and command synthesis: This invention employs distance threshold-triggered segmented servo logic, combined with S-curve weighting and normalized amplitude limiting algorithms, to achieve seamless transition from global approximation to high-frequency visual servoing. This design avoids the mechanical jitter caused by sudden changes in speed commands during mode switching in traditional methods, protecting the robot hardware and ensuring the stability and efficiency of the entire grasping process. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall technical architecture of a robot vision-guided grasping control method and system proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the velocity compensation mapping based on pixel residual closed loop and adaptive Kalman filtering in this invention; Figure 3 This is a logical flowchart of the global approach phase and the switching of high-frequency visual servo mode in this invention. Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow between the end effector and the vision-laser fusion sensing module in this invention; Figure 5 This is a flowchart illustrating the logical flow of dynamic trajectory correction and S-shaped weighted synthesis control command generation in this invention. Detailed Implementation

[0024] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0025] Example 1. This example uses the high-precision gripping of randomly placed metal parts (such as automobile engine cylinder heads) in an industrial automated production line as a typical application scenario. In this scenario, the target object may undergo slight displacement due to conveyor belt vibration, and there are complex working conditions such as strong reflections, partial occlusion, and sudden changes in lighting. To address these challenges, this invention deploys a closed-loop control system that integrates high frame rate visual perception, adaptive filtering, and six-degree-of-freedom dynamic trajectory correction capabilities.

[0026] First, construct a complete hardware and software collaborative platform at the system architecture level. (See...) Figure 1The system consists of five core modules: a visual acquisition module, a feature extraction module, a residual calculation module, a visual servo controller, and a motion execution module. These modules are interconnected via a high-speed real-time communication bus and are uniformly scheduled by an embedded main control unit.

[0027] The vision acquisition module employs a global shutter CMOS industrial camera (model: FLIRBlackflySBFS-U3-50S5C-C) mounted on the robot's end effector. It has a resolution of 2448×2048 pixels, supports a USB 3.0 interface, and a maximum frame rate of 167fps. The camera is fixed to the end effector flange via an M12 lens mount, forming a typical "eye-in-hand" configuration. To assist in depth sensing, the end effector also integrates a high-precision laser displacement sensor (model: KeyenceLK-G5000), with a measurement range of ±5mm and a repeatability of ±0.5μm. Its output signal is connected to the main control unit via an RS-485 interface. The vision acquisition module is powered by the end effector's built-in 24VDC regulated power supply, and the signal level conforms to the TTL standard, ensuring stable operation even in electromagnetic interference environments.

[0028] The feature extraction module runs on a dedicated embedded image processing unit (based on an NVIDIA Jetson AGXXavier SoC), which includes an 8-core ARMv8.2 CPU, a 512-core Volta GPU, and 32GB of LPDDR4x memory. This module is directly connected to the vision acquisition module via a PCIe Gen4x8 interface. After receiving the raw image stream, it uses OpenCV 4.5 and the CUDA acceleration library to perform feature point detection in parallel. Specifically, the system assumes that the target object surface has at least four high-contrast geometric features (such as hole centers, edge corners, or QR code positioning points). The feature extraction algorithm uses an improved FAST corner detection combined with a Shi-Tomasi scoring mechanism to select the N candidate points (N≥4) with the highest response values ​​in each frame of the image, and improves the positioning accuracy to within 0.1 pixels through sub-pixel interpolation. The image coordinates (u,v) of all feature points are stored in 16-bit unsigned integer format and transmitted in real time to the residual calculation module via Gigabit Ethernet (TCP / IP protocol).

[0029] See Figure 4 The residual calculation module and the visual servo controller are jointly deployed on a real-time motion controller (based on Intel x86-64 architecture, equipped with RT-Linux real-time kernel, and with a task scheduling cycle of 1ms). This controller communicates with the robot joint servo drivers via an EtherCAT bus and simultaneously receives feature data streams from the feature extraction module via UDP multicast. The residual calculation module internally maintains a pre-stored desired feature vector. (n≥4), this vector is obtained through teaching during the system initialization phase. The expected pixel coordinates of the nth feature point in the image plane are u (the pixel value in the horizontal direction and v (the pixel value in the vertical direction)); n is the number of feature points (at least 4 are required to ensure positioning accuracy). Let be the dimension space of the vector, indicating that the vector is a 2n-dimensional real vector (each feature point corresponds to 2 pixel coordinates, and n feature points have a total of 2n dimensions).

[0030] See Figure 2 In each control cycle (i.e., a 16.67ms cycle in high-frequency visual servo mode, corresponding to a 60Hz frame rate), the module will change the current feature vector. and Perform element-by-element comparison to generate a multi-dimensional pixel residual vector. This residual vector directly reflects the pose deviation of the end effector relative to the target object, without the need for 3D reconstruction or hand-eye calibration. The current feature vector s represents the feature point position vector extracted from the real-time acquired image. Let n be the actual pixel coordinates of the nth feature point in the current image, and compare them with the above. One-to-one correspondence. The multidimensional pixel residual vector e represents the pixel-level deviation between the current feature point and the desired feature point; This represents an element-wise subtraction operation, which is the actual pixel coordinates of each feature point minus the corresponding expected pixel coordinates. The deviation value directly reflects the pose alignment error.

[0031] The visual servo controller is the core intelligent unit of this system, and its functions include Kalman filtering smoothing, Jacobian mapping, depth estimation, and gain scheduling. The controller integrates an adaptive Kalman filter, whose state vector is the residual vector e, and whose observation equation is... ,in The observed value is for the kth control cycle; The original pixel residual for the k-th control cycle; For observation noise. The state transition equation is e k = Ae k-1 + w k-1 Under the quasi-static capture assumption, the state transition matrix A is taken as the identity matrix I, and the process noise covariance Q is preset as a constant diagonal matrix based on the terminal jitter frequency; k-1 w is the residual vector for the (k-1)th control cycle. k-1 This represents the process noise during the (k-1)th control cycle. The key innovation lies in the observation noise covariance matrix. Designed as a diagonal matrix, and its diagonal elements SNK is the signal-to-noise ratio of the current image (reflecting image sharpness; the greater the noise, the lower the signal-to-noise ratio). These are the diagonal elements of the covariance matrix (corresponding to the noise variances in the u and v directions, respectively). It is a natural exponential function, realizing the inverse mapping between noise variance and signal-to-noise ratio (the higher the signal-to-noise ratio, the smaller the noise variance, and the more reliable the filtered observations are); 0.5 is a preset constant to ensure the basic range of noise variance values.

[0032] The SNK is calculated in real time by the image processing unit: the formula is... ,in It is the square of the grayscale mean. The noise variance is estimated after denoising. First, a 5×5 local window grayscale variance analysis is performed on the image. If the mean variance of all windows is lower than the threshold of 10, it is determined to be a low signal-to-noise ratio state. In this case, all diagonal elements of R are forcibly set to 1.0 to enhance the filter's confidence in the prediction model and suppress feature jumps caused by reflections or blurring. The filtered residual is then input into the Jacobian mapping unit.

[0033] In the Jacobian mapping unit, the system needs to construct the image Jacobian matrix. Each row of the L matrix corresponds to a Jacobian submatrix of a feature point. The construction formula for each element in the L matrix is ​​defined based on the normalized coordinates (x, y) of the feature point and the depth z, and v is specified as the velocity in the camera coordinate system. This velocity then needs to be transformed to the robot's base coordinate system using a hand-eye transformation matrix. Since the construction of L depends on the depth information of the feature points... This system uses the Efficient Perspective-n-Point (EPnP) algorithm to solve the pose online. To improve the convergence speed, the EPnP solver uses the depth value from the previous control cycle. This serves as the initial value for the current iteration. However, when the target object's surface is approximately coplanar (such as a flat plate), EPnP may cause a reprojection error exceeding 2 pixels due to ill-conditioned conditions. In this case, the system triggers a depth correction mechanism: reading the center reference depth measured by the laser displacement sensor. And calculate the angle between each feature point and the image center. (Depend on Confirmed, among which The center pixel coordinates, (focal length) Let be the angle between the i-th feature point and the center of the image.

[0034] The final correction depth is This correction strategy effectively solves the depth uncertainty problem caused by coplanar features. The corrected depth value of the i-th feature point (distance along the camera's optical axis in Cartesian space); The center reference depth is measured by the laser displacement sensor; The cosine of the included angle is used to correct the depth based on the angle at which the feature point deviates from the center (the depth of edge feature points decreases as the angle increases).

[0035] After obtaining the accurate value, the system calculates its Moore-Penrose pseudoinverse. And through the formula Generate six-degree-of-freedom velocity compensation Among them, the gain coefficient Implement dynamic scheduling: when When taking pixels, To ensure stability under large deviations; when pixels At that time, take To accelerate the convergence speed. These are the three linear velocity components of the end effector in Cartesian space (translational velocities along the x, y, and z axes). ω represents the three angular velocity components of the end effector (rotational speeds around the x, y, and z axes); φ is the gain coefficient (dynamically adjusted, 0.6 for large deviations to ensure stability, and 0.8 for small deviations to improve efficiency). The Moore-Penrose pseudoinverse of the image Jacobian matrix L (used to map pixel residuals to velocity quantities). This represents the pixel residuals after Kalman filtering smoothing.

[0036] See Figure 5 The motion execution module is responsible for fusing the speed compensation amount with the original motion command and driving the robot to execute it. Original motion command Generated by the path planner in the global approximation phase, it is typically a Cartesian space velocity vector corresponding to a straight line or spline curve. Upon entering high-frequency visual servoing mode, the system activates an S-curve weighting mechanism: within the first 100 milliseconds after the switch, the weight coefficient φ linearly increases from 0 to 1, and the output velocity is... This design avoids a sudden jump in velocity during mode switching, limiting the rate of change of joint acceleration within a safe range. This is the final output velocity vector; is the original motion command for the global approach phase (generated by the path planner); is the weight coefficient (linearly increases from 0 to 1 within 100ms after mode switching to achieve a smooth transition); is the six-degree-of-freedom velocity compensation amount.

[0037] Subsequently, The inverse kinematics calculation (using the Damped Least Squares method) converts the data into angular velocity commands for each joint. (i=1~6). To prevent overspeeding, the system performs normalized limiting: calculate If it exceeds the rated maximum value (Set by servo drive parameters, typically 150° / s), then calculate the scaling factor. and all Scaling down proportionally The processed commands are sent to each joint servo driver (such as the Yaskawa Σ-7 series) via the EtherCAT bus at a frequency of 1kHz, driving the permanent magnet synchronous motor to track precisely.

[0038] See Figure 3 With the support of the above system architecture, the workflow of this embodiment strictly follows step S in the invention description, as detailed below: Step (1) Global approximation phase: After system startup, the vision acquisition module acquires the initial image at a frame rate of 30Hz. The feature extraction module identifies the approximate outline of the target object and roughly locates its center coordinates through template matching. Based on this, the motion controller generates the original motion command for a straight path. The system drives a six-axis robotic arm (such as the FANUC M-2000iA) to move towards the target area. During this process, a laser displacement sensor continuously monitors the distance between the end effector and the target. When the detected value is ≤100mm, the system determines that a preset distance threshold has been reached and immediately triggers a mode switch.

[0039] Step (2) Extracting Image Feature Vectors: After mode switching, the frame rate of the visual acquisition module is increased to 60Hz, entering the high-frequency visual servo mode. The feature extraction module begins to extract no less than 4 feature points from each frame of the image. If the number of successfully matched feature points m<3 in the current frame due to occlusion or blurring, the system activates the local rescan mechanism: the motion execution module controls the end effector to perform an Archimedean spiral search trajectory with a step size of 2mm, centered on the current Cartesian coordinates (parameter equation: Simultaneously, the vision module performs template matching at a frequency of 120Hz. Here, r is the polar radius of the spiral (distance from the center to the trajectory point); θ is the polar angle (rotation angle between the trajectory point and the center, ranging from 0 to 4π, i.e., 2 rotations); and 0.5 is the step size coefficient, determining the spiral's unfolding density. Once the matching score (normalized cross-correlation coefficient) exceeds 0.85, the system immediately locks onto the target center and resets the Kalman filter's state covariance matrix φ to its initial value (e.g., φ), to quickly adapt to the new positioning state.

[0040] Step (3) Calculate pixel residuals: Under normal servo conditions, the residual calculation module receives the current feature vector s in each control cycle and compares it with the pre-stored... The comparison generates a 2n-dimensional residual vector e. This vector directly represents the deviation on the image plane, completely avoiding the cumulative error caused by the transformation of the three-dimensional coordinate system.

[0041] Step (4) Mapping the speed control vector: The visual servo controller first dynamically adjusts the matrix R of the Kalman filter according to the real-time image signal-to-noise ratio, smooths e, and outputs... Subsequently, combining the depth information obtained from EPnP solving or laser correction, the image Jacobian matrix L is constructed, and its pseudo-inverse is calculated. The gain coefficient φ is dynamically selected based on the current residual norm, ultimately generating the six-degree-of-freedom velocity compensation quantity v. The entire process is completed within a 1ms real-time task cycle, ensuring a control latency of less than 2ms.

[0042] Step (5) Perform dynamic trajectory correction: The motion execution module will combine the motion with the trajectory. By weighted synthesis of S-curve functions, we obtain After inverse kinematics calculation and normalized limiting, safe joint angular velocity commands are generated to drive the robot to adjust its trajectory in real time. This closed-loop process continues until the Euclidean norm of the pixel residual vector is reached. At this point, the system determines that the grasping pose is precisely aligned, triggers the pneumatic gripper to close, and completes the grasping action. An S-curve function (such as a fifth-order polynomial or a sinusoidal transition function) is used for weighting, and the weighting coefficient w(t) follows a predetermined time period T after the switch. Where w(t) is a weighting coefficient that varies with time, t is the real-time, and T is the total transition duration. For normalized time, 10, -15, and 6 are polynomial coefficients.

[0043] Throughout the grasping process, the system achieved a closed-loop operation, dynamic response, and smooth transition. Even if the target object shifts by ±5mm during the robot's approach, the system can correct the deviation within 200ms, achieving a grasping success rate of up to 99.2%. Furthermore, by omitting high-precision hand-eye calibration, the system's on-site deployment time was reduced by 70%, and no accuracy degradation due to calibration drift was observed after 30 days of continuous operation.

[0044] In summary, this embodiment, through precise hardware selection, modular system architecture, and deeply integrated control algorithms, fully realizes the technical solution proposed in this invention, and fully verifies its significant advantages in high precision, strong robustness, and low calibration dependence. The above are merely specific embodiments of this invention, but the technical features of this invention are not limited thereto. Any simple changes, equivalent substitutions, or modifications made based on this invention to solve essentially the same technical problems and achieve essentially the same technical effects are all covered within the protection scope of this invention.

Claims

1. A robot vision-guided grasping control method, characterized in that, Includes the following steps: Step (1) Execute the global approach phase: The initial image of the target object is acquired by the vision acquisition device installed on the robot end effector and coarsely located. The original motion command to move towards the target area is generated so that the end effector approaches the target to within the preset distance threshold range. Step (2) Extracting image feature vectors: After entering the preset distance threshold, the visual acquisition frame rate is increased to the preset high frequency frame rate, and the high frequency visual servo mode is switched to acquire continuous image streams in real time and extract geometric feature vectors of no less than 4 feature points from each frame image. Step (3) Calculate pixel residuals: Compare the currently extracted geometric feature vector with the pre-stored expected feature vector, calculate the pixel-level deviation between the two in the image plane, and form a multi-dimensional pixel residual vector; Step (4) Mapping the velocity control vector: Kalman filtering is introduced to smooth the multidimensional pixel residual vector. The observation noise covariance matrix of the Kalman filter is dynamically adjusted according to the real-time image signal-to-noise ratio, and the observation noise covariance matrix is ​​inversely proportional to the image signal-to-noise ratio. Then, using the image Jacobian matrix and its pseudo-inverse, the smoothed pixel residual is mapped in real time to the six-degree-of-freedom velocity compensation of the robot end effector in Cartesian space, including three linear velocity components and three angular velocity components. Step (5) Perform dynamic trajectory correction: Combine the velocity compensation amount with the original motion command into a vector, and perform normalization and amplitude limiting processing on the angular velocities of each joint after synthesis to generate composite control commands to drive the robot joint actuators to adjust the motion trajectory until the pixel residual is less than the set convergence threshold.

2. The robot vision-guided grasping control method according to claim 1, characterized in that, The vector synthesis process in step (5) uses an S-curve function to weight the speed compensation amount. During a predetermined time period after switching to the high-frequency visual servo mode, the weight coefficient increases linearly from 0 to 1, and the output speed is the sum of the original motion command and the weighted speed compensation amount.

3. The robot vision-guided grasping control method according to claim 1, characterized in that, In step (4), when calculating the six-degree-of-freedom velocity compensation, the EPnP algorithm is used to solve the pose to obtain the feature point depth information, and the depth value of the previous control cycle is used as the initial value of the current iteration to input the solver; when the coplanarity of the feature points causes the reprojection error to be greater than the preset pixel threshold, the center reference depth is obtained by the laser displacement sensor integrated on the end effector, and the depth of each point is corrected by using the cosine value of the angle between the center reference depth and each feature point relative to the center.

4. The robot vision-guided grasping control method according to claim 1, characterized in that, In step (2), if the number of feature points successfully matched in the current frame is less than 3, the end effector is controlled to perform a spiral search trajectory with a predetermined step length centered on the current coordinates, while the vision module performs template matching at a preset high-frequency frame rate; when the matching score exceeds the preset confidence threshold, the target center is locked and the state covariance matrix of the Kalman filter is reset.

5. The robot vision-guided grasping control method according to claim 1, characterized in that, In step (4), when mapping the six-degree-of-freedom velocity compensation amount, the gain coefficient is set to be within a preset range; when the Euclidean norm of the pixel residual is greater than the first preset pixel threshold, the gain coefficient is taken as the first preset value; when the Euclidean norm of the pixel residual is less than or equal to the first preset pixel threshold, the gain coefficient is taken as the second preset value.

6. The robot vision-guided grasping control method according to claim 1, characterized in that, In step (4), the observed noise covariance matrix is ​​a diagonal matrix, and its diagonal elements are all equal to a preset constant plus the negative multiple of the image signal-to-noise ratio of the natural exponential function. When the local gray variance is lower than the preset gray variance threshold, the image signal-to-noise ratio is determined to be in a low signal-to-noise ratio state, and the value of the observed noise covariance matrix is ​​increased to the preset upper limit value.

7. The robot vision-guided grasping control method according to claim 1, characterized in that, The specific process of normalization limiting in step (5) is as follows: determine whether the synthesized joint angular velocity exceeds the rated maximum value; if the maximum absolute value of the joint angular velocity exceeds the rated maximum value, then reduce all joint angular velocities proportionally, with the scaling factor being the ratio of the rated maximum value to the maximum absolute value of the joint angular velocity.

8. A robot vision-guided grasping control system, characterized in that, include: The visual acquisition module is used to acquire the initial image of the target object at the first frame rate during the global approach phase, and to acquire a continuous image stream at the second frame rate after entering the preset distance threshold. The feature extraction module is used to extract geometric feature vectors of no less than 4 feature points from each frame of the image; The residual calculation module is used to compare the current geometric feature vector with the pre-stored expected feature vector to generate a multi-dimensional pixel residual vector. A visual servo controller is used to perform Kalman filtering smoothing on the multidimensional pixel residual vector, wherein the observation noise covariance matrix is ​​dynamically adjusted according to the real-time image signal-to-noise ratio and is inversely proportional to the image signal-to-noise ratio, and the smoothed pixel residual is mapped to a six-degree-of-freedom velocity compensation amount using the image Jacobian matrix and its pseudo-inverse. The motion execution module is used to synthesize the speed compensation amount with the original motion command vector, and perform normalization and amplitude limiting processing on the synthesized joint angular velocities to generate composite control commands to drive the robot joint actuators to adjust the motion trajectory until the pixel residual is less than the set convergence threshold.

9. The robot vision-guided grasping control system according to claim 8, characterized in that, The vision acquisition module and the laser displacement sensor are integrated together on the robot's end effector to form a hand-eye system; the laser displacement sensor is used to provide a center reference depth when the pose solution fails due to the coplanarity of feature points, so as to correct the depth information of each feature point.

10. The robot vision-guided grasping control system according to claim 8, characterized in that, The visual servo controller uses a first gain coefficient when the Euclidean norm of the pixel residual is greater than a first preset pixel threshold, and uses a second gain coefficient when the Euclidean norm of the pixel residual is less than or equal to the first preset pixel threshold; the motion execution module uses an S-curve function to weight the speed compensation amount within a predetermined time period after mode switching, with the weighting coefficient increasing linearly from 0 to 1.

Images