A robot vision servo control method and system

Abstract

The application discloses a kind of robot vision servo control method and system, belong to robot vision servo control technical field.The method includes: the vision characteristic image sequence of target workpiece is collected, includes feature point coordinate matrix, edge contour pixel gradient and surface texture gray distribution data;End effector pose deviation analytical processing is carried out to vision characteristic image sequence, and the vision servo characteristic error containing feature point projection position offset, edge direction angle deviation value and texture matching residual error vector is generated;Pre-compiled servo control law is called to joint space mapping to vision servo characteristic error, and the joint speed correction instruction containing each joint angular velocity increment and acceleration compensation parameter is generated;The joint speed correction instruction is checked to the dynamics constraint of actuator, and the joint speed instruction value after checking is generated and drives robot joint motion, so that end effector tracks target workpiece assembly position.

Description

Technical Field

[0001] This invention relates to the field of robot vision servo control technology, specifically to a robot vision servo control method and system. Background Technology

[0002] Robot vision servo control achieves closed-loop adjustment of the end effector's pose through visual feedback, and is widely used in precision assembly operations. In applications such as electronic assembly or mechanical assembly, industrial cameras acquire images of the target workpiece, extract the coordinates of corner points or feature points as visual features, and map the deviation between the projection position of these features on the image plane and the desired position into motion commands in the robot's joint space, driving the end effector to close to the desired assembly position.

[0003] The limitations of single-type point features are quite prominent in industrial environments. Assembly stations are subject to interference factors such as partial occlusion, ambient light variations, and workpiece surface reflections, which can easily compromise the integrity and accuracy of feature point extraction, leading to distorted pose deviation estimation and jitter or even failure in the servo control process. Visual description capabilities relying solely on point features are limited; they cannot express edge pose deviations and surface state differences along the rotational direction, making it difficult to achieve the required control precision in assembly tasks demanding high alignment accuracy.

[0004] In the servo control command generation stage, the common practice is to directly convert visual errors into joint velocity commands through fixed or adaptive gain, without fully considering the robot's torque and velocity upper limit constraints. When visual deviations suddenly increase, the amplitude of the generated velocity commands may exceed the allowable capacity of the joint drive system, causing servo overshoot or mechanical shock, accelerating joint actuator wear, and affecting the system's operational stability. Therefore, it is necessary to solve the problems of maintaining reliable and accurate pose deviation extraction under complex visual conditions, and ensuring that the generated joint motion commands conform to the actuator's physical constraints and avoid over-limit actuation. Summary of the Invention

[0005] The present invention aims to provide a robot vision servo control method and system to solve the problems of insufficient reliability of pose deviation estimation caused by reliance on a single visual feature in complex industrial assembly scenarios, and the problem of command over-limit impact caused by ignoring actuator dynamics constraints during control command generation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a robot visual servo control method, comprising: acquiring a visual feature image sequence of a target workpiece on an assembly workbench, the visual feature image sequence including feature point coordinate matrix, edge contour pixel gradient, and surface texture grayscale distribution data; performing end effector pose deviation analysis processing on the visual feature image sequence to generate visual servo feature errors of the target workpiece, the visual servo feature errors including feature point projection position offset, edge direction angle deviation value, and texture matching residual vector; calling a pre-compiled servo control law to perform joint space mapping processing on the visual servo feature errors to generate joint velocity correction commands for the end effector, the joint velocity correction commands including joint angular velocity increments and acceleration compensation parameters; performing actuator dynamics constraint verification processing on the joint velocity correction commands to generate verified joint velocity command values; driving robot joint movement according to the verified joint velocity command values, so that the end effector tracks the assembly position of the target workpiece.

[0007] By simultaneously extracting and fusing feature point coordinates, edge contour gradients, and surface texture grayscale distribution information, the constructed visual servo feature error can more comprehensively reflect the spatial pose deviation of the end effector relative to the target workpiece in multiple dimensions, thereby enhancing the sensitivity and anti-interference capability of visual feedback. Preferably, during the end effector pose deviation analysis, the feature point coordinate matrix is ​​compared with a preset reference coordinate matrix to calculate the corresponding point difference, obtaining the feature point projection position offset including horizontal and vertical offset components; based on the directional angle difference between the edge contour pixel gradient and the standard contour gradient, the edge direction angle deviation value is calculated to characterize the pose deviation of the end effector in the rotation direction; the surface texture grayscale distribution data and the reference texture template are compared with pixel-level residual calculations to generate a texture matching residual vector containing grayscale mean deviation and local contrast difference information; the above offset, deviation value, and residual vector are then concatenated to generate the visual servo feature error. This multi-feature fusion error construction method takes into account position, pose, and texture detail deviations, improving servo accuracy in complex lighting and weak texture scenes.

[0008] As a technical solution of the present invention, the calculation of the feature point projection position offset further includes: extracting the current image coordinate value of each feature point in the feature point coordinate matrix, extracting the reference image coordinate value of the corresponding feature point in the preset reference coordinate matrix; calculating the coordinate difference between the current image coordinate value and the reference image coordinate value, generating a horizontal offset component and a vertical offset component, and combining the two into a two-dimensional vector. Preferably, the feature point projection position offset is smoothly estimated using a Kalman filter to effectively suppress image measurement noise, obtain a stable and accurate offset sequence, and improve the stability of servo control.

[0009] When performing joint space mapping on visual servo feature errors by invoking a pre-compiled servo control law, an error attenuation factor is calculated based on the visual servo feature errors at the current moment. This error attenuation factor is positively correlated with the magnitude of the error vector. The visual servo feature errors are multiplied by the error attenuation factor to generate a weighted error vector. This weighted error vector is then multiplied by a preset servo gain matrix to generate a joint velocity increment vector. The dimension of the servo gain matrix matches the number of robot joints. The joint velocity increment vector is then accumulated with the joint velocity command from the previous cycle to generate a joint velocity correction command for the current cycle. This joint velocity correction command includes the angular velocity value and acceleration compensation value for each joint. Through adaptive error weighting and gain mapping, the convergence rate can be dynamically adjusted according to the current deviation magnitude, achieving fast tracking while suppressing overshoot, thus ensuring the speed and stability of the servo response.

[0010] When performing actuator dynamics constraint verification on the joint velocity correction command, the angular velocity increment value of each joint in the joint velocity correction command is extracted, and it is determined whether the angular velocity increment value exceeds the maximum allowable angular velocity threshold of the joint. If it exceeds, the angular velocity increment value is limited to the maximum allowable angular velocity threshold. The acceleration compensation parameter in the joint velocity correction command is extracted, and it is determined whether the acceleration compensation parameter exceeds the maximum allowable acceleration threshold of the joint. If it exceeds, the acceleration compensation parameter is limited to the maximum allowable acceleration threshold. The limited angular velocity increment value and the acceleration compensation parameter are combined to form the verified joint velocity command value. The above constraint verification mechanism ensures that the joint motion command is always limited within the dynamic safety boundary, avoiding joint overspeed, overload, or impact due to excessive visual servo output, thus ensuring the safety of system operation and the service life of the actuator.

[0011] In the step of acquiring the visual feature image sequence, an industrial camera is controlled to acquire continuous image frames of the target workpiece at a fixed sampling frequency; each frame image undergoes region of interest (ROI) cropping to extract a rectangular region image containing the assembly features of the target workpiece; the rectangular region image is then subjected to image enhancement processing including histogram equalization and median filtering for noise reduction; features are extracted from the enhanced image to generate the feature point coordinate matrix, edge contour pixel gradient, and surface texture grayscale distribution data; the above feature data from multiple frames are serialized and stored using timestamp indexes to form the visual feature image sequence. This method can effectively filter out environmental noise and improve image contrast, thereby providing reliable and high-quality visual feature information for subsequent pose deviation analysis.

[0012] When driving robot joints to move according to the verified joint speed command values, the verified joint speed command values ​​are decomposed into position loop speed reference values ​​and current loop torque reference values ​​for each joint. The position loop speed reference values ​​are input to the position control loop of the servo driver to generate joint position tracking errors. The joint position tracking errors are superimposed with the current loop torque reference values ​​to generate current command values ​​for the drive motor windings. The servo motor output torque is controlled according to the current command values ​​to drive the joints to move according to the verified joint speed command values, so that the end effector approaches and tracks the assembly position of the target workpiece. Through the coordinated control of the position loop and the current loop, the tracking accuracy of the joint speed commands and the ability to resist load disturbances are improved, making the movement of the end effector smoother.

[0013] To further eliminate the visual servo steady-state error caused by the tilt of the assembly table plane, this method also utilizes a laser rangefinder to collect distance data between the end effector and the assembly table. This distance data includes the distance values ​​of three non-collinear measurement points. Based on these distance values, the tilt angle and deflection direction of the end effector's end face relative to the assembly table plane are calculated. The tilt angle and deflection direction are then converted into a pose compensation vector, which includes rotational compensation around the x-axis and y-axis. This pose compensation vector is then superimposed on the visual servo feature error adjustment process to correct the feature point projection position offset and edge direction angle deviation. By introducing laser rangefinder information for flatness compensation of visual errors, pose calculation errors caused by table unevenness are effectively suppressed, enabling the end effector to accurately align with the assembly position.

[0014] This invention also provides a robot vision servo control system, including a processor and a memory connected to the processor. The memory stores programs, instructions, or code, and the processor executes the programs, instructions, or code in the memory to implement the aforementioned robot vision servo control method. This system integrates multimodal visual feature error construction, adaptive servo gain mapping, dynamic constraint verification, and laser flatness compensation, enabling high-precision and high-safety robot assembly vision servo control.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: Visual servoing feature errors are constructed by jointly using feature point coordinate matrices, edge contour pixel gradients, and surface texture grayscale distribution data. When analyzing the end effector pose deviation of visual feature image sequences, not only is the translational deviation reflected by the feature point projection position offset, but also the edge direction angle deviation value is introduced to reflect the pose deviation in the rotational direction. Furthermore, the texture matching residual vector supplements the surface grayscale distribution difference information. These three different physical property observations form a multi-dimensional error description. Even if feature points are missing or mismatched due to local occlusion or reflection, edge contours and texture features can still provide effective pose deviation indications. This allows visual servoing to maintain stable deviation signal output under industrial interference conditions such as lighting variations and occlusion, avoiding sudden interruptions or severe oscillations in the control process, and improving alignment accuracy and operational continuity.

[0016] In the joint speed command generation process, a joint speed correction command containing the angular velocity increment and acceleration compensation parameters of each joint is established, and an actuator dynamics constraint verification step is introduced. This step performs threshold judgment on the angular velocity increment value and acceleration compensation parameter of each joint in the joint speed correction command. When the angular velocity increment exceeds the maximum allowable angular velocity threshold of the joint, it is limited to the threshold. When the acceleration compensation parameter exceeds the maximum allowable acceleration threshold, it is also limited. The combined values ​​form the verified joint speed command value, which is then sent to the servo drive. The limiting operation restricts the instantaneous change amplitude of the speed command, eliminates the impact of command spikes on the joint driver when visual errors suddenly increase, and ensures that the changes in torque and speed are always within the dynamic capability boundaries allowed by the motor joint. The joint response curve tends to be smoother, and there will be no robot arm vibration or repeated overshoot around the desired position caused by sudden changes in angular velocity. The end effector movement is smoother during assembly, and it helps to reduce the fatigue risk of the reducer and motor windings in long-term operation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of a robot vision servo control method; Figure 2 This is a flowchart of the actuator dynamics constraint verification process; Figure 3 This is a flowchart of visual feature image sequence acquisition and processing; Figure 4 This is a schematic diagram illustrating the Kalman filter smoothing effect on the feature point projection position offset. Figure 5 This is a schematic diagram of the distribution of edge contour pixels and feature points in the enhanced image; Figure 6 It is a curve showing the change in distance measured by the laser rangefinder and the tilt angle of the end effector face. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See Figure 1 This invention provides a robot visual servo control method, comprising: acquiring multiple frames of original images of a target workpiece on an assembly workbench; extracting feature point coordinate matrices, edge contour pixel gradients, and surface texture grayscale distribution data from each frame of the original image; serializing and storing the feature data of each frame according to the acquisition timestamp index to obtain a visual feature image sequence; performing end effector pose deviation analysis processing on the visual feature image sequence to generate visual servo feature errors of the target workpiece, the visual servo feature errors including feature point projection position offsets, edge direction angle deviation values, and texture matching residual vectors; calling a pre-compiled servo control law to perform joint space mapping processing on the visual servo feature errors to generate joint velocity correction commands for the end effector, the joint velocity correction commands including joint angular velocity increments and acceleration compensation parameters; performing actuator dynamic constraint verification processing on the joint velocity correction commands to generate verified joint velocity command values; and driving robot joint movement according to the verified joint velocity command values ​​to enable the end effector to track the assembly position of the target workpiece.

[0021] Example 1: In practice, the visual feature image sequence is subjected to end effector pose deviation analysis processing to generate the visual servo feature error of the target workpiece. The visual servo feature error is obtained by vector concatenation processing of feature point projection position offset, edge direction angle deviation value and texture matching residual vector.

[0022] To obtain the projection position offset of feature points, the current image coordinates of each feature point in the feature point coordinate matrix are first extracted. The feature point coordinate matrix is ​​stored in the image processing unit of the robot vision system. The current image coordinates of each feature point include a pixel row number and a pixel column number. The pixel row number is denoted as... The pixel column number is recorded as subscript Indicates the index number of the feature point. This represents the frame number in the visual feature image sequence. Simultaneously, reference image coordinate values ​​corresponding to the aforementioned feature points are extracted from a preset reference coordinate matrix placed in the controller. Each reference image coordinate value includes a reference pixel row number. and a reference pixel column number The preset reference coordinate matrix is ​​collected and stored through offline teaching when the target workpiece is in the ideal assembly position.

[0023] Calculate the coordinate difference between the current image coordinates and the reference image coordinates for each feature point to generate a horizontal offset component. and vertical offset components Offset the horizontal component and vertical offset components Combined into a two-dimensional vector, serving as the offset of the feature point projection position. Perform the same operation on all feature points contained in the feature point coordinate matrix to obtain a set of two-dimensional vectors.

[0024] In some embodiments, the feature point projection position offset is smoothly estimated using a Kalman filter to suppress the disturbance caused by image noise in the offset calculation. The Kalman filter uses the feature point projection position offset... As observed values, a state vector is established. The state vector contains smoothed estimates of the horizontal and vertical offset components. The Kalman filter is recursively applied frame by frame, the state transition matrix is ​​set to an identity matrix, and the process noise covariance matrix is ​​pre-calibrated based on the image sampling frequency and the end effector's motion speed. The smoothed offset output after filtering is... Used to replace the original It participates in subsequent vector concatenation processing. Using a Kalman filter can effectively reduce the fluctuation of position offset caused by random noise, but does not change the structure of the feature point projection position offset as a two-dimensional vector.

[0025] In calculating the edge orientation angle deviation, the Sobel operator is first used to calculate the pixel gradient of the edge contour for each frame in the visual feature image sequence. The Sobel operator contains horizontal and vertical templates, which are convolved with the image to obtain the horizontal gradient component. and vertical gradient components ,in , These are the row and column indices of the pixel, respectively. The gradient direction angle of each pixel. From the formula Confirm. Select the set of pixels located on the edge contour of the target workpiece and extract the gradient direction angles corresponding to these pixels. The standard contour gradient direction angle is a set of direction angle values ​​pre-calculated and stored from the standard contour image of the target workpiece. The standard contour image is obtained during the system calibration phase and corresponds to the same ideal assembly posture as the preset reference coordinate matrix. The edge direction angle deviation value is defined as the average of the absolute values ​​of the differences between the gradient direction angle of the edge contour pixels and the gradient direction angle of the corresponding standard contour pixels. If the edge contour contains... The pixel, the The gradient direction angle of each pixel is The corresponding standard profile gradient direction angle is Then the edge direction angle deviation value It can be calculated using the following formula:

[0026] in, This represents the edge direction angle deviation value, in radians, with a range of values ​​of [value missing]. ; This represents the total number of edge contour pixels involved in the calculation; Indicates the first The gradient direction angle of each edge contour pixel, in radians; Indicates the first The standard contour gradient direction angle corresponding to each edge contour pixel is expressed in radians. The formula above uses a smaller value to handle the circumferential angle difference, ensuring the angle deviation is the minimum included angle.

[0027] The generation of the texture matching residual vector is based on the pixel-level residual calculation between the surface texture grayscale distribution data and the reference texture template. The surface texture grayscale distribution data is a matrix of pixel grayscale values ​​of the texture region extracted from a sequence of visual feature images. The reference texture template is a standard grayscale matrix of the same size, which is extracted and stored from the target workpiece image at the ideal assembly position during system calibration. When calculating the texture matching residual vector, the surface texture grayscale distribution data matrix is ​​subtracted pixel-by-pixel from the reference texture template matrix to obtain the residual matrix. The mean grayscale deviation, representing the overall brightness difference, is extracted from the residual matrix. It is the arithmetic mean of all pixel values ​​in the residual matrix; at the same time, local contrast difference information representing texture contrast difference is extracted. Local contrast difference information The residual vector is obtained by calculating the local standard deviation of the residual matrix. Specifically, the residual matrix is ​​divided into several non-overlapping local windows, the standard deviation of pixel values ​​is calculated within each window, and then the average of the standard deviations of all windows is taken. The texture matching residual vector is obtained from the gray-level mean deviation. and local contrast difference information Sequential arrangement constitutes, represented as .

[0028] After acquiring all intermediate error components, the feature point projection position offsets (which can be offsets smoothed by Kalman filtering), edge orientation angle deviations, and texture matching residual vectors are concatenated to generate visual servoing feature errors. The concatenation order is fixed as follows: first, arrange the horizontal and vertical offset components of all feature point projection position offsets sequentially; then, place the edge orientation angle deviations; and finally, place the two components of the texture matching residual vector. The number of feature points is... Taking the case of stitching together the visual servoing feature error vector as an example, The structure is as follows:

[0029] in, and The first The horizontal and vertical offset components of each feature point ( The visual servoing feature error vector This will be used as the input for subsequent servo control laws.

[0030] When generating the feature point projection position offset, if a Kalman filter is used for smooth estimation, the recursive process is as follows: (The rest of the text is incomplete and requires further context.) At frame time, the state prediction is The predicted covariance is ,in The process noise covariance matrix; Kalman gain , To observe the noise covariance matrix; the state is updated as follows: Covariance updated to Observation noise covariance matrix The process noise covariance matrix was determined through offline experiments based on the noise characteristics of the image sensor. The settings are based on the maximum motion speed of the end effector and the image sampling period, with the principle being to ensure a smooth effect without introducing excessive delay.

[0031] See Figure 4 The figure shows three distributions of the coordinates of the target workpiece feature points obtained through visual feature image sequences in Example 1. The horizontal axis represents the pixel column number. The unit is pixels, ranging from approximately 50 to 450; the vertical axis represents the pixel row number. The unit is pixels, and the value ranges from approximately 50 to 350. The coordinate axis direction conforms to the traditional representation of image row and column numbers. In the legend, the three markers represent: the reference image coordinate point marked by black solid dots, which is the coordinate of the feature point of the ideal assembly position in the preset reference coordinate matrix; the red cross represents the original current image coordinate point, which is the coordinate of the feature point extracted in the current visual image frame; and the blue triangle represents the current image coordinate point after Kalman filtering smoothing estimation.

[0032] As can be observed from the image, the black reference points are relatively evenly distributed, covering the typical feature areas of the target workpiece. The original red current points are somewhat scattered and offset compared to the black reference points, reflecting coordinate fluctuations caused by noise and environmental disturbances in the acquired image. The blue filtered current points are generally more concentrated and closer to the reference points than the red points, indicating that the Kalman filter effectively suppresses the influence of random noise on the feature point positions and improves the stability of the feature point projection position offset.

[0033] Specifically, the offset before filtering is in the pixel row number. and pixel column number The directions all show significant fluctuations, and the filtered points undergo slight adjustments in spatial location, moving closer to the reference point, but without excessive smoothing to maintain real-time response capability. This trend verifies the effectiveness of the Kalman filter state prediction and update recursion process in Example 1, where the state transition matrix is ​​set as an identity matrix and the process noise covariance is... and observation noise covariance Proper calibration is necessary to achieve a balance between smoothness and timely response.

[0034] Furthermore, the figure does not show any abnormal phenomena such as filtered points being far from the original point or significantly exceeding the reference point, indicating that the filtering process did not introduce structural errors. The coordinate axis range and data distribution both match the visual characteristics of the workpiece on the assembly table, meeting the requirements for acquiring and processing the offset of the projected position of feature points in visual servo control methods.

[0035] Example 2: In practice, a pre-compiled servo control law is invoked to perform joint space mapping processing on the visual servo feature errors, generating joint velocity correction instructions for the end effector. The pre-compiled servo control law is a function module in the form of a machine instruction set, which is stored in the read-only memory of the robot controller. The input parameter of this function module is the visual servo feature error vector, and the output parameter is the joint velocity correction instructions.

[0036] Within the servo control law, the error attenuation factor for the current moment is first calculated based on the visual servo feature error. The visual servo feature error has already been generated in previous processing steps through vector concatenation, denoted as... Its dimension depends on the number of feature points and whether it includes edge orientation angle deviation values ​​and texture matching residual vectors. The error attenuation factor is positively correlated with the magnitude of the visual servoing feature error vector; that is, as the magnitude of the visual servoing feature error vector increases, the error attenuation factor monotonically increases. One way to calculate the error attenuation factor is as follows:

[0037] in, This represents the error attenuation factor, a dimensionless scalar, with a value range of [value missing]. ; Represents the visual servoing feature error vector of Norm; It is a pre-set positive real constant used to adjust the sensitivity of the error attenuation factor as the modulus changes and to prevent the denominator from being zero. The value is set based on the standard deviation of the noise in image feature extraction. When the standard deviation of the noise at the feature point pixel coordinates is approximately 0.1 pixels, Set to 0.05. When the visual servo feature error vector... The modulus is much larger than a constant. At that time, the error decay factor Approaching 1; when the visual servo feature error vector When the modulus gradually approaches zero, the error decay factor It also approaches zero. The above relationship enables the positive correlation adjustment between the error attenuation factor and the error magnitude.

[0038] In some embodiments, constant The determination process is as follows: Multiple sets of visual servo feature error vectors are collected when the end effector is aligned; the statistical average of their magnitudes is calculated; and one-tenth of this statistical average is selected as... The value of the factor is chosen to ensure that the error attenuation factor is maintained during the small error stage. It has a finite gain. In obtaining the error attenuation factor... Then, the visual servoing feature error vector With error attenuation factor Perform component multiplication, i.e., use scalars. Multiply by vector For each component, generate a weighted error vector. The multiplication operation satisfies .

[0039] The weighted error vector Multiply by a preset servo gain matrix to generate the joint velocity increment vector. The preset servo gain matrix is ​​denoted as... Servo gain matrix The dimension matches the number of robot joints. If the number of robot joints is... Weighted error vector The dimension is Then the servo gain matrix for OK A real matrix of columns. Joint velocity increment vector. It is derived from matrix multiplication that satisfies Servo gain matrix The acquisition method is as follows: at the target assembly position, an image Jacobian matrix is ​​established using the robot kinematics model and camera parameters, and the pseudo-inverse of the image Jacobian matrix is ​​used as the servo gain matrix. Pseudo-inverse calculation adopts In this way, Represents the image Jacobian matrix. Servo gain matrix. The calculations are performed offline during the system calibration phase and stored in the pre-compiled servo control law, which is not recalculated during the control cycle.

[0040] Joint velocity increment vector Each component corresponds to the angular velocity increment value of a robot joint. The joint velocity increment vector... The joint velocity command from the previous cycle is added to the previous cycle's joint velocity command, which is denoted as... The accumulation operation adds the corresponding components to generate the joint velocity correction command for the current cycle. ,satisfy Joint speed correction command Each component represents the target angular velocity value of the corresponding joint in the current control cycle.

[0041] Acceleration compensation parameters are calculated from joint velocity correction commands. Specifically, the robot controller records the joint velocity correction commands in the current cycle. and the joint speed command from the previous cycle. Perform the difference and divide by the control cycle time length. This yields the angular acceleration values ​​for each joint. These angular acceleration values ​​serve as acceleration compensation parameters and are included in the joint velocity correction command along with the angular velocity values. The acceleration compensation parameters are represented as a vector, denoted as [vector name missing]. Its components correspond to the joints and are used for subsequent servo driver feedforward compensation. Therefore, after each call to the pre-compiled servo control law, the output joint velocity correction instruction fully contains the angular velocity increments and acceleration compensation parameters of each joint.

[0042] Example 3: In specific implementation, please refer to Figure 2 The joint velocity correction commands undergo actuator dynamics constraint verification to generate verified joint velocity command values. The joint velocity correction commands, generated by the servo control law in the previous processing step, include the angular velocity increments and acceleration compensation parameters for each joint, stored in joint number order. Before performing the dynamics constraint verification, the maximum permissible angular velocity threshold and maximum permissible acceleration threshold for each joint are read from the robot controller's built-in dynamics parameter configuration file. The maximum permissible angular velocity threshold for each joint is denoted as The unit is radians per second, the first The maximum permissible acceleration threshold for each joint is denoted as . The unit is radians per second squared. For joint serial numbers, , Let be the number of degrees of freedom of the robot.

[0043] During the verification process, the angular velocity increment value of each joint in the joint velocity correction command is extracted. For the first joint... For each joint, the incremental angular velocity value parsed from the unverified joint velocity correction command is denoted as... .get Then, make a judgment Does the absolute value exceed the first? Maximum permissible angular velocity threshold for each joint .like The absolute value exceeds Then Amplitude limit to The sign remains consistent with the original angular velocity increment value; if The absolute value did not exceed Then keep Unchanged. Limiting operation. Use the following expression:

[0044] in, This represents the real number input value to be limited. Indicates the limit boundary value, and ; For a sign function, when The time value is ,when The time value is The increment of angular velocity after amplitude limiting is denoted as... ,satisfy .

[0045] Extract the acceleration compensation parameters from the joint velocity correction command. For the first... For each joint, the acceleration compensation parameters parsed from the unverified joint velocity correction command are denoted as follows: .get Then, make a judgment Does the absolute value exceed the first? Maximum permissible acceleration threshold for each joint .like The absolute value exceeds Then Amplitude limit to The sign remains consistent with the original acceleration compensation parameters; if The absolute value did not exceed Then keep Unchanged. The limit operation will continue to use the above method. function, at this time correspond , correspond The acceleration compensation parameters after amplitude limiting are denoted as ,satisfy .

[0046] After performing limiting checks on the angular velocity increment and acceleration compensation parameters for each joint in sequence, the limited angular velocity increment values ​​for all joints are then calculated. and acceleration compensation parameters after amplitude limiting The joints are rearranged according to their joint numbers and combined to form the verified joint speed command values. During the combination process, the corrected values ​​retain their original data format and corresponding joint indices. The verified joint speed command values ​​can be directly read by the servo driver without changing the timing arrangement of the original control cycle.

[0047] Example 4: In specific implementation, please refer to Figure 3 The process of acquiring visual feature image sequences of the target workpiece on the assembly worktable is completed collaboratively by an industrial camera, an image acquisition card, and a vision processing unit. The industrial camera is fixedly mounted above the assembly worktable or on the robot's end effector, with its optical axis perpendicular to the assembly work surface or at a preset angle. The fixed sampling frequency is set in the system configuration file.

[0048] The industrial camera is controlled to acquire continuous image frames of the target workpiece at a fixed sampling frequency. The fixed sampling frequency is denoted as... The unit is Hertz. The value of is determined based on the speed range of the robot's end effector and the bandwidth of the visual servo control. When the maximum speed of the robot's end effector does not exceed 0.5 meters per second and the visual servo control cycle is 10 milliseconds, the fixed sampling frequency is set to 100 Hz to ensure that at least one new image is available in each control cycle.

[0049] Each time the industrial camera triggers sampling, it outputs one frame of raw image data. This raw image data is temporarily stored as a pixel matrix in the frame buffer of the image acquisition card. The vision processing unit reads each frame of raw image data from the frame buffer and performs region-of-interest (ROI) cropping on each frame. The RIO is defined by pre-defined rectangular region parameters, which include the row number of the top-left pixel. Top left pixel column number Pixel height of rectangular area and the pixel width of the rectangular area Cropping processes extract portions from the original image data from the first... Arrive at the line, number Listed to number A rectangular region image, which contains all pixels of the target workpiece assembly features.

[0050] In some embodiments, the calibration process of the rectangular region parameters is completed during the system initialization phase: the target workpiece standard part is placed on the assembly workbench, the end effector is manually adjusted to the ideal assembly position, and a calibration image is captured; in the calibration image, a rectangular region containing the assembly features of the target workpiece is manually selected, and the upper left corner pixel coordinates, region width, and height of the rectangular region are recorded as region of interest parameters and stored in the parameter memory of the vision processing unit.

[0051] Image enhancement processing is performed on a rectangular region of the image. This process includes two steps: histogram equalization and median filtering for noise reduction. Histogram equalization is applied to the gray-level histogram of the rectangular region of the image, mapping the original gray-level distribution to a uniform distribution. The calculation formula is as follows:

[0052] in, Represents the first rectangular region in the image. line, number The original grayscale value and coordinates of the column pixels The value is arrive integers, coordinates The value is arrive Integers; This represents the grayscale value of the corresponding pixel after histogram equalization; This represents the number of gray levels in an image. For an 8-bit grayscale image, The value is 256; and These represent the pixel height and pixel width of the rectangular region image, respectively. Represents grayscale level Frequency of occurrence in a rectangular region of an image, gray level The value is arrive Integers; summation terms This indicates that the grayscale value in the rectangular region of the image is not greater than... Total number of pixels; symbol This indicates the floor function.

[0053] After histogram equalization, median filtering is applied to the equalized rectangular image for noise reduction. Median filtering uses a sliding window of size 3 pixels by 3 pixels. The window center iterates through every pixel of the rectangular image, and at each window position, the gray values ​​of the nine pixels within the window are sorted in ascending order. The fifth gray value in the sorted sequence is taken as the filtered output of the center pixel of the window. For pixels at the boundaries of the rectangular image, the portion of the median filtering window extending beyond the image boundary is not included in the sorting; only the effective pixel gray values ​​within the overlapping area of ​​the window and the image are used for median calculation.

[0054] After image enhancement, feature extraction is performed on the enhanced rectangular region image to generate a feature point coordinate matrix, edge contour pixel gradients, and surface texture grayscale distribution data. The feature point coordinate matrix is ​​generated using the Harris corner detection algorithm: the gradient components in the row and column directions of the enhanced rectangular region image are calculated, the autocorrelation matrix of each pixel is constructed, the corner response value of each pixel is calculated, and pixels whose corner response values ​​exceed a preset threshold and are local maxima are selected as feature points. The row and column indices of these feature points are arranged sequentially to form the feature point coordinate matrix. Each row of the matrix corresponds to one feature point, the first column stores the row index, and the second column stores the column index. Edge contour pixel gradients are obtained by applying the Sobel edge detection operator to the enhanced rectangular region image. The Sobel operator includes row and column gradient templates, which are convolved with the enhanced image to obtain the row and column gradient components of each pixel. Pixels with gradient magnitudes exceeding a preset edge strength threshold are selected to form an edge contour pixel set. The row and column gradient components of each pixel in the edge contour pixel set are stored sequentially to constitute the edge contour pixel gradient data. Surface texture grayscale distribution data is extracted from a preset texture region of the enhanced rectangular region image. The preset texture region is a uniform surface region without feature points and edge contours. The grayscale values ​​of all pixels in this region are arranged in row-major order into a one-dimensional vector to constitute the surface texture grayscale distribution data.

[0055] The feature point coordinate matrix, edge contour pixel gradient, and surface texture grayscale distribution data of multiple frames of images are serialized and stored using timestamp indexes to form a visual feature image sequence. Each frame's extracted feature point coordinate matrix, edge contour pixel gradient, and surface texture grayscale distribution data are treated as a data unit, coupled with a timestamp accurate to the microsecond level for the industrial camera's exposure time, and stored in the memory buffer of the vision processing unit. Multiple consecutive data units are arranged in ascending order of timestamps to form a visual feature image sequence. The sequence length is dynamically adjusted according to the visual servoing task requirements. The buffer retains the most recent 100 data units, and any excess data is overwritten sequentially.

[0056] See Figure 5 The figure shows the distribution of edge contour pixels and feature point positions of the target workpiece image extracted by the vision processing unit in Example 4. The horizontal axis represents the pixel column number. The value ranges from approximately 0 to 400 pixels; the vertical axis represents the pixel row number. The value ranges from approximately 0 to 300 pixels, and the vertical axis is arranged in reverse to match the image coordinate system.

[0057] The blue point cloud represents the edge contour pixels, distributed in an approximately rectangular outer contour boundary, with densely arranged stripes forming the edge contour of the assembled workpiece. This edge contour is obtained by applying the Sobel operator to the enhanced rectangular region image, selecting pixels with gradient magnitudes exceeding a preset threshold to ensure the continuity and integrity of the edge contour. The edge contour is indicated by pixel row number. The lowest point is close to 20 pixels, and the highest point is close to 280 pixels, pixel column number The range is approximately 50 to 350 pixels, displaying the spatial extent of the region of interest.

[0058] The red dots represent feature points identified by the Harris corner detection algorithm. These points are evenly distributed within the textured region inside the edge contour, indicating a sufficient number and uniform distribution that covers the key assembly features of the target workpiece. The pixel row and column coordinates of the feature points all fall within the pixel range of the blue edge contour, meeting the requirements for region of interest clipping based on the enhanced image in the feature point generation step. The feature point response values ​​exceed the threshold and are local maxima, ensuring the effectiveness and stability of the feature points.

[0059] Example 5: In practice, the process of driving robot joint movement based on the verified joint velocity command values, enabling the end effector to track the assembly position of the target workpiece, involves the coordinated operation of servo drivers and servo motors. The verified joint velocity command values ​​are output by the actuator dynamics constraint verification process, including the angular velocity increment value after limiting for each joint and the acceleration compensation parameters after limiting.

[0060] The verified joint velocity command values ​​are decomposed into position loop velocity reference values ​​and current loop torque reference values ​​for each joint. For the first... One joint, The range of values ​​is to , This represents the total number of robot joints, and the number of joints is extracted from the verified joint velocity command values. The angular velocity increment of the nth joint, The increment of angular velocity of each joint is denoted as The unit is radians per second. The [number]th... Angular velocity increment of each joint With the The current actual angular velocity measurements of each joint are added together to generate the first joint. Reference value for the position ring velocity of the first joint, the first The reference value for the ring velocity of each joint is denoted as... . No. The current actual angular velocity measurement of the first joint is obtained by the joint installed on the second joint. Incremental encoders on each joint servo motor axis detect and feed back data to the servo driver in real time. Simultaneously, the first joint speed command value is extracted from the calibrated joint speed command value. The acceleration compensation parameters of the first joint, the first The acceleration compensation parameters for each joint are denoted as follows: , will the Acceleration compensation parameters for each joint Multiply by the first The motor torque constant and transmission ratio of the first joint are used to generate the second joint. The reference value for the current loop torque of the first joint, the first The reference value for the current loop torque of each joint is denoted as: The motor torque constant is an inherent parameter of the servo motor, obtained from the servo motor nameplate data; the transmission ratio is the reduction ratio of the joint reducer, obtained from the robot structural design parameters.

[0061] The first Reference values ​​for the circumferential velocity of each joint. The position control loop of the servo drive is input to the position control loop. The position control loop of the servo drive includes a proportional controller, whose input is the position loop speed reference value. With the The difference between the actual angular velocity measurements of each joint is output as a proportional control quantity. This proportional control quantity is then integrated to generate the first... The position tracking error of the joint, the ... The position tracking error of each joint is denoted as The proportional gain and integral time constant of the proportional controller were tuned through step response tests during the system commissioning phase.

[0062] The first Joint position tracking error With the Reference value of current loop torque for each joint Superposition, generating the driving force The current command value for each joint motor winding is denoted as... The superposition method is algebraic summation. Current command value The signal is converted into an analog voltage signal by a digital-to-analog converter and input to the current loop regulator of the servo driver. The current loop regulator drives the power amplifier circuit and outputs a three-phase current with corresponding amplitude and frequency to the servo motor windings.

[0063] According to the current command value Controlling the output torque of the servo motor, the output torque of the servo motor is related to the current command value. Proportional to the torque constant of the servo motor. The output torque of the servo motor is amplified by the joint reducer and then drives the first... Each joint moves according to the verified joint speed command value, and the coordinated movement of each joint brings the end effector close to the assembly position of the target workpiece.

[0064] In the vision servoing process, a laser rangefinder is used to collect distance data between the end effector and the assembly table. The laser rangefinder is fixed to the end face of the end effector and includes three non-collinearly arranged laser range probes. The measurement beams of the three probes are emitted perpendicular to the end face of the end effector. The three non-collinear measurement points are selected from three fixed, non-collinear points on the end face of the end effector, and the distance values ​​of the three measurement points are denoted as follows: , , The unit is millimeters. The laser rangefinder sensor synchronously triggers a measurement once per visual servo control cycle, and the distance value is... , , The data is transmitted to the robot controller via a serial communication interface.

[0065] The tilt angle and deflection direction of the end effector face relative to the assembly table plane are calculated based on the distance values ​​of three non-collinear measurement points. An end effector face coordinate system is established in the robot controller, with the x and y axes lying in the end effector face plane and the z axis perpendicular to the end effector face pointing downwards. The x and y coordinates of the three non-collinear measurement points in the end effector face coordinate system are as follows: , , Combined with distance value , , The three-dimensional coordinates of three measurement points in space can be obtained. The plane containing the end effector's face is fitted using these three three-dimensional coordinates, and the plane's normal vector is calculated. The plane's normal vector is expressed in the end effector's face coordinate system as follows: The tilt angle is the angle between the plane normal vector and the z-axis of the end effector's coordinate system. The deflection direction is determined by the projection direction of the plane normal vector onto the xy-plane of the end effector's coordinate system. The formula for calculating the tilt angle is:

[0066] in, This represents the tilt angle of the end effector face relative to the assembly table plane, in radians, with a range of values ​​of [value missing]. ; This represents the component of the plane normal vector along the x-axis in the coordinate system of the end effector face; This represents the component of the plane normal vector along the y-axis in the coordinate system of the end effector face; This represents the component of the plane normal vector along the z-axis in the coordinate system of the end effector face; Represents the inverse cosine function; This represents absolute value operations.

[0067] The tilt angle and yaw direction are converted into a pose compensation vector. The pose compensation vector includes rotation compensation around the x-axis. and rotational compensation around the y-axis , and Determined by the components of the plane normal vector, satisfying , ,in This represents the arctangent function in the four quadrants. The pose compensation vector is superimposed on the visual servo feature error adjustment process to correct the feature point projection position offset and edge direction angle deviation. The correction method is to adjust the rotation compensation amount around the x-axis in the pose compensation vector. and rotational compensation around the y-axis Multiplying by a preset pose-image mapping matrix generates correction increments for the feature point projection position offset and edge direction angle deviation. These correction increments are then superimposed on the feature point projection position offset and edge direction angle deviation, respectively, to obtain the corrected visual servo feature error. The pose-image mapping matrix is ​​calculated offline based on the camera intrinsic parameters and the camera's mounting transformation matrix relative to the end effector face, and is stored in the robot controller's non-volatile memory.

[0068] See Figure 6 In the figure, the horizontal axis represents the visual servo control cycle, ranging from 0 to 500 control cycles; the vertical axis on the left represents the distance values ​​of the three measurement points measured by the non-collinear laser rangefinder. , and The unit is millimeters, and the value ranges from approximately 0 to 13 millimeters; the right vertical axis represents the tilt angle of the end effector face relative to the assembly table plane. The unit is radians, and the range is approximately 0 to 0.022 radians.

[0069] In the curve, the blue solid line, green dashed line, and purple dotted line correspond to the distance values ​​of three non-collinear measurement points, respectively. , and The changing trends of the three distance curves show an overall downward trend, indicating that the end effector gradually approaches the assembly position of the target workpiece on the assembly table. From control cycle 0 to the 250th, the distance value drops rapidly from about 12 mm to about 2 mm, indicating that the robot end effector completes a relatively large approach movement during this stage; from control cycle 250 to the 500th, the distance value tends to stabilize, fluctuating between 0 and 2 mm, reflecting that the end effector has approached the ideal assembly position and is maintaining minor adjustments.

[0070] The black dashed line represents the tilt angle. The tilt angle curve fluctuated throughout the 500 control cycles, with the amplitude mainly maintained in the range of 0 to 0.015 radians, occasionally showing a peak value close to 0.02 radians. This fluctuation reflects the minute attitude adjustments of the end effector face relative to the assembly table plane. The tilt angle did not show obvious monotonic changes, indicating that the robot continuously performed attitude compensation during tracking to keep the end face parallel to the assembly table.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A robot vision servo control method, characterized in that, The method includes: A sequence of visual feature images of the target workpiece on the assembly workbench is acquired. The sequence of visual feature images includes feature point coordinate matrix, edge contour pixel gradient and surface texture grayscale distribution data. The visual feature image sequence is subjected to end effector pose deviation analysis processing to generate the visual servo feature error of the target workpiece. The visual servo feature error includes feature point projection position offset, edge direction angle deviation value and texture matching residual vector. The pre-compiled servo control law is invoked to perform joint space mapping processing on the visual servo feature error, generating the joint velocity correction command of the end effector. The joint velocity correction command includes the angular velocity increment of each joint and acceleration compensation parameters. The joint velocity correction command is subjected to actuator dynamics constraint verification processing to generate a verified joint velocity command value. The robot joints are driven to move according to the verified joint speed command value, so that the end effector tracks the assembly position of the target workpiece.

2. The robot vision servo control method according to claim 1, characterized in that, The step of performing end-effector pose deviation analysis on the visual feature image sequence to generate the visual servo feature error of the target workpiece includes: The feature point coordinate matrix is ​​compared with the preset reference coordinate matrix to calculate the corresponding point difference, and the feature point projection position offset includes a horizontal offset component and a vertical offset component. The edge direction angle deviation value is calculated based on the difference in direction angle between the edge contour pixel gradient and the standard contour gradient. The edge direction angle deviation value characterizes the degree of pose deviation of the end effector in the rotation direction. The surface texture grayscale distribution data and the reference texture template are compared with pixel-level residual calculation to generate a texture matching residual vector, which includes grayscale mean deviation and local contrast difference information. The visual servo feature error is generated by concatenating the feature point projection position offset, edge direction angle deviation value, and texture matching residual vector.

3. The robot vision servo control method according to claim 2, characterized in that, The step of calculating the corresponding point difference between the feature point coordinate matrix and the preset reference coordinate matrix to obtain the feature point projection position offset includes: Extract the current image coordinate value of each feature point in the feature point coordinate matrix, wherein the current image coordinate value includes the pixel row number and the pixel column number; Extract the reference image coordinate values ​​of the corresponding feature points in the preset reference coordinate matrix; Calculate the coordinate difference between the current image coordinates and the reference image coordinates, and generate horizontal and vertical offset components; The horizontal and vertical offset components are combined into a two-dimensional vector of the feature point projection position offset.

4. The robot vision servo control method according to claim 1, characterized in that, The process of calling the pre-compiled servo control law to perform joint space mapping processing on the visual servo feature error and generating the joint velocity correction command for the end effector includes: The error attenuation factor at the current moment is calculated based on the visual servo feature error, and the error attenuation factor is positively correlated with the magnitude of the error vector. The visual servo feature error is multiplied by the error attenuation factor to generate a weighted error vector. The weighted error vector is multiplied by a preset servo gain matrix to generate a joint velocity increment vector. The dimension of the servo gain matrix is ​​matched with the number of robot joints. The joint velocity increment vector is accumulated with the joint velocity command of the previous cycle to generate the joint velocity correction command of the current cycle. The joint velocity correction command includes the angular velocity value and acceleration compensation value of each joint.

5. The robot vision servo control method according to claim 1, characterized in that, The step of performing actuator dynamics constraint verification on the joint velocity correction command to generate verified joint velocity command values ​​includes: Extract the angular velocity increment value of each joint in the joint velocity correction command, and determine whether the angular velocity increment value exceeds the maximum allowable angular velocity threshold of the joint; If the angular velocity increment exceeds the maximum permissible angular velocity threshold, then the angular velocity increment is limited to the maximum permissible angular velocity threshold; Extract the acceleration compensation parameter from the joint velocity correction command, and determine whether the acceleration compensation parameter exceeds the maximum allowable acceleration threshold of the joint. If the acceleration compensation parameter exceeds the maximum allowable acceleration threshold, then the acceleration compensation parameter is limited to the maximum allowable acceleration threshold. The limited angular velocity increment and acceleration compensation parameters are combined to form the verified joint velocity command value.

6. The robot vision servo control method according to claim 1, characterized in that, The sequence of visual feature images of the target workpiece on the assembly workbench includes: The industrial camera is controlled to acquire continuous image frames of the target workpiece at a fixed sampling frequency; Each frame of the image is processed by cropping the region of interest to extract a rectangular region image containing the assembly features of the target workpiece; The rectangular region image is subjected to image enhancement processing, which includes histogram equalization and median filtering for noise reduction; Feature extraction is performed on the enhanced image to generate the feature point coordinate matrix, edge contour pixel gradient, and surface texture grayscale distribution data. The feature point coordinate matrix, edge contour pixel gradient, and surface texture grayscale distribution data of multiple frames of images are serialized and stored using timestamp indexes to form the visual feature image sequence.

7. The robot vision servo control method according to claim 1, characterized in that, The step of driving robot joint movement according to the verified joint speed command value, so that the end effector tracks the assembly position of the target workpiece, includes: The verified joint velocity command value is decomposed into position loop velocity reference value and current loop torque reference value for each joint; The position loop velocity reference value is input to the position control loop of the servo driver to generate the joint position tracking error. The joint position tracking error is superimposed with the current loop torque reference value to generate the current command value of the drive motor winding; The servo motor outputs torque according to the current command value, driving the joint to move according to the verified joint speed command value, so that the end effector approaches the assembly position of the target workpiece.

8. The robot vision servo control method according to claim 1, characterized in that, The method further includes: The distance data between the end effector and the assembly worktable is collected using a laser rangefinder sensor. The distance data includes the distance values ​​of three non-collinear measurement points. The tilt angle and deflection direction of the end effector face relative to the assembly workbench plane are calculated based on the distance values ​​of the three non-collinear measurement points. The tilt angle and deflection direction are converted into a pose compensation vector, which includes rotation compensation amounts around the x-axis and y-axis. By superimposing the pose compensation vector onto the visual servo feature error adjustment process, the feature point projection position offset and edge direction angle deviation are corrected.

9. A robot vision servo control method according to claim 3, characterized in that, The offset of the projected position of the feature point is smoothly estimated using a Kalman filter.

10. A robot vision servo control system, characterized in that, The system includes a processor and a memory, the memory being connected to the processor. The memory is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the memory to implement the robot vision servo control method according to any one of claims 1-9.

Images