A method, system, device and medium for uncalibrated visual servoing for a six degree of freedom robotic arm

Abstract

The application discloses a kind of uncalibrated visual servo method, system, equipment and medium for six degrees of freedom mechanical arm, belong to visual servo technical field, including: the image frame before and after the movement of mechanical arm is acquired, feature point coordinates are extracted based on the image frame before and after movement;Determine initial image Jacobian matrix according to feature point coordinates;The initial image Jacobian matrix is updated, and current image Jacobian matrix is obtained;By comparing target image desired position with current observation position, image feature error is obtained, image feature error and current image Jacobian matrix are combined to calculate, and control increment is obtained;Control increment is converted into six degrees of freedom motion command, and position adjustment is carried out based on six degrees of freedom motion command control mechanical arm;Image frame of the current position of mechanical arm is acquired, feature point is extracted based on the image frame of current position and image feature error is updated.The application realizes high-precision, fast response, dynamic vision control of calibration-free.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual servoing, and in particular to a calibration-free visual servoing method, system, device and medium for a six-degree-of-freedom robot arm. BACKGROUND

[0002] At present, the robot visual servoing system widely adopts a method based on system modeling and calibration, constructs a mapping between an image feature space and a robot motion space by accurately obtaining internal and external parameters of a camera, a robot kinematic model and a hand-eye transformation relationship, and thus realizes target positioning and motion control; in recent years, in order to reduce the dependence on calibration and improve the flexibility of the system, researchers have proposed various uncalibrated visual servoing (UVS) methods, which are usually based on an “eye-in-hand” structure, collect image feature changes before and after motion by controlling the robot end to move along each degree of freedom in a small range, and estimate an image Jacobian matrix online by using a least squares method, a recursive least squares method or a dynamic quasi-Newton method, so as to realize image servo control of a static or dynamic target, typical researches including recursive estimation method and least squares method using multi-degree-of-freedom information.

[0003] Although the existing uncalibrated visual servoing technology alleviates the dependence on traditional system calibration to some extent, there are still the following outstanding problems: firstly, in the initialization process, there is motion coupling between each degree of freedom of the robot, which leads to too strong linear correlation between image feature changes, and thus causes singularity or irreversibility of the image Jacobian matrix, affecting control accuracy and system stability; secondly, the current methods mostly do not effectively compensate for the time delay between image acquisition, processing and mechanical motion, which easily leads to large dynamic target tracking error and response lag; in addition, part of the improved methods depend on neural networks, sample training and other means, although they can improve the adaptability of the system, but the model training is complex and difficult to quickly deploy. SUMMARY

[0004] In view of the above problems, the present application is proposed.

[0005] Therefore, the present application solves the technical problem of how to make a six-degree-of-freedom robot arm realize accurate dynamic tracking and positioning of a target in an unknown or changing environment by constructing a visual servoing control method without system calibration, using an "eye-in-hand" structure, collecting multiple sets of image feature changes by controlling the robot arm to move in the orthogonal directions of each degree of freedom, estimating and iteratively updating an image Jacobian matrix in real time, thereby establishing a mapping relationship between the image space and the motion space, and introducing a predictive compensation proportional-integral (PI) control algorithm based on image errors to effectively offset the system processing delay, realize closed-loop control of dynamic targets, and get rid of the dependence on traditional camera-robot calibration, with the characteristics of fast initialization, stable control, and accurate tracking, and improve the adaptability and engineering practicability of the visual servoing method.

[0006] To solve the above technical problems, the present application provides the following technical solutions: a non-calibration visual servoing method for a six-degree-of-freedom robot arm, comprising,

[0007] Obtaining image frames before and after the movement of the robot arm, extracting feature point coordinates based on the image frames before and after the movement; determining an initial image Jacobian matrix according to the feature point coordinates; updating the initial image Jacobian matrix to obtain a current image Jacobian matrix; obtaining an image feature error by comparing the expected position of the target image with the current observed position, and calculating the control increment by combining the image feature error and the current image Jacobian matrix; converting the control increment into a six-degree-of-freedom motion command, and controlling the robot arm to adjust the position based on the six-degree-of-freedom motion command; obtaining an image frame of the current position of the robot arm, extracting feature points based on the image frame of the current position, and updating the image feature error.

[0008] As a preferred scheme of the non-calibration visual servoing method for a six-degree-of-freedom robot arm, wherein: determining the initial image Jacobian matrix according to the feature point coordinates comprises: calculating the feature change amount before and after the movement of the robot arm based on the feature point coordinates; establishing a corresponding relationship between the feature change amount and the pose change amount of the robot arm; and using the least squares method to estimate the corresponding relationship to obtain the initial image Jacobian matrix.

[0009] As a preferred scheme of the non-calibration visual servoing method for a six-degree-of-freedom robot arm, wherein: updating the initial image Jacobian matrix to obtain the current image Jacobian matrix comprises: recording the image feature change amount and the pose change amount in each control period; and using the dynamic quasi-Newton method to iteratively update the initial image Jacobian matrix by combining the image feature change amount and the pose change amount to obtain the current image Jacobian matrix.

[0010] As a preferred embodiment of the calibration-free visual servoing method for a six-degree-of-freedom robotic arm described in this invention, the control increment is calculated by combining the image feature error and the current image Jacobian matrix, including: processing the image feature error using a predictive compensation PI control algorithm to generate proportional, integral, and predictive compensation control quantities; and converting the proportional, integral, and predictive compensation quantities into control increments through the inverse transformation of the current image Jacobian matrix.

[0011] As a preferred embodiment of the calibration-free visual servoing method for a six-degree-of-freedom robotic arm described in this invention, the method involves: processing the image feature error using a predictive compensation PI control algorithm to generate proportional, integral, and predictive compensation quantities, including: calculating the proportional and integral control quantities based on the image feature error; and determining the predictive compensation quantity using the image feature change rate and compensation coefficient.

[0012] As a preferred embodiment of the calibration-free visual servoing method for a six-degree-of-freedom robotic arm described in this invention, the step of acquiring image frames before and after the robotic arm moves, and extracting feature point coordinates based on the image frames before and after the movement, includes: controlling the robotic arm to move independently along the six degrees of freedom directions in sequence under the eye-hand structure, and acquiring image frames before and after each movement; extracting feature points based on the image frames to obtain feature point coordinates.

[0013] As a preferred embodiment of the calibration-free visual servoing method for a six-degree-of-freedom robotic arm described in this invention, the independent movement along the six degrees of freedom in sequence refers to orthogonal movement along the translational directions of the X-axis, Y-axis, and Z-axis and the rotational directions around the X-axis, Y-axis, and Z-axis, respectively.

[0014] This invention provides a calibration-free vision servo system for a six-degree-of-freedom robotic arm.

[0015] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a calibration-free vision servoing system for a six-degree-of-freedom robotic arm, comprising: a feature point acquisition module, used to acquire image frames before and after the robotic arm moves, and extract feature point coordinates based on the image frames before and after the move; an image Jacobian matrix determination module, used to determine an initial image Jacobian matrix based on the feature point coordinates; a matrix update module, used to update the initial image Jacobian matrix to obtain the current image Jacobian matrix; a control increment module, used to obtain the image feature error by comparing the desired position of the target image with the current observation position, and calculate the control increment by combining the image feature error and the current image Jacobian matrix; a position adjustment module, used to convert the control increment into a six-degree-of-freedom motion command, and control the robotic arm to perform position adjustment based on the six-degree-of-freedom motion command; and a closed-loop feedback module, used to acquire an image frame of the current position of the robotic arm, extract feature points based on the image frame of the current position, and update the image feature error.

[0016] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the aforementioned calibration-free visual servoing method for a six-degree-of-freedom robotic arm.

[0017] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the aforementioned calibration-free visual servoing method for a six-degree-of-freedom robotic arm.

[0018] The beneficial effects of this invention are as follows: By employing orthogonal translation to ensure the linear independence of the motion directions of each degree of freedom, and combining it with the dynamic quasi-Newton method to update the image Jacobian matrix in real time, this invention can continuously adapt to changes in target position, achieving high-precision image feature error convergence and end-effector pose control. The strategy of combining orthogonal initialization with dynamic quasi-Newton iteration effectively avoids the problem of Jacobian matrix invertibility, enabling rapid and accurate tracking. The invention introduces a predictive compensation control term based on the rate of change of image features, calculating the control increment through a predictive compensation PI control algorithm, reducing the delay between image acquisition and execution, minimizing dynamic target tracking errors, and improving overall response capability and real-time performance. It eliminates the need for pre-calibrating camera intrinsic and extrinsic parameters and robotic arm model parameters, adapting to practical application scenarios such as camera or arm position fine-tuning and target changes, facilitating rapid deployment, and achieving high-precision, fast-response, and calibration-free dynamic visual control. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an overall flowchart of a calibration-free vision servoing method for a six-degree-of-freedom robotic arm, provided as an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of an eye-in-hand model provided in one embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of a camera pinhole imaging model provided in one embodiment of the present invention.

[0023] Figure 4 This is a framework diagram of a robot servo system provided in one embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of pixel error of feature points on an image provided in an embodiment of the present invention.

[0025] Figure 6 This is a schematic diagram of the angle error of a robotic arm provided in one embodiment of the present invention.

[0026] Figure 7 This is a schematic diagram of the positional error of an actual feature point provided in an embodiment of the present invention.

[0027] Figure 8 This is a schematic diagram illustrating the dynamic tracking error comparison according to an embodiment of the present invention. Detailed Implementation

[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0030] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0031] Example 1, referring to Figure 1 As one embodiment of the present invention, this embodiment provides a calibration-free visual servoing method for a six-degree-of-freedom robotic arm, comprising:

[0032] S100: Acquire image frames before and after the robotic arm moves, and extract feature point coordinates based on the image frames before and after the movement.

[0033] S200: Determine the initial image Jacobian matrix based on the coordinates of the feature points.

[0034] S300: Update the initial image Jacobian matrix to obtain the current image Jacobian matrix.

[0035] S400: By comparing the expected position of the target image with the current observation position, the image feature error is obtained. The control increment is then calculated by combining the image feature error with the current image Jacobian matrix.

[0036] S500: Converts control increments into six-degree-of-freedom motion commands, and controls the robotic arm to adjust its position based on these six-degree-of-freedom motion commands.

[0037] S600: Acquires an image frame of the current position of the robotic arm, extracts feature points based on the image frame of the current position, and updates the image feature error.

[0038] It should be noted that traditional visual servoing methods require precise calibration of camera intrinsic and extrinsic parameters and robotic arm kinematic parameters. The calibration process is complex and prone to introducing cumulative errors. In practical applications, factors such as changes in camera position, robotic arm wear, and changes in ambient lighting can cause calibration parameters to fail, resulting in decreased control accuracy. At the same time, traditional methods use a fixed image Jacobian matrix, which cannot adapt to changing working environments and target characteristics, leading to insufficient robustness of the control system when facing tasks. Moreover, existing control algorithms lack effective compensation mechanisms for system delays and external interference, often resulting in slow convergence speed and low control accuracy in high-precision operation scenarios, which seriously affects the application effect of six-degree-of-freedom robotic arms in precision assembly, medical surgery, and spatial operations.

[0039] Therefore, to address the aforementioned issues of calibration complexity, poor adaptability, and insufficient control precision, a calibration-free visual servo control method is constructed through steps S100-S600. This method utilizes the robotic arm's motion to establish an initial image Jacobian matrix, avoiding a complex calibration process. The Jacobian matrix is ​​updated in real-time using a dynamic quasi-Newton method, improving adaptability to environmental changes. Furthermore, a predictive compensation PI control algorithm effectively suppresses the impact of system time delay, achieving high-precision closed-loop visual servo control and enhancing the operational accuracy and robustness of the six-degree-of-freedom robotic arm in real-world environments.

[0040] Example 2, refer to Figures 1-4 As one embodiment of the present invention, based on the previous embodiment, a calibration-free visual servoing method for a six-degree-of-freedom robotic arm is provided, comprising:

[0041] It should be noted that, in this embodiment of the invention, the robot platform used is a six-axis industrial robot independently developed by Bozhilin. A Daheng high-speed industrial camera is installed at the end of the robotic arm to collect image information within the working range of the robotic arm. The camera used needs to have a large field of view because the target object cannot leave the camera's field of view during the uncalibrated initialization process; otherwise, the Jacobian matrix error will increase. The camera and robot are installed in an "eye-in-hand" mode, as shown in the schematic diagram below. Figure 2 As shown.

[0042] In this embodiment of the invention, step S100 involves acquiring image frames before and after the robotic arm moves, and extracting feature point coordinates based on the image frames before and after the movement, including the following steps A1-A2:

[0043] A1: Control the robotic arm to move independently along six degrees of freedom in sequence under the eye-hand structure, and acquire image frames before and after each movement.

[0044] A2: Extract feature points based on image frames to obtain feature point coordinates.

[0045] Specifically, in step A1, the robotic arm is controlled to move independently along six degrees of freedom in sequence under the eye-hand structure, and image frames are acquired before and after each movement. The specific operation can be as follows:

[0046] The control program sequentially drives the six-degree-of-freedom robotic arm to perform orthogonal displacements along the X, Y, and Z translation axes and around the X, Y, and Z rotation axes, with the motion amplitude in each direction set to 0.2 mm or 0.2°.

[0047] The camera mounted at the end captures image frames before and after each movement.

[0048] Specifically, in step A2, feature points are extracted based on the image frame to obtain the coordinates of the feature points. The specific operation can be as follows:

[0049] By extracting the position coordinates of three sets of stable feature points (such as corner points or center mass points) from the image frame, the process is completed collaboratively by the robotic arm and the end-effector camera. The target feature points in the image frame are the objects of the operation. The final output is six sets of data pairs corresponding to image features and pose changes, which provides a basis for subsequent Jacobian matrix estimation.

[0050] It should be noted that this invention achieves a precise establishment of the correspondence between image feature changes and robotic arm pose changes by controlling the robotic arm to move independently along six degrees of freedom in sequence and extracting stable feature point coordinates by acquiring image frames before and after each movement. Compared with the problem in the prior art where there is motion coupling between the degrees of freedom of the robotic arm, resulting in excessive linear correlation between image feature changes, this invention ensures the linear independence of each degree of freedom by adopting orthogonal motion, avoiding the problem of singular or irreversible image Jacobian matrix. In particular, by setting the motion amplitude of each direction to an orthogonal displacement of 0.2mm or 0.2°, it ensures that the feature points do not leave the camera's field of view and obtains sufficient image feature change information. This not only improves the stability and reliability of the initialization process but also lays a solid foundation for the accurate estimation of the Jacobian matrix in the future, effectively solving the problem of insufficient decoupling capability in the initialization stage of traditional uncalibrated visual servoing methods.

[0051] It should be noted that the uncalibrated technique, like the traditional calibration technique, is used to describe the relationship between the speed of the robot's end effector and the rate of change of features in the image, including the following steps B1-B5:

[0052] B1: As Figure 3 The image shown is a schematic diagram of a camera pinhole imaging model, which assumes the existence of a point in three-dimensional space. , and In the pixel plane coordinate system axis pixel coordinates and Pixel coordinates in the pixel plane coordinate system are: Based on the traditional camera pinhole imaging model, points can be obtained. The projected coordinates are as follows:

[0053] ;

[0054] in, The focal length of the camera; , and For point In the camera coordinate system axis, shaft and Axis coordinates, point The coordinates in the camera coordinate system are ; and For point In the camera plane coordinate system Axis projection coordinates and Axis projection coordinates, point The projected coordinates in the camera plane coordinate system are: .

[0055] Among them, point Projected coordinates in the camera plane coordinate system pixel coordinates in the pixel plane coordinate system The relationship between them can be specifically represented as:

[0056] ;

[0057] in, and The pixel coordinates of the point where the camera's optical axis passes through the pixel plane; and Each pixel in the pixel plane and The spatial distance represented by the direction.

[0058] B2: Move the points from step B1 Projected coordinates in the camera plane coordinate system pixel coordinates in the pixel plane coordinate system The relationship between them can be converted into a matrix equation, which can be specifically expressed as:

[0059] ;

[0060] in, and In the pixel plane coordinate system axis pixel coordinates and Axis pixel coordinates.

[0061] B3: Assume the camera focal length is... Under the ideal pinhole model of the "eye in hand" system, the transformation relationship between the camera coordinate system and the pixel coordinate system can be specifically expressed as:

[0062] ;

[0063] in, and In the pixel plane coordinate system axis pixel coordinates and Axis pixel coordinates; , and For point In the camera coordinate system axis, shaft and Axis coordinates.

[0064] B4: Based on the motion equations of the robot's end effector Point Coordinates in the camera coordinate system The analysis was conducted, among which The position of the end effector in the base coordinate system. This is the angular velocity vector of the end effector. Given the linear velocity vector of the end effector, we obtain the point. Coordinates in the camera coordinate system The motion relationships are as follows:

[0065] ;

[0066] in, , and For point In the camera coordinate system axis, shaft and Axis coordinates; This represents the translation component of the robotic arm's end effector in the camera coordinate system; This represents the rotational component of the robotic arm's end effector in the camera coordinate system.

[0067] Point Coordinates in the camera coordinate system The kinematic relationships can be converted into matrix equations, which can be specifically expressed as:

[0068] ;

[0069] in, and For point In the pixel plane coordinate system Speed ​​on the shaft and Speed ​​on the shaft; and In the pixel plane coordinate system axis pixel coordinates and Axis pixel coordinates; This is the angular velocity vector of the end effector; This is the linear velocity vector of the end effector.

[0070] B5: In practical applications, it is impossible to obtain the result by measuring every variable in the matrix equation in step B4. and The transformation matrix between the two is used; therefore, the variables in the matrix are considered unknown, and the matrix equation can be specifically expressed as:

[0071] ;

[0072] in, For unknown constants in the matrix; These represent the translational (X / Y / Z axis) and rotational (around the X / Y / Z axis) degrees of freedom in the corresponding Cartesian space.

[0073] In this embodiment of the invention, step S200, which determines the initial image Jacobian matrix based on the feature point coordinates, includes the following steps C1-C3:

[0074] C1: Calculate the changes in features before and after the robotic arm moves based on the coordinates of feature points.

[0075] C2: Establish the correspondence between the feature changes and the pose changes of the robotic arm.

[0076] C3: The correspondence is estimated using the least squares method to obtain the initial image Jacobian matrix.

[0077] Specifically, in step C1, calculating the feature change before and after the robotic arm moves based on the feature point coordinates refers to obtaining the change of feature points (pixel coordinates) when moving in an independent direction. Six sets of robotic arm movements and image feature changes are used as input to form image feature change data.

[0078] Specifically, step C2 establishes the correspondence between the feature changes and the pose changes of the robotic arm, including the following steps C21-C23:

[0079] C21: Due to step B5 These correspond to the degrees of freedom of translation (X / Y / Z axes) and rotation (around X / Y / Z axes) in Cartesian space, respectively. However, on a six-axis robotic arm platform, a single feature pixel does not meet the size requirements, so three feature points are superimposed, which can be represented as follows:

[0080] ;

[0081] in, These are unknown constants in the matrix; pixel coordinates in the pixel plane coordinate system The derivative of .

[0082] set up The rate of change of image features, Let be the Jacobian transformation matrix. Let be the motion vector of the robotic arm's end effector. Then the above equation can be expressed as:

[0083] ;

[0084] in, This represents the rate of change of image features.

[0085] C22: In practical applications, it is necessary to transform the two rates of change of image features. To obtain the motion vector of the robotic arm end effector Therefore, the inverse of the Jacobian transformation matrix is ​​required. , The image Jacobian matrix can be specifically represented as:

[0086] ;

[0087] in, This is the motion vector of the robotic arm's end effector.

[0088] C23: Since the two rates of change of image features are obtained from two adjacent images in practical applications, the equation also needs to be discretized. That is, in the process of high-frequency camera image retrieval, assuming that the Jacobian matrix of two adjacent frames remains approximately unchanged, the discrete equation can be expressed as:

[0089] ;

[0090] ;

[0091] ;

[0092] in, The image features at discrete time point n+1; The image features at discrete time point n; Let be the image Jacobian matrix at discrete time point n; This represents the change in end pose from time point n to n+1. The pose of the robotic arm's end effector at discrete time point n+1; The end effector pose of the robotic arm at discrete time point n; The image Jacobian matrix at discrete time point n. The inverse matrix; The image Jacobian matrix; This represents the change in image features between adjacent frames; This represents the change in pose at the end of adjacent frames; It is the generalized inverse matrix of pose change.

[0093] Specifically, in step C3, the least squares method is used to estimate the correspondence to obtain the initial image Jacobian matrix. This refers to the process after image acquisition and feature point localization, where the main control unit of the servo control system uses the extracted image features to change... pose changes at the end effector of the robotic arm The initial value of the image Jacobian matrix is ​​obtained by using the least squares estimation algorithm. In order to improve the stability and invertibility of the image Jacobian matrix, an orthogonal motion strategy is adopted to make the changes of each degree of freedom linearly independent.

[0094] Furthermore, to obtain a more accurate Jacobian matrix, step C3 decomposes the robot arm's motion into degrees of freedom during the initialization process by standardizing the robot arm's motion direction during initialization. The independent motion of these features makes the resulting feature point sets naturally linearly independent; for example, in independent motion... When moving in a direction, the change in feature points (pixel coordinates) can be specifically represented as:

[0095] ;

[0096] ;

[0097] in, These are unknown constants in the matrix; pixel coordinates in the pixel plane coordinate system The derivative of .

[0098] It should be noted that this invention establishes a precise correspondence between image feature changes and robotic arm pose changes by calculating the changes in feature point coordinates before and after the robotic arm moves, and estimates the initial image Jacobian matrix using the least squares method. Compared with the problem in the prior art where motion coupling between the degrees of freedom of the robotic arm leads to singular or irreversible image Jacobian matrices, this invention ensures the linear independence of changes in each degree of freedom by adopting an orthogonal motion strategy, avoiding numerical instability in the matrix solution process. In particular, by independently processing the translation and rotation degrees of freedom in Cartesian space and using the method of superimposing three feature points to meet the size requirements of a six-axis robotic arm, it not only improves the accuracy and stability of Jacobian matrix estimation, but also provides reliable initial values ​​for subsequent online iterative updates. This effectively solves the problem of excessive linear correlation in the initialization process of traditional uncalibrated visual servoing methods, and improves control accuracy and stability.

[0099] In this embodiment of the invention, step S300 updates the initial image Jacobian matrix to obtain the current image Jacobian matrix, including the following steps D1-D2:

[0100] D1: Records the changes in image features and pose within each control cycle.

[0101] D2: By using the dynamic quasi-Newton method, the Jacobian matrix of the initial image is iteratively updated by combining the changes in image features and pose, and the current image Jacobian matrix is ​​obtained.

[0102] Specifically, recording the changes in image features and pose within each control cycle in step D1 means that during operation, the controller records the changes in image features in real time at a period of 25ms. and pose change at the end of the robotic arm .

[0103] Specifically, in step D2, the iterative update of the initial image Jacobian matrix using the dynamic quasi-Newton method, combined with changes in image features and pose, means that after initializing the image Jacobian matrix, it is necessary to update and iterate the image Jacobian matrix in real time to ensure the accuracy of the robot's operation. This includes the following steps D21-D22:

[0104] D21: In the image plane, actual features With expected characteristics The difference is ,in For joint angle, For time; for the deviation function Perform a Taylor expansion and define a radiation model. Radiation model Specifically, it can be expressed as:

[0105] ;

[0106] Among them, deviation function at time It can be represented as:

[0107] ;

[0108] in, In order to be in The deviation function at time; To describe the changes in the joint angles of the robotic arm from arrive Changes in image features caused by this; In order to be in The deviation function at time; In order to be in Radiation model at time; To describe the changes in the joint angles of the robotic arm Changes in image features caused by this; This is the partial derivative of the characteristic deviation with respect to time; for The time of a moment.

[0109] D22: Combined in Time Deviation Function The Jacobian matrix of the initial image is iteratively updated, which can be specifically expressed as:

[0110] ;

[0111] in, Let be the Jacobian matrix of the image at time k; Let Jacobian matrix be the image at time k-1; This represents the change in characteristic deviation. This represents the change in joint angle. This is the partial derivative of the characteristic deviation with respect to time; For time intervals; This is the transpose of the change in joint angle. This is a transpose.

[0112] It should be noted that this invention updates the image Jacobian matrix in real time using a dynamic quasi-Newton method, combined with high-frequency data acquisition and orthogonal motion strategies within the control cycle, ensuring the stability and invertibility of the matrix. Compared with existing technologies that use fixed Jacobian matrices or simple recursive updates, this invention solves the problem of Jacobian matrix accuracy decaying over time and difficulty adapting to environmental changes in traditional methods through an online iterative update mechanism. In particular, by introducing Taylor expansion and radiation models, it can effectively capture the nonlinear mapping relationship between image features and robotic arm pose, which not only improves the accuracy and stability of visual servo control, but also enhances the adaptability to environmental changes and target motion, ensuring the accurate tracking performance of the robotic arm in real-world scenarios and providing a reliable mathematical foundation for achieving high-precision, calibration-free visual servo control.

[0113] It should be noted that the operation process of the robot vision servo control system is as follows: First, the vision system captures and processes the image, and then inputs the processed image information into the robot controller to start the robot's movement. However, there is a time delay between the vision system capturing the image and the robot starting to move, which will cause system errors when the robot tracks dynamic targets. Therefore, in the robot motion control process, this invention designs a Jacobian matrix PI control algorithm with predictive compensation to reduce system errors caused by time lag.

[0114] In this embodiment of the invention, step S400 involves comparing the desired position of the target image with the current observation position to obtain the image feature error. Specifically, this process can be as follows:

[0115] After the controller receives the real-time image, it will extract the current feature location. With target feature location By comparing the results, the image feature error was calculated. Specifically, it can be expressed as:

[0116] ;

[0117] in, This represents the image feature error.

[0118] In this embodiment of the invention, step S400 combines the image feature error and the current image Jacobian matrix to calculate the control increment, including the following steps E1-E2:

[0119] E1: The predictive compensation PI control algorithm is used to process image feature errors and generate proportional, integral control quantities and predictive compensation quantities.

[0120] E2: Convert the proportional, integral, and predictive compensation terms into control increments through the inverse transformation of the current image Jacobian matrix.

[0121] Specifically, the predictive compensation PI control algorithm in step E1 is constructed, including the following steps E11-E13:

[0122] E11: To improve the real-time performance of the system and ensure effective tracking even at high target speeds, a prediction compensation method is introduced into the visual servoing control algorithm based on the inverse Jacobian matrix. A Jacobian matrix PI control algorithm with prediction compensation is designed, where the system image feature error... Defined as:

[0123] ;

[0124] in, For current image features; The desired image features.

[0125] Predicted compensation amount Defined as:

[0126] ;

[0127] in, is the rate of change of image features; k1 is the compensation coefficient, which is related to the rate of change of image features.

[0128] E12: In the dynamic target tracking process, in order to reduce the system tracking error, a PI control algorithm is introduced into the inverse Jacobian matrix control algorithm. The control quantity can be specifically expressed as:

[0129] ;

[0130] in, n is the control variable; n is the discrete time point; Predict changes in image features; The predicted image features for the next time step; The predicted image features at the current moment.

[0131] E13: To reduce the impact of system image processing time delay on the system, the predicted compensation amount will be... Substituting these values ​​into the control algorithm, we obtain the final visual servo control algorithm, which can be specifically expressed as:

[0132] ;

[0133] in, This is the control quantity used in the final visual servo control algorithm; It is the inverse Jacobian matrix; and These are the proportional coefficient and the integral coefficient, respectively. Let be the system image feature error at time point n; k1 is the compensation coefficient, which is related to the rate of change of the image features; by combining the robot control system and the vision system, a closed-loop robot vision servo system is formed, specifically as follows: Figure 4 As shown.

[0134] Furthermore, in step E1, the image feature error is processed using a predictive compensation PI control algorithm to generate proportional and integral control quantities and predictive compensation quantities, including the following steps E14-E15:

[0135] E14: Calculate the proportional and integral control terms based on the current image feature error.

[0136] E15: Determine the predicted compensation amount by using the rate of change of image features and the compensation coefficient.

[0137] Specifically, in step E14, calculating the proportional and integral control terms based on the current image feature error refers to... After the calculation is completed, the controller will calculate the image feature error. As input, the proportional and integral control terms and the predictive compensation term are substituted into the PI proportional term, integral term, and predictive compensation term to calculate the proportional and integral control terms, which can be specifically expressed as:

[0138] ;

[0139] in, It is the sum of the proportional and integral control terms; This is a proportional control item; This is the proportionality coefficient; For integral control items; For integration variables; is the integral coefficient.

[0140] Specifically, in step E15, determining the predicted compensation amount through the image feature change rate and compensation coefficient refers to compensating for the system delay between image processing and action execution. Step E11 further introduces the predicted compensation amount. .

[0141] Specifically, in step E2, converting the proportional, integral, and predictive compensation terms into control increments through the inverse transformation of the current image Jacobian matrix involves merging the proportional, integral, and predictive compensation terms and inputting the inverse of the image Jacobian matrix to obtain the control increments. This can be expressed as follows:

[0142] ;

[0143] in, To control the increment, it represents the attitude adjustment of the robotic arm end effector in six-dimensional space, which is used to guide its movement towards the desired feature point, thereby achieving high-precision closed-loop visual servo control; a causal closed loop is formed between the image error and the control output, enabling the system to have good stability, accuracy and dynamic response capabilities.

[0144] In this embodiment of the invention, step S500 converts the control increment into a six-degree-of-freedom motion command, and controls the robotic arm to adjust its position based on the six-degree-of-freedom motion command, including the following steps F1-F2:

[0145] F1: Due to control increment It is a six-dimensional fine-tuning vector calculated based on image errors, containing incremental information about the end effector's position and pose in space, thus controlling the increment. With current end pose Add them together to get the target pose. .

[0146] F2: Set the target pose The commands are converted into a six-degree-of-freedom instruction format executable by the industrial robotic arm and sent to the robot controller via a motion control interface (such as MoveL or a path interpolation module). This drives the end effector to perform fine adjustments, achieving high-precision closed-loop tracking control of the target. Then, the next sampling cycle begins, and the industrial camera re-acquires images to achieve closed-loop feedback. The main components of this process are the controller and the actuator, and the objects of action are image errors and mechanical motion commands. The final output is high-precision continuous tracking of the dynamic target.

[0147] In this embodiment of the invention, step S600 involves acquiring an image frame of the current position of the robotic arm, extracting feature points based on the image frame of the current position, and updating the image feature error, including:

[0148] After each control cycle, images are reacquired and the current feature point positions are extracted, and the new image feature error is calculated again. Enter the next control cycle, where image feature error As a closed-loop feedback signal, it continuously drives the system to make high-precision adjustments and is a key parameter for realizing visual servo closed-loop control. This feedback based on image error constitutes the closed-loop adjustment mechanism of visual servo.

[0149] It should be noted that this invention establishes a complete closed-loop feedback control mechanism, converting control increments into six-degree-of-freedom motion commands executable by the robotic arm, and achieving continuous adjustment through real-time image acquisition and error updates. Compared with open-loop control or simple feedback control methods in the prior art, this invention solves the problems of unstable control accuracy and inability to achieve continuous tracking in traditional methods by constructing closed-loop control. This not only improves the stability and reliability of control, but also achieves accurate and continuous tracking of the target, ensuring the closed-loop control performance of the entire visual servo system, and providing a complete solution for high-precision, calibration-free visual servo control of a six-degree-of-freedom robotic arm.

[0150] In summary, this invention ensures linear independence of motion directions for each degree of freedom by employing orthogonal translation, and updates the image Jacobian matrix in real time using a dynamic quasi-Newton method. This allows for continuous adaptation to changes in target position, achieving high-precision image feature error convergence and end-effector pose control. The strategy of combining orthogonal initialization with dynamic quasi-Newton iteration effectively avoids the irreversibility problem of the Jacobian matrix, enabling rapid and accurate tracking. The introduction of a predictive compensation control term based on the rate of change of image features, using a predictive compensation PI control algorithm to calculate the control increment, reduces the delay between image acquisition and execution, minimizes dynamic target tracking errors, and improves overall responsiveness and real-time performance. Furthermore, it eliminates the need for pre-calibrating camera intrinsic and extrinsic parameters and robotic arm model parameters, adapting to practical scenarios such as camera or arm position fine-tuning and target changes, facilitating rapid deployment, and achieving high-precision, fast-response, and calibration-free dynamic vision control.

[0151] Example 3, referring to Figures 5-8 As one embodiment of the present invention, a calibration-free vision servoing method for a six-degree-of-freedom robotic arm is provided. To verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0152] This embodiment uses a six-DOF industrial robotic arm (Bozhilin six-axis robot) as the operating platform, and installs an industrial camera with a resolution of 1024×1024 and a focal length of 8mm at its end effector to construct an "eye-to-hand" visual servo system; firstly, the end effector of the robotic arm is controlled along the six degrees of freedom. Orthogonal displacements (with amplitudes of 0.2 mm or 0.2°) were performed, and images were captured by the camera before and after each movement. Three sets of feature point coordinates were extracted, and the initial value of the image Jacobian matrix was estimated using the least squares method. After orthogonal decoupling optimization, the result was as follows: Figure 5 , Figure 6 and Figure 7 The pixel error, angle error, and position error results shown in the figure demonstrate that the present invention achieves a pose error of 0.32 mm in only 5 iterations. The specific results are shown in Table 1.

[0153] Table 1. Optimized Feature Point Iteration Error Data Table

[0154] ,

[0155] Subsequently, during dynamic target tracking, the control quantity is calculated based on image feature errors, and the end effector motion is adjusted using a PI control algorithm with predictive compensation. Simultaneously, a dynamic quasi-Newton method is used to update the image Jacobian matrix online, thereby achieving real-time target tracking control. The entire process operates in a closed loop within a 25ms control cycle, requiring no system calibration information. As shown in Table 2, the experimental results demonstrate that this invention can reduce the image error from 133.69 pixels to 0.00087 pixels and the pose error from 174.48mm to 0.00047mm within 50 iterations, while keeping the tracking error within 0.3mm. Specifically... Figure 8 As shown, compared with the comparison method, the present invention converges faster, has higher accuracy, and has good dynamic response and robustness.

[0156] Table 2 Feature Point Iteration Error Data Table

[0157] ,

[0158] Example 4 is an embodiment of the present invention, which provides a calibration-free vision servoing system for a six-degree-of-freedom robotic arm, comprising: a feature point acquisition module for acquiring image frames before and after the robotic arm moves, and extracting feature point coordinates based on the image frames before and after the move; an image Jacobian matrix determination module for determining an initial image Jacobian matrix based on the feature point coordinates; a matrix update module for updating the initial image Jacobian matrix to obtain the current image Jacobian matrix; a control increment module for obtaining the image feature error by comparing the desired position of the target image with the current observation position, and calculating the control increment by combining the image feature error and the current image Jacobian matrix; a position adjustment module for converting the control increment into a six-degree-of-freedom motion command, and controlling the robotic arm to adjust its position based on the six-degree-of-freedom motion command; and a closed-loop feedback module for acquiring an image frame of the current position of the robotic arm, extracting feature points based on the image frame of the current position, and updating the image feature error.

[0159] This embodiment also provides an electronic device applicable to a calibration-free visual servoing method for a six-degree-of-freedom robotic arm, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the calibration-free visual servoing method for a six-degree-of-freedom robotic arm as proposed in the above embodiment.

[0160] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a calibration-free visual servoing method for a six-degree-of-freedom robotic arm as proposed in the above embodiments.

[0161] The storage medium proposed in this embodiment and the method for implementing a calibration-free visual servoing method for a six-degree-of-freedom robotic arm proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0162] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0163] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A calibration-free vision servoing method for a six-degree-of-freedom robotic arm, characterized in that: include, Acquire image frames before and after the robotic arm moves, and extract feature point coordinates based on the image frames before and after the movement; The process of acquiring image frames before and after the robotic arm's movement, and extracting feature point coordinates based on the image frames before and after the movement, includes: The robotic arm is controlled to move independently along six degrees of freedom in sequence under the eye-hand structure, and image frames are acquired before and after each movement; Feature points are extracted based on the image frame to obtain the coordinates of the feature points; Determine the initial image Jacobian matrix based on the coordinates of the feature points; The initial image Jacobian matrix is ​​updated to obtain the current image Jacobian matrix; The image feature error is obtained by comparing the expected position of the target image with the current observation position. The control increment is then calculated by combining the image feature error with the Jacobian matrix of the current image. The control increment is converted into a six-degree-of-freedom motion command, and the robotic arm is controlled to adjust its position based on the six-degree-of-freedom motion command. Acquire an image frame of the current position of the robotic arm, extract feature points based on the image frame of the current position, and update the image feature error.

2. The calibration-free vision servoing method for a six-degree-of-freedom robotic arm as described in claim 1, characterized in that: Determining the initial image Jacobian matrix based on the feature point coordinates includes: Calculate the changes in features before and after the robotic arm moves based on the coordinates of the feature points. Establish the correspondence between the aforementioned feature changes and the pose changes of the robotic arm; The correspondence is estimated using the least squares method to obtain the initial image Jacobian matrix.

3. The calibration-free vision servoing method for a six-degree-of-freedom robotic arm as described in claim 2, characterized in that: The initial image Jacobian matrix is ​​updated to obtain the current image Jacobian matrix, including: Record the changes in image features and pose within each control cycle; The initial image Jacobian matrix is ​​iteratively updated by using a dynamic quasi-Newton method, combining the changes in image features and the changes in pose, to obtain the current image Jacobian matrix.

4. The calibration-free vision servoing method for a six-degree-of-freedom robotic arm as described in claim 3, characterized in that: The control increment is calculated by combining the image feature error and the current image Jacobian matrix, including: The image feature error is processed using a predictive compensation PI control algorithm to generate proportional term, integral term control quantity, and predictive compensation quantity; The proportional term, integral term, and prediction compensation term are converted into control increments through the inverse transformation of the current image Jacobian matrix.

5. The calibration-free vision servoing method for a six-degree-of-freedom robotic arm as described in claim 4, characterized in that: The image feature error is processed using a predictive compensation PI control algorithm to generate proportional terms, integral terms, control quantities, and predictive compensation quantities, including: Based on the image feature error, calculate the proportional and integral control terms; The predicted compensation amount is determined by the rate of change of image features and the compensation coefficient.

6. The calibration-free vision servoing method for a six-degree-of-freedom robotic arm as described in claim 1, characterized in that: The independent movement along the six degrees of freedom refers to orthogonal movement along the translational directions of the X-axis, Y-axis, and Z-axis, and rotational directions around the X-axis, Y-axis, and Z-axis, respectively.

7. A calibration-free vision servoing system for a six-degree-of-freedom robotic arm, employing the calibration-free vision servoing method for a six-degree-of-freedom robotic arm as described in any one of claims 1 to 6, characterized in that, include: The feature point acquisition module is used to acquire image frames before and after the robotic arm moves, and extract feature point coordinates based on the image frames before and after the movement; The image Jacobian matrix determination module is used to determine the initial image Jacobian matrix based on the coordinates of feature points. The matrix update module is used to update the initial image Jacobian matrix to obtain the current image Jacobian matrix; The control increment module is used to obtain the image feature error by comparing the expected position of the target image with the current observation position, and to calculate the control increment by combining the image feature error with the current image Jacobian matrix. The position adjustment module is used to convert control increments into six-degree-of-freedom motion commands, and control the robotic arm to adjust its position based on the six-degree-of-freedom motion commands; The closed-loop feedback module is used to acquire image frames of the current position of the robotic arm, extract feature points based on the image frames of the current position, and update the image feature errors.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the uncalibrated visual servoing method for a six-degree-of-freedom robotic arm as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the uncalibrated visual servoing method for a six-degree-of-freedom robotic arm as described in any one of claims 1 to 6.

Images