A dual-rate robust ibvs control method based on predictive observer and kalman filter

Abstract

The application provides a double-rate robust IBVS control method based on a prediction observer and Kalman filtering, relates to the technical field of robot vision servo control, and comprises the following steps: determining a first target time and a second target time; determining whether vision features are updated in a current control period based on the first target time and the second target time; if there is no update, generating a vision feature estimation value corresponding to the current control period by using a prediction observer; if there is an update, correcting a vision feature measurement value by using a Kalman filtering method to obtain a vision feature estimation value corresponding to the current control period; inputting the obtained vision feature estimation value corresponding to the current control period into an IBVS speed controller to obtain camera motion speed in the current control period; and driving a mechanical arm to perform vision servo motion based on the obtained camera motion speed. In this way, the robustness and convergence speed of the IBVS system are improved.

Description

Technical Field

[0001] This application relates to the field of robot vision servo control technology, and in particular to a dual-rate robust IBVS control method based on predictive observer and Kalman filter. Background Technology

[0002] Currently, robots are widely used in various industrial fields, such as drone delivery, robotic arm grasping, and robotic dog inspection. With the continuous development of robotics technology, its application scope is constantly expanding, gradually penetrating multiple industries such as manufacturing, logistics, and healthcare. Especially in tasks requiring precision, the demand for robots is increasing. For example, when robotic arms perform tasks such as tightening bolts, assembling parts, and precision welding, ensuring that the robot can accurately perceive and locate the target object has become one of the core technological challenges in robot applications.

[0003] Visual servoing technology, capable of adjusting a robot's motion and posture in real time based on visual feedback from a camera, has become an ideal technology for solving visual tasks in unstructured and complex robotic environments. Unlike traditional position sensor-based control methods, visual servoing technology provides dynamic feedback on robot motion by directly utilizing image data captured by a camera. This allows robots to accurately perform tasks such as target tracking, grasping, and assembly even when facing environmental changes or unclear target positions. Visual servoing technology can be divided into two main categories: Position-Based Visual Servoing (PBVS) and Image-Based Visual Servoing (IBVS). IBVS, by directly utilizing feature information (such as points, lines, and circles) in a two-dimensional image space as feedback, offers higher control precision and is particularly suitable for high-precision tasks, such as precise bolt alignment using a robotic arm.

[0004] However, in practical industrial applications, IBVS systems typically face two prominent problems. One is the dual-rate control problem. Limited by camera hardware performance and the complexity of image processing algorithms, the frequency of visual feature acquisition is usually significantly lower than the execution frequency of the robot control system. When the robot controller operates at a higher frequency while visual feedback updates lag, the system cannot acquire new visual feature information in adjacent control cycles, easily leading to decreased control input continuity and slower convergence speed. The other problem is visual feature drift caused by noise interference. In complex environments, under changing lighting conditions, or in dynamic environments, the visual features extracted by image processing algorithms are easily affected by noise, manifesting as feature point jitter and drift. When these unstable features are directly used as IBVS control inputs, they will significantly reduce the system's servo performance, for example, causing severe system oscillations or even servo failure. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a dual-rate robust IBVS control method based on predictive observer and Kalman filter, which solves the dual-rate control problem and the problem of visual feature drift caused by noise interference in the existing IBVS system, improves the robustness and convergence speed of the IBVS system, and enhances its practicality in industrial application scenarios.

[0006] In a first aspect, this application provides a dual-rate robust IBVS control method based on a predictive observer and Kalman filter, including its application to the control of a robotic arm, wherein the robotic arm is controlled based on visual features acquired by an image acquisition device, and the method includes: S1: Determine the first target time and the second target time, where the first target time is the control time of the robotic arm in the current control cycle, and the second target time is the image acquisition time of the image acquisition device in the current image acquisition cycle; S2: Determine whether the visual features in the current control cycle have been updated based on whether the first target time and the second target time are the same; S3a: If there is no update, then based on the visual feature estimate of the previous control cycle and the camera speed, use the predictive observer to generate the visual feature estimate corresponding to the current control cycle. S3b: If an update exists, the visual feature measurement value is corrected using the Kalman filter method to obtain the visual feature estimate value corresponding to the current control cycle; S4: Input the estimated visual features corresponding to the current control cycle obtained in S3a or S3b into the IBVS speed controller to obtain the camera motion speed within the current control cycle; S5: Drive the robotic arm to perform visual servo motion based on the camera's movement speed within the current control cycle.

[0007] Optionally, in the above method, in the step of determining whether the visual features of the current control cycle have been updated based on whether the first target time and the second target time are the same, If the first target time and the second target time are the same, then the visual features have been updated; otherwise, there is no update.

[0008] Optionally, in the above method, if there is no update, in the step of generating the visual feature estimate corresponding to the current control cycle using a predictive observer based on the obtained visual feature estimate and camera velocity from the previous control cycle, ; ; This refers to the time interval between each output during the control process; for identity matrix n The number of feature points; for visual features Find the partial derivative at The result of the treatment , For Z, s , v The function, ; for Image Jacobian matrix; For camera speed; Visual features; For feature point depth information; for Observation gain diagonal matrix; for The system outputs an identity matrix; for Camera speed Find the partial derivative The result of the treatment , , , These are the visual features, camera speed, and feature point depth information of the previous control time step, respectively. System state representation Visual feature estimates for the next control time; The system input represents the camera speed; For system output; These are mixed visual feature values; For constant terms, ; These are visual feature measurements obtained from image processing. Predicted visual features generated for the observer; To control the number of times the robotic arm operates; Number of camera samples; This represents the numerical ratio between the camera sampling period and the robotic arm control period.

[0009] Optionally, in the above method, if an update exists, the step of using a Kalman filter to correct the visual feature measurements to obtain the estimated visual features corresponding to the current control cycle is as follows: The Kalman filter result is obtained using the following formula: ; Kalman filter correction results Prior estimates; The result is the Kalman filter correction. ; These are visual feature measurements; Here is the Kalman gain matrix. ; for Measurement error diagonal matrix; for Prior error covariance matrix, ; For the intermediate matrix ; To model the error matrix; The posterior error covariance matrix; The update and iteration process is as follows: .

[0010] Optionally, in the above method, the step of inputting the visual feature estimate corresponding to the current control cycle into the IBVS speed controller to obtain the camera motion speed within the current control cycle is performed by determining the camera motion speed in the following manner: ; The camera's motion speed during the servo process; It is the servo gain; yes Time is determined by current visual features With expected visual features The servo error obtained by subtraction; It is a matrix The Moore-Penrose pseudo-inverse matrix.

[0011] Optionally, the above method further includes: calculating a visual feature error based on the error component between the estimated visual feature value corresponding to the current control cycle and the expected visual feature value corresponding to the current control cycle; if the visual feature error is less than a threshold, then stopping the control of the robotic arm.

[0012] The embodiments of this application bring the following beneficial effects: Addressing the dual-rate control problem faced by the IBVS system, this application proposes using a predictive observer to perform prediction and interpolation between two frames of feature data acquired from the image, compensating for the lack of visual feature information, thereby effectively improving the system's convergence speed and ensuring the smoothness of the motion trajectory. On the other hand, to solve the problem of feature points drifting due to noise interference during image processing, this application employs an extended Kalman filter algorithm to fuse and correct the observer's predicted data with the image data, improving the stability and accuracy of the IBVS control input features, reducing the negative impact of feature point drift on the IBVS system, and further enhancing the system's robustness.

[0013] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application are realized and obtained through the structures particularly pointed out in the description, claims and drawings.

[0014] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart of the IBVS control method provided in this application; Figure 2 A schematic diagram of the robotic arm provided in this application; Figure 3 A schematic diagram of the experimental subject provided in this application; Figure 4 A schematic diagram of the dual-rate IBVS system model provided in this application; Figure 5 A schematic diagram of the dual-rate IBVS system model with a predictive observer provided in this application; Figure 6 This application provides a schematic diagram of an IBVS system model that simultaneously incorporates a predictive observer and a Kalman filter; Figure 7 This is a diagram showing the motion trajectory of feature points; where, Figure 7 (a) shows the feature point motion trajectory diagram using the original method. Figure 7 (b) Feature point motion trajectory diagram using the predictive observer method. Figure 7 (c) Feature point motion trajectory diagram using predictive observer and Kalman filtering method; Figure 8 The total error curve of the IBVS servo; Figure 8 (a) shows the total error curve of the IBVS servo using the original method. Figure 8 (b) shows the total error curve of the IBVS servo using the predictive observer method. Figure 8 (c) shows the total error curve of the IBVS servo using the predictive observer and Kalman filtering method; Figure 9 A graph showing the changes in the visual features of feature point 0 in the x-direction under the condition of using only a predictive observer, provided for this application. Figure 10 A graph showing the change in visual features of feature point 0 in the y-direction under the condition of using only a predictive observer, provided for this application. Figure 11 A graph showing the changes in the visual features of feature point 1 in the x-direction under the condition of using only a predictive observer, provided for this application. Figure 12A graph showing the change in visual features of feature point 1 in the y-direction under the condition of using only a predictive observer, provided for this application. Figure 13 A graph showing the changes in the visual features of feature point 2 in the x-direction under the condition of using only a predictive observer, provided for this application. Figure 14 A graph showing the change in visual features of feature point 2 in the y-direction under the condition of using only a predictive observer, provided for this application. Figure 15 A graph showing the changes in the visual features of feature point 3 in the x-direction under the condition of using only a predictive observer, provided for this application. Figure 16 A graph showing the change in visual features of feature point 4 in the y-direction under the condition of using only a predictive observer, provided for this application. Figure 17 A graph showing the changes in the visual features of feature point 0 in the x-direction under the conditions of a predictive observer and a Kalman filter, provided for this application. Figure 18 A graph showing the changes in the visual features of feature point 0 in the y-direction under the conditions of a predictive observer and a Kalman filter, provided for this application. Figure 19 A graph showing the changes in the visual features of feature point 1 in the x-direction under the conditions of a predictive observer and a Kalman filter, provided for this application. Figure 20 A graph showing the change in visual features of feature point 1 in the y-direction under the conditions of a predictive observer and a Kalman filter, provided for this application. Figure 21 A graph showing the changes in the visual features of feature point 2 in the x-direction under the conditions of a predictive observer and a Kalman filter, provided for this application. Figure 22 A graph showing the changes in the visual features of feature point 2 in the y-direction under the conditions of a predictive observer and a Kalman filter, provided for this application. Figure 23 A graph showing the changes in the visual features of feature point 3 in the x-direction under the conditions of a predictive observer and a Kalman filter, provided for this application. Figure 24 The curves showing the changes in the visual features of feature point 3 in the y-direction under the conditions of a predictive observer and a Kalman filter provided in this application. Detailed Implementation

[0017] The purpose of this application is to provide a dual-rate robust IBVS control method based on predictive observer and Kalman filter, which solves the dual-rate control problem and the problem of visual feature drift caused by noise interference in the existing IBVS system, improves the robustness and convergence speed of the IBVS system, and enhances its practicality in industrial application scenarios.

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] like Figure 1 As shown, this application provides a dual-rate robust IBVS control method based on predictive observer and Kalman filter, which is applied to the control of a robotic arm. The robotic arm is controlled based on visual features acquired by an image acquisition device. The method includes: S1: Determine the first target time and the second target time. The first target time is the control time of the robotic arm within the current control cycle, and the second target time is the image acquisition time of the image acquisition device within the current image acquisition cycle.

[0020] Specifically, an IBVS system model under dual-rate conditions is established. In a certain example, the current image acquisition cycle is longer than the current control cycle. By determining the first target time and the second target time, the dynamic relationship between visual features and robotic arm movement can be described.

[0021] S2: Determine whether the visual features in the current control cycle are updated based on whether the first target time and the second target time are the same.

[0022] Specifically, within each control cycle of the robotic arm, the current control time of the robotic arm is obtained, and the sampling time corresponding to the most recent image data acquired by the image acquisition device is obtained, which serves as the time reference for subsequent control judgment and status update.

[0023] S3a: If no update is available, then based on the visual feature estimates obtained in the previous control cycle and the camera velocity, use the predictive observer to generate the visual feature estimates corresponding to the current control cycle.

[0024] Specifically, based on the visual feature estimates and camera speed from the previous control cycle, a predictive observer is used to predict and interpolate the visual features in time to generate the visual feature estimates for the current control cycle.

[0025] S3b: If an update exists, the visual feature measurement value is corrected using the Kalman filter method to obtain the visual feature estimate value corresponding to the current control cycle.

[0026] Specifically, the visual feature measurement value at the current moment is obtained by the image acquisition device and the image processing algorithm, and the visual feature prediction value at the current moment is obtained by the predictive observer. The visual feature measurement value is filtered and corrected by the Kalman filter method to obtain the visual feature estimate value corresponding to the current control cycle.

[0027] S4: Input the estimated visual features corresponding to the current control cycle obtained from S3a or S3b into the IBVS speed controller to obtain the camera motion speed within the current control cycle.

[0028] Specifically, the visual feature estimate obtained in step S3a or step S3b is input into the IBVS speed controller to calculate the motion speed of the image acquisition device within the current control cycle, so that the visual features gradually converge to the desired value.

[0029] S5: Drives the robotic arm to perform visual servo motion based on the camera's movement speed within the current control cycle.

[0030] Specifically, based on the hand-eye calibration matrix between the camera and the robotic arm end effector, the camera's movement speed is converted into the movement speed of the robotic arm end effector, and control commands are sent to drive the robotic arm accordingly.

[0031] Repeat steps S2 to S4 until the total visual servo error is less than the threshold, then determine that the system has reached convergence and end the closed-loop control process.

[0032] In summary, this application effectively solves the dual-rate control problem and image feature drift problem existing in traditional image-based visual servoing systems by introducing a control strategy that combines a predictive observer and a Kalman filter. This method can perform predictive interpolation when visual features are not updated and filter correction when features are updated, improving the stability and accuracy of the input features controlled by the IBVS, reducing the negative impact of feature point drift on the IBVS system, and thus significantly improving the system's convergence speed, motion smoothness, and robustness. It is suitable for high-precision visual alignment tasks in industrial scenarios.

[0033] It should be noted that, in a specific example, such as Figure 2 As shown, the robotic arm is a UR16e collaborative six-degree-of-freedom robotic arm, and an image acquisition device is set at the end of the robotic arm (near the working surface). The image acquisition device includes, but is not limited to, a motion camera, a CMOS camera, or an industrial camera; the camera can be an Intel RealSene D435 camera.

[0034] In an optional embodiment, in the step of determining whether the visual features are updated in the current control cycle based on whether the first target time and the second target time are the same, if the first target time and the second target time are the same, then the visual features have been updated; otherwise, there is no update.

[0035] Specifically, the control time of the robotic arm is compared with the image acquisition time. When the control time of the robotic arm is consistent with the image acquisition time, it is determined that there is a valid visual feature update within the current control cycle. When the control time of the robotic arm is inconsistent with the image acquisition time, it is determined that there is no new visual feature update within the current control cycle, that is, there is a lack of visual feature information.

[0036] As can be seen, by using simple time comparison logic, real-time judgment of the visual feature update status is achieved, providing accurate triggering conditions for subsequent prediction and filtering processing, and improving system response efficiency and control accuracy.

[0037] In the step of generating the visual feature estimate corresponding to the current control cycle using a predictive observer based on the obtained visual feature estimate and camera velocity from the previous control cycle, if no update is available, specifically, a dual-rate IBVS system model is first established as follows: Figure 4 As shown. Assume the camera's sampling period is... The control system cycle of the robotic arm is For each servo session, there is , And N is a positive integer. For an image acquisition device such as a camera, sampling is performed at a certain moment (solid line circle), using... It indicates that there are A control command (square shape) is issued to the robotic arm at a certain moment, using... It indicates that there are , This represents the number of times the robotic arm controls the system. If between any two time steps sampled by the camera ( ),have The dual-rate IBVS system model is represented as follows: (1) in, for The result of taking the derivative with respect to time, The visual feature values ​​of feature points represent the system state. The system input represents the camera speed. for The system input matrix represents the image Jacobian matrix. For system output, for The system outputs an identity matrix. This represents the number of feature points.

[0038] The current system control frequency uses the lowest operating frequency of each module, i.e., the camera sampling frequency. This means that the output between the two time steps of camera sampling is missing. Under the condition of low-frequency camera sampling and high-frequency controller execution, if control is based solely on the low-frequency visual feedback of the camera, a dual-rate control problem will inevitably occur, thus adversely affecting the dynamic performance of the system.

[0039] As a preferred example, in this application, the IBVS motion model contained in equation (1) is represented as follows: (2) in, It's camera speed. It is the image Jacobian matrix composed of all feature points, while visual features... s With corresponding depth value Obtained through the corresponding processing algorithm. It is related to Z , s , v The function. i The Jacobian matrix of the image for each feature point is represented as follows: (3) in, x and y These are the two-dimensional coordinates of the feature points in the image coordinate system; the combination of the two constitutes the visual feature. s .

[0040] Acquire and record the control time of the robotic arm within the current control cycle. and the camera's most recent image acquisition time .

[0041] right and Compare and judge. When and The inconsistency indicates that the camera cannot update visual features within the current control cycle. In this case, a predictive observer is needed to perform predictive processing to interpolate the missing visual feature values. and At the same time, it indicates that within the current control cycle, the camera can update the visual features used for visual servoing through image processing algorithms. At this time, only the noise interference present in the image feature extraction process needs to be considered, and Kalman filtering is performed on it to obtain the filtered and corrected visual feature values.

[0042] In order to predict missing visual features, a predictive observer is introduced, which requires linearizing the IBVS motion model first. For equation (2) at the operation point... Performing a Taylor expansion and retaining the first two terms, we get: (4) in, , , , , , These are the visual features, camera speed, and feature point depth information from the previous control time step.

[0043] For the i There are 1 feature points The results are expressed as follows: (5) In equations (4) and (5) express visual features s Find the partial derivative at The result at the first place, for the first i There are 1 feature points The specific expression is as follows: (6) In equations (4) and (5) express Camera speed v Find the partial derivative at The result at the first place, for the first i There are 1 feature points The specific expression is as follows: (7) Will n Feature points , The result of stacking them together , .

[0044] Substituting the above results into equation (4) and transforming them, and combining the constant terms, we obtain the linearized IBVS motion model as follows: (8) The constant term is .

[0045] Next, the linearized IBVS motion model of equation (8) is discretized using the forward difference method, and the results are as follows: (9) Expanding and combining equation (9), we obtain the linearized discrete-time IBVS motion model as follows: (10) in, for Visual feature values ​​at the next control time. This represents the time interval between each output during the system control process. yes Identity matrix.

[0046] Finally, the dual-rate IBVS system model in equation (1) becomes the following after linearization and discretization: (11) in, It is the output value of the visual feature measurement system obtained from image processing. yes The system output for the next control time.

[0047] Subsequently, based on the linear model of Equation (8), the observer is designed in the following form: (12) Since equation (12) is still a continuous-time model, the above discretization process is still required. The final result is as follows: (13) in, yes Observation gain diagonal matrix, yes The estimated value. These are mixed visual feature values, specifically represented as follows: (14) in, These are visual feature measurements obtained from image processing. These are visual feature predictions generated by the observer.

[0048] Therefore, the previous control time was... and Substituting into equation (13), the estimated visual features for predicting the current control time can be calculated. ,in .

[0049] Between the two image detection processing time steps, the IBVS system model in equation (11) becomes as follows: (15) like Figure 5This is a model of a dual-rate IBVS system with an observer. Unlike traditional feedback systems, where the controller can only output a control command (square) once within the camera sampling period, limiting the feedback update frequency, the introduction of an observer allows the system to generate multiple predictive feedbacks (dashed circles) between adjacent image sampling periods (solid circles), enabling high-frequency estimation and continuous updating of visual features. This mechanism allows the controller to operate at a higher frequency, theoretically significantly improving the response speed of the IBVS system.

[0050] It is evident that an efficient predictor interpolation formula is designed based on the linearized and discretized visual servoing model. This formula can continuously generate high-frequency visual feature estimates during periods of missing image features, effectively alleviating the system response lag problem caused by dual-rate control and improving the servo convergence speed.

[0051] Using the above method, the estimated visual features from the previous control period are used to predict and interpolate the estimated visual features for the current control period.

[0052] If an update exists, a Kalman filter is introduced to correct visual features with noise interference. In the step of obtaining the estimated visual features corresponding to the current control cycle, the Extended Kalman Filter (EKF) algorithm is specifically used. Because predictive observer interpolation is performed when there is no visual feature update, this process has already linearized the IBVS motion model. Based on this, a Kalman filter can be directly designed. Assuming that... The visual features at time point are obtained by an image detection algorithm, denoted as . The prediction result obtained by processing the visual features from the previous control time using the observer is denoted as... From equation (13), the result can be obtained as follows: (16) in, This is the result of Kalman filter correction. Prior estimates, .

[0053] Kalman filter correction results The specific representation is as follows: (17) in, The Kalman gain matrix is ​​expressed as follows: (18) in, yes The prior error covariance matrix needs to be initialized at time zero. yes The measurement error diagonal matrix is ​​jointly determined by the vision camera and the image processing algorithm.

[0054] Calculate the prior error covariance matrix The formula is as follows: (19) in, , It is the posterior error covariance matrix. yes The modeling error matrix is ​​determined by the mathematical model used.

[0055] The update and iteration process is as follows: (20) in, yes Identity matrix. (Using...) and Updated This leads to the Kalman filter correction process in the next control time.

[0056] This design employs an extended Kalman filter framework, which effectively handles nonlinear observation models in IBVS systems. By dynamically adjusting the Kalman gain, it adaptively balances the confidence levels of predicted and measured values: when image noise is high, more trust is placed on model predictions; when the image is clear and stable, more consideration is given to measured values. This data fusion mechanism significantly reduces the impact of visual feature drift on control performance and improves the system's robustness in complex environments such as varying illumination, occlusion, or dynamic backgrounds.

[0057] like Figure 6 The diagram shows a schematic of an IBVS system model that simultaneously incorporates an observer and a Kalman filter. At any given moment, if Kalman filtering is not used, the system will directly use the output of the image processing algorithm. (Solid circle) as the system state estimate However, when image feature points drift, the state estimate obtained based on this instability will have a large deviation, leading to a decrease in system control performance. In contrast, if equation (17) is used... Perform Kalman filtering correction and obtain the new... (Solid circle) represents the current state. Because of the introduction of prior state estimates provided by the observer (Dashed circle) Dynamic data fusion and updating can effectively suppress feature drift, thereby improving the servo convergence performance of the IBVS system.

[0058] Furthermore, in the step of inputting the estimated visual features corresponding to the current control cycle into the IBVS speed controller to obtain the camera motion speed within the current control cycle, the camera motion speed is determined in the following manner, and the speed of the linearized discrete-time IBVS system is designed. as follows: (twenty one) The camera's motion speed during the servo process; It is the servo gain; yes Time is determined by current visual features With expected visual features The servo error obtained by subtraction; It is a matrix The Moore-Penrose pseudo-inverse matrix.

[0059] Thus, the calculation method of the IBVS speed controller is clarified: the camera motion speed is calculated based on the error between the current estimated and expected visual feature values, combined with the servo gain. Since the visual features input to this controller have been optimized using the aforementioned predictive observer or Kalman filter mechanism, control deviations caused by using original noisy or outdated features are avoided. Therefore, the speed command is more accurate and smoother, enabling the robotic arm to converge quickly and stably to the target pose, reducing overshoot and oscillation, and improving operational efficiency and safety.

[0060] In an optional embodiment, the visual feature error is calculated based on the error component between the estimated visual feature value corresponding to the current control cycle and the expected visual feature value corresponding to the current control cycle. If the visual feature error is less than a threshold, the control of the robotic arm is stopped.

[0061] The camera's motion speed during the servoing process is calculated based on the visual servoing control model. Since the camera is fixedly mounted on the end effector of the robotic arm, its relative pose relationship with the end effector can be predetermined using a hand-eye extrinsic parameter calibration algorithm. Based on this relative pose relationship, the camera's motion speed is converted into the motion speed of the end effector relative to the robotic arm's base coordinate system. Furthermore, according to the robotic arm's Real-Time Data Exchange (RTDE) protocol, corresponding motion control commands are sent to the robotic arm, thereby driving the robotic arm to perform visual servoing motion.

[0062] The total visual servo error is calculated based on the estimated visual features obtained within the current control cycle. This total error is defined as the sum of the squares of the feature error components of all feature points, representing the overall deviation of all visual feature points relative to the desired feature point. When the total error is less than a threshold value, the system is considered to have reached convergence, and the closed-loop control process ends.

[0063] It is evident that this method achieves automatic termination of closed-loop control, preventing systems using the IBVS control method from continuing to execute ineffective movements after reaching the target. This saves energy, reduces mechanical wear, and prevents continuous jitter caused by minute residual errors. Furthermore, the judgment logic is simple, easy to implement in engineering, and enhances the system's intelligence and practicality.

[0064] This threshold can be flexibly set according to the accuracy requirements of the task, balancing control precision and real-time performance. For example, a larger threshold can be set in the coarse positioning stage to speed up the response, while a smaller threshold is used in the fine alignment stage to ensure positioning accuracy. This quantification criterion provides a clear and repeatable engineering standard for convergence judgment, avoiding subjective or ambiguous judgments and improving system reliability and consistency.

[0065] To verify the feasibility and effectiveness of the method proposed in this invention, three IBVS alignment bolt experiments were conducted. The experimental objects were as follows: Figure 3 As shown: 1. IBVS Alignment Screw Experiment: A robotic arm carrying a camera servo-converges from a distance of 35cm from the bolt (camera plane distance from bolt surface) to a distance of 20cm from the bolt. During this process, no predictive observer or Kalman filter is used; 2. IBVS alignment experiment with an observer: The robotic arm carrying a camera servo converges from a distance of 35cm from the bolt (the distance between the camera plane and the bolt surface) to a distance of 20cm from the bolt to verify whether the use of an observer can improve the convergence speed of the system; 3. IBVS alignment experiment with an observer and a Kalman filter: The robotic arm carrying a camera servo converges from a distance of 35cm from the bolt (the distance between the camera plane and the bolt surface) to a distance of 20cm from the bolt to verify whether the use of a Kalman filter can correct image feature drift and make the motion trajectory smoother.

[0066] Figure 7 , Figure 8 , Figure 9 - Figure 24 This is a demonstration of the experimental results.

[0067] Figure 7 Three methods are demonstrated to illustrate the movement trajectory of feature points during the alignment of bolts using IBVS. Figure 7 (c) and Figure 7 (a) and Figure 7Comparing (b) with the original method, it is clear that the feature points exhibit significant jumps or fluctuations during motion when using the original method and the observer-only method. However, after applying the Kalman filter, the motion trajectory becomes smoother, without significant fluctuations. This indicates that the Kalman filter effectively corrects the drift of image feature points, thereby significantly enhancing the stability and robustness of the system.

[0068] Figure 8 The total error curves during the servo process of the three IBVS methods are shown. Through comparison... Figure 8 (b) and Figure 8 As shown in (a), due to feature point drift, the total error curve during the servoing process exhibits significant fluctuations, but overall shows a convergence trend. When using the predictive observer method, the convergence time is 6.283 seconds, significantly faster than the original IBVS method's convergence time of 7.964 seconds, a reduction of 1.681 seconds and an improvement in convergence speed of approximately 21.1%. This result verifies that the predictive observer can effectively compensate for the lack of image feature information, thereby improving the system's convergence speed. Further comparison... Figure 8 (c) and Figure 8 In Figure (b), it can be observed that the total error curve becomes smoother and shows no significant fluctuations after using the observer and Kalman filter, indicating that the Kalman filter effectively corrects image noise interference, thereby enhancing the robustness of the system. Furthermore, its convergence time of 6.288 seconds is almost the same as the convergence time of 6.283 seconds using only the observer, indicating that the predictive observer plays a dominant role in improving the system's convergence speed, while the introduction of the Kalman filter does not significantly increase the system's dynamic response time.

[0069] Figures 9-16 The results describe the changes of four feature points in the x and y directions when only a predictive observer is used. Figures 17-24 The changes of four feature points in the x and y directions are described under the condition of using a predictive observer and a Kalman filter. Circles represent the image features actually acquired by the camera, and solid lines represent the overall result after merging the actual image features with the predicted features after observer interpolation (and Kalman filtering). Comparing the changes of the same feature points in the corresponding directions, it is evident that after introducing the Kalman filter, the high-frequency fluctuations and abrupt changes in the visual feature curve are significantly reduced, and the overall trend becomes smoother and more continuous. This further verifies the effectiveness of the Kalman filter in suppressing measurement noise and feature drift, thereby improving the stability and robustness of the system.

[0070] Furthermore, in the description of the embodiments of this application, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0071] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0072] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A dual-rate robust IBVS control method based on predictive observer and Kalman filter, characterized in that, This includes methods for controlling a robotic arm, which is controlled based on visual features acquired by an image acquisition device. The methods include: S1: Determine the first target time and the second target time, where the first target time is the control time of the robotic arm in the current control cycle, and the second target time is the image acquisition time of the image acquisition device in the current image acquisition cycle; S2: Determine whether the visual features in the current control cycle have been updated based on whether the first target time and the second target time are the same; S3a: If there is no update, then based on the visual feature estimate of the previous control cycle and the camera speed, use the predictive observer to generate the visual feature estimate corresponding to the current control cycle. S3b: If an update exists, the visual feature measurement value is corrected using the Kalman filter method to obtain the visual feature estimate value corresponding to the current control cycle; S4: Input the estimated visual features corresponding to the current control cycle obtained in S3a or S3b into the IBVS speed controller to obtain the camera motion speed within the current control cycle; S5: Drive the robotic arm to perform visual servo motion based on the camera's movement speed within the current control cycle.

2. The method according to claim 1, characterized in that, In the step of determining whether the visual features of the current control cycle have been updated based on whether the first target time and the second target time are the same, If the first target time and the second target time are the same, then the visual features have been updated; otherwise, there is no update.

3. The method according to claim 2, characterized in that, If no update is available, in the step of generating the visual feature estimate corresponding to the current control cycle using the predictive observer based on the obtained visual feature estimate from the previous control cycle and the camera velocity, The visual feature estimate corresponding to the current control cycle is generated using the following formula: ； ； This refers to the time interval between each output during the control process; for identity matrix n The number of feature points; for visual features Find the partial derivative at The result of the treatment , For Z, s , v The function, ; for Image Jacobian matrix; For camera speed; Visual features; For feature point depth information; for Observation gain diagonal matrix; for The system outputs an identity matrix; for Camera speed Find the partial derivative The result of the treatment , , , These are the visual features, camera speed, and feature point depth information of the previous control time step, respectively. System state representation Visual feature estimates for the next control time; The system input represents the camera speed; For system output; These are mixed visual feature values; For constant terms, ; These are visual feature measurements obtained from image processing. Visual feature predictions generated for the observer; To control the number of times the robotic arm operates; Number of camera samples; This represents the numerical ratio between the camera sampling period and the robotic arm control period.

4. The method according to claim 3, characterized in that, If an update exists, the visual feature measurements are corrected using a Kalman filter to obtain the estimated visual feature value corresponding to the current control cycle. The Kalman filter result is obtained using the following formula: ； Kalman filter correction results Prior estimates; The result is the Kalman filter correction. ; These are visual feature measurements; Here is the Kalman gain matrix. ; for Measurement error diagonal matrix; for Prior error covariance matrix, ; For the intermediate matrix ; To model the error matrix; The posterior error covariance matrix; The update and iteration process is as follows: 。 5. The method according to claim 4, characterized in that, In the step of inputting the visual feature estimate corresponding to the current control cycle into the IBVS speed controller to obtain the camera motion speed within the current control cycle, the camera motion speed is determined in the following manner: ； The camera's motion speed during the servo process; It is the servo gain; yes Time is determined by current visual features With expected visual features The servo error obtained by subtraction; It is a matrix The Moore-Penrose pseudo-inverse matrix.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes calculating a visual feature error based on the error component between the estimated visual feature value corresponding to the current control cycle and the expected visual feature value corresponding to the current control cycle; if the visual feature error is less than a threshold, then control of the robotic arm is stopped.

Images