A Robot Motion Control Method Based on Real-Time Visual Feedback

By establishing a time delay prediction model based on content complexity and a hierarchical joint compensation mechanism, the problem of the singularity of time delay compensation in visual servo control is solved, thereby improving the accuracy and stability of robot motion control.

CN123060701APending Publication Date: 2026-11-10HANGZHOU TURUI INTELLIGENT EQUIP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611466466.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-20
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing visual servo control, the compensation methods for image processing latency and communication latency are singular and cannot take into account dynamic changes, resulting in lagging joint correction, which affects the robot's tracking accuracy and system stability. Moreover, the cost of upgrading multi-sensor fusion solutions is high.

Method used

An image processing latency prediction model is established based on the actual processing latency and content complexity of historical frames. Joint sensitivity is divided by Jacobi matrix, and a hierarchical compensation mechanism is adopted. Combined with dynamic prediction and feedforward compensation of image processing and communication latency, the final control command is generated.

Benefits of technology

It improves the time delay compensation accuracy and computational resource allocation efficiency of visual servo control, reduces the risk of attitude oscillation, enhances control stability, and adapts to changes in image processing time under different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123060701A_ABST
    Figure CN123060701A_ABST
Patent Text Reader

Abstract

This invention discloses a robot motion control method based on real-time visual feedback, relating to the field of robot visual servoing technology. The method includes: establishing an image processing latency prediction model based on the correspondence between actual processing latency and content complexity of historical frames; and predicting the image processing latency of the current frame based on the content complexity of the current frame image. The proposed robot motion control method dynamically predicts image processing latency based on image content complexity, and combines dual-path differential compensation and joint hierarchical compensation mechanisms, improving both the latency compensation accuracy and computational resource allocation efficiency in visual servoing control. By performing latency prediction, the predicted output is earlier than the pose calculation completion time, reserving a lead time window for compensation calculation. During the operation of the latency prediction model, the collected actual processing latency data is continuously iterated, making it more adaptable to image processing time variations in different scenarios compared to fixed latency compensation methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of robot vision servo technology, specifically to a robot motion control method based on real-time visual feedback. Background Technology

[0002] Visual servo control is one of the core technologies for industrial robots to achieve autonomous operation. It uses visual sensors to acquire target images in real time, and then uses pose calculation to drive the robot to complete error correction. It is widely used in assembly, handling, and flexible manufacturing scenarios. However, there is an inherent delay in the entire link from visual information acquisition to effect, including image processing links such as feature extraction and pose calculation, as well as communication links such as image data transmission and command issuance. This causes joint correction to lag behind the observation results, affecting tracking accuracy and system stability.

[0003] Regarding image processing latency compensation, existing solutions mostly use fixed latency or moving average latency as the compensation benchmark. However, image processing latency fluctuates dynamically with the complexity of image content, and latency estimation deviations are significant in scenarios such as target occlusion and abrupt texture changes, limiting compensation accuracy. In terms of latency compensation architecture, existing solutions typically employ a single compensation method for the total latency across the entire link. However, the generation mechanisms and variation patterns of image processing latency and communication latency differ, making it difficult to consider the characteristics of both types of latency in a unified compensation approach. Regarding the dimensions of compensation effect, existing solutions mostly perform full-degree-of-freedom compensation uniformly in the end-effector pose domain, failing to consider the differences in sensitivity of different joints to visual latency, resulting in a mismatch between computational resource allocation and actual compensation requirements. Some solutions improve compensation effectiveness by introducing multi-sensor fusion, but this incurs high costs for retrofitting existing robots.

[0004] To address this, we propose a robot motion control method based on real-time visual feedback. Summary of the Invention

[0005] To address the problems raised in the background art, this invention provides a robot motion control method based on real-time visual feedback, specifically including: An image processing latency prediction model is established based on the correspondence between the actual processing latency of historical frames and the content complexity. Based on the content complexity of the current frame image, the image processing latency of the current frame is predicted. Using the dynamically estimated image processing latency as the prediction step size, the position error is extrapolated over time to obtain the image processing latency compensation amount; the feedforward compensation amount is calculated based on the expected end velocity of the original motion trajectory and the pre-calibrated communication latency; the two types of compensation amounts are superimposed to obtain the total latency compensation amount, which is mapped to the joint latency compensation increment via Jacobian transpose; the attitude error is corrected using an independent fixed gain ratio and does not participate in latency prediction compensation, and its correction amount is mapped to the joint correction amount via Jacobian transpose; the joint corresponding to the attitude error is assigned to the low-sensitivity compensation level; Based on the current joint angle, the visual time delay sensitivity coefficient of each joint is calculated using the Jacobian matrix, and the joints are divided into at least two compensation levels according to their sensitivity. For high-sensitivity joints, the joint time delay compensation increment is superimposed on the proportional correction corresponding to the original position error, and full compensation is applied. For low-sensitivity joints, only the proportional correction is made based on the original position error, and the time delay compensation increment is not superimposed. The correction values ​​of each joint after layered compensation are superimposed on the joint commands corresponding to the original motion trajectory to generate the final control command for issuance; at the same time, the actual processing latency data of each frame is recorded for iterative updating of the image processing latency prediction model.

[0006] Preferably, the content complexity is a quantified value representing the computational load of image processing, calculated by weighting at least two of the following dimensions: The number of effective feature points, image texture richness, occlusion area ratio, and target area ratio; The weight coefficients for each dimension are determined based on the contribution of each dimension to the image processing latency, which is calibrated offline.

[0007] Preferably, the image processing latency prediction model is a linear mapping relationship between content complexity and processing latency, established through regression analysis based on multiple sets of offline collected sample data; The content complexity is extracted synchronously from the intermediate calculation results in the image feature extraction and pose calculation process. After the complexity calculation is completed in the feature extraction stage, the model outputs the corresponding image processing delay prediction result. The output time is earlier than the time when the pose calculation of the current frame is completed.

[0008] Preferably, when extrapolating the position error over time, the prediction step size is adjusted in real time according to the prediction results of each frame; A linear extrapolation method based on the error change rate is adopted, wherein the error change rate is obtained by dividing the difference in the original position error of adjacent frames by the time interval of the corresponding image acquisition time.

[0009] Preferably, when calculating the feedforward compensation amount, the expected end velocity vector of the original motion trajectory at the current moment is taken and multiplied by the pre-calibrated communication delay scalar to obtain the feedforward compensation amount; The communication delay is obtained by calibrating the round-trip delay of the heartbeat packet between the controller and the vision sensor, and the communication delay includes the delay of issuing control commands.

[0010] Preferably, when calculating the visual time delay sensitivity coefficient of each joint, the column vector magnitude of each joint corresponding to the Jacobian matrix under the current pose is taken as the sensitivity coefficient. When dividing the compensation levels, each joint is sorted from largest to smallest according to its sensitivity coefficient, and divided into at least two compensation levels according to a preset ratio or preset threshold.

[0011] Preferably, based on the image processing delay prediction result output earlier than the current frame pose calculation completion time, the position error time extrapolation and feedforward compensation calculation are executed in parallel after the prediction output and before the pose calculation is completed, and the compensation calculation result is output to the hierarchical compensation stage synchronously with the pose calculation result.

[0012] Preferably, the attitude error is corrected only by a fixed gain ratio, without time delay prediction extrapolation or feedforward compensation; the attitude error ratio correction frequency is synchronized with the control cycle and is not affected by visual frame rate fluctuations.

[0013] Preferably, the visual confidence level is calculated in real time, and the visual confidence level is determined based on the feature point effectiveness or reprojection error; when the visual confidence level is lower than a preset threshold, the time delay compensation output of low-sensitivity joints is turned off first, while the time delay compensation output of high-sensitivity joints is retained.

[0014] Preferably, the method also includes the step of iteratively updating the image processing latency prediction model: Using a sliding window sample set, after accumulating a predetermined number of actual processing delay data, new data is added to the sample set and the oldest data is removed. Then, regression analysis is re-executed to update the model parameters. Before adding data to the sample set, abnormal delay data are removed based on the Grubbs criterion or the 3σ criterion.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: The robot motion control method proposed in this invention dynamically predicts image processing latency based on image content complexity, and combines dual-path differential compensation and joint hierarchical compensation mechanisms to improve both latency compensation accuracy and computational resource allocation efficiency in visual servo control. By reusing intermediate calculation results during pose solving for latency prediction, the predicted output is earlier than the pose solving completion time, reserving a lead time window for compensation calculation. During the operation of the latency prediction model, the collected actual processing latency data is continuously iterated, making it more adaptable to image processing time variations in different scenarios compared to fixed latency compensation methods. Position and posture errors are compensated through independent paths, which helps reduce the interference of noise introduced by latency prediction extrapolation on the posture channel and reduces the risk of posture oscillation. Quantifying the latency sensitivity of each joint based on the Jacobian matrix and applying compensation hierarchically helps avoid unnecessary compensation jitter introduced by low-sensitivity joints. When visual confidence is insufficient, the compensation output is adjusted hierarchically, and the model update mechanism with sliding window and outlier screening improves control stability under visual condition fluctuations. Attached Figure Description

[0016] Figure 1 This is a flowchart of the robot motion control method of the present invention. Detailed Implementation

[0017] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0018] Reference Figure 1 As shown, a robot motion control method based on real-time visual feedback includes: An image processing latency prediction model is established based on the correspondence between the actual processing latency of historical frames and the content complexity. The image processing latency of the current frame is dynamically predicted according to the content complexity of the current frame image.

[0019] The content complexity is a quantitative value representing the computational load of image processing, which is calculated by weighting at least two of the following dimensions: number of effective feature points, image texture richness, proportion of occluded areas, and proportion of target area.

[0020] The four dimensions correspond to the four core time-consuming steps in the image processing pipeline: the number of effective feature points corresponds to the iterative computation scale of feature matching and pose solving; the image texture richness corresponds to the computation density of feature detection and descriptor generation; the proportion of occluded regions corresponds to the triggering frequency of out-of-point culling and occlusion completion logic; and the proportion of the target region area determines the total scale of pixels participating in the computation.

[0021] These four dimensions characterize the image processing load from four orthogonal perspectives: number of computational points, time per point, exception handling overhead, and region range, covering the main factors affecting latency. The inclusion of at least two dimensions, rather than requiring all, is based on the following considerations: The effectiveness of each dimension varies in different application scenarios. For example, in scenarios where the target is always fully visible, the occlusion dimension is always zero and does not carry effective information. Forcing its inclusion will dilute the weight accuracy of other dimensions. When the computational cost of a certain dimension is negligible under a specific hardware or algorithm configuration, such as when hardware-accelerated target detection makes the area ratio of the target region have a minimal impact on the total latency, it can be removed from the complexity calculation to simplify the model. The number of dimensions is positively correlated with the sample size requirement of the regression model; reducing redundant dimensions can reduce the amount of data collected for offline calibration.

[0022] Dimension selection employs a two-round calibration method. The first round involves a complete calibration of all dimensions to obtain the independent contribution of each dimension. Non-dominant dimensions are eliminated using 10% of the average contribution of all dimensions as a threshold. The second round uses the retained dimensions to perform a complete calibration again, yielding the final weight coefficients. The contribution coefficients of non-dominant dimensions are recorded as zero, and their corresponding weight coefficients are also zero; they are not included in the normalization calculation.

[0023] Specifically, the quantification method for the four dimensions is as follows: The number of effective feature points is extracted using a corner detection algorithm. The Harris corner detection algorithm is employed, and the specific process is as follows: For the input grayscale image, the grayscale gradient of each pixel in the x and y directions is calculated, a second-order moment matrix is ​​constructed, and its response function value is calculated. Local extrema of the response function are used as candidate feature points. Non-maximum suppression is applied to the candidate points to remove those with lower response values ​​in neighboring regions. Further, candidate points within a set pixel range at the image edge and candidate points with grayscale contrast below a set threshold are eliminated. The number of remaining candidate points is the number of effective feature points. The set threshold is determined based on the noise level of the image acquisition device. Specifically, multiple uniform grayscale images are acquired, and the standard deviation of the corner detection response values ​​in the images is statistically analyzed. The response value corresponding to three times the standard deviation is used as the contrast threshold; candidate points below this threshold are considered false feature points caused by noise.

[0024] Image texture richness is characterized by the entropy value of the image gray-level co-occurrence matrix. The higher the entropy value, the more complex the image texture. The specific calculation process is as follows: within the target area of ​​the image, the joint occurrence probability of gray values ​​is statistically calculated according to a set step size and direction to generate a gray-level co-occurrence matrix; the logarithm of the probability value of each element in the co-occurrence matrix is ​​taken, multiplied by the probability value, and the sum is obtained by taking the negative value.

[0025] The gray-level co-occurrence matrix has 8 to 16 gray levels, obtained by uniformly quantizing the gray values ​​of the original gray-level image. The number of quantization levels is determined based on the trade-off between the image's dynamic range and computational cost. The generation step size and direction are determined based on the neighborhood size of the feature matching algorithm. The step size is selected as half the number of pixels in the feature matching neighborhood. The reason for selecting this step size is that when the statistical window of the texture descriptor is consistent with the scale of the feature matching neighborhood, the correlation between texture richness and the computation time of the descriptor is strongest. Four directions are selected: 0 degrees, 45 degrees, 90 degrees, and 135 degrees. The average of the entropy values ​​of the four directions is taken as the final texture richness index. Taking the average of multiple directions can eliminate the directional bias of a single direction, making the texture richness index rotationally invariant.

[0026] The proportion of the occluded area is determined by spatial abrupt change detection of feature point distribution density. The specific process is as follows: the target area is divided into grid cells of a set size, and the number of effective feature points within each grid cell is counted to obtain the feature point density distribution; the side length of the grid cell is determined by the following formula: in, The side length of the grid cell; The pixel area of ​​the target region; This represents the total number of grid cells, an integer between 20 and 50, determined based on the target region size and occlusion detection resolution requirements. Too few cells result in insufficient spatial resolution, making it impossible to detect local occlusions; too many cells lead to large fluctuations in the number of feature points within a single cell, increasing the false detection rate.

[0027] The average density of all grid cells within the target area is calculated, and grid cells with densities lower than a predetermined proportion of the average density are identified as occluded cells. The ratio of the total area of ​​occluded cells to the area of ​​the target area is the occluded area ratio. The predetermined proportion is determined by a benchmark calibration method. A benchmark image of the target object is acquired in an unoccluded state, and the density of feature points per unit area is calculated as the benchmark density. In multiple sets of sample images with known occlusion ratios, the corresponding density reduction ratio is calculated. Using the manually labeled actual occlusion ratio as a benchmark, the density reduction ratio with the highest determination accuracy is selected as the predetermined proportion.

[0028] In the target region area ratio, the target region is the pixel region of the target in the image, determined by a template matching algorithm. The offline-acquired target template image and the current input image are subjected to normalized cross-correlation calculation, and the position with the largest correlation response is taken as the target center. The bounding rectangle corresponding to the template size is used as the target region. When the target undergoes rotation or scale changes, feature matching combined with homography matrix estimation is used to obtain the target's contour region. The ratio of the number of pixels in the target region to the total number of pixels in the image is the target region area ratio.

[0029] The formula for calculating content complexity is: in, This represents the content complexity value, with a dimension of 1. Let be the weight coefficient of the i-th complexity dimension, satisfying ; is the normalized value of the i-th complexity dimension. Each dimension is normalized to the range [0,1] by dividing by the maximum value of the corresponding dimension; i is the dimension index, and its value range is the set of dimensions involved in the calculation.

[0030] The weight coefficients for each dimension are determined based on the contribution of each dimension to the image processing latency based on offline calibration. The calibration process includes: calibration data acquisition, sample preprocessing, collinearity test, contribution calculation, and weight normalization.

[0031] During calibration, a hardware platform that is completely identical to that used in actual operation is built, including the same processor, the same vision sensor, the same operating system and algorithm version, and calibration image samples are collected within the typical operation scenarios of the robot.

[0032] Sample collection is organized by dimension for typical scenarios. For each complexity dimension, several sets of image samples are constructed where the value of that dimension changes significantly and the values ​​of other dimensions remain basically unchanged.

[0033] For the dimension of effective feature point quantity, multiple sets of images of the same target object at different distances are selected. The change in distance causes a significant increase or decrease in the total number of feature points, while the texture entropy value, occlusion ratio, and relative target area remain basically stable. For the dimension of texture richness, multiple sets of images of the same scene under different lighting angles or different surface texture attachment conditions are selected. The texture contrast changes significantly, while the number of feature points, target area, and occlusion ratio remain basically unchanged. For the dimension of occlusion ratio, multiple sets of images of the same target partially covered by occluders of different sizes are selected. The occlusion ratio changes significantly, while the other dimensions remain basically unchanged. For the dimension of target area ratio, multiple sets of images of the same target under different field-of-view ratio conditions are selected. The shooting distance or target scaling ratio is adjusted to make the target area ratio change significantly, while the other dimensions remain basically unchanged.

[0034] The sample sets of each typical scenario are merged to form a complete training sample set for partial least squares regression. The purpose of organizing typical scenarios by dimension is to ensure that the samples have sufficient independent variability in their respective dimensions. Combined with the principal component extraction mechanism of partial least squares regression, it is possible to separate the independent contribution of each dimension even when there is a certain degree of coupling between dimensions, thus avoiding the coefficient estimation instability problem of ordinary least squares under collinearity conditions.

[0035] The actual processing latency of each image frame is measured using a high-precision timer of the processor. The measurement starts at the moment when the image data is transferred from the sensor to the memory and ends at the moment when the pose calculation result is output to the controller. Each image frame simultaneously records the raw values ​​of four dimensions and the corresponding actual processing latency, forming a set of samples.

[0036] The total number of samples is determined by the error convergence method. Initially, a fixed independent validation set is defined, which accounts for 20% of the initial 200 samples and is not used in training. The remaining samples are used as the training set. After every 50 new samples are added, they are added to the training set, while the validation set remains unchanged. The root mean square error of the model is calculated using the same validation set. If, after three consecutive additions of new samples, the decrease in the validation set error fails to pass the F-test at a significance level of 0.05, the sample size is considered sufficient. The F-test is used to compare whether there is a significant difference in the mean squared error of the predictions of the two models before and after the addition of new samples on the same independent validation set. The numerator and denominator degrees of freedom are both the number of validation set samples minus one. The null hypothesis is that the variance of the prediction error of the model after the addition of new samples is not significantly different from that before the addition. A one-sided test is used to determine whether the decrease in error is significant.

[0037] Two preprocessing operations are performed after sample collection. The first is outlier removal. Under stable operating conditions, the image processing latency is approximately normally distributed due to random noise. However, factors such as operating system scheduling jitter and sudden changes in cache hit rate may introduce abnormally large latency values.

[0038] Outliers are removed using the Grubbs criterion. The mean and standard deviation of the time delay for all samples are calculated. For each sample, a statistic is calculated, which is the absolute deviation of the sample's time delay from the mean time delay divided by the standard deviation. The largest statistic is compared with the Grubbs critical value with a significance level of 0.05. If it is greater than the critical value, the corresponding sample is removed. The above process is repeated until there are no outliers.

[0039] The verification of the normality hypothesis uses different methods depending on the sample size. When the sample size is less than 50, the Shapiro-Wilk test is used; when the sample size is greater than 50, the Kolmogorov-Smirnov test is used. If the p-value is less than 0.05, it is considered a deviation from the normal distribution. In this case, the box plot method (interquartile range method) is used to identify outliers. Samples exceeding the upper quartile plus 1.5 times the interquartile range or falling below the lower quartile minus 1.5 times the interquartile range are considered outliers. The second step is dimensional normalization. For each dimension's original value, it is divided by the maximum value of that dimension in all valid samples to normalize it to the interval between zero and one. The purpose of normalization is to eliminate differences in the dimensions and value ranges of each dimension, making the weight coefficients comparable.

[0040] There may be statistical correlations, i.e., collinearity, between different complexity dimensions. For example, images with high texture richness are usually accompanied by a large number of effective feature points. Here, high texture richness means that the normalized result of the gray-level co-occurrence matrix entropy value is greater than 0.7, that is, the texture entropy of the image exceeds the level of the 70th percentile in the calibration sample. Under this condition, the normalized value of the number of effective feature points is usually also greater than 0.6.

[0041] Directly weighting and summing highly correlated dimensions can lead to information redundancy, making the coefficient estimates of the regression model unstable and the physical meaning of the weights unclear. Therefore, a collinearity test must be performed before weight labeling.

[0042] The test first calculates the Pearson correlation coefficients between each pair of the four normalized dimensions. The correlation coefficient is calculated by dividing the covariance of the two dimensions by the product of their respective standard deviations, with a value ranging from -1 to 1. Then, the variance inflation factor is calculated for each dimension. A linear regression is established with the remaining dimensions as independent variables to obtain the coefficient of determination for that dimension. The variance inflation factor is equal to one minus the reciprocal of the coefficient of determination.

[0043] If the absolute value of the correlation coefficient between any two dimensions is greater than 0.7, or the variance inflation factor of any dimension is greater than 5, then significant collinearity is determined to exist. In this case, ordinary least squares (OLS) is not suitable for directly calculating the independent contribution of each dimension; partial least squares (PLS) regression must be used. PLS regression extracts latent principal components that maximize both the explanatory power of the independent variables and the predictive power of the dependent variable, and can still obtain stable coefficient estimates even when collinearity exists among the independent variables.

[0044] In typical industrial vision servoing scenarios, the correlation coefficient between the number of effective feature points and texture richness is usually in the range of 0.65 to 0.85, the correlation coefficient between the proportion of occluded area and the proportion of target area is usually in the range of -0.5 to -0.7, and the correlation coefficients between other dimensions are generally below 0.4. Therefore, the collinearity condition is usually met in four-dimensional scenarios, and it is recommended to use partial least squares regression for weight calibration by default.

[0045] The contribution calculation employs partial least squares regression, aiming to extract several latent principal components from the independent variable matrix and dependent variable vector. These principal components should adequately summarize the information of the independent variables while maximizing their explanation of the variations in the dependent variable. The number of rows in the independent variable matrix represents the number of samples, and the number of columns represents the number of dimensions. The length of the dependent variable vector equals the number of samples, and each element represents the processing delay for the corresponding sample. The specific solution utilizes a nonlinear iterative partial least squares algorithm, as follows: Data standardization involves performing z-score standardization on the independent variable matrix and dependent variable vector column by column. This means subtracting the mean of each column and then dividing by the standard deviation of that column. After standardization, the mean of each column is zero and the variance is one. The purpose of standardization is to eliminate differences in dimensions and make the regression coefficients of each dimension comparable.

[0046] Then, the independent variable residuals are initialized as a standardized independent variable matrix, and the dependent variable residuals are initialized as a standardized dependent variable vector, and the first principal component extraction begins. For each principal component extracted, the current dependent variable residual is used as the initial vector for iteration, and the weights and scores of that principal component are calculated iteratively, including: calculating the independent variable weight vector and normalizing it, calculating the independent variable score vector, calculating the dependent variable loading coefficients, and calculating the independent variable loading vector; The iterative vector of the dependent variable is updated using the current independent variable score vector. The weight vector of the independent variable is then recalculated using the updated iterative vector of the dependent variable. This process is repeated until the change in the independent variable score vector is less than the convergence threshold. The convergence threshold is determined using the relative error threshold method; that is, convergence is considered complete when the relative change in the independent variable score vector is less than a set value. The relative change is calculated by dividing the L2 of the difference between the score vectors before and after the update by the L2 of the updated score vector. The set value is selected based on the accuracy requirement of the principal component's explanatory power for the dependent variable. When the cumulative explanatory power calculation accuracy is required to reach three decimal places, the corresponding threshold for the relative change in the score vector is approximately on the order of 10 to the power of negative 6. The derivation relationship is as follows: For every order of magnitude change in the score vector, its explanatory power for the dependent variable changes by approximately the square of the same order of magnitude. Therefore, to achieve an explanatory power accuracy of one-thousandth, the relative change in the score vector must be less than the square root of 10 to the power of negative three, or approximately 10 to the power of negative one and a half. Considering the cumulative effect of multiple principal components and leaving a margin, in engineering practice, 10 to the power of negative six is ​​usually used to ensure sufficient convergence.

[0047] In actual calibration, the threshold can be gradually reduced and the stability of the regression coefficient can be observed. When the relative change of the coefficient is less than one percent, the corresponding threshold is the convergence threshold that meets the accuracy requirements.

[0048] After convergence, the residuals of the independent variables are updated with the score vector and loading vector of the principal component, and the residuals of the dependent variable are updated with the score vector and loading coefficients of the dependent variable. The essence of residual update is to subtract the part explained by the principal component from the original data, and the remaining part is used to extract the next principal component.

[0049] Determine whether to continue extracting principal components. When the cumulative explanatory power of the extracted principal components for the dependent variable, i.e., image processing delay, exceeds a set threshold, such as 95%, or when the number of principal components reaches the number of dimensions of the independent variable, stop the extraction.

[0050] The cumulative explained rate equals the sum of squares of the residuals of the dependent variable under the current number of principal components divided by the sum of squares of the original dependent variables after standardization, and then subtracted from this ratio. After stopping the extraction, back substitution yields the standardized regression coefficients. The standardized regression coefficient vector is obtained by operating on the weight matrix, loading matrix, and dependent variable loading vector. The weight matrix is ​​the column-wise concatenation of the weight vectors of each principal component, the loading matrix is ​​the column-wise concatenation of the loading vectors of each principal component, and the dependent variable loading vector is the vector composed of the dependent variable loading coefficients corresponding to each principal component. The standardized regression coefficient equals the weight matrix multiplied by the inverse of the product of the transpose of the loading matrix and the weight matrix, and then multiplied by the dependent variable loading vector.

[0051] Standardized regression coefficients reflect the degree of influence of each dimension on the standard change in time delay. Specifically, each standard deviation change in a dimension corresponds to a change in the number of standard deviations in the dependent variable, and the weighting can be directly obtained by normalizing their absolute values. When the standardized coefficients are restored to their original dimensions, the original regression coefficient for each dimension is equal to the ratio of the corresponding standardized coefficient multiplied by the standard deviation of the dependent variable to the standard deviation of the independent variable for that dimension. This original regression coefficient is the contribution coefficient of that dimension, representing the time delay increment corresponding to each unit increment in that dimension.

[0052] After obtaining the contribution coefficients for each dimension, weight normalization is performed. The contribution coefficients should be positive, meaning that higher dimension values ​​correspond to longer processing latency. This aligns with the fundamental physical logic that higher content complexity leads to greater computational load. The contribution coefficients for each dimension are directly normalized to obtain their corresponding weight coefficients. The weight of a dimension equals its contribution coefficient divided by the sum of the contribution coefficients of all dimensions involved in the calculation, with all weights equal to one. A larger contribution coefficient indicates a stronger independent impact of that dimension on image processing latency, resulting in a higher weight. If the contribution coefficient of a dimension in the calibration results is negative, it indicates that the dimension has no positive correlation with latency, contradicting the physical meaning of complexity, and should be directly removed from the complexity calculation dimensions.

[0053] The following uses a four-dimensional scenario as an example to illustrate the complete calculation process of weight calibration. The values ​​involved are for illustrative purposes only and are not intended to limit the scope of protection: Assuming 100 sets of calibration samples have been collected, after preprocessing, the mean number of effective feature points is 120, the standard deviation is 50, the mean texture richness entropy value is 4.5, the standard deviation is 1.2, the mean occlusion area ratio is 0.15, the standard deviation is 0.10, the mean target area ratio is 0.40, the standard deviation is 0.15, and the processing latency is 8.0 milliseconds, with a standard deviation of 2.5 milliseconds.

[0054] When performing the collinearity test, the correlation coefficients between the number of effective feature points and texture richness were calculated to be 0.72, -0.25, and 0.35 respectively; the correlation coefficients between texture richness and occluded region proportion were -0.18 and 0.28 respectively; and the correlation coefficients between occluded region proportion and target region area proportion were -0.58. It is evident that the correlation coefficient between the number of feature points and texture richness is greater than 0.7, indicating a significant correlation. Further calculation of the variance inflation factors for each dimension revealed a variance inflation factor of 2.13 for the number of effective feature points, 1.98 for texture richness, 1.55 for occlusion proportion, and 1.73 for target area proportion. The maximum variance inflation factor was 2.13, which, although not exceeding the strict threshold of 5, showed that the correlation coefficients between feature points and texture richness exceeded 0.7. To ensure the stability of the weight estimation, partial least squares regression was used.

[0055] After iterative solving using the partial least squares algorithm, the cumulative explained rate reached 96.3% when extracting the two principal components, exceeding the 95% threshold, so extraction was stopped. The obtained standardized regression coefficients were as follows: number of effective feature points 0.45, texture richness 0.32, occlusion region percentage 0.28, and target region area percentage 0.18. The standardized regression coefficients reflect the degree of influence of each dimension on the standard change in latency, and all coefficients are positive, consistent with physical expectations. Normalization was performed directly based on the standardized coefficients, and the sum of the coefficients was 1.23. After normalization, the weight of the number of effective feature points was 0.45 divided by 1.23, approximately equal to 0.366; the weight of texture richness was 0.32 divided by 1.23, approximately equal to 0.260; the weight of occlusion region percentage was 0.28 divided by 1.23, approximately equal to 0.228; and the weight of target region area percentage was 0.18 divided by 1.23, approximately equal to 0.146. The sum of the four weights is 1.000, satisfying the normalization condition. The final content complexity calculation formula is: the normalized value of the number of effective feature points multiplied by 0.366, plus the normalized value of texture richness multiplied by 0.260, plus the normalized value of the occluded region proportion multiplied by 0.228, plus the normalized value of the target region area proportion multiplied by 0.146. The above values ​​are exemplary calculation results; the actual weights need to be obtained through calibration based on the specific hardware platform and algorithm configuration.

[0056] The image processing latency prediction model is a linear mapping relationship between content complexity and processing latency, established through regression analysis based on multiple sets of offline collected sample data. The rationale for using a linear mapping is that the computational load of an image increases approximately linearly with the number of feature points and pixels involved in the calculation. Given a fixed algorithm structure, the mapping relationship between processing latency and content complexity is close to linear. Using a linear model ensures fitting accuracy while offering advantages such as low computational load and convenient updates, making it suitable for real-time invocation within the control cycle. The model formula is: in, This is an estimated image processing latency, in seconds. The slope of the linear regression is expressed in seconds. This is the intercept of the linear regression, in seconds; This represents the content complexity value, with a dimension of 1.

[0057] During offline modeling, images with varying content complexity and their corresponding actual processing latency are collected to form sample pairs. The sample size is determined using an error convergence statistical test. Initial samples are collected stratified according to content complexity, and a regression model is established. Subsequently, new sample batches are gradually added. After each batch of new samples, the model is recalculated, and an F-test is performed on the independent validation set to evaluate the prediction error. The F-test is used to compare whether there is a significant difference in the mean squared error of the predictions of the two models before and after the addition of new samples on the same independent validation set. The numerator and denominator degrees of freedom are both the number of validation set samples minus one. The null hypothesis is that "the variance of the prediction error of the model after the addition of new samples is not significantly different from that before the addition." A one-sided test is used to determine whether the decrease in error is significant. The significance level α is set to 0.05, which is a conventional significance level value in statistical tests, balancing the risks of Type I and Type II errors.

[0058] When the decrease in prediction error fails to pass the F-test at a significance level of α after three consecutive additions of new samples, the current sample size is taken as the minimum sample size. The statistical basis for selecting three consecutive tests as the stopping condition is that a single F-test has randomness, and the error may not decrease significantly due to the randomness of sample batches, leading to a misjudgment of convergence. Failing the test three times consecutively can reduce the probability of misjudgment to the order of α³, significantly improving the reliability of the convergence determination. Stratified sampling is used during sample construction to ensure that the sample size of each complexity interval is proportional to the interval width.

[0059] The training data collection conditions cover the range of illumination intensity, target pose, and background complexity under typical operational scenarios to ensure the representativeness of the samples. During data collection, the robot is controlled to move along a trajectory, and the vision sensor synchronously acquires images and records the processing latency of each frame, with the acquisition frame rate consistent with the actual running frame rate. All collected samples are randomly divided into training and validation sets according to stratified sampling principles, with the training set accounting for 70% to 80% and the validation set accounting for 20% to 30%. The stratification is based on the content complexity range to ensure that the sample distribution ratio of the training and validation sets is consistent across each complexity range.

[0060] The initial parameters of the model are obtained by least squares solution during the offline modeling phase and are directly used as the initial values ​​for online operation. For scenarios without historical samples after a scene change, the initial slope is the ratio of the latency variation range to the complexity variation range under typical scenarios, and the initial intercept is the minimum latency value, i.e., the basic processing latency corresponding to zero content complexity. The convergence criterion for model updates is: after N consecutive updates, if the relative change in model parameters is less than a set threshold, the model is considered to have converged to the current scene, where N is half the window sample size, and the set threshold is 5% of the initial parameter values.

[0061] The intercept and slope of linear regression are solved using the least squares method. The formula is as follows: in, The total number of samples; Let i be the content complexity value of the i-th sample; This represents the actual processing delay for the i-th sample.

[0062] The initial accuracy of the model is verified using the root mean square error of the predicted error on a reserved validation sample set. The formula for calculating the root mean square error of the predicted error is as follows: in, To verify the sample size; The estimated latency for the i-th validation sample; Let be the actual time delay of the i-th verification sample. The model is considered qualified when the root mean square error is lower than the calibration tolerance. The calibration tolerance is determined by the error equal contribution allocation method: the allowable deviation of the system end position is divided by the typical motion speed of the end to obtain the equivalent total time delay allowable deviation, and then it is evenly allocated according to the number of each time delay error source. The share allocated to the image processing time delay prediction error is the calibration tolerance.

[0063] The content complexity is extracted synchronously from the intermediate calculation results during image feature extraction and pose calculation. After the complexity calculation is completed in the feature extraction stage, the model outputs the corresponding image processing latency prediction result, with the output time being earlier than the completion time of the pose calculation for the current frame. Specifically, the number of effective feature points and texture richness are taken from the calculation results in the feature extraction stage, while the occlusion region ratio and target region area ratio are taken from the region statistics results in the early stages of target detection and pose calculation. The values ​​for each dimension are all intermediate data naturally generated during the pose calculation process, requiring no additional independent complexity calculation module.

[0064] Using the dynamically estimated image processing delay as the prediction step size, the image processing delay compensation amount is obtained by extrapolating the position error over time. When extrapolating the position error over time, the prediction step size is adjusted in real time according to the prediction result of each frame, and a linear extrapolation method based on the error change rate is adopted.

[0065] The rationale for choosing linear extrapolation over higher-order extrapolation is that the order of magnitude of the image processing delay is much smaller than the time scale at which the end motion state changes significantly. Within the processing delay of a single frame, the end motion can be approximated as a uniform state, and the truncation error of linear extrapolation is within the allowable range. Higher-order extrapolation methods, such as quadratic extrapolation or polynomial fitting, require more historical frame data to support them. This not only increases the computational load but also introduces larger estimation errors in scenarios with fluctuations in visual frame intervals or frame loss, thus reducing the compensation accuracy. Linear extrapolation only requires the current error and the error of the previous frame to calculate. It has fast initialization, good robustness to frame loss, and the overall balance between computational load and compensation accuracy is optimal.

[0066] The formula for calculating the error change rate is: in, The error rate is expressed in meters per second. For the first The original position error of the frame, in meters; For the first The original position error of the frame, in meters; For the first The image acquisition time of a frame, in seconds; For the first The frame is the time of image acquisition, in seconds.

[0067] The formula for calculating the time extrapolation of position error is: in, This is the amount of image processing delay compensation, expressed in meters. This represents the original position error of the previous frame, in meters. The error rate is expressed in meters per second. This is a dynamic estimate of the image processing latency for the current frame, in seconds.

[0068] This formula represents the compensation amount extrapolated from the known position error of the previous frame to the moment when the pose calculation of the current frame is completed. The extrapolation starting point is the position error corresponding to the image acquisition time of the (k-1)th frame. The total extrapolation step size is the sum of the frame interval and the image processing delay, that is, the time difference from the acquisition time of the (k-1)th frame to the completion time of the pose calculation of the kth frame is... This fully covers the entire time difference from the known error moment to the target compensation moment, eliminating the influence of the inherent lag in the next frame of the parallel computing architecture. The reason for using the error of the previous frame as the starting point for extrapolation is that the pose calculation of the current frame has not yet been completed, and the position error of the current frame cannot be obtained. This is an inherent constraint under the parallel computing architecture. The inter-frame error change is estimated by linear extrapolation of the error change rate.

[0069] The extrapolation formula should be modified accordingly as follows: in, is the average frame interval of the visual sensor.

[0070] In the first frame or initialization phase, the error rate of change is initialized to zero, and the image processing delay compensation is equal to the current position error. When frame loss or frame interval exceeding a set multiple of the normal visual frame interval is detected, the calculation result of the current error rate of change is discarded, and the error rate of the previous valid frame is reused. The set multiple is determined based on the normal frame rate fluctuation range of the visual sensor. Specifically, it involves collecting the frame interval of multiple consecutive frames and calculating their average value and standard deviation. The frame interval corresponding to the average value plus three times the standard deviation is used as the judgment threshold. Frame intervals exceeding this threshold are judged as abnormal. When consecutive frame loss exceeds a set number of frames, the error rate is reset to zero. Error rate calculation resumes only after two consecutive frames of valid visual data are obtained again. Calculate using the following formula: In the formula, Allowable deviation for the position at the end of the system; This represents the statistical mean of the error change rate under typical operating conditions. The average interval between visual frames is denoted as ; the statistical mean of the error change rate is taken from the moving average of historical data during continuous system operation.

[0071] The feedforward compensation is calculated based on the expected end velocity of the original trajectory and the pre-calibrated communication delay. To calculate the feedforward compensation, the expected end velocity vector of the original trajectory at the current moment is taken and multiplied by the pre-calibrated communication delay scalar. The calculation formula is as follows: in, This is the feedforward compensation amount, in meters; The expected end velocity of the original trajectory at the current moment, in meters per second; This is the pre-calibrated communication delay, in seconds.

[0072] The communication delay is obtained by calibrating the round-trip delay of the heartbeat packets between the controller and the vision sensor. The calibration method is as follows: the controller periodically sends heartbeat packets to the vision sensor and records the sending time. The heartbeat packet period is determined by the allowable relative error of the communication delay measurement. The communication delay is measured continuously multiple times and spectrum analysis is performed. The frequency at which the delay power spectrum drops by 3dB is taken as the delay fluctuation bandwidth. The heartbeat packet frequency is taken to be more than twice this bandwidth to ensure that the Nyquist sampling condition is met. The heartbeat packet period is taken as the time interval corresponding to this frequency.

[0073] After receiving the heartbeat packet, the vision sensor immediately returns a response frame, which records the reception time. The controller records the time of receiving the response frame, and the difference between the controller's reception time and the controller's transmission time is used as the round-trip delay. The average and standard deviation of the round-trip delay measured multiple times are calculated. After removing abnormal measurement values ​​that exceed three times the standard deviation, the average is taken again, and the resulting average value is used as the calibration value of the communication delay.

[0074] The communication delay includes the control command issuance delay. The transmission delay of the control command from the controller to the joint actuator is much smaller than the image processing delay and the image transmission delay. In engineering, it can be separately calibrated and then superimposed on the communication delay calibration value to be included in the total communication delay. The heartbeat packet round-trip delay covers the transmission overhead of visual data uplink and command response downlink, and is the bidirectional transmission delay between the controller and the visual sensor.

[0075] When the feedforward compensation and position error compensation are in different coordinate systems, coordinate transformation is required: the position error is in the camera coordinate system, and the desired end velocity is in the base coordinate system. The feedforward compensation needs to be superimposed on the position error compensation after a coordinate transformation from base to camera. The coordinate transformation relationship is as follows: In the formula, This is the feedforward compensation amount in the camera coordinate system; This is the feedforward compensation amount in the base coordinate system; Let be the rotation matrix from the base coordinate system to the camera coordinate system, and be the rotation submatrix of the homogeneous transformation matrix obtained from hand-eye calibration. Hand-eye calibration adopts the Tsai-Lenz two-step method, which solves the transformation matrix from the camera end effector to the robot end effector by acquiring multiple sets of calibration board images with different poses and corresponding robot joint angles; when the eye is on the hand, the calibration result is the transformation from the end flange to the camera, and when the eye is outside the hand, the calibration result is the transformation from the base to the camera.

[0076] The total delay compensation is obtained by superimposing the two types of compensation amounts, and then mapped to the joint delay compensation increment via the Jacobian transpose. The formula for calculating the total delay compensation is: in, This represents the total delay compensation for the operating space, expressed in meters. This is the amount of image processing delay compensation, expressed in meters. This represents the feedforward compensation, expressed in meters. Both types of compensation are superimposed using the pose calculation completion time as the time reference and the camera coordinate system as the spatial reference. The image processing delay compensation describes the increment of the end position error caused by the image processing delay from the current acquisition time to the pose calculation completion time. Its coordinate system is the camera coordinate system, and the time end point is the pose calculation completion time. The feedforward compensation describes the displacement generated by the end effector moving at the desired speed during the communication delay from the issuance of the control command to the execution of the joint. After coordinate transformation, it is also in the camera coordinate system, and its time starting point is the moment when the pose calculation is completed. The two types of compensation cover the image processing delay and the communication delay, which are two consecutive stages in time. The compensation direction is to reduce the end effector error, so the total compensation can be obtained by direct algebraic addition.

[0077] The Jacobian transpose is mapped to the joint delay compensation increment, and the mapping formula is: in, This represents the time delay compensation increment in the joint space, expressed in radians. This represents the total delay compensation for the operating space, expressed in meters. Let Jacobian matrix be the value at the current pose. This is the transpose of the Jacobian matrix; the i-th row of the matrix corresponds to the i-th joint, and the j-th column corresponds to the j-th degree of freedom in the operation space. The Jacobian transpose mapping uses the Jacobian matrix corresponding to the pose of the previous visual frame. The Jacobian matrix is ​​updated synchronously with each frame of visual data and remains unchanged within the control period of adjacent visual frames.

[0078] In visual servoing scenarios, the dimensions of the operation space and the joint space are usually not equal. The Jacobian inverse matrix either does not exist or has singularity issues and cannot be solved directly. The Jacobian transpose is essentially equivalent to the gradient descent method, which distributes the error compensation amount of the operation space to each joint along the direction in which each joint contributes the most to the end effector motion. Under redundant degrees of freedom, it can naturally obtain the minimum norm solution of the joint increment, and the computational cost is much less than that of inversion or pseudo-inversion. In addition, the transpose mapping does not produce infinite joint increments when approaching singular poses, and its numerical stability is better than that of the inverse Jacobian method, avoiding control command divergence.

[0079] It should be noted that the Jacobian transpose mapping is essentially equivalent to gradient descent, and its convergence speed is slower than methods based on the Jacobian pseudoinverse, which may lead to a slower convergence process in the initial stage of large errors. To address this characteristic, this scheme adopts the following compensation strategy: The basic scaling correction stage of visual servoing works together with the latency compensation increment. The scaling correction provides a continuous driving force for error convergence, and the latency compensation adds an increment on this basis to eliminate steady-state lag. It does not rely entirely on transpose mapping to independently complete error convergence. The proportional correction gain is tuned according to the system bandwidth requirements to ensure that the convergence speed of the overall closed-loop system meets the operation cycle requirements, and the convergence rate of the transpose mapping does not constitute a bottleneck for the system convergence speed. In typical industrial vision servo systems, vision compensation is only responsible for millimeter-level deviation correction at the end of the trajectory. The initial position error is usually within the centimeter level, which is a small error. The linear approximation and convergence speed of the Jacobi transpose fully meet the accuracy requirements.

[0080] Based on the image processing delay prediction results output earlier than the current frame pose calculation completion time, the position error time extrapolation and feedforward compensation calculations are performed in parallel after the prediction output and before the pose calculation is completed. Specifically, the position error time extrapolation uses the end position error and error change rate of the previous frame as input. The position error of the previous frame is obtained by the difference between the previous pose calculation result and the expected pose of the previous frame. The feedforward compensation calculation uses the expected end velocity of the original motion trajectory at the current acquisition time as input. The expected end velocity is obtained by differentiating the original motion trajectory at the current acquisition time.

[0081] The specific implementation of parallel execution is divided into two categories according to the controller hardware architecture: On a multi-core processor platform, position error extrapolation and feedforward compensation calculations run in an independent calculation thread. This thread is woken up and starts calculation after the complexity calculation and latency estimation are completed. Pose calculation runs in the main processing thread. The two exchange data through shared memory, and the calculation results are synchronized with the pose calculation results through a semaphore mechanism. On a single-core processor platform, a time-slice pipeline approach is adopted. After feature extraction is completed, the compensation calculation step is inserted first, and then the subsequent process of pose solving continues. The compensation calculation is completed within the time window between the completion of feature extraction and the end of pose solving, which is equivalent to parallel execution in terms of functional timing. On a single-core platform, the time cost of compensation calculation must be less than the remaining time of pose solving to ensure that compensation calculation is completed before pose solving.

[0082] Neither of them requires waiting for the complete pose calculation result of the current frame to be calculated. After the compensation calculation is completed, the calculated compensation value is temporarily stored. After the pose calculation result of the current frame is output, the calculated compensation value and the pose calculation result are sent into the layered compensation stage at the same time, using the moment when the pose calculation is completed as the synchronization point.

[0083] Attitude error is corrected using an independent fixed gain ratio and does not participate in time delay prediction compensation. Its correction amount is mapped to the joint correction amount via the Jacobian transpose. The design rationale for using an independent correction channel is that the convergence speed of attitude error in visual servoing operations is usually slower than that of position error, and the attitude channel correction gain is low. Attitude deviations caused by time delay can be stably converged through proportional correction. Simultaneously, time delay prediction of attitude error is prone to introducing noise amplification, leading to end-effector oscillations. Using an independent proportional channel can meet the accuracy requirements and simplifies the computational chain. The formula for calculating the proportional correction of attitude error is: in, This is the attitude correction amount for the operating space; is the fixed gain matrix for attitude correction, and is a diagonal matrix, where each element on the diagonal corresponds to the correction gain for each degree of freedom of attitude in the operation space. The pose error for the current frame, in camera coordinates, is determined by the difference between the end-effector pose obtained from visual measurements and the desired pose. The pose correction is mapped to the joint-space pose correction increment via the pose Jacobian transpose: In the formula, Incremental pose correction for joint space; For the attitude Jacobian matrix, Its transpose represents the mapping relationship between end-effector posture motion and joint motion.

[0084] The attitude correction gain is determined based on the expected response bandwidth of the attitude channel and the system damping characteristics. It is obtained through simulation or experimental debugging, with the constraint of ensuring stable convergence of attitude error and no overshoot oscillation.

[0085] Joints corresponding to attitude errors are assigned to the low-sensitivity compensation level. Attitude adjustment is primarily accomplished by the wrist joints at the end effector of the robotic arm. These joints have a lower time-delay sensitivity coefficient to end-effector position errors than the arm joints, and therefore naturally fall into the low-sensitivity level in the hierarchical structure sorted by time-delay sensitivity. For joints involved in both position and attitude adjustment, their level assignment is determined by their time-delay sensitivity to position errors. The attitude correction is calculated independently and added to the joint's total correction, regardless of the level assignment.

[0086] The proportional correction frequency for attitude error is synchronized with the control cycle and is unaffected by fluctuations in the visual frame rate. When the visual frame rate is lower than the control frequency, the attitude error value of the current visual frame remains unchanged in each control cycle between adjacent visual frames. Each control cycle multiplies the currently maintained attitude error by a fixed gain to obtain the attitude correction amount. When the visual frame rate is higher than the control frequency, the attitude error corresponding to the visual data of the most recent frame at the arrival time of each control cycle is used as the correction input for that cycle. Attitude correction and position compensation are superimposed in joint space and output together, and they are time-aligned through the control cycle beat.

[0087] Based on the current joint angle, the visual time delay sensitivity coefficient of each joint is calculated using the Jacobian matrix, and the joints are divided into at least two compensation levels according to their sensitivity.

[0088] When calculating the visual time delay sensitivity coefficients for each joint, the column vector magnitude of each joint corresponding to the Jacobian matrix under the current pose is taken as the sensitivity coefficient. The calculation formula is as follows: in, Let be the visual time delay sensitivity coefficient of the i-th joint, with dimensions of length / angle; is the column vector in the Jacobian matrix corresponding to the i-th joint; The L2 norm of the vector is the square root of the sum of the squares of the degrees of freedom elements at each position of the column vector. The sensitivity coefficient characterizes the contribution of a single joint's unit angular motion to the end-effector displacement; the larger the coefficient, the greater the end-effector position deviation caused by the same time delay error.

[0089] The i-th column of the Jacobian matrix characterizes the displacement produced by a unit motion of the i-th joint in each degree of freedom at the end effector. The magnitude of the column vector reflects the contribution of a unit joint motion to the total displacement of the end effector position. Under the same time delay error, the joint with a larger contribution magnitude introduces a larger deviation in the end effector position, i.e., a higher time delay sensitivity. Therefore, using the L2 norm of the column vector as a quantitative indicator of sensitivity coefficient has clear physical significance. When dividing the compensation levels, each joint is sorted from largest to smallest according to its sensitivity coefficient, and divided into at least two compensation levels according to a preset ratio or preset threshold. Different joints contribute significantly to the end effector error. Applying delay compensation to all joints indiscriminately would introduce unnecessary prediction noise and computational overhead on low-sensitivity joints. After dividing by sensitivity, high-contribution joints receive full compensation to ensure accuracy, while low-contribution joints only retain basic ratio correction to suppress jitter. Overall, the compensation effect can be optimized under the condition of limited computational resources.

[0090] The number of strata is determined using the within-group variance elbow method. The specific calculation process is as follows: First, sort the joints according to their time delay sensitivity coefficient from largest to smallest; second, set the number of layers to k, starting from 2 and gradually increasing until the total number of joints is reduced by one; third, for each value of k, divide the sorted joints into k layers according to the first k-1 partition positions, and calculate the sum of squares within each group for each partitioning method: In the formula, Let be the sensitivity coefficient of the j-th joint. Given the mean sensitivity coefficient of layer l, the partition that minimizes the sum of squares within a group is taken as the optimal partition corresponding to that value k; fourth step, calculate the total sum of squares: In the formula, The mean of all joint sensitivity coefficients is used to obtain the proportion of variance within each group. Fifth step: Plot the relationship curve between k and r(k), and calculate the second difference of the curve. Step 6: Take the absolute value of the second difference. The k value corresponding to the maximum value is taken as the final number of strata. The point where the absolute value of the second difference is the maximum corresponds to the inflection point where the curve curvature is the largest, that is, the turning point where the marginal benefit of increasing the number of strata to reduce within-group variance changes from rapid decay to slow decay.

[0091] Starting from layer 2, gradually increase the number of layers. Calculate the ratio of the sum of squares within each group to the total sum of squares of the joint delay sensitivity coefficients in each layer for each layer number. Plot the relationship curve between the number of layers and the proportion of within-group variance, and take the layer number corresponding to the maximum absolute value of the second difference of the curve as the final number of layers.

[0092] When using a preset ratio for division, joints with the highest predicted ratio are designated as high-sensitivity levels, and the rest are designated as low-sensitivity levels. When using a preset threshold for division, joints with sensitivity coefficients higher than the preset threshold are designated as high-sensitivity levels, and the rest are designated as low-sensitivity levels. The preset threshold is determined using the sensitivity distribution gap method, with the following specific steps: The time delay sensitivity coefficients of all joints are sorted from largest to smallest. The relative difference between the sensitivity coefficients of two adjacent joints is calculated, which is the difference between the two coefficients divided by the sensitivity coefficient of the next joint. The gap corresponding to the largest relative difference among the first few joints after sorting is taken as the stratification threshold. The threshold is the geometric mean of the sensitivity coefficients on both sides of the gap. The basis for selecting the gap with the largest relative difference as the stratification boundary is that the existence of a significant gap in the sensitivity distribution means that there is a natural boundary between the magnitude of the time delay influence of the two joints. Dividing the stratification based on this boundary can minimize the difference within the layer and maximize the difference between the layers, avoiding the subjectivity of manually selecting the threshold. When the largest relative difference is less than 20%, it indicates that the sensitivity distribution of each joint is relatively uniform and there is no natural boundary. In this case, the threshold division method is not used, but a preset ratio method is used instead.

[0093] The preset ratio is determined using a precision reduction method: Starting with all joints assigned to the high-sensitivity level, they are sequentially assigned to the low-sensitivity level based on their time delay sensitivity coefficients, from smallest to largest. After assigning each joint, the additional end-effector error introduced by omitting time delay compensation for that joint is calculated; the additional end-effector error is calculated using the following formula: in, The additional end-effector position error introduced by omitting delay compensation for the i-th joint; Let be the time delay sensitivity coefficient of the i-th joint; Let be the time delay compensation increment for the i-th joint, in radians, obtained by mapping the compensation amount. This calculation is based on the physical meaning of the sensitivity coefficient, the end-effector displacement corresponding to a unit change in joint angle, and directly estimates the end-effector error increment after omitting compensation. It does not require closed-loop system simulation and can be calculated offline based on nominal parameters during the system design phase.

[0094] When the additional error exceeds the allowable error allocation value of the delay compensation stage, the allocation stops, and the ratio corresponding to the previous allocation is used as the preset ratio; the allowable error allocation value of the delay compensation stage is obtained by allocating the total allowable deviation at the end according to the principle of equal effect.

[0095] For highly sensitive joints, the joint delay compensation increment is superimposed on the proportional correction corresponding to the original position error, and full compensation is applied. For low-sensitive joints, only the proportional correction is based on the original position error, without superimposing the delay compensation increment. The proportional correction corresponding to the original position error is the basic control element of visual servoing. After calculating the proportional correction amount in the operating space, it is mapped to the joint space correction amount through Jacobian transpose. All joints participate, and its gain is tuned according to the following steps: Under conditions without visual compensation, given an end-step position error, measure the end-error response curve under pure proportional correction; The gain is gradually increased proportionally from small to large. After each adjustment, the overshoot and settling time of the system are recorded. The overshoot is the difference between the peak response and the steady-state value divided by the steady-state value. The settling time is the time required for the error to enter and remain within 5% of the steady-state value. The maximum gain that meets the requirements of overshoot not exceeding 10% and settling time being less than the cycle time is selected as the initial value. Closed-loop verification is performed on a typical operating trajectory. If continuous oscillations occur at the end of the trajectory, the gain is gradually reduced until the oscillations are eliminated and the steady-state error meets the tracking accuracy requirements. The gain at this point is the final tuning result. This tuning method is based on the time-domain performance index system of second-order systems in classical control theory. The upper limit of the gain is guaranteed by the overshoot constraint to ensure stability margin, and the lower limit is guaranteed by the settling time constraint to ensure operating efficiency. The final value is obtained after balancing the two.

[0096] Visual confidence is calculated in real time, based on either the feature point effectiveness rate or the reprojection error. The feature point effectiveness rate is the ratio of the number of effective feature points to the total number of detected feature points; a higher ratio indicates higher confidence. The total number of feature points is the total number of feature points detected after corner detection, after non-maximum suppression but before validity screening. The reprojection error is the average pixel deviation between the calculated pose (projecting the 3D feature points back to the image plane) and the actual detected image coordinates; a smaller deviation indicates higher confidence. The feature point effectiveness rate is suitable for scenes with dense features and rich textures, while the reprojection error is suitable for scenes with well-defined target geometric constraints.

[0097] When the visual confidence level is lower than a preset threshold, the delay compensation output of low-sensitivity joints is turned off first, while the delay compensation output of high-sensitivity joints is retained. The preset threshold is determined based on the calibration accuracy of the visual sensor and the allowable tracking error during operation, specifically as follows: The effective feature point rate or reprojection error value corresponding to the pose calculation accuracy just meeting the allowable error of the operation is used as the benchmark. After considering a certain safety margin, it is determined as the threshold to ensure that the remaining effective feature points can still support the minimum accuracy requirement of pose calculation under the confidence threshold condition.

[0098] After disabling time delay compensation for low-sensitivity joints, if visual confidence continues to decline and falls below the second-level threshold, time delay compensation for high-sensitivity joints will be further disabled, with only the basic proportional correction retained. When confidence is downgraded, only the incremental time delay compensation will be disabled, while the basic proportional correction and posture proportional correction will be retained throughout.

[0099] The second-level threshold is determined by the compensation effectiveness calibration method: at different visual confidence levels, the end error after applying time delay compensation and the end error without applying compensation are collected simultaneously; when the error after applying compensation is greater than the error without applying compensation for the first time, it indicates that the time delay compensation has lost its positive effect, and the corresponding confidence level is the second-level threshold.

[0100] When the target is completely lost, pose calculation fails, or the Jacobian matrix is ​​close to singular, the system enters protection mode according to the anomaly type. These three are independent triggering conditions, each corresponding to a different protection level: When the target is completely lost or pose calculation fails, it enters the first protection mode: all visual compensation calculations are stopped, only the joint commands corresponding to the original motion trajectory are executed, and target detection and pose calculation are continuously attempted. After the target is recaptured and the pose calculation is successful, the visual compensation output is gradually restored.

[0101] When the Jacobian matrix approaches singularity, a secondary protection mode is entered: visual compensation is not stopped, but the latency compensation increment of high-sensitivity joints is frozen, retaining only the basic proportional correction and the proportional correction of low-sensitivity joints. Simultaneously, the trajectory planning module adjusts the joint increments based on the condition number gradient, guiding the robot to move in the direction of decreasing Jacobian matrix condition number, gradually exiting the singular pose region. The singularity of the Jacobian matrix is ​​determined by the condition number; when the condition number exceeds a set threshold, it is considered close to singularity. This set threshold is determined based on the controller's numerical stability requirements, taking the condition number corresponding to a 1% relative error in matrix inversion as the threshold. When the condition number recovers to below the threshold, the secondary protection is released and full compensation is restored.

[0102] Level 1 protection has a higher priority than Level 2 protection. When multiple abnormal conditions are triggered simultaneously, the protection mode with the highest priority will be executed.

[0103] The correction values ​​of each joint after layered compensation are superimposed on the joint commands corresponding to the original motion trajectory to generate the final control command for issuance; at the same time, the actual processing latency data of each frame is recorded for iterative updating of the image processing latency prediction model.

[0104] It also includes the step of iteratively updating the image processing latency prediction model: when the sample size has not reached the sliding window size in the initial stage of system startup, all collected samples are used for regression update, and the regression solution is re-executed for each new valid sample; when the sample size fills the window, the sliding window update mechanism is switched. A sliding window sample set is used. The size of the sliding window is determined based on the balance between model update speed and fitting stability. The larger the window, the more stable the model fitting, but the slower the response to scene changes. The smaller the window, the faster the response, but the more susceptible to instantaneous fluctuations. The window size is determined by comparing the root mean square error of the model prediction under different window lengths with the convergence speed after scene switching. A window length that achieves a balance between steady-state accuracy and dynamic response is selected.

[0105] After accumulating the actual processing latency data for each predetermined number of frames, new data is added to the sample set and the oldest data is removed. The regression analysis is then re-executed to update the model parameters. The solution method for the regression analysis is the same as that used in offline modeling, still employing the least squares method for full-window recalculation. The reason for using full-window least squares instead of recursive least squares during online updates is that the linear regression model in this scheme has low dimensionality, the sliding window sample size is limited, and the computational cost of full-window recalculation is within the limits of the controller's computing power. Compared to recursive least squares, full-window least squares is unaffected by the selection of the forgetting coefficient, making it more adaptable to scenarios with sudden time delays and preventing model drift due to numerical accumulation errors during the recursive process. The sliding window mechanism achieves online model tracking through data input and output replacement, resolving the least squares solution using all samples within the window each time, resulting in superior numerical stability compared to the pointwise recursive method. The predetermined number of frames is determined using the tracking bandwidth matching method. Spectral analysis is performed on historical image processing delay data. The spectral analysis uses the Fast Fourier Transform (FFT) algorithm. The FFT is calculated on the mean-removed delay sequence, and the square of the amplitude of each frequency component is taken as the power spectrum estimate. The resolution is determined by the data length; the longer the data length, the higher the resolution. When implemented on an embedded controller, the recursive average periodogram method can be used. The delay sequence is segmented, the periodogram is calculated separately for each segment, and then the average is taken to reduce the variance of the spectrum estimate. The segment length is taken as an integer power of 2 to facilitate FFT calculation.

[0106] The frequency at which the power spectrum drops by 3 dB is taken as the bandwidth of the time delay variation, that is, the frequency at which the power spectrum amplitude drops from half of the DC component amplitude. The tracking bandwidth of the sliding window model is directly proportional to the update frequency and inversely proportional to the window sample size. The approximate formula for calculating the tracking bandwidth is: in, The model tracking bandwidth is expressed in Hertz. The model update frequency, measured in Hertz; This represents the sample size of the sliding window.

[0107] The tracking bandwidth is required to be greater than the latency change bandwidth. Based on this, the minimum update frequency is calculated, and then converted into the number of frames between two adjacent updates, which is the predetermined number of frames.

[0108] Before adding data to the sample set, abnormal delay data are removed based on the Grubbs criterion or the 3σ criterion.

[0109] Under stable operating conditions, image processing latency is mainly affected by the superposition of multiple independent random factors such as random image noise and operating system scheduling jitter. According to the central limit theorem, its distribution approximately follows a normal distribution. In the early stage of system operation or during scene switching, the normality of latency data can be verified by the Shapiro-Wilke test or skewness-kurtosis test. When the test results support the normal distribution hypothesis, the Grubbs criterion or 3σ criterion is used for outlier detection.

[0110] If the normality test fails, it indicates that the time delay distribution has skewed or heavy-tailed characteristics. In this case, the box plot method can be used for outlier detection: the upper quartile plus 1.5 times the interquartile range is used as the upper bound, and the lower quartile minus 1.5 times the interquartile range is used as the lower bound. Data outside the range are judged as outliers. The box plot method does not rely on distribution assumptions and is suitable for non-normally distributed data scenarios. The 3σ criterion is the simplest to calculate, requiring only the mean and standard deviation to complete the judgment, making it suitable for embedded controllers with limited computing resources. The Grubbs criterion introduces a sample size correction factor, and its detection accuracy is higher than that of the 3σ criterion and the false detection rate is lower when the sliding window sample size is small. Two criteria are provided to adapt to controllers with different computing power levels. Both are parametric anomaly detection methods, with a much lower computational cost than non-parametric methods, meeting the time constraints of real-time control.

[0111] When using the Grubbs criterion, the mean and standard deviation of all time-delay data in the sample set are calculated. The deviation of the data to be tested from the mean is divided by the standard deviation to obtain the statistic. When the statistic is greater than the critical value at the corresponding significance level, it is judged as outlier and removed. Determine by the following formula: In the formula, The value represents the sample size of the sliding window; this value ensures that the expected number of normal samples misclassified as abnormal does not exceed 1 during each model update, achieving a balance between anomaly detection sensitivity and false detection rate.

[0112] When using the 3σ criterion, the normal range is defined as the sample mean plus or minus three standard deviations. Data outside this range is considered outlier and removed. After outlier removal, if the current frame is also removed, the sample set is not updated until the next frame arrives, at which point the determination is made again.

[0113] When multiple exceptions are triggered simultaneously, they are handled according to the following priority: Communication timeout and pose calculation failure have the highest priority and will switch open-loop protection immediately upon triggering; visual confidence level is downgraded and controls the on / off state of compensation output; frame loss and error rate reset have the lowest priority and only affect the validity of extrapolation calculation without changing the system control mode; outlier removal is independent of the above priorities and only affects the sample selection stage of model update without affecting the control output of the current frame.

[0114] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A robot motion control method based on real-time visual feedback, characterized in that, include: An image processing latency prediction model is established based on the correspondence between the actual processing latency of historical frames and the content complexity. Based on the content complexity of the current frame image, the image processing latency of the current frame is predicted. Using the dynamically estimated image processing delay as the prediction step size, the position error is extrapolated over time to obtain the image processing delay compensation amount; the feedforward compensation amount is calculated based on the expected end velocity of the original motion trajectory and the pre-calibrated communication delay; the two types of compensation amounts are superimposed to obtain the total delay compensation amount, which is then mapped to the joint delay compensation increment via Jacobian transpose. The attitude error is corrected by an independent fixed gain ratio and does not participate in the time delay prediction compensation. Its correction amount is mapped to the joint correction amount through the Jacobian transpose. The joints corresponding to the posture error are assigned to the low-sensitivity compensation level; Based on the current joint angle, the visual time delay sensitivity coefficient of each joint is calculated using the Jacobian matrix, and the joints are divided into at least two compensation levels according to their sensitivity. For highly sensitive joints, the joint delay compensation increment is superimposed on the proportional correction corresponding to the original position error, and full compensation is applied. For low-sensitivity joints, only proportional correction is made based on the original position error, without adding time delay compensation increments; The correction values ​​of each joint after layered compensation are superimposed on the joint commands corresponding to the original motion trajectory to generate the final control command for issuance; at the same time, the actual processing latency data of each frame is recorded for iterative updating of the image processing latency prediction model.

2. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, The content complexity is a quantified value representing the computational load of image processing, calculated by weighting at least two of the following dimensions: The number of effective feature points, image texture richness, occlusion area ratio, and target area ratio; The weight coefficients for each dimension are determined based on the contribution of each dimension to the image processing latency, which is calibrated offline.

3. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, The image processing latency prediction model is a linear mapping relationship between content complexity and processing latency, and is established through regression analysis based on multiple sets of offline collected sample data. The content complexity is extracted synchronously from the intermediate calculation results in the image feature extraction and pose calculation process. After the complexity calculation is completed in the feature extraction stage, the model outputs the corresponding image processing delay prediction result. The output time is earlier than the time when the pose calculation of the current frame is completed.

4. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, When extrapolating the position error over time, the prediction step size is adjusted in real time according to the prediction results of each frame; A linear extrapolation method based on the error change rate is adopted, wherein the error change rate is obtained by dividing the difference in the original position error of adjacent frames by the time interval of the corresponding image acquisition time.

5. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, When calculating the feedforward compensation, the expected end velocity vector of the original motion trajectory at the current moment is taken and multiplied by the pre-calibrated communication delay scalar to obtain the feedforward compensation. The communication delay is obtained by calibrating the round-trip delay of the heartbeat packet between the controller and the vision sensor, and the communication delay includes the delay of issuing control commands.

6. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, When calculating the visual time delay sensitivity coefficient of each joint, the column vector magnitude of each joint corresponding to the Jacobian matrix under the current pose is taken as the sensitivity coefficient. When dividing the compensation levels, each joint is sorted from largest to smallest according to its sensitivity coefficient, and divided into at least two compensation levels according to a preset ratio or preset threshold.

7. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, Based on the image processing delay prediction results output earlier than the current frame pose calculation completion time, the position error time extrapolation and feedforward compensation calculations are executed in parallel after the prediction output and before the pose calculation is completed. The compensation calculation results are output to the hierarchical compensation stage synchronously with the pose calculation results.

8. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, The attitude error is corrected using only a fixed gain ratio, without time delay prediction extrapolation or feedforward compensation; the proportional correction frequency of the attitude error is synchronized with the control cycle and is not affected by visual frame rate fluctuations.

9. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, Visual confidence is calculated in real time, and the visual confidence is determined based on the feature point effectiveness or reprojection error. When the visual confidence is lower than a preset threshold, the time delay compensation output of low-sensitivity joints is turned off first, while the time delay compensation output of high-sensitivity joints is retained.

10. The robot motion control method based on real-time visual feedback according to claim 1, characterized in that, It also includes the step of iteratively updating the image processing latency prediction model: Using a sliding window sample set, after accumulating a predetermined number of actual processing delay data, new data is added to the sample set and the oldest data is removed. Then, regression analysis is re-executed to update the model parameters. Before adding data to the sample set, abnormal delay data are removed based on the Grubbs criterion or the 3σ criterion.