Mechanical arm visual servo control method and system based on self-learning disturbance observer

Abstract

The application provides a kind of mechanical arm visual servo control method and system based on self-learning disturbance observer, it is related to robot control technical field, the method comprises: obtaining real-time image data;Extract image feature vector;State estimation is carried out by extended Kalman filtering, and the filtered image feature state is obtained;Disturbance estimation value is obtained by kernel least mean square algorithm online learning;Based on the ideal model without disturbance, nominal control sequence and nominal state trajectory are generated;According to the deviation and disturbance estimation value, the final control command is generated and the robot arm is driven to move.The application combines extended Kalman filtering and kernel least mean square algorithm to construct a composite disturbance observer, realizes adaptive online learning and high-precision estimation of unknown disturbance, and forms a double-layer robust control structure combined with tube model predictive control, which significantly improves the control accuracy and robust stability of the mechanical arm visual servo system in complex disturbance environment.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a visual servo control method and system for a robotic arm based on a self-learning perturbation observer. Background Technology

[0002] Visual servoing technology is an important branch of robot control. It uses visual sensors to acquire target image information and guides robotic arms to complete precise positioning and tracking tasks. In existing technologies, visual servoing systems for robotic arms generally employ model-based control methods, which suppress external disturbances by constructing a fixed-structure observer or using model-based feedforward compensation strategies. These methods rely on accurate system modeling and prior knowledge of disturbances, treating unmodeled dynamics, parameter uncertainties, and external environmental disturbances as lumped disturbances for estimation and compensation.

[0003] However, when faced with complex disturbances that are nonlinear, strongly coupled, and have unknown dynamic characteristics, the traditional observers lack the ability to learn from the disturbance characteristics and cannot adjust the estimation strategy autonomously according to the system's operating state. This results in insufficient disturbance estimation accuracy, which in turn leads to a decline in control performance, an increase in tracking error, and even serious system instability when operating conditions change abruptly. Summary of the Invention

[0004] This invention provides a visual servo control method and system for robotic arms based on a self-learning disturbance observer, in order to solve the technical problem of decreased control accuracy caused by unknown disturbances in the prior art.

[0005] To achieve the above objectives, a first aspect of the present invention provides a visual servo control method for a robotic arm based on a self-learning perturbation observer, comprising:

[0006] Acquire real-time image data;

[0007] The real-time image data is subjected to visual processing to extract image features and obtain image feature vectors;

[0008] Using the image feature vector as the actual measured pixel coordinates, combined with the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle, the state is estimated by extended Kalman filtering to obtain the filtered image feature state of the current control cycle.

[0009] Based on the filtered image feature state and control input of the current control cycle, online learning is performed using the kernel least mean square algorithm to obtain the disturbance estimate of the current control cycle.

[0010] Based on the filtered image feature state of the current control cycle and the disturbance estimate of the current control cycle, a nominal control sequence and a nominal state trajectory are generated based on an unperturbed ideal model.

[0011] Based on the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory, as well as the disturbance estimate of the current control cycle, a final control command is generated and converted into a joint drive command to drive the robotic arm to move.

[0012] Furthermore, the step of obtaining the filtered image feature state of the current control cycle through extended Kalman filtering for state estimation includes:

[0013] Obtain the filtered image feature state, control input, and disturbance estimate of the previous control cycle;

[0014] Prior estimation is performed based on the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle to obtain the prior estimate of the current image feature state.

[0015] Based on the prior estimate of the current image feature state and the actual measured pixel coordinates, a posterior update is performed to obtain the image feature state after filtering in the current control period.

[0016] Furthermore, the step of obtaining the disturbance estimate for the current control cycle through online learning using the kernel least mean square algorithm includes:

[0017] Obtain the filtered image feature state and the control input for the current control cycle, and construct an extended state vector;

[0018] The extended state vector is mapped to the regenerative kernel Hilbert space, and the perturbation estimate for the current control cycle is obtained through online learning using the kernel least mean square algorithm.

[0019] Furthermore, the step of obtaining the disturbance estimate for the current control cycle through online learning using the kernel least mean square algorithm includes:

[0020] Based on the weight vector of the previous control cycle and the high-dimensional eigenvector of the current moment, the disturbance estimate for the current cycle is calculated using the following formula:

[0021] ;

[0022] in, To extend the state vector to a feature vector in a higher-dimensional space, This is the weight vector from the previous control cycle;

[0023] The estimation error between the filtered image feature state of the current control cycle and the actually measured pixel coordinates is calculated using the following formula:

[0024] ;

[0025] in, This is a priori estimate of the image feature state in the current control cycle. These are the actual measured pixel coordinates;

[0026] The weight vector is updated based on the estimation error and step size factor, using the following formula:

[0027] ;

[0028] in, Step size factor;

[0029] By employing kernel functions, the inner product in high-dimensional space is achieved by computing a kernel function in the original input space, thus avoiding the explicit computation of high-dimensional mappings.

[0030]

[0031] in, Indicates the inner product. This indicates that in higher-dimensional space, For kernel functions;

[0032] A Gaussian kernel function is used as the kernel function to calculate the inner product in high-dimensional space. The Gaussian kernel function is:

[0033] ;

[0034] in, , To expand the elements in the state vector, For kernel parameters;

[0035] Transforming the inner product operation in high-dimensional space into kernel function form yields the perturbation estimate, which is then expressed in the original space. The specific calculation formula is as follows:

[0036] ;

[0037] in, , These are the extended state vectors at different times in the original space;

[0038] Obtain the invariant set of the perturbation and tighten the constraints:

[0039]

[0040]

[0041] in, For the actual state constraint set, This is the tightened set of state constraints. Actual input constraint set, This is the tightened set of input constraints. Minkowski's difference For the tubular feedback gain matrix, For the set of disturbances.

[0042] Furthermore, the generation of the nominal control sequence and nominal state trajectory based on the perturbationless ideal model includes:

[0043] Constructing an unperturbed nominal model based on the image Jacobian matrix;

[0044] Define the prediction time domain and construct the objective function;

[0045] Solve the quadratic programming problem of the objective function under the constraints to obtain the nominal control sequence;

[0046] Generate a nominal state trajectory based on the nominal control sequence;

[0047] The objective function is:

[0048] ;

[0049] in, , For the desired image features, The state weight matrix is... To control the weight matrix, This is the terminal weight matrix. To predict the time domain, To control the time domain;

[0050] The constraints must be applied under the tightened conditions.

[0051] Furthermore, the generation of final control instructions includes:

[0052] Obtain the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory;

[0053] A compensation control quantity is generated based on the deviation and the disturbance estimate of the current control cycle;

[0054] The compensation control quantity is fused with the nominal control sequence and the disturbance compensation term to obtain the final control command.

[0055] A second aspect of the present invention provides a visual servo control system for a robotic arm based on a self-learning perturbation observer, comprising:

[0056] The visual sensing module is used to acquire real-time image data;

[0057] The visual processing module is used to perform visual processing on the real-time image data, extract image features, and obtain image feature vectors.

[0058] The disturbance observation module includes an extended Kalman filter and a kernel least mean square adaptive filter. It is used to take the image feature vector as the actual measured pixel coordinates, combine the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle, and perform state estimation through extended Kalman filtering to obtain the filtered image feature state of the current control cycle. Based on the filtered image feature state and control input of the current control cycle, it performs online learning through the kernel least mean square algorithm to obtain the disturbance estimate of the current control cycle.

[0059] The motion decision module is used to generate a nominal control sequence and a nominal state trajectory based on a disturbance-free ideal model according to the filtered image feature state of the current control cycle and the disturbance estimate of the current control cycle; and to generate a final control command according to the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory, as well as the disturbance estimate of the current control cycle.

[0060] A motion drive module is used to convert the final control command into joint drive commands;

[0061] The robotic arm execution module is used to drive the robotic arm to move according to the joint drive commands.

[0062] A third aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps in the visual servo control method for a robotic arm based on a self-learning perturbation observer as described in the first aspect of the present invention.

[0063] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the robotic arm visual servo control method based on a self-learning perturbation observer as described in the first aspect of the present invention.

[0064] A fifth aspect of the present invention provides a computer program product comprising software code, wherein the program in the software code performs the steps of the robotic arm visual servo control method based on a self-learning perturbation observer as described in the first aspect of the present invention.

[0065] Compared with existing technologies, the present invention provides a robotic arm visual servo control method and system based on a self-learning disturbance observer, which has the following advantages: This method constructs a composite disturbance observer by combining the Extended Kalman Filter (EKF) and Kernel Least Mean Square (KLMS) algorithm, achieving linear estimation of the system state and online learning of lumped disturbances in a high-dimensional space. Simultaneously, it generates a nominal trajectory through tubular model predictive control and integrates disturbance compensation, constraining the actual system trajectory within a robust invariant set. This system can autonomously adjust the estimation strategy according to the operating state, continuously improving the disturbance estimation accuracy, and achieving highly robust and high-precision control of the robotic arm's visual servo system in complex disturbance environments. Attached Figure Description

[0066] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0067] Figure 1 A flowchart of a robotic arm visual servo control method based on a self-learning perturbation observer provided in Embodiment 1 of the present invention;

[0068] Figure 2 This is a schematic diagram of the camera imaging geometry provided in Embodiment 1 of the present invention;

[0069] Figure 3 This is a flowchart of the visual servo control algorithm provided in Embodiment 1 of the present invention;

[0070] Figure 4 This is a diagram of the kernel adaptive filter structure provided in Embodiment 1 of the present invention;

[0071] Figure 5 This is a block diagram of the composite disturbance observer structure provided in Embodiment 1 of the present invention;

[0072] Figure 6 This is a block diagram of the Tube-MPC controller structure provided in Embodiment 1 of the present invention;

[0073] Figure 7 This is a schematic diagram of the visual servo closed-loop control principle provided in Embodiment 1 of the present invention;

[0074] Figure 8 This is a block diagram of a robotic arm vision servo control system based on a self-learning perturbation observer, provided in Embodiment 2 of the present invention. Detailed Implementation

[0075] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0076] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0077] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0078] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0079] Example 1

[0080] like Figure 1 This embodiment provides a visual servo control method for a robotic arm based on a self-learning perturbation observer. The specific process of this method includes steps S1 to S6. Figure 1 The complete control flow from acquiring real-time image data to generating joint drive commands is demonstrated, and the detailed implementation of each step is as follows:

[0081] S1. Acquire real-time image data.

[0082] Specifically, such as Figure 2 The diagram shows the geometric relationship of camera imaging. During the movement of the robotic arm, a vision sensor installed at the end of the robotic arm continuously collects real-time image data containing the target object. Figure 2 This demonstrates the transformation process from the world coordinate system to the pixel coordinate system, including rigid body transformation and perspective projection. Before performing visual servo control, camera calibration and hand-eye calibration are required: N sets of calibration board images are acquired using an end-effector vision sensor (such as a depth camera). Using Zhang Zhengyou's calibration method, the checkerboard calibration board is placed in different poses within the robotic arm's workspace. A calibration algorithm is then used to solve for the camera's intrinsic and extrinsic parameters. .

[0083]

[0084] in,( ( ) is the camera focal length, ( ) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system. The extrinsic parameter matrix includes the rotation matrix R and the translation matrix T, which represent the spatial attitude mapping relationship between the camera coordinate system and the world coordinate system.

[0085] Then, the robotic arm is controlled to drive the end-effector camera along a preset trajectory. The pose matrix of the robotic arm end-effector and the pose matrix of the calibration board captured by the camera are recorded for each movement. Based on the hand-eye calibration equation AX=XB, the X matrix is ​​optimized to obtain the transformation matrix between the camera and the end-effector.

[0086]

[0087] Establish a real-time transformation relationship between the camera coordinate system and the end coordinate system.

[0088] S2. Perform visual processing on the real-time image data, extract image features, and obtain image feature vectors.

[0089] Specifically, such as Figure 2 As shown, the key corner points of the target object are extracted through the vision processing module, and the world coordinate system coordinates of the corner points are obtained by combining the depth camera ranging function. The world coordinates are projected into pixel coordinates using the camera intrinsic and extrinsic parameter matrices. The following is the specific process:

[0090]

[0091] in, Let be the depth of the corner point in the camera coordinate system. The projection formula involves a depth factor, intrinsic matrix, extrinsic matrix, and homogeneous form of world coordinates. In pixel coordinates ( Using ) as the core, construct image feature vectors .

[0092] The relationship between the pixel change rate of feature points and the generalized velocity of the end effector is established using the image Jacobian matrix. The pixel change rate is equal to the image Jacobian matrix multiplied by the camera's velocity in its own coordinate system. The calculation formula is as follows:

[0093]

[0094] in, Let the velocity of the camera be in its own coordinate system. The image Jacobian matrix is ​​given by the following formula:

[0095]

[0096] S3. Using the image feature vector as the actual measured pixel coordinates, and combining the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle, the state is estimated by extended Kalman filtering to obtain the filtered image feature state of the current control cycle.

[0097] Specifically, such as Figure 3 The flowchart of the visual servo control algorithm shown illustrates the synergistic relationship between EKF filtering, KLMS disturbance learning, and Tube-MPC control. The sum of unmodeled dynamics and external disturbances is defined as... Using image feature vectors as the system's state variables Camera speed As input to the system Construct a first-order affine nonlinear system model:

[0098]

[0099]

[0100] in, Given the observed function, This is the system output, i.e., the actual measured pixel coordinates. The noise is measured to conform to a Gaussian distribution with a mean of 0 and a covariance of R.

[0101] Considering the discrete characteristics of the control system, the sampling period is... The system is discretized using the Euler method, resulting in discrete state equations:

[0102]

[0103] In each control cycle k, based on the filtered state from the previous time step... Control input Previous period disturbance estimate (This can be obtained later via KLMS) and the measurement value at the current time. The optimal estimate of image feature state is obtained after the extended Kalman filter is applied. The specific steps are as follows:

[0104] (1) Prior estimation:

[0105]

[0106]

[0107] in, This is the prior estimate of the error covariance matrix at the current time. This is an estimate of the error covariance matrix from the previous time step. The process noise covariance matrix is... Let be the Jacobian matrix of the state transition function, which is formed by the state transition function at the estimated point. It unfolds from there:

[0108]

[0109]

[0110] in, It is an identity matrix.

[0111] (2) Posterior update:

[0112]

[0113]

[0114]

[0115] in, For the calculated Kalman gain, Let be the Jacobian matrix of the observation function, and let be the observation function. , exist Perform a first-order Taylor expansion; The covariance matrix of the measurement noise; This is the error covariance matrix estimated at the current time.

[0116] S4. Based on the filtered image feature state and control input of the current control cycle, online learning is performed using the kernel least mean square algorithm to obtain the disturbance estimate of the current control cycle.

[0117] Specifically, such as Figure 4 The kernel adaptive filtering structure diagram shown illustrates how the KLMS algorithm uses kernel functions to map the input to a high-dimensional space and perform online learning. Figure 5 The block diagram of the composite perturbation observer shown illustrates the collaborative workflow of EKF and KLMS. The filtered state is represented by the EKF output. With this as the core, combined with the input values ​​of the IBVS system Construct extended state vector As the input vector to the KLMS perturbation observer, the extended state vector will be used. Mapping to the regenerating kernel Hilbert space, the kernel least mean square algorithm from kernel adaptive filtering is introduced to learn the unknown perturbation online in the high-dimensional space. The specific steps are as follows:

[0118] (1) Assumption It is the original space To high-dimensional feature space The feature mapping is then used This represents the corresponding characteristic vector after nonlinear transformation, where the input vector at time k is... , simplified to The weight vector is Then, by applying an adaptive filtering algorithm in high-dimensional space, the perturbation value of the predicted output is obtained as follows:

[0119]

[0120] (2) Calculate the estimation error:

[0121] (3) Minimize the estimation error using the gradient descent rule. The update formula for the weight vector is:

[0122]

[0123] Where μ is the step size factor of the algorithm. Repeating the above equation, we get:

[0124]

[0125] (4) Define the kernel function: given a function : ,exist If the following conditions are met:

[0126]

[0127] Then it is called a function This refers to a kernel function. A kernel function takes two vectors as input and returns a value that is plotted against each of the two vectors. The result of mapping followed by dot product is the same. This invention uses a Gaussian kernel function, whose mathematical form is as follows:

[0128]

[0129] (5) For each new signal vector input into the system, its corresponding output can be represented as the inner product of the mapped input signal:

[0130]

[0131] The inner product operation relation in the RKHS space can be transformed into that in the original space. The perturbation estimate is in the form of the kernel function:

[0132]

[0133] in, for The extended state vector in the original space at each time step. for The extended state vector in the original space at time , The estimation error is defined above.

[0134] S5. Based on the filtered image feature state of the current control cycle and the disturbance estimate of the current control cycle, generate a nominal control sequence and a nominal state trajectory based on an unperturbed ideal model.

[0135] Specifically, such as Figure 6 The diagram shown illustrates the Tube-MPC controller architecture, illustrating a two-layer control structure consisting of an upper-level nominal MPC controller and a lower-level error feedback controller. The prediction time domain is defined based on an unperturbed ideal model. Design the objective function Solve the quadratic programming problem to generate the nominal control sequence. With nominal state trajectory .

[0136] First, obtain the nominal model:

[0137]

[0138] in, It is the nominal image feature state. It is nominal control input.

[0139] Obtain the invariant set of the disturbance and tighten the constraints:

[0140]

[0141]

[0142] in, For the actual state constraint set, This is the tightened set of state constraints. Actual input constraint set, This is the tightened set of input constraints. Minkowski's difference For the tubular feedback gain matrix, It is the set of state deviations.

[0143] Among them, the objective function of the design for:

[0144] ;

[0145] in , For the desired image features, It is the state weight matrix, used to adjust the system's ability to suppress visual servo tracking errors; The weight matrix is ​​used to control the magnitude of control commands. This is the terminal weight matrix, used to enhance the control performance at the end of the prediction time domain.

[0146] Solving the optimization problem requires minimizing the objective function J while also satisfying the tightened constraints.

[0147] The length obtained after solving is nominal control sequence To predict the control sequence, and the length is nominal state trajectory , .

[0148] S6. Based on the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory, and the disturbance estimate of the current control cycle, generate the final control command and convert the final control command into a joint drive command to drive the robotic arm to move.

[0149] Specifically, such as Figure 7 The diagram shown illustrates the visual servo closed-loop control principle, illustrating the complete closed-loop circuit of the robotic arm system, where the desired and actual features are compared, and the control input is generated by the EKF-KLMS observer and controller to drive the system. The deviation between the actual and nominal image features is calculated, and a lower-level error feedback controller is designed. The nominal and compensation control inputs are then fused to obtain the final control input.

[0150] (1) Calculate the actual image features With nominal characteristics Deviation:

[0151]

[0152] in, By combining the discrete state equations of the actual system and the nominal system model, the dynamic equations of the error trajectory can be derived:

[0153]

[0154] in, .

[0155] (2) There exists an error set That is, if the current actual state And the input obtained by Tube MPC optimization was applied. The actual state at the next moment It will inevitably fall into the next pipe Internally, a feedback gain K is designed to compensate for the control input. Designed as follows:

[0156]

[0157] in, For interference compensation, The pseudo-inverse of the image Jacobian matrix, This is a tubular feedback control law. for Tube-type feedback gain matrix.

[0158] Camera velocity in camera coordinate system After a series of transformations, the final velocity is converted into a base coordinate system. Then, the end effector velocity is converted into the joint velocity of the robotic arm through the velocity Jacobian matrix. .

[0159] (1) Camera velocity transformation in camera coordinate system End velocity in the end coordinate system The conversion between these two is a velocity conversion between different objects. The angular velocity of the camera has a pulling effect on the linear velocity of the robotic arm's end effector; rotation cannot be considered in isolation. Therefore:

[0160]

[0161] in, Let be a skew-symmetric matrix, assuming ,So:

[0162]

[0163] but:

[0164]

[0165] in, and These are the transformation matrices from the camera coordinate system to the end coordinate system. Rotation vectors and translation vectors in the equation.

[0166] (2) Convert the end velocity from the end coordinate system to the base coordinate system. These two describe the same velocity, only their representation differs in different coordinate systems. Since the base coordinates are not affected by the motion, only the viewing angle is different, so only the rotation angle needs to be considered:

[0167] ;

[0168] (3) Once the end-effector velocity in the base coordinates is obtained, the joint velocities of the robotic arm can be solved using the velocity Jacobian matrix:

[0169]

[0170] Will The pulse commands are converted into pulse commands that can be recognized by the joint actuators, driving the movement of the robotic arm joints.

[0171] Example 2

[0172] like Figure 8 This embodiment provides a visual servo control system for a robotic arm based on a self-learning disturbance observer. The system includes a visual sensing module, a visual processing module, a disturbance observation module, a motion decision module, a motion drive module, and a robotic arm execution module. Figure 8 The data flow and collaboration relationships between the modules are illustrated, and the specific implementation of each module is as follows:

[0173] The visual sensing module is used to acquire real-time image data.

[0174] Specifically, the vision sensing module uses a vision sensor installed at the end of the robotic arm to acquire images of the working scene in real time, and achieves data synchronization through communication protocols such as USB 3.0, EtherCAT, or ROS2. Before performing visual servo control, the vision sensing module works together to complete camera calibration and hand-eye calibration: by acquiring images of the calibration board, the camera's intrinsic and extrinsic parameter matrices are solved using the Zhang Zhengyou calibration method. Then, the robotic arm is controlled to drive the end-effector camera to move along a preset trajectory. Based on the hand-eye calibration equation AX=XB, the transformation matrix between the camera and the end-effector is solved, and a real-time transformation relationship between the camera coordinate system and the end-effector coordinate system is established.

[0175] The visual processing module is used to perform visual processing on the real-time image data, extract image features, and obtain image feature vectors.

[0176] Specifically, the vision processing module processes the raw image information acquired by the vision sensor through real-time target detection and recognition, feature point extraction, and coordinate transformation. It ultimately outputs the real-time pose or feature error of the dynamic target relative to the robotic arm's end effector, providing core visual feedback information for the entire control system. The vision processing module extracts key corner points of the target object, obtains the world coordinate system coordinates of these corner points using the depth camera's ranging function, projects these world coordinates into pixel coordinate system coordinates using the camera's intrinsic and extrinsic parameter matrices, constructs an image feature vector based on these pixel coordinates, and establishes the relationship between the pixel change rate of feature points and the generalized velocity of the end effector using the image Jacobian matrix.

[0177] The disturbance observation module includes an extended Kalman filter and a kernel least mean square adaptive filter. It uses the image feature vector as the actual measured pixel coordinates, combines the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle, and performs state estimation through the extended Kalman filter to obtain the filtered image feature state of the current control cycle. Based on the filtered image feature state and control input of the current control cycle, it performs online learning through the kernel least mean square algorithm to obtain the disturbance estimate of the current control cycle.

[0178] Specifically, the disturbance observation module constructs a composite disturbance observer based on the extended Kalman filter and kernel least mean square algorithm to perform online estimation and learning of unmodeled dynamics, parameter uncertainties, and external disturbances in the system. The extended Kalman filter part defines the sum of unmodeled dynamics and external disturbances as... The system uses image feature vectors as state variables and camera velocity as input, constructing and discretizing a first-order affine nonlinear system model. In each control cycle, based on the filtered state from the previous time step, control input, the estimated disturbance from the previous cycle, and the current measurement, the optimal estimate of the image feature state is obtained through prior estimation and posterior update. The kernel least mean square algorithm uses the filtered state output by the EKF as the core, combining it with the system input to construct an extended state vector. This extended state vector is mapped to the regenerating kernel Hilbert space. Through kernel adaptive filtering, the unknown disturbance is learned online in the high-dimensional space, the disturbance estimate is calculated, and the weight vector is updated.

[0179] The motion decision module is used to generate a nominal control sequence and a nominal state trajectory based on a disturbance-free ideal model according to the filtered image feature state of the current control cycle and the disturbance estimate of the current control cycle; and to generate a final control command according to the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory, as well as the disturbance estimate of the current control cycle.

[0180] Specifically, the motion decision module introduces a Tube-MPC controller. Based on an unperturbed ideal model, it sets the prediction time domain, designs an objective function, solves a quadratic programming problem, and generates a nominal control sequence and a nominal state trajectory. By calculating the deviation between actual image features and nominal features, a lower-level error feedback controller is designed, fusing the nominal control quantity and the compensation control quantity to obtain the final control input. The control input is designed as the nominal control quantity plus a disturbance compensation term plus a tubular feedback control law, where the disturbance compensation term equals the negative pseudo-inverse of the image Jacobian matrix multiplied by the disturbance estimate, and the tubular feedback control law equals the feedback gain multiplied by the deviation.

[0181] A motion drive module is used to convert the final control command into joint drive commands.

[0182] Specifically, the motion drive module transforms the camera velocity in the camera coordinate system into the end-effector velocity in the base coordinate system through a series of transformations. Then, it uses a velocity Jacobian matrix to convert the end-effector velocity into the joint velocity of the robotic arm. Finally, it converts the joint velocity into pulse commands recognizable by the joint actuators. The velocity transformation includes: converting the camera velocity in the camera coordinate system to the end-effector velocity in the end-effector coordinate system, considering the traction motion of the camera's angular velocity on the linear velocity of the robotic arm's end-effector; the transformation formula involves a rotation matrix and a skew-symmetric matrix of a translation vector. Converting the end-effector velocity in the end-effector coordinate system to the end-effector velocity in the base coordinate system only requires considering the rotation angle; the transformation formula involves a rotation matrix.

[0183] The robotic arm execution module is used to drive the robotic arm to move according to the joint drive commands.

[0184] Specifically, the robotic arm execution module is responsible for completing the end effector movement according to the control commands. It drives the motors of each joint of the robotic arm through the underlying servo driver, and finally accurately completes the movement of the end effector.

[0185] Example 3

[0186] Embodiment 3 of the present invention provides an electronic device.

[0187] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. The processor includes, but is not limited to, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a tensor processor (TPU), or an artificial intelligence acceleration chip. The program is used to implement the steps in the robotic arm visual servo control method based on a self-learning perturbation observer as described in Embodiment 1 of the present invention when executing the program.

[0188] The detailed steps are the same as those of the robotic arm vision servo control method based on a self-learning perturbation observer provided in Example 1, and will not be repeated here.

[0189] Example 4

[0190] Embodiment 4 of the present invention provides a computer-readable storage medium.

[0191] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the robotic arm visual servo control method based on a self-learning perturbation observer as described in Embodiment 1 of the present invention.

[0192] The detailed steps are the same as those of the robotic arm vision servo control method based on a self-learning perturbation observer provided in Example 1, and will not be repeated here.

[0193] Example 5

[0194] Embodiment 5 of the present invention provides a computer program product.

[0195] A computer program product includes software code, wherein the program in the software code performs the steps of the robotic arm visual servo control method based on a self-learning perturbation observer as described in Embodiment 1 of the present invention.

[0196] The detailed steps are the same as those of the robotic arm vision servo control method based on a self-learning perturbation observer provided in Example 1, and will not be repeated here.

[0197] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages. For example, in one implementation, the methods and systems can be developed based on deep learning frameworks (such as TensorFlow, PyTorch, etc.) and using the Python language. Those skilled in the art will understand that other suitable programming languages ​​or tools can also be used for implementation without departing from the core ideas of the present invention.

[0198] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A method for visual servoing control of a robot arm based on a self-learning disturbance observer, the method comprising: include: Acquire real-time image data; The real-time image data is subjected to visual processing to extract image features and obtain image feature vectors; Using the image feature vector as the actual measured pixel coordinates, combined with the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle, the state is estimated by extended Kalman filtering to obtain the filtered image feature state of the current control cycle. Based on the filtered image feature state and control input of the current control cycle, online learning is performed using the kernel least mean square algorithm to obtain the disturbance estimate of the current control cycle. Based on the filtered image feature state of the current control cycle and the disturbance estimate of the current control cycle, a nominal control sequence and a nominal state trajectory are generated based on an unperturbed ideal model. Based on the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory, as well as the disturbance estimate of the current control cycle, a final control command is generated and converted into a joint drive command to drive the robotic arm to move.

2. The method of claim 1, wherein, The process of obtaining the filtered image feature state of the current control cycle through extended Kalman filtering for state estimation includes: Obtain the filtered image feature state, control input, and disturbance estimate of the previous control cycle; Prior estimation is performed based on the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle to obtain the prior estimate of the current image feature state. Based on the prior estimate of the current image feature state and the actual measured pixel coordinates, a posterior update is performed to obtain the image feature state after filtering in the current control period.

3. The method of claim 1, wherein, The process of obtaining the disturbance estimate for the current control cycle through online learning using the kernel least mean square algorithm includes: Obtain the filtered image feature state and the control input for the current control cycle, and construct an extended state vector; The extended state vector is mapped to the regenerative kernel Hilbert space, and the perturbation estimate for the current control cycle is obtained through online learning using the kernel least mean square algorithm.

4. The method of claim 3, wherein, The process of obtaining the disturbance estimate for the current control cycle through online learning using the kernel least mean square algorithm includes: Based on the weight vector of the previous control cycle and the high-dimensional eigenvector of the current moment, the disturbance estimate for the current cycle is calculated using the following formula: ； wherein, is a feature vector of the extended state vector mapped to a high dimensional space, is a weight vector of the previous control period; The estimation error between the filtered image feature state of the current control cycle and the actually measured pixel coordinates is calculated using the following formula: ； wherein, is the a priori estimate of the image feature state in the current control cycle, is the actual measured pixel coordinates; The weight vector is updated based on the estimation error and step size factor, using the following formula: ； wherein is a step factor; By employing kernel functions, the inner product in high-dimensional space is achieved by computing a kernel function in the original input space, thus avoiding the explicit computation of high-dimensional mappings. wherein, denotes an inner product, denotes in high dimensional space, is a kernel function; A Gaussian kernel function is used as the kernel function to calculate the inner product in high-dimensional space. The Gaussian kernel function is: ； in, , To expand the elements in the state vector, For kernel parameters; The inner product operation in high-dimensional space is transformed into a kernel function form to obtain the perturbation estimate. The specific calculation formula is as follows: ； in, , These are the extended state vectors at different times in the original space; Obtain the invariant set of the perturbation and tighten the constraints: ； ； in, For the actual state constraint set, This is the tightened set of state constraints. Actual input constraint set, This is the tightened set of input constraints. Minkowski's difference For the tubular feedback gain matrix, For the set of disturbances.

5. The method according to claim 1, characterized in that, The generation of nominal control sequences and nominal state trajectories based on the unperturbed ideal model includes: Constructing an unperturbed nominal model based on the image Jacobian matrix; Define the prediction time domain and construct the objective function; Solve the quadratic programming problem of the objective function under the constraints to obtain the nominal control sequence; Generate a nominal state trajectory based on the nominal control sequence; The objective function is: ； in, , For the desired image features, The state weight matrix is... To control the weight matrix, This is the terminal weight matrix. To predict the time domain, To control the time domain; The constraints must be applied under the tightened conditions.

6. The method according to claim 1, characterized in that, The generation of the final control command includes: Obtain the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory; A compensation control quantity is generated based on the deviation and the disturbance estimate of the current control cycle; The compensation control quantity is fused with the nominal control sequence and the disturbance compensation term to obtain the final control command.

7. A visual servo control system for a robotic arm based on a self-learning perturbation observer, characterized in that, include: The visual sensing module is used to acquire real-time image data; The visual processing module is used to perform visual processing on the real-time image data, extract image features, and obtain image feature vectors. The disturbance observation module includes an extended Kalman filter and a kernel least mean square adaptive filter. It is used to take the image feature vector as the actual measured pixel coordinates, combine the filtered image feature state of the previous control cycle, the control input, and the disturbance estimate of the previous control cycle, and perform state estimation through extended Kalman filtering to obtain the filtered image feature state of the current control cycle. Based on the filtered image feature state and control input of the current control cycle, it performs online learning through the kernel least mean square algorithm to obtain the disturbance estimate of the current control cycle. The motion decision module is used to generate a nominal control sequence and a nominal state trajectory based on a disturbance-free ideal model according to the filtered image feature state of the current control cycle and the disturbance estimate of the current control cycle; and to generate a final control command according to the deviation between the filtered image feature state of the current control cycle and the nominal state trajectory, as well as the disturbance estimate of the current control cycle. A motion drive module is used to convert the final control command into joint drive commands; The robotic arm execution module is used to drive the robotic arm to move according to the joint drive commands.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the robotic arm visual servo control method based on a self-learning perturbation observer according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the robotic arm visual servo control method based on a self-learning perturbation observer as described in any one of claims 1 to 6.

10. A computer program product, comprising software code, characterized in that, The program in the software code executes the steps of the robotic arm visual servo control method based on a self-learning perturbation observer according to any one of claims 1 to 6.

Images