Mechanical arm visual servo tracking method and device, medium, product and robot

Abstract

The invention discloses a mechanical arm visual servo tracking method and device, a medium, a product and a robot, and relates to the field of robot visual tracking, and the method comprises the steps: obtaining a target position in real time based on machine vision, and generating target position data; adopting a CARIMA model to construct a position prediction model; generating a predicted value of the target position based on the target position data by adopting a position prediction model, and correcting the predicted value of the target position according to real-time feedback of target operation to obtain a predicted target position; and according to the error between the predicted target position and the current target position of the mechanical arm, the movement track of the mechanical arm is adjusted. According to the invention, the precision of target tracking and the stability of control can be improved in a dynamic environment, so that the problem that a robot cannot accurately grab a dynamic target when performing a visual servo tracking task is solved.

Description

Technical Field

[0001] The present application relates to the field of robot vision tracking, and particularly to a robotic arm vision servo tracking method, device, medium, product, and robot. Background Art

[0002] With the development of industrial automation and intelligent manufacturing technologies, robotic arms are widely used in various tasks, especially in target tracking and grasping in dynamic environments. However, traditional vision servo control methods often have difficulty in dealing with dynamic targets with fast movement or irregular motion, and there are problems of position prediction errors and target mismatches.

[0003] The CARIMA (Controlled Autoregressive Integrated Moving Average) model is a method that combines the characteristics of autoregressive (AR) and integrated moving average (IMA). It can model the historical data of the system, predict the future position of the target, and make real-time corrections according to the error. However, in the field of robot vision control, there is no relevant application of the CARIMA model. Based on this, how to use the CARIMA model to accurately track the target in a dynamic environment and ensure that the robot accurately grasps the target has become a major challenge in vision servo control technology. Summary of the Invention

[0004] The purpose of the present application is to provide a robotic arm vision servo tracking method, device, medium, product, and robot, which can improve the accuracy of target tracking and the stability of control in a dynamic environment, and thus solve the problem that the robot cannot accurately grasp dynamic targets during vision servo tracking tasks.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In the first aspect, the present application provides a robotic arm vision servo tracking method, including:

[0007] Based on machine vision, obtain the target position in real time and generate target position data;

[0008] Use the CARIMA model to construct a position prediction model;

[0009] Use the position prediction model to generate a predicted value of the target position based on the target position data, and correct the predicted value of the target position according to the real-time feedback of the target movement to obtain the predicted target position;

[0010] Adjust the movement trajectory of the robotic arm according to the error between the predicted target position and the current target position of the robotic arm.

[0011] Optionally, the target position is obtained in real time based on machine vision to generate target position data, including:

[0012] The target position is obtained in real time based on machine vision, and the target position is preprocessed;

[0013] The target position data is formed based on the preprocessed target position.

[0014] Optionally, the preprocessing includes: noise filtering and smoothing processing.

[0015] Optionally, according to the real-time feedback of the target operation, the predicted value of the target position is corrected to obtain the predicted target position, including:

[0016] According to the real-time feedback of the target operation, the increment of the current target position is determined in combination with the historical increment;

[0017] Based on the increment of the current target position, the predicted value of the current target position is corrected to obtain the predicted target position.

[0018] Optionally, the increment of the target position is expressed as:

[0019] Δy(t) = θ1Δy(t - 1) + θ2Δy(t - 2) + … + θ q Δy(t - q) + η(t);

[0020] In the formula, Δy(t) represents the increment of the target position at time t, θ1, θ2, …, θ q are all coefficients of the integral sliding average part of the CARIMA model, η(t) is the prediction error of the position prediction model, and Δy(t - 1), Δy(t - 2),..., Δy(t - q) respectively represent the historical increments at times t - 1, t - 2,..., t - p.

[0021] Optionally, according to the error between the predicted target position and the current target position of the robotic arm, the motion trajectory of the robotic arm is adjusted, including:

[0022] The trajectory correction amount is determined according to the error between the predicted target position and the current target position of the robotic arm;

[0023] Based on the trajectory correction amount, the motion trajectory of the robotic arm is adjusted.

[0024] In a second aspect, the present application provides a robot, including: a robot body, a vision sensor, a linear parameter time-varying prediction controller, and an incremental prediction controller;

[0025] The visual sensor, the linear parameter time-varying predictive controller, and the incremental predictive controller are all disposed on the robot body; the visual sensor is connected to the linear parameter time-varying predictive controller; the linear parameter time-varying predictive controller is connected to the incremental predictive controller; the linear parameter time-varying predictive controller and the incremental predictive controller are integrated into the intelligent control unit of the robot body;

[0026] The visual sensor is used to obtain the target position in real time and generate target position data; the linear parameter time-varying predictive controller is implanted with a position prediction model, which is used to generate a predicted value of the target position based on the target position data by using the position prediction model, and is used to correct the predicted value of the target position according to the real-time feedback of the target operation to obtain the predicted target position; the position prediction model is constructed by using the CARIMA model; the incremental predictive controller is used to adjust the motion trajectory of the robotic arm according to the error between the predicted target position and the current target position of the robotic arm.

[0027] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the robotic arm visual servo tracking method provided above.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the robotic arm visual servo tracking method provided above are implemented.

[0029] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the robotic arm visual servo tracking method provided above are implemented.

[0030] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0031] The present application provides a robotic arm visual servo tracking method, device, medium, product, and robot. By using the CARIMA model to construct a position prediction model to generate and correct the predicted value of the target position, the time-varying characteristics of dynamic targets can be effectively processed, and the accuracy of target tracking can be improved in a dynamic environment; according to the error between the predicted target position and the current target position of the robotic arm, adjusting the motion trajectory of the robotic arm can improve the control accuracy of the robotic arm for grasping the target, so as to be applicable to the target grasping and tracking tasks in a high-dynamic environment, thereby solving the problem that the robot cannot accurately grasp dynamic targets when performing visual servo tracking tasks. Description of the Drawings

[0032] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0033] Figure 1 It is a schematic flowchart of a robotic arm visual servo tracking method provided by an embodiment of the present application;

[0034] Figure 2 It is a schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0036] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0037] In an exemplary embodiment, the present application provides a robotic arm visual servo tracking method, which is executed by a computer device. Specifically, it can be executed independently by a computer device such as a terminal or a server, or jointly executed by a terminal and a server. In the embodiments of the present application, this method is described by taking it as an example applied to a server. As Figure 1 shown, this method includes:

[0038] Step 100: Based on machine vision, obtain the target position in real time and generate target position data.

[0039] Step 101: Use the CARIMA model to construct a position prediction model.

[0040] Step 102: Use the position prediction model to generate a predicted value of the target position based on the target position data, and correct the predicted value of the target position according to the real-time feedback of the target operation to obtain the predicted target position.

[0041] Step 103: Adjust the motion trajectory of the robotic arm according to the error between the predicted target position and the current target position of the robotic arm.

[0042] In another exemplary embodiment of the present application, in order to ensure that the robotic arm can accurately track the target in a dynamic environment and perform grasping, in this embodiment, the implementation process of step 100 can be replaced by the following steps 200 to 203.

[0043] Step 200: Obtain the target position in real time based on machine vision. For example, use machine vision devices such as cameras, lidar, or depth sensors to capture the position information of the target in real time. The position information of these targets is the basis for establishing the position prediction model and predictive control.

[0044] Step 201: Preprocess the target position. For example, perform preprocessing operations such as noise filtering and smoothing on the collected target position to remove external interference and improve the accuracy of the data. And, the preprocessing method can adopt Kalman filtering, low-pass filtering, etc.

[0045] Step 202: Form target position data based on the preprocessed target position.

[0046] In another exemplary embodiment of the present application, the CARIMA model is mainly used to describe a dynamic system with time-varying characteristics. Especially in target trajectory prediction, it can predict the future motion state of the target based on historical data. Among them, the autoregressive (AR) part of the CARIMA model mainly infers the motion pattern of the target through current and past observation data. The integrated moving average (IMA) part of the CARIMA model mainly models the accumulation of errors to correct the deviation in prediction. Based on this, in order to accurately predict the future motion state of the target based on historical data, in the implementation process of the above step 101, the position data of the target is set as y(t). According to the CARIMA model, the prediction equation of the target position of the constructed position prediction model is:

[0047] y(t) = φ1y(t - 1) + φ2y(t - 2) + … + φ p y(t - p) + ∈(t).

[0048] In the formula, φ1, φ2, …, φ p are the autoregressive coefficients of the CARIMA model, p is the order of autoregression, ∈(t) is the error term, and has the characteristics of moving average.

[0049] Furthermore, for incremental predictive control, in step 102, the increment of the current target position can be determined according to the real-time feedback of the target operation and combined with the historical increment. The predicted value of the current target position is corrected based on the increment of the current target position to obtain the predicted target position.

[0050] For example, the increment Δy(t) of the target position at time t of the position prediction model can be predicted by the historical increments Δy(t-1), Δy(t-2), …, Δy(t-q), and we have:

[0051] Δy(t) = θ1Δy(t-1) + θ2Δy(t-2) + … + θ q Δy(t-q) + η(t);

[0052] In the formula, θ1, θ2, …, θ q are all coefficients of the integrated moving average part of the CARIMA model, and η(t) is the prediction error of the position prediction model.

[0053] When predicting the future position of the target through historical position data and incremental changes, assuming that the target position at time t is y(t), through the calculation of the autoregressive and moving average parts, the increment Δy(t+1) of the target position is obtained, and then the predicted target position y(t+1) is calculated:

[0054] y(t+1) = y(t) + Δy(t+1).

[0055] Based on the above description, in the process of correcting the predicted value of the current target position based on the increment of the current target position to obtain the predicted target position, the predicted value is corrected according to the real-time feedback of the target movement. Whenever new target movement feedback information arrives, the predicted position of the target is recalculated and corrected according to the increment of the current target position. This real-time correction mechanism can implement a continuous adjustment control strategy in a dynamically changing environment.

[0056] In another exemplary embodiment of the present application, if the position of the target deviates, the robot should adjust its motion trajectory to eliminate the error. Based on this, the implementation process of step 103 can be replaced by the following step 300-step 301.

[0057] Step 300: Determine the trajectory correction amount according to the error between the predicted target position and the current target position of the robotic arm. Among them, it is set that the current pose of the robot is p(t), the predicted value of the target position is y(t+1), and the required trajectory correction amount is Δp(t):

[0058] Δp(t) = K(y(t+1) - p(t)).

[0059] Among them, K is the gain matrix, and the response sensitivity of the system is optimized by adjusting K.

[0060] Step 301: Adjust the motion trajectory of the robotic arm based on the trajectory correction amount. Based on the adjusted motion trajectory information, the execution device (such as a robot, etc.) adjusts the motion of the robotic arm through a motion control algorithm (such as inverse kinematics or dynamics control), so as to ensure that the robotic arm can accurately track the target and perform tasks such as target grasping, item handling, and assembly.

[0061] In an exemplary embodiment, based on the same inventive concept, the present application also provides a robot. The robot includes: a robot body, a vision sensor, a linear parameter time-varying predictive controller, and an incremental predictive controller.

[0062] The vision sensor, the linear parameter time-varying predictive controller, and the incremental predictive controller are all disposed on the robot body. The vision sensor is connected to the linear parameter time-varying predictive controller. The linear parameter time-varying predictive controller is connected to the incremental predictive controller. The linear parameter time-varying predictive controller and the incremental predictive controller are integrally arranged as an intelligent control unit of the robot body.

[0063] The vision sensor is used to obtain the target position in real time and generate target position data. The linear parameter time-varying predictive controller is implanted with a position prediction model, and is used to generate a predicted value of the target position based on the target position data by using the position prediction model, and is used to correct the predicted value of the target position according to the real-time feedback of the target operation to obtain the predicted target position. The position prediction model is constructed by using the CARIMA model. The incremental predictive controller is used to adjust the motion trajectory of the robotic arm according to the error between the predicted target position and the current target position of the robotic arm.

[0064] Among them, the linear parameter time-varying predictive controller can be an incremental form of controller. This controller can predict the future motion position of the target by analyzing the historical motion trajectory of the target, and adjust the motion trajectory of the robotic arm according to these predicted values.

[0065] Furthermore, through incremental prediction, the linear parameter time-varying predictive controller can eliminate the cumulative error in the system, avoid the prediction deviation caused by the uncertainty of the target motion, and enhance the tracking accuracy of the robotic arm to the target.

[0066] The incremental predictive controller adjusts the motion trajectory of the robotic arm. When the position of the target deviates, it controls the robot body to adjust its motion trajectory to eliminate the error. Based on the corrected trajectory information, the robot body adjusts the motion of the robotic arm through a motion control algorithm (such as inverse kinematics or dynamics control), so as to ensure that the robotic arm can accurately track the target and perform the grasping task.

[0067] As a preferred embodiment, in the actual application process, the linear parameter time-varying predictive controller and the incremental predictive controller can be encapsulated into a computer program unit according to the robotic arm visual servo tracking method provided above in this application, so as to be implanted into the existing robot intelligent control device to adjust and control the motion trajectory of the robotic arm of the existing robot.

[0068] In summary, through the CARIMA model, this application can effectively handle the time-varying characteristics of dynamic targets. Traditional servo control methods usually rely only on fixed models, while the CARIMA model can continuously update predictions based on historical data and perform error correction. Therefore, the control accuracy of this application is greatly improved, especially suitable for target grasping and tracking tasks in high-dynamic environments, solving the problem of grasping dynamic targets when the robot performs visual servo tracking tasks, and improving the accuracy of target tracking and the stability of control.

[0069] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structural diagram can be as Figure 2 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store robotic arm visual servo tracking data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a robotic arm visual servo tracking method.

[0070] Those skilled in the art can understand that Figure 2 the structure shown in

[0071] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which when executed by a processor, implements the steps in the above method embodiments.

[0072] In an exemplary embodiment, a computer program product is provided, including a computer program, which when executed by a processor, implements the steps in the above method embodiments.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0074] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0075] In each of the embodiments provided in this application, the database involved may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., and is not limited thereto. In each of the embodiments provided in this application, the processor may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., and is not limited thereto.

[0076] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0077] Specific examples are used in this article to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A robot arm visual servo tracking method, characterized in that: include: Acquire the target position in real time based on machine vision and generate target position data; The CARIMA model is used to build a location prediction model; Using a position prediction model, generating a predicted value of the target position based on the target position data, and correcting the predicted value of the target position according to real-time feedback of the target operation to obtain a predicted target position; According to the error between the predicted target position and the current target position of the robot arm, the motion trajectory of the robot arm is adjusted.

2. The robot arm visual servo tracking method according to claim 1, characterized in that: Based on machine vision, the target position is acquired in real time and the target position data is generated, including: Acquire the target position in real time based on machine vision, and pre-process the target position; The target position data is formed based on the pre-processed target position.

3. The robot arm visual servo tracking method according to claim 2, characterized in that: The preprocessing includes: noise filtering and smoothing.

4. The robot arm visual servo tracking method according to claim 1, characterized in that: According to the real-time feedback of the target operation, the predicted value of the target position is corrected to obtain the predicted target position, including: Based on the real-time feedback of the target operation, the increment of the current target position is determined in combination with the historical increments; The predicted value of the current target position is corrected based on the increment of the current target position to obtain the predicted target position.

5. The robot arm visual servo tracking method according to claim 4, characterized in that: The increment of the target position is expressed as: Δy(t)=θ1Δy(t-1)+θ2Δy(t-2)+…+θ q Δy(tq)+η(t); Where Δy(t) represents the increment of the target position at time t, θ1,θ2,…,θ q are all coefficients of the integral moving average part of the CARIMA model, η(t) is the prediction error of the position prediction model, Δy(t-1), Δy(t-2)..., Δy(tq) represent the historical increments at t-1, t-2,..., tp, respectively.

6. The robot arm visual servo tracking method according to claim 1, characterized in that: According to the error between the predicted target position and the current target position of the robot arm, the motion trajectory of the robot arm is adjusted, including: Determine the trajectory correction amount based on the error between the predicted target position and the current target position of the robot arm; Based on the trajectory correction amount, the motion trajectory of the robot arm is adjusted.

7. A robot, characterized in that: include: Robot body, visual sensor, linear parameter time-varying predictive controller and incremental predictive controller; The visual sensor, the linear parameter time-varying prediction controller and the incremental prediction controller are all arranged on the robot body; the visual sensor is connected to the linear parameter time-varying prediction controller; the linear parameter time-varying prediction controller is connected to the incremental prediction controller; the linear parameter time-varying prediction controller and the incremental prediction controller are integrated to form an intelligent control unit of the robot body; The visual sensor is used to obtain the target position in real time and generate target position data; The linear parameter time-varying prediction controller is implanted with a position prediction model, and is used to generate a predicted value of the target position based on the target position data using the position prediction model, and is used to correct the predicted value of the target position according to real-time feedback of the target operation to obtain a predicted target position; The position prediction model is constructed by adopting the CARIMA model; the incremental prediction controller is used to adjust the motion trajectory of the robotic arm according to the error between the predicted target position and the current target position of the robotic arm.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot arm visual servo tracking method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot arm visual servo tracking method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the robot arm visual servo tracking method described in any one of claims 1 to 6 is implemented.

Images