Robotic arm control method, device, equipment and storage medium

Abstract

The present invention relates to a robot control method, device, equipment and storage medium. The robot control method comprises: acquiring a target image, and calculating an image Jacobian matrix according to the image feature information of the target image; calculating an image feature error according to a first coordinate of the end point of the robot in a target pixel plane and a second coordinate of the target point in the target pixel plane, wherein the target pixel plane is constructed based on the target image; determining a moving distance of the end point of the robot in the target pixel plane, and calculating a joint angle according to the moving distance, the image feature error and the image Jacobian matrix, so as to control the robot to move toward the target point based on the joint angle. The method provided in the present application effectively improves the convergence speed, thereby achieving the purpose of efficiently and accurately controlling the robot to reach the target point.

Description

Technical Field

[0001] The present invention relates to the technical field of visual servoing and robotic arm motion control, and in particular to a robotic arm control method, device, equipment and storage medium. Background Art

[0002] In the field of visual servoing, a feature deviation is typically calculated based on the expected and actual image features, and then used to guide a robotic arm to the target position. However, this method, which controls the robotic arm's movement to the target position by directly sending joint positions, has a slow convergence rate, significantly affecting the efficiency of the robotic arm's motion control. Summary of the Invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a robotic arm control method, device, equipment and storage medium, which effectively improve the convergence speed, thereby achieving the purpose of efficiently and accurately controlling the robotic arm to reach the target point.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for controlling a robotic arm, comprising:

[0005] Acquire a target image, and calculate an image Jacobian matrix based on image feature information of the target image;

[0006] Calculating an image feature error based on a first coordinate of the end point of the robot arm on a target pixel plane and a second coordinate of the target point on the target pixel plane, wherein the target pixel plane is constructed based on the target image;

[0007] Determine the movement distance of the end point of the robotic arm in the target pixel plane, and calculate the joint angle according to the movement distance, the image feature error and the image Jacobian matrix, so as to control the movement of the robotic arm toward the target point based on the joint angle.

[0008] Optionally, the target image includes a current frame image and an initial frame image.

[0009] Optionally, calculating the image feature error according to the first coordinate of the end point of the robot arm on the target pixel plane and the second coordinate of the target point on the target pixel plane includes:

[0010] Obtaining the initial coordinates of the end point of the robotic arm on the initial pixel plane constructed based on the initial frame image;

[0011] Calculating an image feature error based on the initial coordinates, a first coordinate of the end point of the robotic arm in a current pixel plane constructed based on the current frame image, and a second coordinate of a target point in the current pixel plane, wherein the target point refers to at least one pixel point of a target object to be controlled by the robotic arm in the current frame image;

[0012] The target pixel plane includes the initial pixel plane and the current pixel plane.

[0013] Optionally, calculating the image feature error according to the initial coordinates, the first coordinates of the robotic arm end point in the current pixel plane constructed based on the current frame image, and the second coordinates of the target point in the current pixel plane includes:

[0014] Calculating a first characteristic error of the initial frame image according to the initial coordinates and the second coordinate of the target point on the current pixel plane;

[0015] Calculating a second characteristic error of the current frame image according to the second coordinate and the first coordinate of the end point of the robotic arm on the current pixel plane;

[0016] The image feature error includes the first feature error and the second feature error.

[0017] Optionally, calculating the joint angle according to the movement distance, the image feature error, and the image Jacobian matrix includes:

[0018] Calculate a control coefficient matrix according to the first characteristic error and the second characteristic error;

[0019] The joint angle is calculated according to the movement distance, the control coefficient matrix and the image Jacobian matrix.

[0020] Optionally, the first characteristic error includes a first error in a first axial direction and a second error in a second axial direction, and the second characteristic error includes a third error in the first axial direction and a fourth error in the second axial direction.

[0021] Optionally, calculating a control coefficient matrix according to the first characteristic error and the second characteristic error includes:

[0022] Calculating a ratio of the third error to the first error to obtain a first ratio in the first axial direction;

[0023] Calculating a ratio of the second error to the fourth error to obtain a second ratio in the second axis direction;

[0024] A multi-order control coefficient matrix is ​​generated according to the first ratio and the second ratio through a preset function, wherein the preset function is a monotonically increasing function passing through the origin.

[0025] Optionally, calculating the joint angle according to the movement distance, the control coefficient matrix and the image Jacobian matrix includes:

[0026] Calculating the moving speed of the end point of the robotic arm in the target pixel plane according to the moving distance and the sampling interval of the target image;

[0027] The joint angle is calculated according to the robot arm control period, the moving speed, the control coefficient matrix and the image Jacobian matrix.

[0028] Optionally, after controlling the robotic arm to move toward the target point based on the joint angle, the method further includes:

[0029] Acquire a next frame of image, and calculate the image feature error of the next frame of image;

[0030] When the image feature error is greater than a preset threshold, the operation of calculating the joint angle based on the currently acquired image to control the movement of the robotic arm to the target point is repeated until the image feature error of the currently acquired image is less than or equal to the preset threshold, and then the control of the robotic arm is terminated.

[0031] In a second aspect, an embodiment of the present disclosure provides a robotic arm control device, comprising:

[0032] An acquisition unit, configured to acquire a target image and calculate an image Jacobian matrix based on image feature information of the target image;

[0033] a first calculation unit, configured to calculate an image feature error based on a first coordinate of a manipulator end point on a target pixel plane and a second coordinate of a target point on the target pixel plane, wherein the target pixel plane is constructed based on the target image;

[0034] The second calculation unit is used to determine the movement distance of the end point of the robotic arm in the target pixel plane, and calculate the joint angle according to the movement distance, the image feature error and the image Jacobian matrix, so as to control the movement of the robotic arm toward the target point based on the joint angle.

[0035] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0036] Memory;

[0037] processor; and

[0038] computer programs;

[0039] The computer program is stored in the memory and is configured to be executed by the processor to implement the robotic arm control method as described above.

[0040] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the robotic arm control method as described above when executed by a processor.

[0041] The present application provides a robotic arm control method including: acquiring a target image and calculating an image Jacobian matrix based on image feature information of the target image; calculating an image feature error based on a first coordinate of the robotic arm end point in a target pixel plane and a second coordinate of the target point in the target pixel plane, wherein the target pixel plane is constructed based on the target image; determining a moving distance of the robotic arm end point in the target pixel plane, and calculating a joint angle based on the moving distance, the image feature error and the image Jacobian matrix, so as to control the robotic arm to move toward the target point based on the joint angle. In each control period, the present application dynamically adjusts the calculated joint angle through the moving distance of the robotic arm end point in the pixel plane, the image Jacobian matrix and the image feature error, thereby effectively improving the convergence speed, thereby achieving the purpose of efficiently and accurately controlling the robotic arm to reach the target point. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0043] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic flow chart of a robotic arm control method provided in an embodiment of the present disclosure;

[0045] Figure 2 for Figure 1 FIG. 4 is a schematic diagram of a detailed process of S103 in a method for controlling a robotic arm;

[0046] Figure 3 A schematic flow chart of another robotic arm control method provided by an embodiment of the present disclosure;

[0047] Figure 4A schematic structural diagram of a robotic arm control device provided in an embodiment of the present disclosure;

[0048] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0051] In response to the above technical problems, the embodiments of the present disclosure provide a method for controlling a robotic arm. This method introduces a dynamic proportional control coefficient when calculating joint angles, dynamically adjusts the joint angles calculated in each control cycle, and thereby controls the motion behavior of the robotic arm in each control cycle. This method avoids the disadvantage of slow convergence speed of traditional visual servoing, effectively improves the control efficiency of the robotic arm, enables the robotic arm to reach the target point quickly and accurately, facilitates subsequent operations of the robotic arm, and improves the user experience. This will be described in detail through one or more of the following embodiments.

[0052] Specifically, the robotic arm control method can be executed by a terminal or a server. For example, in one application scenario, the server calculates the joint angle based on the acquired target image, and sends the joint angle to the robotic arm to control the movement of the robotic arm. The target image can be obtained by taking a picture by the terminal. Alternatively, the target image is obtained by the terminal from other devices. Alternatively, the target image is an image obtained after the terminal performs image processing on a preset image, and the preset image can be obtained by taking a picture by the terminal, or the preset image can be obtained by the terminal from other devices. Other devices are not specifically limited here. In another application scenario, the terminal calculates the joint angle based on the acquired target image, and sends the joint angle to the robotic arm to control the movement of the robotic arm. It can be understood that the robotic arm control method provided by the embodiments of the present disclosure is not limited to the several possible scenarios described above.

[0053] The following describes in detail the method using the terminal executing the self-image acquisition and execution of the robotic arm control method as an example.

[0054] Figure 1This is a flow chart of a robotic arm control method provided by an embodiment of the present disclosure, which is applied to a robotic arm motion control scenario. The terminal includes an acquisition module, a robotic arm, and a control module. The acquisition module is used to acquire a target image and upload it to the control module. The control module is used to calculate the joint angle based on the target image and send the joint angle to the robotic arm. The robotic arm is used to move based on the joint angle. The robotic arm control method specifically includes the following steps: Figure 1 Steps S101 to S103 shown:

[0055] S101 , acquiring a target image, and calculating an image Jacobian matrix based on image feature information of the target image.

[0056] It is understood that the acquisition module can be at least one camera, which can be configured on and / or around the robotic arm. The camera periodically acquires images, resulting in a series of periodically acquired image sets. The target image can be understood as at least a portion of the images in the image set. After acquiring the target image, image feature information of the target image is extracted. The specific extraction method is not described in detail here. In one possible scenario, the image feature information is acquired by the camera, which then directly transmits the image feature information to the control module, or the image feature information is acquired by the control module.

[0057] It is understandable that the image Jacobian is calculated based on the image feature information, and the image Jacobian is recorded as J I , which can be a multi-order matrix of 2*7. The specific calculation method of the image Jacobian is not limited here and can be selected according to user needs.

[0058] S102 : Calculate an image feature error based on a first coordinate of the end point of the robot arm on a target pixel plane and a second coordinate of the target point on the target pixel plane.

[0059] The target pixel plane is constructed based on the target image.

[0060] It is understandable that, based on the above S101, a target pixel plane is constructed based on the target image. The target pixel plane can be understood as a plane under the camera coordinates, and each pixel point has a corresponding two-dimensional coordinate in the target pixel plane. Alternatively, a target pixel space is constructed based on the captured image, and each pixel point has a corresponding three-dimensional coordinate in the space. The following embodiment is explained in detail using the construction of the pixel plane as an example. Subsequently, the first coordinate of the end point of the manipulator in the target pixel plane is determined. In one possible case, the camera can capture the end of the manipulator, and the target image will include the end of the manipulator. The coordinate of the pixel point corresponding to the end of the manipulator (hereinafter referred to as the manipulator end point) in the target pixel plane is the first coordinate. In another possible case, the positional relationship between the manipulator and the camera is known, and the coordinate of the end point of the manipulator in the target pixel plane can be determined based on the camera parameters of the target image. Determine the second coordinate of the target point in the target pixel plane. The second coordinate can also be understood as the target position / expected position, etc. Specifically, the target point refers to a pixel point of the target object included in the target image in the target pixel plane. The position of the pixel point is the second coordinate. One application scenario is that the camera shoots the target object to generate a target image, and controls the robotic arm to move to the target object according to the target image for subsequent operations. The target object refers to the object to be operated by the robotic arm, and the target point can be understood as the specific part when the robotic arm operates the target object.

[0061] It is understandable that the image feature error is calculated based on the first coordinate and the second coordinate, which is specifically described in detail through the following embodiments.

[0062] The target image includes a current frame image and an initial frame image.

[0063] It can be understood that the target image includes at least an initial frame image and a current frame image. The initial frame image can be understood as the image obtained by the camera for the first time, or the image collected in the first sampling period, or a customized frame of historical image, or the previous frame image. The current frame image refers to a frame image collected in the current sampling period.

[0064] Optionally, the calculating of the image feature error according to the first coordinate of the end point of the manipulator on the target pixel plane and the second coordinate of the target point on the target pixel plane is specifically implemented by the following steps:

[0065] Obtain the initial coordinates of the end point of the robotic arm in an initial pixel plane constructed based on the initial frame image; calculate the image feature error based on the initial coordinates, the first coordinates of the end point of the robotic arm in a current pixel plane constructed based on the current frame image, and the second coordinates of the target point in the current pixel plane, wherein the target point refers to at least one pixel point of the target object to be controlled by the robotic arm in the current frame image; wherein the target pixel plane includes the initial pixel plane and the current pixel plane.

[0066] It can be understood that the initial pixel plane is constructed based on the initial frame image, and the current pixel plane is constructed based on the current frame image. The specific construction method is not limited, wherein the target pixel plane includes the initial pixel plane and the current pixel plane. Determine the initial coordinates of the end point of the robotic arm on the initial pixel plane, the initial coordinates are marked as P0[u0,v0]. The initial coordinates of a certain point P on the initial pixel plane can also be determined. This certain point P exists in all subsequently acquired images, that is, each acquired image includes a fixed point P. The end point of the robotic arm can also be understood as point P, wherein u and v refer to the first axial coordinate and the second axial coordinate of the end point of the robotic arm in the camera coordinate system. Determine the first coordinate of the end point of the robotic arm on the current pixel plane and the second coordinate of the target point on the current pixel plane. The first coordinate can be understood as the real-time coordinate of the end point of the robotic arm on the pixel plane at time t, recorded as P t [u,v], the second coordinate can be understood as the real-time coordinate of the target point on the pixel plane at time t, denoted as T[u t ,v t ]. Subsequently, the image feature error / image feature deviation is calculated according to the initial coordinates, the first coordinates, and the second coordinates, which can be specifically described in detail through the following embodiments.

[0067] Optionally, the above-mentioned calculation of the image feature error based on the initial coordinates, the first coordinates of the end point of the robotic arm in the current pixel plane constructed based on the current frame image, and the second coordinates of the target point in the current pixel plane can be specifically implemented by the following steps:

[0068] The first characteristic error of the initial frame image is calculated based on the initial coordinates and the second coordinates of the target point in the current pixel plane; the second characteristic error of the current frame image is calculated based on the second coordinates and the first coordinates of the end point of the robotic arm in the current pixel plane; wherein the image characteristic error includes the first characteristic error and the second characteristic error.

[0069] The initial coordinates include a first value in the first axis and a second value in the second axis, the first coordinates include a third value in the first axis and a fourth value in the second axis, and the second coordinates include a fifth value in the first axis and a sixth value in the second axis.

[0070] It is understandable that the difference between the fifth value and the first value is calculated to obtain a first difference along the first axis, and the difference between the sixth value and the second value is calculated to obtain a second difference along the second axis. The first characteristic error of the initial frame image is obtained based on the first difference and the second difference. The first characteristic error can be understood as the initial image characteristic error. The difference between the fifth value and the third value is calculated to obtain a third difference along the first axis, and the difference between the sixth value and the fourth value is calculated to obtain a fourth difference along the second axis. The second characteristic error of the current frame image is obtained based on the third difference and the fourth difference. The second characteristic error can be understood as the real-time image characteristic error. It is understandable that the order of calculating the first characteristic error and the second characteristic error is not limited and can be determined according to user needs. As the image is acquired in real time, the initial coordinates remain unchanged, while the first coordinates and the second coordinates change with the acquired image.

[0071] It can be understood that the calculation formulas for the first characteristic error and the second characteristic error are as shown in the following formulas (1) and (2):

[0072] Δuv0=[u t -u0,v t -v0]=[u e0 ,v e0 ]Formula (1)

[0073] Δuv=[u t -u,v t -v]=[u e ,v e ]Formula (2)

[0074] Where Δuv0 is the first characteristic error, Δuv is the second characteristic error, P0[u0,v0] is the initial coordinate, P t [u,v] is the first coordinate, T[u t ,v t ] is the second coordinate, u is the coordinate value along the first axis, and v is the coordinate value along the second axis.

[0075] S103. Determine the movement distance of the end point of the robotic arm in the target pixel plane, and calculate the joint angle according to the movement distance, the image feature error and the image Jacobian matrix, so as to control the robotic arm to move toward the target point based on the joint angle.

[0076] It is understandable that, based on the above S101 and S102, the movement distance of the end point of the manipulator on the target pixel plane is determined. The movement distance can be specifically calculated based on the initial coordinates and the first coordinates in the same camera coordinate system, that is, the distance moved by the end point of the manipulator on the initial frame image and the current frame image in the camera coordinate system. Subsequently, the joint angle is calculated based on the movement distance, the image feature error (the first feature error and the second feature error), and the image Jacobian matrix, and a control instruction is generated based on the joint angle. The control instruction is used to control the manipulator to move to various joint positions / joint angles, and then the manipulator moves to the target point through multiple control movements.

[0077] Optionally, after controlling the robotic arm to move toward the target point based on the joint angle, the method further includes:

[0078] Acquire the next frame image and calculate the image feature error of the next frame image; when the image feature error is greater than a preset threshold, repeatedly calculate the joint angle based on the currently acquired image to control the movement of the robotic arm to the target point until the image feature error of the currently acquired image is less than or equal to the preset threshold, then end the control of the robotic arm.

[0079] It is understandable that after controlling the movement of the robotic arm based on the joint angle, the next frame of image captured by the camera is obtained, and the image feature error is calculated based on the image feature information of the next frame of image to obtain an updated image feature error. For the specific calculation steps, see the above embodiment. It is determined whether the updated image feature error is greater than a preset threshold, wherein the preset threshold is used to characterize the relative distance between the end point of the robotic arm and the target point. Specifically, it can be determined by the user according to the control accuracy requirement. For example, the preset threshold is 0. If the updated image feature error is greater than the preset threshold, it means that the robotic arm has not yet moved to the target point, that is, it has not converged, and it is necessary to continue to capture images to control the movement of the robotic arm. Specifically, the above-mentioned operations of capturing images, calculating image Jacobians, image feature errors, and joint angles are repeated until the latest calculated image feature error is less than or equal to the preset threshold, indicating that the robotic arm has moved to the target position, that is, the robotic arm meets the convergence condition, then the motion control of the robotic arm is terminated, and the robotic arm can be subsequently controlled to perform other operations at the target position.

[0080] A robotic arm control method provided by an embodiment of the present disclosure calculates the image feature error through the coordinate change of the robotic arm end point between the initial image and the actual image and the coordinate difference between the robotic arm end point and the target point in the actual image, and then dynamically adjusts the generated joint angle according to the image feature error and the image Jacobian matrix, that is, dynamically controls the movement behavior / movement amplitude / movement status of the robotic arm, effectively improves the convergence speed of visual servoing, improves the control efficiency of the robotic arm, enables the robotic arm to move to the target point more accurately and quickly, facilitates the subsequent execution of related operations at the target point, and effectively improves the user experience.

[0081] Currently, to address the shortcomings of traditional visual servoing's slow convergence speed, existing technologies have introduced proportional control methods, namely, adding a control coefficient to the control law. However, the control coefficient is usually a constant, that is, the control coefficient is fixed. However, fixed coefficients do not effectively solve the problem of control efficiency and are difficult to achieve ideal control effects. Therefore, to address the above technical issues, this embodiment proposes the concept of a dynamic proportional control coefficient, that is, the control coefficient is adjustable and can be adjusted in real time based on the collected image. This dynamic proportional control method not only improves control efficiency, but also greatly increases the smoothness of the robot arm's motion control. Specific examples are described in detail below.

[0082] Based on the above embodiments, Figure 2 for Figure 1 The detailed flow diagram of S103 in a robot arm control method shown in FIG. 1 is shown. Optionally, the calculation of the joint angle according to the moving distance, the image feature error and the image Jacobian matrix specifically includes the following steps: Figure 2 Steps S201 to S202 shown:

[0083] S201. Calculate a control coefficient matrix according to the first characteristic error and the second characteristic error.

[0084] It can be understood that the first characteristic error is calculated based on the initial coordinates of the end point of the robotic arm in the initial frame image and the second coordinates of the target point in the current frame image, the second characteristic error is calculated based on the first coordinates of the end point of the robotic arm in the current frame image and the second coordinates of the target point in the current frame image, and the control coefficient matrix is ​​calculated based on the first characteristic error and the second characteristic error. The control coefficient matrix is ​​dynamically determined with full consideration of the situation of the robotic arm relative to the target point, that is, different control coefficients are calculated based on the actual situation of the robotic arm and the target point in the acquired image, so as to facilitate more efficient and accurate calculation of the joint angle.

[0085] The first characteristic error includes a first error in a first axial direction and a second error in a second axial direction, and the second characteristic error includes a third error in the first axial direction and a fourth error in the second axial direction.

[0086] It can be understood that, based on the above example, the first characteristic error (Δuv0) includes the first error (u e0 ) and the second error (v in the second axial direction (v) e0 ), the second characteristic error (Δuv) includes the third error (u e ) and the fourth error (v e ).

[0087] Optionally, the calculation of the control coefficient matrix according to the first characteristic error and the second characteristic error in S201 can be specifically implemented by the following steps:

[0088] Calculate the ratio of the third error to the first error to obtain a first ratio on the first axis; calculate the ratio of the second error to the fourth error to obtain a second ratio on the second axis; generate a multi-order control coefficient matrix according to the first ratio and the second ratio through a preset function, wherein the preset function refers to a monotonically increasing function that passes through the origin.

[0089] It can be understood that the ratio of the third error to the first error is calculated to obtain a first ratio on the first axis (u), and the ratio of the fourth error to the second error is calculated to obtain a second ratio on the second axis (v), wherein the first ratio and the second ratio are within a preset range, which can be [0, 1]. Subsequently, the first ratio is used as the input of a preset function to obtain a first coefficient, and the second ratio is used as the input of the preset function to obtain a second coefficient. That is, the first ratio and the second ratio are used as the input of the preset function respectively to obtain the coefficient corresponding to each ratio, wherein the preset function refers to a monotonically increasing function passing through the origin. The preset function can be a hyperbolic function, for example, a hyperbolic tangent function. The hyperbolic tangent function is generally written as tanh in mathematical language, and can also be abbreviated as th. By introducing the tanh function for dynamic proportional control, the calculation is relatively simple and convenient. Subsequently, the first coefficient and the second coefficient are combined to obtain a multi-order control coefficient matrix. For example, the control coefficient matrix can be a 2*2 matrix, and the first coefficient and the second coefficient are diagonally distributed. It is understandable that the image feature error between the target point and the end point of the manipulator in three-dimensional space can also be calculated, thereby increasing the ratio along the third axis and expanding the order of the control coefficient matrix. The specific order of the control coefficient matrix is ​​not limited and can be determined according to user needs. The specific formula of the control coefficient matrix is ​​shown in the following formula (3):

[0090]

[0091] Wherein, K is the control coefficient matrix, ku is the first coefficient, kv is the second coefficient, tanh is the preset function, ux is the first ratio, and vx is the second ratio.

[0092] It is understandable that the farther point P is from the target point, the larger K is. When point P is at the initial point, K is at its maximum value of 1; when point P reaches the target point, K is at its minimum value, possibly zero.

[0093] S202. Calculate the joint angle according to the moving distance, the control coefficient matrix, and the image Jacobian matrix.

[0094] It can be understood that, based on the above S201, with each movement of the robotic arm, the moving distance between the end point of the robotic arm and the target point in the image also changes accordingly, and the control coefficient matrix calculated in each cycle also changes accordingly. Therefore, the joint angle is dynamically calculated based on the moving distance, the control coefficient matrix and the image Jacobian to dynamically control the movement of the robotic arm.

[0095] Optionally, the calculation of the joint angle according to the movement distance, the control coefficient matrix and the image Jacobian matrix in S202 can be specifically implemented by the following steps:

[0096] The moving speed of the end point of the robotic arm in the target pixel plane is calculated according to the moving distance and the sampling interval of the target image; the joint angle is calculated according to the robotic arm control period, the moving speed, the control coefficient matrix and the image Jacobian matrix.

[0097] It is understandable that the moving speed of the end point of the manipulator on the pixel plane is calculated based on the moving distance and the sampling time / sampling period / sampling interval of the target image, or the moving speed of a certain image feature point on the pixel plane can also be calculated. The joint angle is then calculated based on the manipulator control period, moving speed, control coefficient matrix and image Jacobian matrix, where the manipulator control period and the camera sampling period can be the same or different, and the control period can be in ms or s. The calculation formulas for moving speed and joint angle are shown in formulas (4) to (6):

[0098]

[0099]

[0100]

[0101] Where, is the moving speed, J I is the image Jacobian matrix, is the joint velocity of the robot arm, J -1is the image Jacobian inverse matrix, K is the control coefficient matrix, t is the control period, and q is the joint angle.

[0102] In a possible embodiment, the image Jacobian matrix is ​​a 2*7 matrix, the control coefficient matrix is ​​a 2*2 matrix, the joint velocity is a 7*1 matrix, and the joint angle is also a 7*1 matrix.

[0103] A robotic arm control method provided by an embodiment of the present disclosure introduces a dynamic proportional control method, that is, the control coefficient matrix is ​​automatically adjusted according to the collected image, and a function is introduced for dynamic proportional control, which is simple and convenient to calculate. The joint angle is adjusted by the dynamic control coefficient calculated in each control cycle, which not only improves the convergence speed and the control efficiency of the robotic arm, but also greatly increases the smoothness of the robotic arm motion control, facilitates the immediate subsequent operations after the robotic arm reaches the target point, controls the robotic arm to reach the target point efficiently and accurately, and improves the user experience to a certain extent.

[0104] Based on the above embodiments, Figure 3 A flow chart of another robot arm control method provided by an embodiment of the present disclosure, specifically including the following steps: Figure 3 The following steps are shown:

[0105] (1) Obtain image features: collect images and obtain image feature information; (2) Calculate image Jacobian: calculate image Jacobian based on image feature information; (3) Calculate image feature deviation: calculate image feature error based on the initial coordinates of a point P in the pixel plane, the real-time coordinates of a point P in the pixel plane, and the real-time coordinates of the target point in the pixel plane; (4) Calculate proportional control coefficient: adjust the proportional control coefficient based on the image feature deviation; (5) Calculate joint speed: calculate the joint speed of the manipulator in each control cycle; (6) Send joint position: calculate the moving speed of the image feature point in the pixel plane based on the joint speed and image Jacobian, and calculate the joint angle based on the dynamic proportional control coefficient, image Jacobian, moving speed and control cycle; (7) Determine whether the image feature deviation is 0: obtain the feature information of the next frame image and recalculate the image feature deviation, and determine whether the recalculated image feature deviation is less than 0; (8) If the image feature deviation is less than 0, end the process; if the image feature deviation is not less than 0, return to step (1).

[0106] It is understandable that the specific implementation steps of the above process can be found in the above embodiment and will not be described in detail here.

[0107] Figure 4 The schematic diagram of the structure of a robot arm control device provided by the embodiment of the present disclosure. The robot arm control device provided by the embodiment of the present disclosure can execute the processing flow provided by the embodiment of the robot arm control method, such as Figure 4As shown, the apparatus 400 includes a collection unit 401, a first calculation unit 402, and a second calculation unit 403, wherein:

[0108] An acquisition unit 401 is configured to acquire a target image and calculate an image Jacobian matrix based on image feature information of the target image.

[0109] A first calculation unit 402 is configured to calculate an image feature error based on a first coordinate of a manipulator end point on a target pixel plane and a second coordinate of a target point on the target pixel plane, wherein the target pixel plane is constructed based on the target image;

[0110] The second calculation unit 403 is used to determine the movement distance of the end point of the robotic arm in the target pixel plane, and calculate the joint angle according to the movement distance, the image feature error and the image Jacobian matrix, so as to control the movement of the robotic arm toward the target point based on the joint angle.

[0111] Optionally, the target image in the device 400 includes a current frame image and an initial frame image.

[0112] Optionally, the first calculation unit 402 is configured to:

[0113] Obtaining the initial coordinates of the end point of the robotic arm on the initial pixel plane constructed based on the initial frame image;

[0114] Calculating an image feature error based on the initial coordinates, a first coordinate of the end point of the robotic arm in a current pixel plane constructed based on the current frame image, and a second coordinate of a target point in the current pixel plane, wherein the target point refers to at least one pixel point of a target object to be controlled by the robotic arm in the current frame image;

[0115] The target pixel plane includes the initial pixel plane and the current pixel plane.

[0116] Optionally, the first calculation unit 402 is configured to:

[0117] Calculating a first characteristic error of the initial frame image according to the initial coordinates and the second coordinate of the target point on the current pixel plane;

[0118] Calculating a second characteristic error of the current frame image according to the second coordinate and the first coordinate of the end point of the robotic arm on the current pixel plane;

[0119] The image feature error includes the first feature error and the second feature error.

[0120] Optionally, the second calculation unit 403 is configured to:

[0121] Calculate a control coefficient matrix according to the first characteristic error and the second characteristic error;

[0122] The joint angle is calculated according to the movement distance, the control coefficient matrix and the image Jacobian matrix.

[0123] Optionally, in the device 400 , the first characteristic error includes a first error in a first axial direction and a second error in a second axial direction, and the second characteristic error includes a third error in the first axial direction and a fourth error in the second axial direction.

[0124] Optionally, the second calculation unit 403 is configured to:

[0125] Calculating a ratio of the third error to the first error to obtain a first ratio in the first axial direction;

[0126] Calculating a ratio of the second error to the fourth error to obtain a second ratio in the second axis direction;

[0127] A multi-order control coefficient matrix is ​​generated according to the first ratio and the second ratio through a preset function, wherein the preset function is a monotonically increasing function passing through the origin.

[0128] Optionally, the second calculation unit 403 is configured to:

[0129] Calculating the moving speed of the end point of the robotic arm in the target pixel plane according to the moving distance and the sampling interval of the target image;

[0130] The joint angle is calculated according to the robot arm control period, the moving speed, the control coefficient matrix and the image Jacobian matrix.

[0131] Optionally, the apparatus 400 is further configured to:

[0132] Acquire a next frame of image, and calculate the image feature error of the next frame of image;

[0133] When the image feature error is greater than a preset threshold, the operation of calculating the joint angle based on the currently acquired image to control the movement of the robotic arm to the target point is repeated until the image feature error of the currently acquired image is less than or equal to the preset threshold, and then the control of the robotic arm is terminated.

[0134] Figure 4 The robotic arm control device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0135] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the embodiment of the present disclosure. Figure 5, which shows a schematic structural diagram of an electronic device 500 suitable for implementing the embodiments of the present disclosure. The electronic device 500 in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, and the like, as well as fixed terminals such as digital TVs, desktop computers, smart home devices, and the like. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0136] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes to implement the robot arm control method of the embodiment described in the present disclosure according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage device 508 to the random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0137] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0138] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart, thereby implementing the robot arm control method described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0139] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0140] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0141] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0142] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0146] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0147] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or gateway that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or gateway. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or gateway that includes the elements.

[0149] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A robot arm control method, characterized in that: include: Acquire a target image, and calculate an image Jacobian matrix according to image feature information of the target image, wherein the target image includes a current frame image and an initial frame image; Calculate an image feature error according to a first coordinate of the end point of the robot arm in a target pixel plane and a second coordinate of the target point in the target pixel plane, wherein the target pixel plane is constructed based on the target image, and the image feature error is calculated by a change in the coordinate of the end point of the robot arm between the initial frame image and the current frame image and a coordinate difference between the end point of the robot arm and the target point in the current frame image; Determine the movement distance of the end point of the robotic arm in the target pixel plane, and calculate the joint angle according to the movement distance, the image feature error and the image Jacobian matrix, so as to control the robotic arm to move toward the target point based on the joint angle; Wherein, calculating the joint angle according to the moving distance, the image feature error and the image Jacobian matrix includes: Calculating a control coefficient matrix according to the image feature error includes: calculating a first feature error according to the initial coordinates of the end point of the robot arm in the initial frame image and the second coordinates of the target point in the current frame image; calculating a second feature error according to the first coordinates and the second coordinates of the end point of the robot arm in the current frame image, and calculating a control coefficient matrix according to the first feature error and the second feature error, wherein the control coefficient matrix is ​​dynamically determined based on the situation of the end point of the robot arm relative to the target point; The joint angle is calculated according to the moving distance, the control coefficient matrix and the image Jacobian matrix.

2. The method according to claim 1, characterized in that The calculating the image feature error according to the first coordinate of the end point of the robot arm on the target pixel plane and the second coordinate of the target point on the target pixel plane comprises: Obtaining the initial coordinates of the end point of the robotic arm on an initial pixel plane constructed based on the initial frame image; Calculate the image feature error according to the initial coordinates, the first coordinates of the end point of the robotic arm in the current pixel plane constructed based on the current frame image, and the second coordinates of the target point in the current pixel plane, wherein the target point refers to at least one pixel point of the target object to be controlled by the robotic arm in the current frame image; The target pixel plane includes the initial pixel plane and the current pixel plane.

3. The method according to claim 2, characterized in that The calculating the image feature error according to the initial coordinates, the first coordinates of the end point of the robot arm in the current pixel plane constructed based on the current frame image, and the second coordinates of the target point in the current pixel plane includes: Calculate a first characteristic error of the initial frame image according to the initial coordinates and the second coordinate of the target point on the current pixel plane; Calculate a second characteristic error of the current frame image according to the second coordinate and the first coordinate of the end point of the robotic arm on the current pixel plane; The image feature error includes the first feature error and the second feature error.

4. The method according to claim 1, characterized in that: The first characteristic error includes a first error in a first axial direction and a second error in a second axial direction, the second characteristic error includes a third error in the first axial direction and a fourth error in the second axial direction, and the control coefficient matrix is ​​calculated according to the first characteristic error and the second characteristic error, including: Calculating a ratio of the third error to the first error to obtain a first ratio in the first axial direction; Calculating a ratio of the second error to the fourth error to obtain a second ratio in the second axial direction; A multi-order control coefficient matrix is ​​generated according to the first ratio and the second ratio through a preset function, wherein the preset function is a monotonically increasing function passing through the origin.

5. The method according to claim 1, characterized in that The calculating of the joint angle according to the moving distance, the control coefficient matrix and the image Jacobian matrix comprises: Calculating the moving speed of the end point of the robot arm in the target pixel plane according to the moving distance and the sampling interval of the target image; The joint angle is calculated according to the robot arm control period, the moving speed, the control coefficient matrix and the image Jacobian matrix.

6. The method according to claim 1, characterized in that After controlling the robot arm to move toward the target point based on the joint angle, the method further includes: Acquire a next frame of image, and calculate the image feature error of the next frame of image; When the image feature error is greater than a preset threshold, the operation of calculating the joint angle based on the currently acquired image to control the movement of the robotic arm to the target point is repeated until the image feature error of the currently acquired image is less than or equal to the preset threshold, and then the control of the robotic arm is terminated.

7. A robot arm control device, characterized in that: include: An acquisition unit, used for acquiring a target image and calculating an image Jacobian matrix according to image feature information of the target image, wherein the target image includes a current frame image and an initial frame image; A first calculation unit is used to calculate an image feature error according to a first coordinate of a manipulator end point in a target pixel plane and a second coordinate of a target point in the target pixel plane, wherein the target pixel plane is constructed based on the target image, and the image feature error is calculated by a coordinate change of the manipulator end point between the initial frame image and the current frame image and a coordinate difference between the manipulator end point and the target point in the current frame image; a second calculation unit, used to determine a moving distance of the end point of the robot arm in the target pixel plane, and calculate a joint angle according to the moving distance, the image feature error and the image Jacobian matrix, so as to control the robot arm to move toward the target point based on the joint angle; Wherein, the second computing unit is used for: A control coefficient matrix is ​​calculated according to the image feature error, including: obtaining a first feature error according to the initial coordinates of the end point of the robot arm in the initial frame image and the second coordinates of the target point in the current frame image; obtaining a second feature error according to the first coordinates and the second coordinates of the end point of the robot arm in the current frame image, and calculating a control coefficient matrix according to the first feature error and the second feature error, wherein the control coefficient matrix is ​​dynamically determined based on the situation of the end point of the robot arm relative to the target point; and calculating a joint angle according to the moving distance, the control coefficient matrix and the image Jacobian matrix.

8. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; Wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the robot arm control method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the robot arm control method according to any one of claims 1 to 6 are implemented.

Images