A robot control method, device, computer equipment and storage medium

Abstract

The embodiment of the present invention relates to a robot control method, device, computer equipment and storage medium, including: obtaining multiple expected images of a second part through multiple cameras; extracting multiple line features and point features from each expected image; determining a first correlation relationship between a first parameter change of multiple line features in each expected image and a posture change of a robot end coordinate system; determining a second correlation relationship between a second parameter change of multiple point features in each expected image and a position change of a robot end coordinate system; determining a rotation control rate of the robot according to the first correlation set, and determining a translation control rate of the robot according to the second correlation set; controlling the movement of the robot according to the rotation control rate and the translation control rate, so that the first part is in a set position. The robot can be guided by multiple cameras to operate the first part and the second part to align, reduce part alignment errors, expand the robot's operating space, and improve part alignment efficiency and accuracy.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of robot control, and in particular, to a robot control method, device, computer equipment, and storage medium. Background Art

[0002] When a robot operates a part for alignment, it is usually necessary to capture the image of the part through a monocular camera or a binocular camera, and guide the robot to operate the part for alignment based on the camera.

[0003] Due to the limitation of the field of view, the monocular camera cannot accurately obtain the parameters of point features and line features in a large range. Due to the limitation of the number of binocular cameras, there are errors in the part alignment calculation. Therefore, in the existing part alignment method, the robot has a small operating space, the part alignment error is large, and the calculation process is easily limited by the part morphology. Summary of the invention

[0004] In view of this, in order to solve the technical problem of large errors in the alignment of parts controlled by the above-mentioned robot, an embodiment of the present invention provides a robot control method, device, computer equipment and storage medium.

[0005] In a first aspect, an embodiment of the present invention provides a robot control method, the method comprising:

[0006] Acquire multiple desired images of a second part by using multiple cameras, wherein the multiple cameras are arranged on a mechanical arm of the robot, the second part corresponds to the first part, and the first part is arranged at the end of the mechanical arm;

[0007] Extracting a plurality of line features and a plurality of point features from each of the desired images;

[0008] Determine a first association relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images to obtain a first association relationship set;

[0009] Determine a second association relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images to obtain a second association relationship set;

[0010] Determine a rotation control rate of the robotic arm according to the first association relationship set, and determine a translation control rate of the robotic arm according to the second association relationship set;

[0011] The movement of the robot arm is controlled according to the rotation control rate and the translation control rate so that the first part is in a set position.

[0012] In a possible implementation, the line features are extracted in the following manner:

[0013] Extracting a center point and a target straight line in the desired image;

[0014] Obtaining the coordinates of the point on the target straight line that is closest to the origin;

[0015] Determine, according to the intrinsic parameter matrix of the camera, a first coordinate of the coordinate of the point closest to the origin on the focal length normalized imaging plane;

[0016] Determining a second coordinate of the center point on a focal length normalized imaging plane;

[0017] Determining the line feature of the target straight line on a focal length normalized imaging plane according to the first coordinate and the second coordinate;

[0018] The point features are extracted in the following way:

[0019] Obtaining coordinates of a target point in the desired image;

[0020] The third coordinate of the coordinate of the target point on the focal length normalized imaging plane is determined according to the intrinsic parameter matrix of the camera as the point feature.

[0021] In a possible implementation, determining a first correlation relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images includes:

[0022] Determine the line feature interaction matrix of each of the desired images before the robot arm moves, and obtain a set of line feature interaction matrices;

[0023] Determine the first parameter change amount of the plurality of line features in each of the expected images after the movement of the robotic arm, obtain a first parameter change amount set, and determine the posture change amount of the robotic arm end coordinate system;

[0024] Determine the attitude transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain an attitude transformation matrix set;

[0025] Determine a target line feature interaction matrix from the line feature interaction matrix set according to the target expected image, determine a set of target first parameter variation from the first parameter variation set, and determine a target posture transformation matrix from the posture transformation matrix set;

[0026] The first association relationship is determined according to the target line feature interaction matrix, the set of target first parameter changes, the target posture transformation matrix and the posture change.

[0027] In a possible implementation, the determining of a second correlation relationship between a second parameter change amount of the plurality of point features and a position change amount of the robot end coordinate system in each of the expected images includes:

[0028] Determine a point feature interaction matrix of each of the desired images before the robot arm moves, and obtain a set of point feature interaction matrices;

[0029] Determine the second parameter change amount of the plurality of point features in each of the expected images after the movement of the robotic arm, obtain a second parameter change amount set, and determine the position change amount of the robotic arm end coordinate system;

[0030] Determine the position transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain a set of position transformation matrices;

[0031] Determine a target point feature interaction matrix from the point feature interaction matrix set according to the target expected image, determine a set of target second parameter variation from the second parameter variation set, and determine a target position transformation matrix from the position transformation matrix set;

[0032] The second association relationship is determined according to the target point feature interaction matrix, the set of target second parameter changes, the target position transformation matrix and the position change.

[0033] In a possible implementation, the posture transformation matrix is ​​determined in the following manner:

[0034] Determine a history line feature interaction matrix of the desired image when the robotic arm is in an initial state;

[0035] Controlling the robotic arm to complete multiple rotational movements;

[0036] Determine the historical rotation change amount corresponding to each rotational movement, and obtain a set of historical rotation change amounts;

[0037] Determine the historical first parameter change amount corresponding to each rotational movement, and obtain a historical first parameter change amount set;

[0038] The posture transformation matrix is ​​determined according to the historical rotation variation set, the historical first parameter variation set and the historical line feature interaction matrix.

[0039] In a possible implementation, the position transformation matrix is ​​determined in the following manner:

[0040] Determine a historical point feature interaction matrix of the desired image when the robotic arm is in an initial state, and obtain a historical point feature interaction matrix set;

[0041] Controlling the robotic arm to complete multiple translation movements;

[0042] Determine the historical position change amount corresponding to each translation movement, and obtain a set of historical position change amounts;

[0043] Determine the historical second parameter change amount corresponding to each translation movement, and obtain a historical second parameter change amount set;

[0044] The position transformation matrix is ​​determined according to the historical position change amount set, the historical second parameter change amount set and the historical point feature interaction matrix.

[0045] In a possible implementation manner, controlling the movement of the robot arm according to the rotation control rate and the translation control rate so that the first part is in a set position includes:

[0046] The rotation control rate and the translation control rate are adjusted to control the movement of the robot arm. When it is detected that the absolute values ​​of the position change and the posture change of the robot arm end coordinate system are less than a set threshold, it is determined that the first part is in a set position.

[0047] In a second aspect, an embodiment of the present invention provides a robot control device, comprising:

[0048] an acquisition module, configured to acquire a plurality of desired images of a second part through a plurality of cameras, wherein the plurality of cameras are disposed on a mechanical arm of the robot, the second part corresponds to the first part, and the first part is disposed at the end of the mechanical arm;

[0049] A processing module, used for extracting a plurality of line features and a plurality of point features from each of the desired images;

[0050] The processing module is further used to determine a first association relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images, to obtain a first association relationship set;

[0051] The processing module is further used to determine a second association relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images, to obtain a second association relationship set;

[0052] The processing module is further used to determine a rotation control rate of the mechanical arm according to the first association relationship set, and determine a translation control rate of the mechanical arm according to the second association relationship set;

[0053] A control module is used to control the movement of the robot arm according to the rotation control rate and the translation control rate so that the first part is in a set position.

[0054] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a processor and a memory, wherein the processor is used to execute a robot control program stored in the memory to implement the robot control method described in any one of the first aspects above.

[0055] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the robot control method described in any one of the first aspects above.

[0056] The robot control scheme provided by the embodiment of the present invention obtains multiple expected images of the second part through multiple cameras; extracts multiple line features and multiple point features from each of the expected images; determines the first association relationship between the first parameter change amount of the multiple line features and the posture change amount of the robot end coordinate system in each of the expected images to obtain a first association relationship set; determines the second association relationship between the second parameter change amount of the multiple point features and the position change amount of the robot end coordinate system in each of the expected images to obtain a second association relationship set; determines the rotation control rate of the robot arm according to the first association relationship set, and determines the translation control rate of the robot arm according to the second association relationship set; controls the movement of the robot arm according to the rotation control rate and the translation control rate to make the first part in a set position. In order to realize the alignment and assembly of the first part and the second part by the robot under the guidance of multiple cameras, the images are taken by multiple cameras, and the position of the parts is adjusted in combination with the construction of the point-line feature interaction matrix in the image, which can reduce the error of part alignment, expand the robot's operable space, and improve the efficiency and accuracy of part alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of an application scenario of a robot control method provided by an embodiment of the present invention;

[0058] Figure 2 A schematic flow chart of a robot control method provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of the structure of a robot control device provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0063] The following first describes an exemplary application scenario of the robot control method provided by an embodiment of the present invention:

[0064] See also Figure 1 , is a schematic diagram of an application scenario of a robot control method provided by an embodiment of the present invention, such as Figure 1 The application scenario shown includes: a robot arm, part 1, part 2, and cameras 1 to K (K ≥ 2);

[0065] The robotic arm is provided with multiple cameras, which are placed at part 2 to capture images of part 2 at different angles. Part 1 is provided at the end of the robotic arm, and part 2 is installed in other places. Part 1 can be connected with part 2. The movement of the robotic arm drives part 1 to move at the same time. During the movement of the robotic arm, only the position change and posture change of the coordinate system of the end of the robotic arm are changed, which is used to adjust the posture and position of part 1 so that part 1 can be assembled with part 2, that is, to control the alignment of part 1 and the insertion of part 2.

[0066] Figure 2 A schematic diagram of a flow chart of a robot control method provided by an embodiment of the present invention, such as Figure 2 As shown, the method specifically includes:

[0067] S11, acquiring multiple desired images of the second part through multiple cameras;

[0068] The robot control method provided in this embodiment is applied to a robot control device, which may include: a computer, a host computer, etc. Specifically, the image of the second part is acquired through the robot's multiple cameras, and the movement of the robot's mechanical arm is controlled based on the point features and line features in the image to align the first part with the second part.

[0069] In this embodiment, the second part is Figure 1For part 2 shown in the figure, the user first manually controls the movement of the robotic arm and performs a pre-alignment action on part 1 and part 2 so that each camera can capture a complete image of the second part as the desired image, thereby obtaining multiple desired images captured by multiple cameras.

[0070] S12, extracting a plurality of line features and a plurality of point features from each of the desired images;

[0071] In this embodiment, the line features are determined in the following manner: first, the calibrated center point in each desired image is extracted, and the image coordinates of the center point are determined to be (u0, v0);

[0072] Secondly, extract multiple straight lines from each desired image, obtain the i-th straight line as the target straight line, determine the image coordinates (upi, vpi) of the point on the target straight line closest to the image coordinate origin; determine the coordinates (x1pi, y1pi) of (upi, vpi) on the focal length normalized imaging plane according to the camera's intrinsic parameter matrix as the first coordinate:

[0073]

[0074] In the above formula, (x1pi, y1pi) is the coordinate on the focal length normalized imaging plane corresponding to the image coordinate closest to the image coordinate origin on the i-th line, (u0, v0) is the image coordinate of the center point, and kx and ky are the magnification coefficients from the imaging plane to the image plane.

[0075] Determine the coordinates (x1p0, y1p0) of the center point (u0, v0) on the focus normalized imaging plane as the second coordinate, and determine the line features of the target straight line on the focus normalized imaging plane according to the first coordinate and the second coordinate, including: taking (x1p0, y1p0) as the coordinate origin on the focus normalized imaging plane, and calculating the line features of each straight line, and the calculation method includes:

[0076]

[0077]

[0078] Thus, the line feature (ρ) of the i-th line in the focal length normalized imaging plane is obtained. i ,θ i ), determine the line features of each straight line in each expected image respectively, and obtain multiple line features.

[0079] Point features are determined in the following way: first, multiple feature intersection points are extracted from each desired image, the i-th intersection point is set as the target point, the image coordinates of the target point are (ui, vi), and the coordinates of the target point in the focal length normalized imaging plane are determined according to the camera's intrinsic parameter matrix:

[0080]

[0081] Through the above formula, the coordinates (x1ci, y1ci) of the i-th point on the focal length normalized imaging plane can be obtained as the third coordinate, and the third coordinate is used as the point feature.

[0082] S13, determining a first association relationship between first parameter changes of the plurality of line features and rotation changes of the robot end coordinate system in each of the expected images, and obtaining a first association relationship set;

[0083] In this embodiment, the line feature interaction matrix of each desired image before the robot arm moves is first determined, and the determination method includes:

[0084] Extract the line features of the jth straight line in the desired image of the kth camera ( k ρ 0j , k θ 0j );Let L Cki is the line feature interaction matrix before the i-th motion in the expected image corresponding to the k-th camera, L Cki For the i-th posture control of the first part at the end of the robot arm, i∈(0,m),j∈(0,n),k∈(0,K),m,n,K≥2,L Cki Calculated by:

[0085]

[0086] in, k ρ ij and k θ ij Represents the parameters of the line features of the jth straight line before the i-th motion in the k-th camera image, k θ ij As the first parameter, the line feature interaction matrix of each expected image is determined respectively to obtain a set of line feature interaction matrices.

[0087] Secondly, determine a set of first parameter changes of multiple line features in each expected image after the robot arm moves for the i-th time A plurality of groups of first parameter variation amounts constitute a first parameter variation amount set.

[0088] And determine the posture change of the robot end coordinate system Among them, Δ k θ ij = k θ ij - k θ 0j , Δθ xi ,Δθ yi,Δθ zi is the i-th rotational motion of the camera.

[0089] Furthermore, the attitude transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm is determined. Let K CkR is the attitude transformation matrix between the coordinate system of the kth camera and the coordinate system of the end of the robot arm. The robot arm is manually controlled to complete l rotations, l>10, and the historical first parameter change of each rotation is recorded to obtain the historical first parameter change set. The historical attitude change of each rotation is recorded to obtain the historical rotation change set, and the historical line feature interaction matrix L of the robot arm at the initial position is determined. Ck0 - The posture transformation matrix K is determined based on the historical rotation variation set, the historical first parameter variation set and the historical line feature interaction matrix CkR :

[0090]

[0091] Among them, L Ck0 + =(L Ck0 T L Ck0 ) -1 L Ck0 T , T R + =(T R T T R ) -1 T R T .

[0092] According to the target line feature interaction matrix corresponding to the target expected image, a set of target first parameter changes, the target posture transformation matrix and the posture change, the relationship between the first parameter change and the posture change of the robot end coordinate system is generated as the first association relationship:

[0093]

[0094] A plurality of first association relationships corresponding to the plurality of expected images are calculated to obtain a first association relationship set.

[0095] S14, determining a second association relationship between the second parameter change amounts of the plurality of point features and the translation change amounts of the robot end coordinate system in each of the expected images, to obtain a second association relationship set;

[0096] In this embodiment, the point feature interaction matrix of each desired image before the robot arm moves is first determined, and the determination method includes:

[0097] Extract the point features of the i-th point in the desired image of the k-th camera ( k x 1c0i , k y 1c0i ), let L Pki is the point feature interaction matrix before the i-th motion in the expected image corresponding to the k-th camera, L Pki Used for the i-th position control of the first part at the end of the robot arm, where i∈(0,m),j∈(0,n),k∈(0,K), m,n,K≥2, L Pki Calculated by:

[0098]

[0099] in,( k x 1cij , k y 1cij ) represents the coordinates of the jth point feature in the normalized imaging plane before the i-th movement in the k-th camera image as the second parameter. The point feature interaction matrix of each expected image is determined respectively to obtain a set of point feature interaction matrices.

[0100] Secondly, determine a set of second parameter changes of multiple point features in each expected image after the robot moves for the i-th time Multiple sets of second parameter changes constitute a first parameter change set. And determine the translation change of the robot end coordinate system Among them, Δ k x 1cij = k x 1cij - k x 1c0j ,Δ k y 1cij = k y 1cij - k y 1c0j , Δx i ,Δy i ,Δz i is the i-th translation motion of the camera.

[0101] Furthermore, the position transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm is determined. Let K PkRis the position transformation matrix between the coordinate system of the kth camera and the coordinate system of the end of the robotic arm. The robotic arm is manually controlled to complete l translation movements, l>10, and the historical second parameter change of each translation movement is recorded to obtain the historical second parameter change set. The historical position change of each translation movement is recorded to obtain the historical position change set, and the historical point feature interaction matrix L of the robotic arm at the initial position is determined. Pk0 The position transformation matrix K is determined based on the historical position change set, the historical second parameter change set and the historical point feature interaction matrix PkR :

[0102]

[0103] Among them, L Pk0 + =(L Pk0 T L Pk0 ) -1 L Pk0 T , T P + =(T P T T P ) -1 T P T .

[0104] According to the target point feature interaction matrix corresponding to the target expected image, a set of target second parameter changes, the target position transformation matrix and the translation change, the relationship between the second parameter change and the position change of the robot end coordinate system is generated as the second association relationship:

[0105]

[0106] A plurality of second association relationships corresponding to the plurality of expected images are calculated to obtain a second association relationship set.

[0107] S15. Determine a rotation control rate of the robotic arm according to the first association relationship set, and determine a translation control rate of the robotic arm according to the second association relationship set;

[0108] In this embodiment, the rotation control rate of the end of the robot arm is determined according to the line feature interaction matrix and the first association relationship:

[0109]

[0110] Among them, λ w ∈(0,1) is the attitude control coefficient,

[0111] The translation control rate of the end of the robot arm is determined based on the point feature interaction matrix and the second association relationship:

[0112]

[0113] Among them, λ v ∈(0,1) is the translation control coefficient,

[0114] S16. Control the movement of the robot arm according to the rotation control rate and the translation control rate so that the first part is in a set position.

[0115] In this embodiment, the set position is used to indicate the position where the first part is aligned with the second part. Specifically, the movement of the robot arm is controlled according to the rotation control rate and the translation control rate, so that the end position of the robot arm is iteratively adjusted until Δθ xi ,Δθ yi ,Δθ zi and Δx i ,Δy i ,Δz i The absolute value of is less than the set threshold, and it is determined that the first part is in the set position, thereby completing the part alignment and achieving successful assembly of the first part and the second part.

[0116] The robot control method provided in this embodiment obtains multiple expected images of the second part through multiple cameras; extracts multiple line features and point features from each expected image; determines the first correlation between the first parameter change of multiple line features in each expected image and the posture change of the robot end coordinate system; determines the second correlation between the second parameter change of multiple point features in each expected image and the position change of the robot end coordinate system; determines the rotation control rate of the robot according to the first correlation set, and determines the translation control rate of the robot according to the second correlation set; controls the movement of the robot according to the rotation control rate and the translation control rate, so that the first part is in a set position. The robot operates the parts for alignment based on the guidance of multiple cameras, reduces the part alignment error, expands the robot's operable space, and improves the efficiency and accuracy of part alignment.

[0117] Figure 3 A schematic diagram of the structure of a robot control device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, including:

[0118] an acquisition module 31, configured to acquire a plurality of desired images of a second part through a plurality of cameras, wherein the plurality of cameras are arranged on a mechanical arm of the robot, the second part corresponds to the first part, and the first part is arranged at the end of the mechanical arm;

[0119] A processing module 32, configured to extract a plurality of line features and a plurality of point features from each of the desired images;

[0120] The processing module 32 is further used to determine a first association relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images, to obtain a first association relationship set;

[0121] The processing module 32 is further used to determine a second association relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images, to obtain a second association relationship set;

[0122] The processing module 32 is further configured to determine a rotation control rate of the robotic arm according to the first association relationship set, and determine a translation control rate of the robotic arm according to the second association relationship set;

[0123] The control module 33 is used to control the movement of the robot arm according to the rotation control rate and the translation control rate so that the first part is in a set position.

[0124] In a possible implementation, the acquisition module 31 is specifically used to extract the center point and the target straight line in the desired image;

[0125] Obtaining the coordinates of the point on the target straight line that is closest to the origin;

[0126] The processing module 32 is specifically configured to determine the first coordinate of the coordinate of the point closest to the origin on the focal length normalized imaging plane according to the intrinsic parameter matrix of the camera;

[0127] Determining a second coordinate of the center point on a focal length normalized imaging plane;

[0128] Determining the line feature of the target straight line on a focal length normalized imaging plane according to the first coordinate and the second coordinate;

[0129] The acquisition module 31 is specifically used to acquire the coordinates of the target point in the desired image;

[0130] The processing module 32 is specifically configured to determine the third coordinate of the coordinate of the target point on the focal length normalized imaging plane as the point feature according to the intrinsic parameter matrix of the camera.

[0131] In a possible implementation, the processing module 32 is specifically configured to determine a line feature interaction matrix of each of the desired images before the robot arm moves, and obtain a set of line feature interaction matrices;

[0132] Determine the first parameter change amount of the plurality of line features in each of the expected images after the movement of the robotic arm, obtain a first parameter change amount set, and determine the posture change amount of the robotic arm end coordinate system;

[0133] Determine the attitude transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain an attitude transformation matrix set;

[0134] Determine a target line feature interaction matrix from the line feature interaction matrix set according to the target expected image, determine a set of target first parameter variation from the first parameter variation set, and determine a target posture transformation matrix from the posture transformation matrix set;

[0135] The first association relationship is determined according to the target line feature interaction matrix, the set of target first parameter changes, the target posture transformation matrix and the posture change.

[0136] In a possible implementation manner, the processing module 32 is specifically configured to determine a point feature interaction matrix of each of the desired images before the robot arm moves, and obtain a point feature interaction matrix set;

[0137] Determine the second parameter change amount of the plurality of point features in each of the expected images after the movement of the robotic arm, obtain a second parameter change amount set, and determine the position change amount of the robotic arm end coordinate system;

[0138] Determine the position transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain a set of position transformation matrices;

[0139] Determine a target point feature interaction matrix from the point feature interaction matrix set according to the target expected image, determine a set of target second parameter variation from the second parameter variation set, and determine a target position transformation matrix from the position transformation matrix set;

[0140] The second association relationship is determined according to the target point feature interaction matrix, the set of target second parameter changes, the target position transformation matrix and the position change.

[0141] In a possible implementation, the processing module 32 is specifically configured to determine a history line feature interaction matrix of the desired image when the robot arm is in an initial state;

[0142] Controlling the robotic arm to complete multiple rotational movements;

[0143] Determine the historical rotation change amount corresponding to each rotational movement, and obtain a set of historical rotation change amounts;

[0144] Determine the historical first parameter change amount corresponding to each rotational movement, and obtain a historical first parameter change amount set;

[0145] The posture transformation matrix is ​​determined according to the historical rotation variation set, the historical first parameter variation set and the historical line feature interaction matrix.

[0146] In a possible implementation, the processing module 32 is specifically configured to determine a historical point feature interaction matrix of the desired image when the robot arm is in an initial state, and obtain a historical point feature interaction matrix set;

[0147] Controlling the robotic arm to complete multiple translation movements;

[0148] Determine the historical position change amount corresponding to each translation movement, and obtain a set of historical position change amounts;

[0149] Determine the historical second parameter change amount corresponding to each translation movement, and obtain a historical second parameter change amount set;

[0150] The position transformation matrix is ​​determined according to the historical position change amount set, the historical second parameter change amount set and the historical point feature interaction matrix.

[0151] In one possible embodiment, the control module 33 is specifically used to adjust the rotation control rate and the translation control rate to control the movement of the robotic arm. When it is detected that the absolute values ​​of the position change and the posture change of the robotic arm end coordinate system are less than a set threshold, it is determined that the multiple first parts are in the set position.

[0152] The robot control device provided in this embodiment can be as follows Figure 3 The device shown in FIG. 1 may perform the following steps: Figure 2 All steps of the robot control method in order to achieve Figure 2 For details, please refer to Figure 2 For the sake of brevity, the relevant description is not repeated here.

[0153] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention, Figure 4 The computer device 400 shown includes: at least one processor 401, a memory 402, at least one network interface 404 and other user interfaces 403. The various components in the electronic device 400 are coupled together via a bus system 405. It is understood that the bus system 405 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 405 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 405 is not shown in FIG. Figure 4 Various buses are labeled as bus system 405 .

[0154] The user interface 403 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0155] It can be understood that the memory 402 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 402 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0156] In some implementations, the memory 402 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 4021 and application programs 4022 .

[0157] The operating system 4021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 4022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 4022.

[0158] In the embodiment of the present invention, by calling the program or instruction stored in the memory 402, specifically, the program or instruction stored in the application 4022, the processor 401 is used to execute the method steps provided by each method embodiment, for example, including:

[0159] Acquire multiple desired images of a second part by using multiple cameras, wherein the multiple cameras are arranged on a mechanical arm of the robot, the second part corresponds to the first part, and the first part is arranged at the end of the mechanical arm;

[0160] Extracting a plurality of line features and a plurality of point features from each of the desired images;

[0161] Determine a first association relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images to obtain a first association relationship set;

[0162] Determine a second association relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images to obtain a second association relationship set;

[0163] Determine a rotation control rate of the robotic arm according to the first association relationship set, and determine a translation control rate of the robotic arm according to the second association relationship set;

[0164] The movement of the robot arm is controlled according to the rotation control rate and the translation control rate so that the first part is in a set position.

[0165] The method disclosed in the above embodiment of the present invention can be applied to the processor 401, or implemented by the processor 401. The processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 401 or the instruction in the form of software. The above processor 401 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 402, and the processor 401 reads the information in the memory 402 and completes the steps of the above method in combination with its hardware.

[0166] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0167] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0168] The computer device provided in this embodiment may be Figure 4 The computer device shown in , can execute Figure 2 All steps of the robot control method in order to achieve Figure 2 For details, please refer to Figure 2 For the sake of brevity, the relevant description is not repeated here.

[0169] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0170] When one or more programs in the storage medium can be executed by one or more processors to implement the above-mentioned robot control method executed on the device side.

[0171] The processor is used to execute the robot control program stored in the memory to implement the following steps of the robot control method executed on the device side:

[0172] Acquire multiple desired images of a second part by using multiple cameras, wherein the multiple cameras are arranged on a mechanical arm of the robot, the second part corresponds to the first part, and the first part is arranged at the end of the mechanical arm;

[0173] Extracting a plurality of line features and a plurality of point features from each of the desired images;

[0174] Determine a first association relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images to obtain a first association relationship set;

[0175] Determine a second association relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images to obtain a second association relationship set;

[0176] Determine a rotation control rate of the robotic arm according to the first association relationship set, and determine a translation control rate of the robotic arm according to the second association relationship set;

[0177] The movement of the robot arm is controlled according to the rotation control rate and the translation control rate so that the first part is in a set position.

[0178] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0179] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0180] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A robot control method, characterized in that: The method comprises: Acquire multiple desired images of a second part by using multiple cameras, wherein the multiple cameras are arranged on a mechanical arm of the robot, the second part corresponds to the first part, and the first part is arranged at the end of the mechanical arm; Extracting a plurality of line features and a plurality of point features from each of the desired images; Determine a first association relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images to obtain a first association relationship set; Determine a second association relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images to obtain a second association relationship set; Determine a rotation control rate of the robotic arm according to the first association relationship set, and determine a translation control rate of the robotic arm according to the second association relationship set; Control the movement of the robot arm according to the rotation control rate and the translation control rate so that the first part is in a set position; The determining of a first correlation relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images comprises: Determine the line feature interaction matrix of each of the desired images before the robot arm moves, and obtain a set of line feature interaction matrices; Determine the first parameter change amount of the plurality of line features in each of the expected images after the movement of the robotic arm, obtain a first parameter change amount set, and determine the posture change amount of the robotic arm end coordinate system; Determine the attitude transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain an attitude transformation matrix set; Determine a target line feature interaction matrix from the line feature interaction matrix set according to the target expected image, determine a set of target first parameter variation from the first parameter variation set, and determine a target posture transformation matrix from the posture transformation matrix set; Determine the first association relationship according to the target line feature interaction matrix, the set of target first parameter changes, the target posture transformation matrix and the posture change; The determining of a second correlation relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images comprises: Determine a point feature interaction matrix of each of the desired images before the robot arm moves, and obtain a set of point feature interaction matrices; Determine the second parameter change amount of the plurality of point features in each of the expected images after the movement of the robotic arm, obtain a second parameter change amount set, and determine the position change amount of the robotic arm end coordinate system; Determine the position transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain a set of position transformation matrices; Determine a target point feature interaction matrix from the point feature interaction matrix set according to the target expected image, determine a set of target second parameter variation from the second parameter variation set, and determine a target position transformation matrix from the position transformation matrix set; The second association relationship is determined according to the target point feature interaction matrix, the set of target second parameter changes, the target position transformation matrix and the position change.

2. The method according to claim 1, characterized in that The line features are extracted in the following way: Extracting a center point and a target straight line in the desired image; Obtaining the coordinates of the point on the target straight line that is closest to the origin; Determine, according to the intrinsic parameter matrix of the camera, a first coordinate of the coordinate of the point closest to the origin on the focal length normalized imaging plane; Determining a second coordinate of the center point on a focal length normalized imaging plane; Determining the line feature of the target straight line on a focal length normalized imaging plane according to the first coordinate and the second coordinate; The point features are extracted in the following way: Obtaining coordinates of a target point in the desired image; The third coordinate of the coordinate of the target point on the focal length normalized imaging plane is determined according to the intrinsic parameter matrix of the camera as the point feature.

3. The method according to claim 1, characterized in that The posture transformation matrix is ​​determined by: Determine a history line feature interaction matrix of the desired image when the robotic arm is in an initial state; Controlling the robotic arm to complete multiple rotational movements; Determine the historical rotation change amount corresponding to each rotational movement, and obtain a set of historical rotation change amounts; Determine the historical first parameter change amount corresponding to each rotational movement, and obtain a historical first parameter change amount set; The posture transformation matrix is ​​determined according to the historical rotation variation set, the historical first parameter variation set and the historical line feature interaction matrix.

4. The method according to claim 1, characterized in that The position transformation matrix is ​​determined by: Determine a historical point feature interaction matrix of the desired image when the robotic arm is in an initial state, and obtain a historical point feature interaction matrix set; Controlling the robotic arm to complete multiple translation movements; Determine the historical position change amount corresponding to each translation movement, and obtain a set of historical position change amounts; Determine the historical second parameter change amount corresponding to each translation movement, and obtain a historical second parameter change amount set; The position transformation matrix is ​​determined according to the historical position change amount set, the historical second parameter change amount set and the historical point feature interaction matrix.

5. The method according to claim 1, characterized in that The step of controlling the movement of the robot arm according to the rotation control rate and the translation control rate so that the first part is in a set position comprises: The rotation control rate and the translation control rate are adjusted to control the movement of the robot arm. When it is detected that the absolute values ​​of the position change and the posture change of the robot arm end coordinate system are less than a set threshold, it is determined that the first part is in a set position.

6. A robot control device, characterized in that: include: an acquisition module, configured to acquire a plurality of desired images of a second part through a plurality of cameras, wherein the plurality of cameras are disposed on a mechanical arm of the robot, the second part corresponds to the first part, and the first part is disposed at the end of the mechanical arm; A processing module, used for extracting a plurality of line features and a plurality of point features from each of the desired images; The processing module is further used to determine a first association relationship between first parameter changes of the plurality of line features and posture changes of the robot end coordinate system in each of the expected images, to obtain a first association relationship set; The processing module is further used to determine a second association relationship between a second parameter change amount of a plurality of the point features and a position change amount of the robot end coordinate system in each of the expected images, to obtain a second association relationship set; The processing module is further used to determine a rotation control rate of the mechanical arm according to the first association relationship set, and determine a translation control rate of the mechanical arm according to the second association relationship set; A control module, used for controlling the movement of the robot arm according to the rotation control rate and the translation control rate, so that the first part is in a set position; The processing module is specifically used to determine the line feature interaction matrix of each of the expected images before the robot arm moves, and obtain a set of line feature interaction matrices; Determine the first parameter change amount of the plurality of line features in each of the expected images after the movement of the robotic arm, obtain a first parameter change amount set, and determine the posture change amount of the robotic arm end coordinate system; Determine the attitude transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain an attitude transformation matrix set; Determine a target line feature interaction matrix from the line feature interaction matrix set according to the target expected image, determine a set of target first parameter variation from the first parameter variation set, and determine a target posture transformation matrix from the posture transformation matrix set; Determine the first association relationship according to the target line feature interaction matrix, the set of target first parameter changes, the target posture transformation matrix and the posture change; The processing module is specifically used to determine the point feature interaction matrix of each of the expected images before the robot arm moves, and obtain a set of point feature interaction matrices; Determine the second parameter change amount of the plurality of point features in each of the expected images after the movement of the robotic arm, obtain a second parameter change amount set, and determine the position change amount of the robotic arm end coordinate system; Determine the position transformation matrix between the coordinate system of each camera and the coordinate system of the end of the robot arm to obtain a set of position transformation matrices; Determine a target point feature interaction matrix from the point feature interaction matrix set according to the target expected image, determine a set of target second parameter variation from the second parameter variation set, and determine a target position transformation matrix from the position transformation matrix set; The second association relationship is determined according to the target point feature interaction matrix, the set of target second parameter changes, the target position transformation matrix and the position change.

7. A computer device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a robot control program stored in the memory to implement the robot control method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the robot control method according to any one of claims 1 to 5.

Images