Robot control method, device, equipment, robot and storage medium

Abstract

The application discloses a control method of a robot, the robot comprising a mechanical arm and a 2D camera, a target object is in the working space of the mechanical arm, one side of the target object is provided with a positioning code which is relatively fixed with the position, the 2D camera is arranged on the mechanical arm and is relatively fixed with the position of the end, and is used for collecting the image of the positioning code; the method comprises the following steps: collecting the image of the positioning code by the 2D camera to obtain the three-dimensional pose of the positioning code in the base coordinate system, and determining the user coordinate system established based on the positioning code according to the three-dimensional pose; determining the three-dimensional working pose of the end of the mechanical arm according to the user coordinate system and the pre-stored working pose information of the end of the mechanical arm in the user coordinate system; controlling the end of the mechanical arm to move to the three-dimensional working pose to perform the work on the target object. The robot adopts 2D vision, obtains the three-dimensional working pose of the end of the mechanical arm, realizes the 3D positioning of the work of the target object by the mechanical arm, and has high positioning precision and low cost.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot control method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, with the rapid development of science and technology, robot products have been continuously updated and iterated, and have been widely used in many fields such as medicine, education, and industry.

[0003] During operation, robots typically use visual localization to determine the target object's position within the workspace before performing tasks such as grasping or welding. Currently, robot visual localization usually involves using 2D cameras to capture images of the target object, which outputs the object's planar coordinates and angles. While this method is inexpensive and has low manufacturing costs, it suffers from low positioning accuracy. Summary of the Invention

[0004] The main objective of this invention is to propose a robot control method that aims to solve the technical problems pointed out in the background art.

[0005] To achieve the above objectives, the present invention proposes a robot control method, wherein the robot includes a robotic arm and a 2D camera. The robotic arm has a target object in its workspace, and a positioning code is provided on one side of the target object with its position fixed relative to it. The 2D camera is mounted on the robotic arm and its position is fixed relative to the end of the robotic arm, and is used to acquire images of the positioning code.

[0006] Robot control methods include:

[0007] The image of the positioning code is acquired by the 2D camera to obtain the three-dimensional pose of the positioning code in the base coordinate system, and the user coordinate system established based on the positioning code is determined according to the three-dimensional pose.

[0008] The three-dimensional working pose of the robotic arm's end effector is determined based on the user coordinate system and the pre-stored working pose information of the robotic arm's end effector in the user coordinate system.

[0009] Control the end effector of the robotic arm to move to the three-dimensional working pose in order to perform the operation on the target object.

[0010] The steps of acquiring images of the positioning code using a 2D camera to obtain the 3D pose of the positioning code in a base coordinate system, and determining the user coordinate system established based on the positioning code according to the 3D pose of the positioning code in the base coordinate system include:

[0011] The positioning code is captured by a 2D camera to obtain its position information in the camera coordinate system, and the current position information of the end effector of the robotic arm in the base coordinate system is recorded at the same time.

[0012] The three-dimensional pose of the positioning code in the base coordinate system is calculated based on the current position information of the end effector of the robotic arm in the base coordinate system, the pre-measured relative position relationship between the end effector of the robotic arm and the 2D camera in the base coordinate system, and the position information of the positioning code in the camera coordinate system.

[0013] Prior to the step of determining the three-dimensional working pose of the robotic arm's end effector based on the determined user coordinate system and the pre-stored working pose information of the robotic arm's end effector in the user coordinate system, the robot control method further includes:

[0014] The positioning code is captured by a 2D camera to obtain its position information in the camera coordinate system, and the current position information of the end effector of the robotic arm in the base coordinate system is recorded at the same time.

[0015] The three-dimensional pose of the positioning code in the base coordinate system is calculated based on the current position information of the end effector of the robot arm in the base coordinate system, the pre-measured relative position relationship between the end effector of the robot arm and the 2D camera in the base coordinate system, and the position information of the positioning code in the camera coordinate system.

[0016] A user coordinate system based on the positioning code is established according to the three-dimensional pose of the positioning code in the base coordinate system.

[0017] The end effector of the teaching robot moves to the teaching position for performing the operation on the target object, and at the same time records the teaching pose information of the end effector in the user coordinate system as the operation pose information of the end effector in the user coordinate system for storage.

[0018] The three-dimensional pose of the positioning code in the base coordinate system is calculated according to the following formula:

[0019] object_matrix=robot_matrix*RT_matrix*image_matrix;

[0020] Wherein, object_matrix is ​​the pose matrix of the positioning code in the base coordinate system, robot_matrix is ​​the pose matrix of the end effector of the robot arm in the base coordinate system, RT_matrix is ​​the matrix of the relative position relationship between the end effector of the robot arm and the 2D camera in the base coordinate system, and image_matrix is ​​the pose matrix of the positioning code in the camera coordinate system.

[0021] The robotic arm's workspace also includes a calibration plate;

[0022] Robot control methods also include:

[0023] The robotic arm is controlled to move sequentially to multiple different photo-taking points. At each photo-taking point, a 2D camera is used to capture an image of the calibration board to obtain its position information, and the position information of the robotic arm is recorded at the same time.

[0024] Based on the position information of multiple calibration plates and the position information of the robotic arm, the relative position relationship matrix between the end effector of the robotic arm and the 2D camera is calculated to obtain the relative position relationship between the end effector of the robotic arm and the 2D camera in the base coordinate system.

[0025] Before the step of calculating the relative positional relationship matrix between the end effector of the robotic arm and the 2D camera based on the positional information of multiple calibration plates and the positional information of the robotic arm, in order to measure the relative positional relationship between the end effector of the robotic arm and the 2D camera in the base coordinate system, the robot control method also includes:

[0026] Determine whether the number of times the calibration board's image is acquired is less than the preset number;

[0027] If so, control the robotic arm to move to other photo-taking points and continue to collect images of the calibration board;

[0028] If not, then stop acquiring images of the calibration board.

[0029] The present invention also proposes a robot control device, wherein the robot includes a robotic arm and a 2D camera. The robotic arm has a target object in its work space. A positioning code is provided on one side of the target object and its position is fixed relative to it. The 2D camera is mounted on the robotic arm and its position is fixed relative to the end of the robotic arm, and is used to acquire images of the positioning code.

[0030] The robot's control device includes:

[0031] The user coordinate system determination module is used to acquire images of the positioning code through a 2D camera to obtain the three-dimensional pose of the positioning code in the base coordinate system, and to determine the user coordinate system based on the positioning code according to the three-dimensional pose.

[0032] The job position determination module is used to determine the three-dimensional job pose of the end effector of the robotic arm based on the user coordinate system and the pre-stored job pose information of the end effector of the robotic arm in the user coordinate system.

[0033] The robotic arm operation control module is used to control the end effector of the robotic arm to move to the three-dimensional operation pose in order to perform operations on the target object.

[0034] The present invention also proposes a robot control device, which includes:

[0035] Memory, used to store computer programs;

[0036] A processor is used to execute computer programs to implement the robot control method as described above.

[0037] The present invention also proposes a robot comprising:

[0038] A robotic arm has a target object in its workspace, and a positioning code with a fixed position relative to the target object is provided on one side of the target object.

[0039] A 2D camera is mounted on the robotic arm and its position is fixed relative to the end of the robotic arm. It is used to capture images of the positioning code.

[0040] A controller is used to execute the steps of the robot control method as described above.

[0041] The present invention also proposes a storage medium storing a computer program, which, when executed by a processor, implements the steps of the robot control method described above.

[0042] The robot control method of this invention is applied to a robot including a robotic arm and a 2D camera. A target object is located in the workspace of the robotic arm, and a positioning code is fixed to one side of the target object. The 2D camera is mounted on the robotic arm and fixed to its end effector to acquire images of the positioning code. Based on this, the robot control method first acquires images of the positioning code using the 2D camera to obtain the three-dimensional pose of the positioning code in a base coordinate system, and then determines a user coordinate system based on the positioning code according to the three-dimensional pose. Next, based on the user coordinate system and pre-stored work pose information of the robotic arm's end effector in the user coordinate system, the three-dimensional work pose of the robotic arm's end effector is determined. Finally, the robotic arm's end effector is controlled to move to the three-dimensional work pose to perform a task on the target object. The control method of the robot of the present invention uses a positioning code with a fixed positional relationship with the target object as the identification object. A 2D camera is used to identify the positioning code and acquire its two-dimensional planar image to determine the three-dimensional working pose of the end effector of the robotic arm in the three-dimensional working space for performing the operation on the target object. Then, the end effector of the robotic arm is controlled to move to the three-dimensional working pose to perform the operation on the target object. That is, the robot of the present invention uses 2D vision to obtain the three-dimensional working pose of the end effector of the robotic arm, realizes 3D positioning of the robotic arm to the target object, and improves the positioning accuracy compared with the prior art. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the robot's working scenario in this invention;

[0044] Figure 2 This is a flowchart of a robot control method according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of a robot control method according to an embodiment of the present invention;

[0046] Figure 4 This is a flowchart of a robot control method according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the robot hand-eye calibration scenario in this invention;

[0048] Figure 6 This is a flowchart of a robot control method according to an embodiment of the present invention;

[0049] Figure 7 This is a flowchart of a robot control method according to an embodiment of the present invention;

[0050] Figure 8 This is a block diagram of the robot control device according to an embodiment of the present invention;

[0051] Figure 9 This is a schematic diagram of the architecture of the robot control device in one embodiment of the present invention. Detailed Implementation

[0052] The solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0053] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0054] It should also be noted that when a component is described as "fixed to" or "set on" another component, it can be directly on the other component or there may be an intervening component present. When a component is described as "connected to" another component, it can be directly connected to the other component or there may be an intervening component present.

[0055] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0056] Reference Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of the robot's working scenario in this invention. Figure 2 Here is a flowchart of a robot control method according to an embodiment of the present invention:

[0057] This invention proposes a robot control method, wherein the robot 100 includes a robotic arm 110 and a 2D camera 120. The robotic arm 110 has a target object 10 in its work space. A positioning code 20 is provided on one side of the target object 10 and is fixed relative to its position. The 2D camera 120 is mounted on the robotic arm 110 and is fixed relative to the end 111 of the robotic arm 110, and is used to acquire images of the positioning code 20.

[0058] Robot control methods include:

[0059] Step S100: Acquire images of the positioning code using the 2D camera 120 to obtain the three-dimensional pose of the positioning code 20 in the base coordinate system, and determine the user coordinate system established based on the positioning code 20 according to the three-dimensional pose;

[0060] Step S200: Determine the three-dimensional working pose of the end effector 111 of the robotic arm 110 based on the user coordinate system and the pre-stored working pose information of the end effector 111 of the robotic arm 110 in the user coordinate system.

[0061] Step S300: Control the end effector 111 of the robotic arm 110 to move to the three-dimensional working pose to perform the operation on the target object 10.

[0062] In this embodiment, during operation, the robot 100 determines the three-dimensional working pose of the end effector 111 of the robotic arm 110 in the work space, thereby controlling the end effector 111 of the robotic arm 110 to move to the three-dimensional working pose and perform the operation on the target object 10. In practical applications, the end effector 111 of the robotic arm 110 is equipped with an actuator for performing the operation on the target object 10. The type of operation performed by the robotic arm 110 is not limited, while the actuator can be configured according to its working type. For example, when the robotic arm 110 is used to grasp the target object 10, the actuator can be a pneumatic suction cup, a clamping cylinder, etc.; or when the robotic arm 110 is used to perform dispensing, welding, or screw fastening operations on the target object 10, the actuator can be a dispensing head, a welding head, or an electric screwdriver, etc., and so on.

[0063] Specifically, such as Figure 1The robot 100 shown has a target object 10 in the workspace of its robotic arm 110. A positioning code 20 is fixed to one side of the target object 10. A 2D camera 120 is mounted on the robotic arm 110 and fixed to the end effector 111 of the robotic arm 110, used to capture images of the positioning code 20. Optionally, the 2D camera 120 is located on one side of the end effector 111 of the robotic arm 110, and the positioning code 20 can be an Apriltag code, etc. Taking the robotic arm 110 grasping the target object 10 as an example, the working range of the robot 100 can be fixed. For example, the robotic arm 110 is fixedly set on the work table, and a positioning code 20 is set on the work table. The target object 10 is placed on the work table, and the positioning code 20 is located on one side of the target object 10 and its position is fixed relative to it. The positioning code is identified by the 2D camera 120 to determine the three-dimensional grasping pose of the robotic arm 110 in the work space. The end effector 111 of the robotic arm 110 moves to the three-dimensional grasping pose and then grasps the target object 10. Furthermore, the robot 100's operating scenario may include a conveyor line that transports the target object 10 via a fixture. The fixture is equipped with a positioning code 20 and carries the target object 10. The positioning code 20 is located on one side of the target object 10 and its position is fixed relative to it. The conveyor line transports the fixture into the working space of the robotic arm 110. The positioning code is identified by a 2D camera 120 to determine the three-dimensional grasping pose of the robotic arm 110 in the working space. The end effector 111 of the robotic arm 110 moves to the three-dimensional grasping pose and then grasps the target object 10.

[0064] Alternatively, the working range of robot 100 can be non-fixed. For example, robot 100 may also include a mobile carrier, on which robotic arm 110 is mounted. The mobile carrier moves to bring robotic arm 110 to the target object 10. A positioning code 20 is fixed to the target object 10 on one side. When the target object 10 enters the working space of robotic arm 110, a 2D camera 120 identifies the positioning code to determine the three-dimensional grasping pose of robotic arm 110 within the working space. Robotic arm 110 then moves to this three-dimensional grasping pose to perform three-dimensional positioning and grasping of the target object 10. The mobile carrier can be an AGV (Automated Guided Vehicle).

[0065] Based on this, the steps S100 to S300 of the robot control method are as follows: Figure 2As shown, firstly, an image of the positioning code 20 is acquired using a 2D camera to obtain the three-dimensional pose of the positioning code 20 in the base coordinate system, and the user coordinate system established based on the positioning code 20 is determined according to the three-dimensional pose; then, the three-dimensional working pose of the end effector 111 of the robotic arm 110 is determined according to the user coordinate system and the pre-stored working pose information of the end effector 111 of the robotic arm 110 in the user coordinate system; finally, the end effector 111 of the robotic arm 110 is controlled to move to the 3D working pose to perform the operation on the target object 10.

[0066] The coordinate systems commonly used by robot 100 include the geodetic coordinate system, base coordinate system, tool coordinate system, workpiece coordinate system, joint coordinate system, and user coordinate system. The base coordinate system is a Cartesian coordinate system based on the robot 100's mounting base, used to describe the robot 100's body motion. The user coordinate system is a Cartesian coordinate system customized by the user for each workspace, used for teaching and executing position registers, executing position compensation commands, etc. In this embodiment, the user coordinate system is established based on the positioning code 20. Because the positions of the target object 10 and the positioning code 20 are relatively fixed, the positional relationship of the target object 10 in the user coordinate system established based on the positioning code 20 is fixed. Furthermore, the positional relationship of the three-dimensional work pose of the end effector 111 of the robotic arm 110 performing operations on the target object 10 in the user coordinate system is also fixed. Specifically, during the operation, the robotic arm 110 is controlled to move so that the 2D camera 120 is aligned with the positioning code 20. The positioning code 20 enters the field of view of the 2D camera 120. The 2D camera 120 captures a two-dimensional planar image of the positioning code 20 and performs calculations to obtain the three-dimensional pose of the positioning code 20 in the base coordinate system. After obtaining the three-dimensional pose of the positioning code 20 in the base coordinate system, the user coordinate system established based on the positioning code 20 can be determined. Combined with the pre-stored working pose information of the end effector 111 of the robotic arm 110 in the user coordinate system, the three-dimensional working pose position that the end effector 111 of the robotic arm 110 needs to reach can be further determined. Thus, the end effector 111 of the robotic arm 110 can be controlled to move to the three-dimensional working pose position to perform the operation on the target object 10. The working pose information of the end effector 111 of the robotic arm 110 in the user coordinate system is obtained and stored in advance. This process will be described in detail in subsequent embodiments and will not be described in detail here. The end effector 111 of the robotic arm 110 has working pose information in the user coordinate system, including position coordinates and position orientation, which belongs to three-dimensional pose information. Therefore, when the robotic arm 110 is working, it can realize three-dimensional pose operation on the target object 10, thereby adapting to the working effect that cannot be achieved by 2D vision.

[0067] The control method of the robot of the present invention uses a positioning code 20 with a fixed positional relationship to the target object 10 as the identification object. A 2D camera is used to identify the positioning code 20 and acquire its two-dimensional planar image to determine the three-dimensional working pose of the end effector 111 of the robotic arm 110 in the three-dimensional work space for performing tasks on the target object 10. Then, the end effector 111 of the robotic arm 110 is controlled to move to the three-dimensional working pose to perform tasks on the target object 10. In other words, the robot 100 of the present invention uses 2D vision to obtain the three-dimensional working pose of the end effector 111 of the robotic arm 110, realizing 3D positioning of the robotic arm 110 on the target object 10, which improves positioning accuracy compared to existing technologies. Furthermore, the 2D camera 120 used is small in size and less restricted by the working environment, allowing the robot 100 to be applied to more work scenarios.

[0068] The embodiments of this application will be further described in detail below with reference to the accompanying drawings:

[0069] Reference Figure 3 , Figure 3 Here is a flowchart of a robot control method according to an embodiment of the present invention:

[0070] In some embodiments, step S100 includes:

[0071] Step S110: Acquire images of positioning code 20 using 2D camera 120 to obtain position information of positioning code 20 in camera coordinate system, and simultaneously record the current position information of end effector 111 of robotic arm 110 in base coordinate system.

[0072] Step S120: Calculate the three-dimensional pose of the positioning code 20 in the base coordinate system based on the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system, the pre-measured relative position relationship between the end effector 111 of the robotic arm 110 in the base coordinate system and the 2D camera 120, and the position information of the positioning code 20 in the camera coordinate system.

[0073] During the operation of robot 100, robotic arm 110 moves to move 2D camera 120 to the working imaging position. At this time, positioning code 20 is within the imaging field of view of 2D camera 120. The coordinates and orientation (Xr, Yr, Zr, RXr, RYr, RZr) of the end effector 111 of robotic arm 110 in the base coordinate system are recorded and converted into a pose matrix, that is, the current position information of the end effector 111 of robotic arm 110 in the base coordinate system is recorded. Image information of positioning code 20 is acquired by 2D camera 120, and then the pose information of positioning code 20 is detected and converted into a pose matrix, that is, the position information of positioning code 20 in the camera coordinate system is obtained.

[0074] Then, based on the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system, the pre-measured relative position relationship between the end effector 111 of the robotic arm 110 in the base coordinate system and the 2D camera 120, and the position information of the positioning code 20 in the camera coordinate system, the pose matrix of the positioning code 20 in the base coordinate system is calculated, thus obtaining the three-dimensional pose of the positioning code 20 in the base coordinate system.

[0075] Specifically, the three-dimensional pose of the positioning code 20 in the base coordinate system is calculated according to the following formula:

[0076] object_matrix=robot_matrix*RT_matrix*image_matrix;

[0077] Wherein, object_matrix is ​​the pose matrix of positioning code 20 in the base coordinate system, robot_matrix is ​​the pose matrix of end effector 111 of robotic arm 110 in the base coordinate system, RT_matrix is ​​the pre-measured relative position relationship matrix between end effector 111 of robotic arm 110 and 2D camera 120 in the base coordinate system, and image_matrix is ​​the pose matrix of positioning code 20 in the camera coordinate system.

[0078] Reference Figure 4 , Figure 4 Here is a flowchart of a robot control method according to an embodiment of the present invention:

[0079] In some embodiments, prior to step S200, the robot control method further includes:

[0080] Step S400: The 2D camera 120 acquires the image of the positioning code 20 to obtain the position information of the positioning code 20 in the camera coordinate system, and at the same time records the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system.

[0081] Step S500: Calculate the three-dimensional pose of the positioning code 20 in the base coordinate system based on the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system, the pre-measured relative position relationship between the end effector 111 of the robotic arm 110 in the base coordinate system and the 2D camera 120, and the position information of the positioning code 20 in the camera coordinate system.

[0082] Step S600: Establish a user coordinate system based on the three-dimensional pose of the positioning code 20 in the base coordinate system;

[0083] Step S700: The end effector 111 of the teaching robot arm 110 moves to the teaching position for performing the operation on the target object 10, and at the same time records the teaching pose information of the end effector 111 of the robot arm 110 in the user coordinate system as the operation pose information of the end effector 111 of the robot arm 110 in the user coordinate system for storage.

[0084] In this embodiment, steps S400 to S700 are implemented to obtain the operational pose information of the end effector 111 of the robotic arm 110 in the user coordinate system. Specifically, during the determination of the operational position information of the end effector 111 of the robotic arm 110 in the user coordinate system, firstly, the robotic arm 110 moves to move the 2D camera 120 to the measurement and imaging position. At this time, the positioning code 20 is within the imaging field of view of the 2D camera 120. The coordinates and orientation (Xr, Yr, Zr, RXr, RYr, RZr) of the end effector 111 of the robotic arm 110 in the base coordinate system are recorded and converted into a pose matrix, that is, the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system is recorded. The image information of the positioning code 20 is captured by the 2D camera 120, and then the pose information of the positioning code 20 is detected and converted into a pose matrix, that is, the position information of the positioning code 20 in the camera coordinate system is obtained.

[0085] Then, based on the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system, the pre-measured relative position relationship between the end effector 111 of the robotic arm 110 in the base coordinate system and the 2D camera 120, and the position information of the positioning code 20 in the camera coordinate system, the pose matrix of the positioning code 20 in the base coordinate system is calculated, thus obtaining the three-dimensional pose of the positioning code 20 in the base coordinate system.

[0086] Specifically, the position information of the positioning code 20 in the base coordinate system is calculated according to the following formula:

[0087] object_matrix=robot_matrix*RT_matrix*image_matrix;

[0088] Wherein, object_matrix is ​​the pose matrix of positioning code 20 in the base coordinate system, robot_matrix is ​​the pose matrix of end effector 111 of robotic arm 110 in the base coordinate system, RT_matrix is ​​the pre-measured relative position relationship matrix between end effector 111 of robotic arm 110 and 2D camera 120 in the base coordinate system, and image_matrix is ​​the pose matrix of positioning code 20 in the camera coordinate system.

[0089] Finally, based on the three-dimensional pose of the positioning code 20 in the base coordinate system, a user coordinate system based on the positioning code 20 is established. Specifically, the relationship between the user coordinate system established based on the positioning code 20 and the base coordinate system is as follows:

[0090] user_matrix=object_matrix*robot_matrix;

[0091] Wherein, user_matrix is ​​the transformation matrix of the user coordinate system, object_matrix is ​​the pose matrix of the positioning code 20 in the base coordinate system, and robot_matrix is ​​the pose matrix of the end effector 111 of the robot arm 110 in the base coordinate system.

[0092] Subsequently, in the user coordinate system, the end effector 111 of the robotic arm 110 is taught to move to the working pose for performing the operation on the target object 10. This teaching process can be performed by manually dragging the robotic arm 110 or by using a teach pendant to control the movement of the robotic arm 110, depending on the actual situation, and there are no restrictions on this. The teaching pose for performing the operation on the target object 10 is the working pose of the end effector 111 of the robotic arm 110 when the robotic arm 110 performs the operation on the target object 10 in the actual working scenario. During teaching, the end effector 111 of the robotic arm 110 moves to the teaching position for performing the operation on the target object 10, the teaching pose information of the end effector 111 of the robotic arm 110 in the user coordinate system is recorded, that is, the working pose information of the end effector 111 of the robotic arm 110 in the user coordinate system is recorded, and it is stored. Here, the teaching pose is set as P1 in the user coordinate system, which corresponds to converting the coordinate pose into a homogeneous matrix P1_matrix representing the spatial position. This relationship P1_matrix is ​​fixed in the user coordinate system determined by the positioning code 20.

[0093] Depending on the specific work scenario, the working pose for performing tasks on the target object 10 can be one or multiple. When there are multiple working poses, after establishing a user coordinate system based on the positioning code 20, the end effector 111 of the robotic arm 110 can be taught multiple times to move to multiple teaching positions for performing tasks on the target object 10, while simultaneously recording multiple teaching pose information of the end effector 111 of the robotic arm 110 in the user coordinate system. This allows for obtaining multiple working pose information of the end effector 111 of the robotic arm 110 in the user coordinate system, which can then be selected and used according to the actual work situation. Therefore, in practical applications, if the robotic arm 110 needs to perform tasks on the target object 10 from a certain working pose, it can be selected and set accordingly. Furthermore, in practical applications, the teaching position for the robotic arm 110 to perform tasks on the target object 10 can be set to the central imaging position of the workspace, thereby adjusting the attitude of the 2D camera and narrowing the field of view to improve detection accuracy and achieve high-precision operation on the target object 10.

[0094] Reference Figure 5 and Figure 6 , Figure 5 This is a schematic diagram of the robot hand-eye calibration scenario in this invention. Figure 6 Here is a flowchart of a robot control method according to an embodiment of the present invention:

[0095] In some embodiments, the robotic arm 110 also has a calibration plate 30 in its workspace;

[0096] Robot control methods also include:

[0097] Step S800: Control the robotic arm 110 to move sequentially to multiple different photo-taking points. At each photo-taking point, the 2D camera 120 captures the image of the calibration plate 30 to obtain its position information, and at the same time records the position information of the robotic arm 110.

[0098] Step S900: Based on the position information of multiple sets of calibration plates 30 and the position information of the robotic arm 110, calculate the relative position relationship matrix between the end effector 111 of the robotic arm 110 and the 2D camera 120, so as to obtain the relative position relationship between the end effector 111 of the robotic arm 110 and the 2D camera 120 in the base coordinate system.

[0099] The determination of the relative positional relationship between the end effector 111 of the robotic arm 110 and the 2D camera 120 in the base coordinate system is the hand-eye calibration of the robotic arm 110 and the 2D camera 120. During calibration, the robotic arm 110 is first controlled to move to the initial image capture position. The 2D camera 120 is aligned with the calibration plate 30, which is located within the field of view of the 2D camera 120 and remains stationary. The current coordinate position of the robotic arm 110 is recorded, i.e., the position information of the robotic arm 110. By acquiring images of the calibration plate 30 through the 2D camera 120, its position information can be obtained. Based on the image information of the calibration plate 30, the coordinate position of the calibration plate 30 can be converted, i.e., the position information of the calibration plate 30. Thus, a set of position information of the calibration plate 30 and the position information of the robotic arm 110 are obtained.

[0100] Then, the robotic arm 110 is controlled to move to the next shooting point to take pictures of the calibration plate 30 and record the position of the robotic arm 110. When taking pictures, the 2D camera 120 is aimed at the calibration plate 30. The calibration plate 30 is located within the shooting field of the 2D camera 120 and is fixed, thereby obtaining the position information of another set of calibration plates 30 and the position information of the robotic arm 110.

[0101] By obtaining the position information of multiple sets of calibration plates 30 and robotic arm 110, the relative positional relationship matrix between the end effector 111 of robotic arm 110 and 2D camera 120 can be calculated to obtain the relative positional relationship between the end effector 111 of robotic arm 110 and 2D camera 120 in the base coordinate system. Specifically, firstly, the intrinsic parameters and distortion of 2D camera 120 are calibrated based on the collected position information of calibration plates 30. Then, hand-eye calibration is further performed by combining the collected position information of calibration plates 30 and position information of robotic arm 110, and the relative positional relationship matrix between the end effector 111 of robotic arm 110 and 2D camera 120 is calculated, that is, the hand-eye relationship between the end effector 111 of robotic arm 110 and 2D camera 120 in the base coordinate system is obtained.

[0102] Reference Figure 7 , Figure 7 Here is a flowchart of a robot control method according to an embodiment of the present invention:

[0103] In some embodiments, prior to step S800, the robot control method further includes:

[0104] Step S1000: Determine whether the number of times the image information of the calibration board 30 is acquired is less than the preset number;

[0105] Step S1100: If yes, control the robotic arm 110 to move to other photo capture points and continue to collect images from the calibration board 30;

[0106] Step S1200: If not, end the acquisition of images from calibration board 30.

[0107] In the hand-eye calibration process between the robotic arm 110 and the 2D camera 120, the robotic arm 110 needs to move to a certain number of image capture points. By collecting and recording a certain amount of data, the accuracy of the hand-eye calibration is ensured. Therefore, before calculating the relative positional relationship between the end effector 111 of the robotic arm 110 and the 2D camera 120, it is determined whether the number of image information acquisitions from the calibration plate 30 is less than a preset number. If the number of acquisitions is less than the preset number, the robotic arm 110 is controlled to move to the next image capture point to acquire image information from the calibration plate 30. When the number of acquisitions reaches the preset number, the acquisition of image information from the calibration plate 30 ends. Then, based on the image information from multiple calibration plates 30 and the pose information of the robotic arm 110, the relative positional relationship matrix between the end effector 111 of the robotic arm 110 and the 2D camera 120 can be calculated to obtain the relative positional relationship between the end effector 111 of the robotic arm 110 and the 2D camera 120 in the base coordinate system. The preset number of acquisitions can be set according to actual conditions, such as 15 times, and is not limited thereto.

[0108] Reference Figure 1 and Figure 8 , Figure 8 Here is a block diagram of the robot control device according to an embodiment of the present invention:

[0109] The present invention also proposes a robot control device, wherein the robot 100 includes a robotic arm 110 and a 2D camera 120. The robotic arm 110 has a target object 10 in its working space. A positioning code 20 is provided on one side of the target object 10 and is fixed relative to its position. The 2D camera 120 is mounted on the robotic arm 110 and is fixed relative to the end 111 of the robotic arm 110, and is used to acquire images of the positioning code 20.

[0110] The robot's control device includes:

[0111] The user coordinate system determination module is used to acquire images of the positioning code 20 through a 2D camera to obtain the three-dimensional pose of the positioning code 20 in the base coordinate system, and to determine the user coordinate system established based on the positioning code 20 according to the three-dimensional pose.

[0112] The job position determination module is used to determine the three-dimensional job pose of the end effector 111 of the robotic arm 110 based on the user coordinate system and the pre-stored job pose information of the end effector 111 of the robotic arm 110 in the user coordinate system.

[0113] The robotic arm operation control module is used to control the end effector 111 of the robotic arm 110 to move to the three-dimensional operation pose so as to perform operations on the target object 10.

[0114] In some embodiments, the user coordinate system determination module includes:

[0115] The acquisition and recording unit is used to acquire images of the positioning code 20 through a 2D camera to obtain the position information of the positioning code 20 in the camera coordinate system, and at the same time record the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system.

[0116] The calculation unit is used to calculate the three-dimensional pose of the positioning code 20 in the base coordinate system based on the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system, the pre-measured relative position relationship between the end effector 111 of the robotic arm 110 in the base coordinate system and the 2D camera 120, and the position information of the positioning code 20 in the camera coordinate system.

[0117] In some embodiments, the robot's control device further includes:

[0118] The acquisition and recording module is used to acquire images of the positioning code 20 through a 2D camera to obtain the position information of the positioning code 20 in the camera coordinate system, and at the same time record the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system.

[0119] The calculation module is used to calculate the three-dimensional pose of the positioning code 20 in the base coordinate system based on the current position information of the end effector 111 of the robotic arm 110 in the base coordinate system, the pre-measured relative position relationship between the end effector 111 of the robotic arm 110 in the base coordinate system and the 2D camera 120, and the position information of the positioning code 20 in the camera coordinate system.

[0120] The user coordinate system establishment module is used to establish a user coordinate system based on the three-dimensional pose of the positioning code 20 in the base coordinate system.

[0121] The teaching operation position recording module is used to teach the end effector 111 of the robotic arm 110 to move to the teaching position for performing operations on the target object 10, and at the same time record the teaching pose information of the end effector 111 of the robotic arm 110 in the user coordinate system for storage as the operation pose information of the end effector 111 of the robotic arm 110 in the user coordinate system.

[0122] In some embodiments, the three-dimensional pose of the positioning code 20 in the base coordinate system is calculated according to the following formula:

[0123] object_matrix=robot_matrix*RT_matrix*image_matrix;

[0124] Wherein, object_matrix is ​​the pose matrix of positioning code 20 in the base coordinate system, robot_matrix is ​​the pose matrix of end effector 111 of robotic arm 110 in the base coordinate system, RT_matrix is ​​the pre-measured relative position relationship matrix between end effector 111 of robotic arm 110 and 2D camera 120 in the base coordinate system, and image_matrix is ​​the pose matrix of positioning code 20 in the camera coordinate system.

[0125] In some embodiments, the robotic arm 110 also has a calibration plate 30 in its workspace;

[0126] The robot control device also includes:

[0127] The first execution module is used to control the robotic arm 110 to move sequentially to multiple different photo-taking points. At each photo-taking point, the 2D camera 120 captures the image of the calibration plate 30 to obtain its position information, and at the same time records the position information of the robotic arm 110.

[0128] The second execution module is used to calculate the relative position relationship matrix between the end effector 111 of the robotic arm 110 and the 2D camera 120 based on the position information of multiple sets of calibration plates 30 and the position information of the robotic arm 110, so as to measure the relative position relationship between the end effector 111 of the robotic arm 110 and the 2D camera 120 in the base coordinate system.

[0129] In some embodiments, the robot's control device further includes:

[0130] The judgment module is used to determine whether the number of image acquisitions of the calibration board 30 is less than the preset number;

[0131] The third execution module is used to control the robotic arm 110 to move to other shooting points and continue to collect images of the calibration board 30 when the number of times the images of the calibration board 30 are collected is less than the preset number.

[0132] The fourth execution module is used to terminate the acquisition of images of the calibration board 30 when the number of image acquisitions of the calibration board 30 is not less than a preset number.

[0133] The present invention also proposes a robot control device, which includes:

[0134] Memory 1005 is used to store computer programs;

[0135] The processor 1001 is used to execute a computer program to implement the steps of the robot control method as described above.

[0136] The control device for the robot proposed in this embodiment of the invention can be a robot or a PC. For example... Figure 9 As shown, the robot's control device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit, such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0137] Those skilled in the art will understand that Figure 9 The robot control device structure shown does not constitute a limitation on the robot control device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0138] like Figure 9 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a robot control program.

[0139] exist Figure 9 In the robot control device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the robot control program stored in the memory 1005.

[0140] The present invention also proposes a robot 100, which includes:

[0141] The robotic arm 110 has a target object 10 in its working space, and a positioning code 20 with its position fixed on one side of the target object 10.

[0142] A 2D camera 120 is mounted on the robotic arm 110 and is fixed relative to the end of the robotic arm 110 111, and is used to acquire images of the positioning code 20.

[0143] A controller is used to execute the steps of the robot control method as described above.

[0144] The present invention also proposes a storage medium storing a computer program, which, when executed by a processor, implements the steps of the robot control method described above.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0146] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0147] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0148] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The above description is only a part or preferred embodiment of the present invention. Neither the text nor the drawings should limit the scope of protection of the present invention. All equivalent structural transformations made using the content of the present invention specification and drawings under the overall concept of the present invention, or direct / indirect applications in other related technical fields, are included within the scope of protection of the present invention.

Claims

1. A method for controlling a robot, characterized in that, The robot includes a robotic arm and a 2D camera. The robotic arm has a target object in its workspace. A positioning code is fixed to one side of the target object. The 2D camera is mounted on the robotic arm and is fixed to the end of the robotic arm to capture images of the positioning code. The robot control method includes: The 2D camera acquires an image of the positioning code to obtain the three-dimensional pose of the positioning code in the base coordinate system, and the user coordinate system established based on the positioning code is determined according to the three-dimensional pose. The three-dimensional working pose of the end effector of the robotic arm is determined based on the user coordinate system and the pre-stored working pose information of the end effector of the robotic arm in the user coordinate system. Control the end effector of the robotic arm to move to the three-dimensional working pose in order to perform the operation on the target object; The step of acquiring an image of the positioning code using the 2D camera to obtain the three-dimensional pose of the positioning code in a base coordinate system, and determining the user coordinate system established based on the positioning code according to the three-dimensional pose of the positioning code in the base coordinate system includes: The 2D camera captures images of the positioning code to obtain the position information of the positioning code in the camera coordinate system, and records the current position information of the end effector of the robotic arm in the base coordinate system. The three-dimensional pose of the positioning code in the base coordinate system is calculated based on the current position information of the end of the robotic arm in the base coordinate system, the pre-measured relative position relationship between the end of the robotic arm and the 2D camera in the base coordinate system, and the position information of the positioning code in the camera coordinate system.

2. The robot control method according to claim 1, characterized in that, Before the step of determining the three-dimensional working pose of the robotic arm's end effector based on the determined user coordinate system and the pre-stored working pose information of the robotic arm's end effector in the user coordinate system, the robot control method further includes: The 2D camera captures images of the positioning code to obtain the position information of the positioning code in the camera coordinate system, and simultaneously records the current position information of the end effector of the robotic arm in the base coordinate system. The three-dimensional pose of the positioning code in the base coordinate system is calculated based on the current position information of the end of the robotic arm in the base coordinate system, the pre-measured relative position relationship between the end of the robotic arm and the 2D camera in the base coordinate system, and the position information of the positioning code in the camera coordinate system. Based on the three-dimensional pose of the positioning code in the base coordinate system, a user coordinate system based on the positioning code is established; The robotic arm's end effector is taught to move to the taught position for performing the operation on the target object, and the taught pose information of the robotic arm's end effector in the user coordinate system is recorded and stored as the operation pose information of the robotic arm's end effector in the user coordinate system.

3. The robot control method according to claim 1 or 2, characterized in that, The three-dimensional pose of the positioning code in the base coordinate system is calculated according to the following formula: object_matrix=robot_matrix RT_matrix image_matrix; Wherein, object_matrix is ​​the pose matrix of the positioning code in the base coordinate system, robot_matrix is ​​the pose matrix of the end effector of the robot arm in the base coordinate system, RT_matrix is ​​the pre-measured relative position relationship matrix between the end effector of the robot arm and the 2D camera in the base coordinate system, and image_matrix is ​​the pose matrix of the positioning code in the camera coordinate system.

4. The robot control method according to claim 1 or 2, characterized in that, The robotic arm's workspace also includes a calibration plate; The robot control method further includes: The robotic arm is controlled to move sequentially to multiple different photo-taking points. At each photo-taking point, the 2D camera captures an image of the calibration plate to obtain its position information, and the position information of the robotic arm is recorded simultaneously. Based on the position information of the calibration plates and the position information of the robotic arm, the relative position relationship matrix between the end of the robotic arm and the 2D camera is calculated to obtain the relative position relationship between the end of the robotic arm and the 2D camera in the base coordinate system.

5. The robot control method according to claim 4, characterized in that, Before the step of calculating the relative positional relationship matrix between the end effector of the robotic arm and the 2D camera based on the positional information of multiple sets of calibration plates and the positional information of the robotic arm, in order to measure the relative positional relationship between the end effector of the robotic arm and the 2D camera in the base coordinate system, the robot control method further includes: Determine whether the number of times the image of the calibration board is acquired is less than a preset number; If so, the robotic arm is controlled to move to other imaging points to continue acquiring images of the calibration board; If not, then stop acquiring images of the calibration board.

6. A control device for a robot, characterized in that, The robot includes a robotic arm and a 2D camera. The robotic arm has a target object in its workspace. A positioning code is fixed to one side of the target object. The 2D camera is mounted on the robotic arm and is fixed to the end of the robotic arm to capture images of the positioning code. The robot's control device includes: The user coordinate system determination module is used to acquire an image of the positioning code through the 2D camera to obtain the three-dimensional pose of the positioning code in the base coordinate system, and determine the user coordinate system based on the positioning code according to the three-dimensional pose. The job position determination module is used to determine the three-dimensional job pose of the end effector of the robotic arm based on the user coordinate system and the pre-stored job pose information of the end effector of the robotic arm in the user coordinate system. The robotic arm operation control module is used to control the end effector of the robotic arm to move to the three-dimensional operation pose in order to perform operations on the target object; The user coordinate system determination module includes: The acquisition and recording unit is used to acquire images of the positioning code through the 2D camera to obtain the position information of the positioning code in the camera coordinate system, and simultaneously record the current position information of the end of the robotic arm in the base coordinate system. The calculation unit is used to calculate the three-dimensional pose of the positioning code in the base coordinate system based on the current position information of the end of the robotic arm in the base coordinate system, the pre-measured relative position relationship between the end of the robotic arm and the 2D camera in the base coordinate system, and the position information of the positioning code in the camera coordinate system.

7. A robot control device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the control method for the robot as described in any one of claims 1 to 5.

8. A robot, characterized in that, include: A robotic arm, wherein the working space of the robotic arm contains a target object, and one side of the target object is provided with a positioning code that is fixed relative to its position; A 2D camera, which is mounted on the robotic arm and fixed relative to the end of the robotic arm, is used to acquire images of the positioning code; A controller for performing the steps of the control method for the robot as described in any one of claims 1 to 5.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the robot control method as described in any one of claims 1 to 5.

Images