Trajectory positioning method based on 2d vision, robot system and storage medium

Abstract

The application discloses a trajectory positioning method based on 2D vision, comprising the following steps: acquiring an initial image of a reference object captured by a 2D camera; establishing a corresponding relationship between a three-dimensional model of a workpiece and a reference coordinate system; acquiring a new image of the reference object captured by the 2D camera; comparing the new image with the initial image to determine position change information of the reference object in a robot coordinate system; and determining coordinates of a point corresponding to a target trajectory on the three-dimensional model in the robot coordinate system according to the position change information of the reference object and the corresponding relationship between the three-dimensional model and the reference coordinate system. The application further discloses a robot system and a storage medium. The application acquires an image of a reference object captured by a 2D camera, and when a target trajectory on a surface of a workpiece needs to be positioned, the position of the target trajectory on the surface of the workpiece can be obtained by using the image captured by the 2D camera and the three-dimensional model of the workpiece, so that high-precision positioning of the trajectory on the surface of the workpiece is realized by using a low-cost 2D camera, and the hardware cost is reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of robotics technology, and in particular to a trajectory localization method, robot system and storage medium based on 2D vision. Background Technology

[0002] Industrial robots, as one of the important symbols of the development of modern manufacturing technology and an emerging technology industry, have been widely used in various industries and have had a significant impact on all fields of modern high-tech industries and even people's lives.

[0003] In industrial applications, industrial robots can apply adhesive to the contoured surface of workpieces or perform welding at weld seams, depending on the user's needs. To accurately capture the preset welding or adhesive application trajectories on the workpiece surface and control the industrial robot to position itself accordingly, some manufacturers use 3D (3D) cameras to capture images of the workpiece and its surface trajectory in real time and feed them back to the industrial robot. However, 3D camera-based visual positioning methods heavily rely on the quality of the 3D imaging point cloud, requiring high precision. For example, for workpieces with complex materials and surfaces, the imaging quality of ordinary 3D cameras is often poor, severely affecting the positioning accuracy of the workpiece surface trajectory. This limits the applicability of 3D camera-based visual positioning and results in high hardware costs. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a trajectory positioning method, robot system and storage medium based on 2D vision, which can balance the positioning accuracy of workpiece surface trajectory and hardware cost, and has a wide range of applications.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A 2D vision-based trajectory localization method for locating target trajectories on a workpiece surface includes:

[0007] Acquire an initial image of a reference object captured by a 2D camera, wherein the pose of the reference object is relatively fixed relative to the target trajectory on the workpiece surface;

[0008] Establish a correspondence between the 3D model of the workpiece and the reference coordinate system O_ref determined based on the reference object;

[0009] Acquire a new image of the reference object captured by a 2D camera;

[0010] Based on the new image and the initial image, determine the position change information of the reference object in the robot coordinate system;

[0011] Based on the position change information of the reference object and the correspondence between the three-dimensional model and the reference coordinate system O_ref, the coordinates of the point on the three-dimensional model corresponding to the target trajectory in the robot coordinate system are determined.

[0012] As one implementation, the step of determining the position change information of the reference object in the robot coordinate system based on the new image and the initial image includes:

[0013] Obtain the first coordinate transformation relationship RT1 between the camera coordinate system where the 2D camera is located and the robot coordinate system;

[0014] Based on the initial image of the reference object and the first coordinate transformation relationship RT1, determine the initial pose of the reference object in the robot coordinate system;

[0015] Based on the new image of the reference object and the first coordinate transformation relationship RT1, determine the new pose of the reference object in the robot coordinate system;

[0016] Based on the new pose and the initial pose, determine the pose change of the reference object in the robot coordinate system.

[0017] As one implementation, the step of obtaining the first coordinate transformation relationship RT1 between the camera coordinate system and the robot coordinate system where the 2D camera is located includes:

[0018] Obtain the first coordinates of at least three points on the calibration object in the camera coordinate system and the second coordinates in the robot coordinate system;

[0019] Based on the first and second coordinates of each point, the first coordinate transformation relationship RT1 is derived.

[0020] As one implementation method, the step of determining the pose of the reference object in the robot coordinate system includes:

[0021] Based on the image of the reference object captured by the 2D camera, determine the coordinates (x0, y0) of the reference object in the camera coordinate system and the horizontal rotation angle Rz_ref;

[0022] According to the first coordinate transformation relationship RT1, the coordinates (x0, y0) of the reference object in the camera coordinate system are transformed into the coordinates in the robot coordinate system;

[0023] The step of determining the pose change of the reference object in the robot coordinate system based on the new pose and the initial pose includes:

[0024] By comparing the horizontal rotation angle determined based on the initial image of the reference object and the horizontal rotation angle determined based on the new image of the reference object, the relative angle value R of the rotation of the reference coordinate system O_ref is determined;

[0025] Based on the new image of the reference object, the coordinates of the reference object in the robot coordinate system and the relative angle value R are determined, and the new coordinates of the reference coordinate system O_ref in the robot coordinate system are determined.

[0026] In one implementation, the reference coordinate system O_ref uses the coordinates of the reference object in the robot coordinate system at the initial pose as the origin of the XY coordinate axis, and the origin of the Z coordinate axis and the directions of the XYZ coordinate axes of the reference coordinate system O_ref are consistent with the robot coordinate system.

[0027] As one implementation method, the step of establishing a correspondence between the three-dimensional model of the workpiece and the reference coordinate system O_ref determined according to the reference object includes:

[0028] Obtain the second coordinate transformation relationship RT2 between the coordinate system of the 3D model and the robot coordinate system;

[0029] Based on the second coordinate transformation relationship RT2, the coordinates (x, y, z, Rx, Ry, Rz) of all trajectory points in the coordinate system of the 3D model in the robot coordinate system are obtained;

[0030] Unify the coordinates (x, y, z, Rx, Ry, Rz) of the trajectory point in the robot coordinate system to the reference coordinate system O_ref to obtain the initial coordinates P1 of the trajectory point in the reference coordinate system O_ref;

[0031] The step of determining the coordinates of the point corresponding to the target trajectory on the 3D model in the robot coordinate system based on the position change information of the reference object and the correspondence between the 3D model and the reference coordinate system O_ref includes:

[0032] After determining the new pose of the reference object in the robot coordinate system, the initial coordinates P1 of the trajectory point in the reference coordinate system O_ref are converted into new coordinates P2 in the robot coordinate system.

[0033] As one implementation, the step of obtaining the second coordinate transformation relationship RT2 between the coordinate system of the 3D model and the robot coordinate system includes:

[0034] Obtain the coordinates (X, Y, Z) of multiple points on the workpiece in the robot coordinate system;

[0035] The second coordinate transformation relationship RT2 is obtained by using the coordinates (x, y, z) of each point on the workpiece in the coordinate system of the three-dimensional model and the coordinates (X, Y, Z) of each point in the robot coordinate system.

[0036] In one embodiment, the reference object is a workpiece or a marker set independently of the workpiece.

[0037] Another object of the present invention is to provide a robot system comprising:

[0038] 2D camera;

[0039] Robotic arm, including the actuator for performing the operation;

[0040] A controller, communicatively connected to the 2D camera and the robotic arm, is configured to control the robotic arm according to the aforementioned 2D vision-based trajectory positioning method, so that the actuator of the robotic arm moves along a target trajectory on the workpiece surface.

[0041] Another object of the present invention is to provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the above-described 2D vision-based trajectory localization method.

[0042] This invention uses an initial image of a reference object captured by a 2D camera as a reference image. When it is necessary to locate the target trajectory on the surface of the workpiece, the position change of the reference object is determined based on the updated image captured by the 2D camera. By combining the position change of the reference object with the three-dimensional model of the workpiece, the position of the target trajectory on the surface of the workpiece can be obtained. High-precision positioning of the workpiece surface trajectory is achieved using a low-cost 2D camera, reducing hardware costs. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a trajectory localization method based on 2D vision according to an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of a method for determining the position change information of a reference object in a robot coordinate system according to an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of a method for obtaining the coordinate transformation relationship between the camera coordinate system and the robot coordinate system according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of a method for determining the pose of a reference object in a robot coordinate system according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of a method for determining the pose change of a reference object in a robot coordinate system according to an embodiment of the present invention;

[0048] Figure 6 This is a flowchart illustrating another trajectory localization method based on 2D vision according to an embodiment of the present invention.

[0049] Figure 7This is a schematic diagram illustrating a method for obtaining the coordinate transformation relationship between the coordinate system of a 3D model and the robot coordinate system according to an embodiment of the present invention.

[0050] Figure 8 This is a structural block diagram of a robot system according to an embodiment of the present invention;

[0051] Figure 9 This is a structural block diagram of a robot according to an embodiment of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] In this invention, the terms "set up," "equipped with," and "connected" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, elements, or components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0054] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0055] Furthermore, in addition to indicating direction or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in certain situations to indicate a dependency or connection. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0057] like Figure 1 As shown, this embodiment of the invention provides a trajectory localization method based on 2D vision. This method is used to locate the target trajectory on the surface of a workpiece and mainly includes:

[0058] S01. Acquire an initial image of a reference object captured by a 2D camera, wherein the pose of the reference object is relatively fixed relative to the target trajectory on the workpiece surface;

[0059] S02, Acquire a new image of the reference object captured by the 2D camera;

[0060] S03. Based on the new image and the initial image, determine the position change information of the reference object in the robot coordinate system;

[0061] S04. Establish a correspondence between the three-dimensional model of the workpiece and the reference coordinate system O_ref determined based on the reference object;

[0062] S05. Based on the position change information of the reference object and the correspondence between the 3D model and the reference coordinate system O_ref, determine the coordinates of the point on the 3D model corresponding to the target trajectory in the robot coordinate system.

[0063] It is understandable that step S04 is performed after step S01. A reference coordinate system O_ref is established by using the initial image of the reference object captured by the 2D camera in step S01. In steps S02 to S03, the position change of the reference object in the robot coordinate system is determined by the new image of the reference object captured by the 2D camera. The correspondence between the initial image of the reference object, the 3D model of the workpiece obtained in S01 and S04, and the reference coordinate system O_ref is used as the reference basis for determining the target trajectory on the surface of the workpiece in the subsequent step S05.

[0064] Preferably, step S04 can be performed in the preparation stage after step S01 and before S02-S03. That is, steps S01 and S04 are completed in advance before the robot performs its operation on the current workpiece, so that the robot can call them during the subsequent operation stages S02-S03 and S05, without having to re-execute the above steps S01 and S04 for each operation during the operation stage. For example, the above steps S01 and S04 can be performed after the reference object is placed in the robot coordinate system and before the operation of the first workpiece begins.

[0065] In this system, the pose of the reference object is relatively fixed relative to the target trajectory on the workpiece surface. The reference object can be the workpiece itself or a part of it, or it can be a marker placed independently of the workpiece. For example, the marker could be a tray supporting the workpiece, or an object placed at intervals from the workpiece. When the reference object is the workpiece itself or a part of it, the robot directly adjusts the position of the target trajectory based on the placement differences between the preceding and following workpieces. When the reference object is a marker independent of the workpiece, the robot indirectly adjusts the position of the target trajectory on the workpiece based on the pose differences of the same marker.

[0066] It is understandable that the position change information of the reference object in the robot coordinate system includes changes in position and orientation. In step S04, after establishing a correspondence between the 3D model of the workpiece and the reference coordinate system O_ref determined based on the reference object, each point on the 3D model of the workpiece has a one-to-one and unique coordinate in the reference coordinate system O_ref. When the reference object is shifted and / or rotated relative to the robot coordinate system, it means that the workpiece is shifted and / or rotated relative to the robot coordinate system. The coordinates of the reference coordinate system O_ref and each point on the 3D model are also shifted and / or rotated synchronously. In step S05, the new coordinates of each point on the 3D model after synchronous shift and / or rotation can be obtained.

[0067] For example, the 2D camera is mounted on the robot, such as on one of the robotic arms or on an actuator at the end of the robotic arm. In other embodiments, the 2D camera may also be mounted on an object other than the robot, such as a wall.

[0068] This application uses a 2D camera to capture 2D images of a reference object and compares the newly acquired image with an initial reference image to determine the positional changes of the reference object in the robot coordinate system. Simultaneously, this application pre-acquires a 3D model of the workpiece and establishes a correspondence between it and the reference coordinate system determined based on the reference object, thus associating the pose of the reference object with the pose of the 3D model. Finally, based on the positional changes of the reference object in the robot coordinate system and the correspondence between the 3D model of the workpiece and the reference coordinate system, the coordinates of the points on the 3D model corresponding to the target trajectory in the robot coordinate system can be obtained. This facilitates the robot's precise positioning to the target trajectory point on the workpiece for operations such as welding or applying adhesive, ensuring operational accuracy and yield.

[0069] Since the embodiments of this application only require a 2D camera and a 3D model of the workpiece to achieve accurate positioning of the target trajectory on the workpiece surface, without the need for an expensive 3D camera, and can guarantee positioning accuracy, the cost of the robot is reduced. Furthermore, the workpiece does not need to be placed in a specific posture or position, which reduces the requirements for workpiece placement and also improves work efficiency to a certain extent.

[0070] like Figure 2 In one embodiment, step S03, which determines the position change information of the reference object in the robot coordinate system based on the new image and the initial image, may specifically include:

[0071] S031. Obtain the first coordinate transformation relationship RT1 between the camera coordinate system where the 2D camera is located and the robot coordinate system;

[0072] S032. Based on the initial image of the reference object obtained in step S01 and the first coordinate transformation relationship RT1 mentioned above, determine the initial pose of the reference object in the robot coordinate system.

[0073] S033. Based on the new image of the reference object obtained in step S02 and the first coordinate transformation relationship RT1 mentioned above, determine the new pose of the reference object in the robot coordinate system.

[0074] S034. Based on the new pose determined in step S033 and the initial pose determined in step S032, determine the pose change of the reference object in the robot coordinate system.

[0075] like Figure 3 As shown, in one embodiment, step S031, which involves obtaining the first coordinate transformation relationship RT1 between the camera coordinate system and the robot coordinate system of the 2D camera, may specifically include:

[0076] S0311. Obtain the first coordinates (x, y) of M points on the calibration object in the camera coordinate system and the second coordinates (x, y) in the robot coordinate system; where M is an integer and M≥3;

[0077] S0312. Based on the first coordinates (x, y) and second coordinates (X, Y) of each point, derive the first coordinate transformation relationship RT1.

[0078] In one embodiment, in step S04, the reference coordinate system O_ref can be determined based on the initial image of the reference object captured by the 2D camera. Specifically, the reference coordinate system O_ref is established using the coordinates (X1_ref, Y1_ref) of the reference object in the robot coordinate system at its initial pose in the initial image as the origin of the XY coordinate axes, and the origin of the Z coordinate axis and the directions of the XYZ coordinate axes of the reference coordinate system O_ref are aligned with the robot coordinate system. This simplifies the calculation process for coordinate transformation and calibration. It is understood that in other embodiments, the origin of the Z coordinate axis and the directions of the XYZ coordinate axes of the reference coordinate system O_ref can be set differently from the robot coordinate system. It is also understood that in other embodiments, the reference coordinate system O_ref can be determined based on other images of the reference object preceding the new image.

[0079] This embodiment uses the coordinates of at least three points on the calibration object in both the camera coordinate system and the robot coordinate system as data to calculate the first coordinate transformation relationship RT1. Based on this, the formula for the first coordinate transformation relationship RT1 can be derived for use in subsequent steps, improving the robot's reaction speed during trajectory localization. According to this first coordinate transformation relationship RT1, every point in the image acquired in the camera coordinate system can find a unique corresponding coordinate value in the robot coordinate system.

[0080] like Figure 4 In one embodiment, the process of determining the pose of the reference object in the robot coordinate system can be obtained in the following way:

[0081] S11. Based on the image of the reference object captured by the 2D camera, determine the coordinates (x0, y0) of the reference object in the camera coordinate system and the horizontal rotation angle Rz_ref;

[0082] S12. According to the first coordinate transformation relationship RT1, convert the coordinates (x0, y0) of the reference object in the camera coordinate system to the coordinates (X0_ref, Y0_ref) in the robot coordinate system.

[0083] Thus, combined Figure 5 As shown, in one embodiment, step S034, which determines the pose change of the reference object in the robot coordinate system based on the new pose and the initial pose, specifically includes:

[0084] (1) In the preparation stage, according to the above steps S11 and S12, obtain the initial pose of the reference object:

[0085] S0341. Based on the initial image of the reference object captured by the 2D camera, determine the coordinates (x1, y1) and horizontal rotation angle Rz_ref1 of the reference object in the camera coordinate system. The coordinates (x1, y1) and horizontal rotation angle Rz_ref1 of the reference object in the camera coordinate system obtained from the initial image can be used as a reference benchmark for subsequent judgment of the pose changes of the reference object and the workpiece.

[0086] S0342. According to the first coordinate transformation relationship RT1, convert the coordinates (x1, y1) of the reference object in the camera coordinate system to the coordinates (X1_ref, Y1_ref) in the robot coordinate system.

[0087] (2) During the operation phase, according to steps S11 and S12 above, obtain the new pose of the reference object:

[0088] S0343. Based on the new image of the reference object captured by the 2D camera, determine the coordinates (x2, y2) of the reference object in the camera coordinate system and the horizontal rotation angle Rz_ref2.

[0089] S0344. According to the first coordinate transformation relationship RT1, convert the coordinates (x2, y2) of the reference object in the camera coordinate system to the coordinates (X2_ref, Y2_ref) in the robot coordinate system.

[0090] (3) During the operation phase, determine the new coordinates of the reference coordinate system O_ref in the robot coordinate system based on the front and rear poses of the reference object:

[0091] S0345. By comparing the horizontal rotation angle Rz_ref1 determined based on the initial image of the reference object and the horizontal rotation angle Rz_ref2 determined based on the new image of the reference object, determine the relative angle value R of the rotation of the reference coordinate system O_ref.

[0092] S0346. Based on the new image of the reference object and the coordinates (X2_ref, Y2_ref) and relative angle value R of the reference object in the robot coordinate system, determine the new coordinates of the reference coordinate system O_ref in the robot coordinate system. In this way, the new pose of the reference object in the robot coordinate system can be determined, serving as the calculation benchmark for the new coordinates of the trajectory points in the robot coordinate system.

[0093] For example, when the reference coordinate system O_ref takes the coordinates (X1_ref, Y1_ref) of the reference object in the robot coordinate system at the initial pose as the origin of the XY coordinate axis, and the origin of the Z coordinate axis and the directions of the XYZ coordinate axes of the reference coordinate system O_ref are consistent with the robot coordinate system, in the above step S0346, when determining the new coordinates of the reference coordinate system O_ref in the robot coordinate system, the new coordinates of the reference coordinate system O_ref in the robot coordinate system are (X2_ref, Y2_ref, 0, 0, 0, R), which is denoted as RT3 here.

[0094] like Figure 6 In one embodiment, step S04, which establishes a correspondence between the three-dimensional model of the workpiece and the reference coordinate system O_ref determined based on the reference object, specifically includes:

[0095] S041. Obtain the second coordinate transformation relationship RT2 between the coordinate system of the 3D model and the robot coordinate system;

[0096] S042. According to the second coordinate transformation relationship RT2, obtain the coordinates (x, y, z, Rx, Ry, Rz) of all trajectory points in the coordinate system of the 3D model in the robot coordinate system.

[0097] S043. Unify the coordinates (x, y, z, Rx, Ry, Rz) of the trajectory point in the robot coordinate system to the reference coordinate system O_ref to obtain the initial coordinates P1 of the trajectory point in the reference coordinate system O_ref;

[0098] Similar to the processes S0341 and S0342 described above for obtaining the initial pose of the reference object, step S04 in this embodiment can also be completed in advance during the preparation stage and called directly during the running stage.

[0099] Once steps S01, S0341, S0342, and S04 of the preparation phase are completed, step S05 can be executed. Based on the position change information of the reference object and the correspondence between the 3D model and the reference coordinate system O_ref, the coordinates of the points on the 3D model corresponding to the target trajectory in the robot coordinate system are determined.

[0100] In step S05, which involves determining the coordinates of the point corresponding to the target trajectory on the 3D model in the robot coordinate system, the specific steps may include:

[0101] S051. After determining the new pose of the reference object in the robot coordinate system (i.e., after determining the new coordinates RT3 of the reference coordinate system O_ref in the robot coordinate system in step S0346 of step S03), the initial coordinates P1 of the trajectory point in the reference coordinate system O_ref are converted into new coordinates P2 in the robot coordinate system. Here, the new coordinates P2 of the trajectory point in the robot coordinate system are P2 = RT3 * P1.

[0102] like Figure 7 In one embodiment, step S041, which involves obtaining the second coordinate transformation relationship RT2 between the coordinate system of the 3D model and the robot coordinate system, may specifically include:

[0103] S0411. Obtain the coordinates (X, Y, Z) of multiple points on the workpiece in the robot coordinate system;

[0104] S0412. Based on the coordinates (x, y, z) of each point on the workpiece in the coordinate system of the three-dimensional model and the coordinates (X, Y, Z) of each point in the robot coordinate system, the second coordinate transformation relationship RT2 is obtained.

[0105] In steps S0411 and S0412 above, a unique second coordinate transformation relationship RT2 can be obtained by acquiring the coordinates of at least four points on the workpiece in the robot coordinate system. After the second coordinate transformation relationship RT2 is determined, the coordinates (x, y, z, Rx, Ry, Rz) of the trajectory points of the 3D model in the robot coordinate system are obtained. By using the coordinates of the reference coordinate system O_ref in the robot coordinate system, the coordinates of the trajectory points of the 3D model in the robot coordinate system are unified to the reference coordinate system O_ref. Thus, the initial coordinates P1 of the trajectory points in the reference coordinate system O_ref can be easily obtained for use during the runtime phase.

[0106] In summary, this application acquires the 3D model of the workpiece in advance during the preparation stage before the robot operates on the current workpiece, extracts the target trajectory points on the 3D model, and obtains the first coordinate transformation relationship RT1 between the camera coordinate system and the robot coordinate system; uses 2D camera vision to identify the initial pose of the reference object in the camera coordinate system, and transforms it according to the first coordinate transformation relationship RT1 to obtain the initial pose of the reference object in the robot coordinate system, thereby establishing a reference coordinate system O_ref; and by calibrating the second coordinate transformation relationship RT2 between the coordinate system of the 3D model of the workpiece and the robot coordinate system, the target trajectory points on the 3D model are sequentially transformed to the robot coordinate system and the reference coordinate system O_ref, thus obtaining a unique and definite relationship between the target trajectory points and the reference coordinate system O_ref. During the operation phase of the robot working on the current workpiece, this application uses a 2D camera to visually identify the new pose of the reference object in the camera coordinate system, determines the new coordinates of the reference coordinate system O_ref in the robot coordinate system based on the changes in the pose of the reference object in the robot coordinate system, and obtains the new coordinates of the target trajectory point in the robot coordinate system based on the unique and definite relationship between the new coordinates and the target trajectory point obtained in the preparation phase and the reference coordinate system O_ref.

[0107] Furthermore, this invention also provides a control method for a robotic arm. This method controls the robotic arm according to the aforementioned 2D vision-based trajectory positioning method, enabling the actuator of the robotic arm to move along a target trajectory on the workpiece surface, thereby completing operations such as welding or applying adhesive on the workpiece surface along a predetermined trajectory. The movement along the target trajectory on the workpiece surface can be sequentially moving from one endpoint of the target trajectory to the other, or it can be dividing the target trajectory into several segments and moving along each segment separately.

[0108] like Figure 8 This invention also provides a robot system, comprising:

[0109] 2D camera 1, which can be used to acquire 2D images containing reference objects and / or workpieces;

[0110] Robotic arm 2, including an execution end for performing operations;

[0111] Controller 3 is communicatively connected to the 2D camera and the robotic arm. The controller is configured to control the robotic arm according to any of the above-mentioned 2D vision-based trajectory positioning methods, so that the actuator of the robotic arm moves along the target trajectory on the workpiece surface.

[0112] For example, the 2D camera is mounted on the robot, such as on one of the robotic arms or on an actuator at the end of the robotic arm. In other embodiments, the 2D camera may also be mounted on an object other than the robot, such as a wall.

[0113] Another embodiment of the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the above-described 2D vision-based trajectory localization method.

[0114] Another embodiment of the present invention provides a robot, such as Figure 9 The robot includes a memory 10, a processor 20, and program instructions 30 for a 2D vision-based trajectory localization method stored in the memory 10 and executable on the processor 20. When the program instructions 30 for the 2D vision-based trajectory localization method are executed by the processor 20, the aforementioned 2D vision-based trajectory localization method is implemented.

[0115] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of an electronic device. In this embodiment, the processor is used to run program code stored in a storage medium or to process data.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0117] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A trajectory localization method based on 2D vision for locating a target trajectory on a workpiece surface, characterized in that, include: Acquire an initial image of a reference object captured by a 2D camera, wherein the pose of the reference object is relatively fixed relative to the target trajectory on the workpiece surface; Establish a correspondence between the 3D model of the workpiece and the reference coordinate system O_ref determined based on the reference object; Acquire a new image of the reference object captured by a 2D camera; Based on the new image and the initial image, determine the position change information of the reference object in the robot coordinate system; Based on the position change information of the reference object and the correspondence between the three-dimensional model and the reference coordinate system O_ref, the coordinates of the point on the three-dimensional model corresponding to the target trajectory in the robot coordinate system are determined.

2. The 2D vision based trajectory localization method of claim 1, wherein, The step of determining the position change information of the reference object in the robot coordinate system based on the new image and the initial image includes: Obtain the first coordinate transformation relationship RT1 between the camera coordinate system where the 2D camera is located and the robot coordinate system; Based on the initial image of the reference object and the first coordinate transformation relationship RT1, determine the initial pose of the reference object in the robot coordinate system; Based on the new image of the reference object and the first coordinate transformation relationship RT1, determine the new pose of the reference object in the robot coordinate system; Based on the new pose and the initial pose, determine the pose change of the reference object in the robot coordinate system.

3. The 2D vision based trajectory localization method of claim 2, wherein, The step of obtaining the first coordinate transformation relationship RT1 between the camera coordinate system and the robot coordinate system where the 2D camera is located includes: Obtain the first coordinates of at least three points on the calibration object in the camera coordinate system and the second coordinates in the robot coordinate system; Based on the first and second coordinates of each point, the first coordinate transformation relationship RT1 is derived.

4. The trajectory localization method based on 2D vision according to claim 2, characterized in that, The steps for determining the pose of a reference object in the robot coordinate system include: Based on the image of the reference object captured by the 2D camera, determine the coordinates (x0, y0) of the reference object in the camera coordinate system and the horizontal rotation angle Rz_ref; According to the first coordinate transformation relationship RT1, the coordinates (x0, y0) of the reference object in the camera coordinate system are transformed into the coordinates in the robot coordinate system; The step of determining the pose change of the reference object in the robot coordinate system based on the new pose and the initial pose includes: By comparing the horizontal rotation angle Rz_ref determined based on the initial image of the reference object with the horizontal rotation angle determined based on the new image of the reference object, the relative angle value R of the rotation of the reference coordinate system O_ref is determined. Based on the new image of the reference object, the coordinates of the reference object in the robot coordinate system and the relative angle value R are determined, and the new coordinates of the reference coordinate system O_ref in the robot coordinate system are determined.

5. The trajectory localization method based on 2D vision according to claim 2, characterized in that, The reference coordinate system O_ref takes the coordinates of the reference object in the robot coordinate system at the initial pose as the origin of the XY coordinate axis. The origin of the Z coordinate axis and the directions of the XYZ coordinate axes of the reference coordinate system O_ref are consistent with the robot coordinate system.

6. The trajectory localization method based on 2D vision according to claim 1, characterized in that, The step of establishing a correspondence between the three-dimensional model of the workpiece and the reference coordinate system O_ref determined based on the reference object includes: Obtain the second coordinate transformation relationship RT2 between the coordinate system of the 3D model and the robot coordinate system; Based on the second coordinate transformation relationship RT2, the coordinates (x, y, z, Rx, Ry, Rz) of all trajectory points in the coordinate system of the 3D model in the robot coordinate system are obtained; Unify the coordinates (x, y, z, Rx, Ry, Rz) of the trajectory point in the robot coordinate system to the reference coordinate system O_ref to obtain the initial coordinates P1 of the trajectory point in the reference coordinate system O_ref; The step of determining the coordinates of the point corresponding to the target trajectory on the 3D model in the robot coordinate system based on the position change information of the reference object and the correspondence between the 3D model and the reference coordinate system O_ref includes: After determining the new pose of the reference object in the robot coordinate system, the initial coordinates P1 of the trajectory point in the reference coordinate system O_ref are converted into new coordinates P2 in the robot coordinate system.

7. The trajectory localization method based on 2D vision according to claim 6, characterized in that, The step of obtaining the second coordinate transformation relationship RT2 between the coordinate system of the 3D model and the robot coordinate system includes: Obtain the coordinates (X, Y, Z) of multiple points on the workpiece in the robot coordinate system; The second coordinate transformation relationship RT2 is obtained by using the coordinates (x, y, z) of each point on the workpiece in the coordinate system of the three-dimensional model and the coordinates (X, Y, Z) of each point in the robot coordinate system.

8. The trajectory localization method based on 2D vision according to claim 1, characterized in that, The reference object is the workpiece or an identifier set independently of the workpiece.

9. A robot system, characterized in that, include: 2D camera; Robotic arm, including the actuator for performing the operation; A controller, communicatively connected to the 2D camera and the robotic arm, is configured to control the robotic arm using the 2D vision-based trajectory positioning method according to any one of claims 1 to 8, so that the actuator of the robotic arm moves along a target trajectory on the surface of the workpiece.

10. A computer-readable storage medium, characterized in that, The system stores multiple instructions adapted for loading by a processor and executing the steps of the 2D vision-based trajectory localization method according to any one of claims 1 to 8.

Images