Track extraction method and device, readable storage medium and robot control system

Abstract

The invention provides a track extraction method and device, a readable storage medium and a robot control system. The track extraction method comprises the following steps: controlling a first camera to shoot a first image; drawing a first track based on the first image, wherein the first track comprises M first track points; performing pixel coordinate extraction on the drawn first image to obtain M first pixel coordinates corresponding to the M first track points; determining M first physical coordinates corresponding to the M first pixel coordinates according to a coordinate system conversion relation between a first coordinate system of the first camera and a second coordinate system of the robot; and based on the M first physical coordinates, determining a first moving track when the robot machines the to-be-machined surface. In the process, it can be ensured that the extracted trajectory is not distorted, and the physical coordinate system of the trajectory points and the coordinate system of real production equipment are unified, and meanwhile the trajectory extraction cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of control technology, and more specifically, to a trajectory extraction method, apparatus, readable storage medium, and robot control system. Background Technology

[0002] In related technical solutions, robot trajectory extraction in a plane is mostly achieved through three methods: teaching, offline programming, and online extraction. However, the above trajectory extraction methods have problems such as trajectory distortion, incompatibility between the physical coordinate system of the trajectory points and the coordinate system of the actual production equipment, and high extraction costs.

[0003] Specifically, trajectories extracted using the teaching-based trajectory extraction method are prone to distortion. If the robot operates according to the trajectory extracted using the teaching-based trajectory extraction method, it will lead to a decline in product quality.

[0004] Offline programming trajectory extraction methods often use 3D models to extract trajectories and then discretize them to obtain trajectory points. However, the physical coordinate system of the trajectory points cannot be unified with the coordinate system of the actual production equipment.

[0005] Online trajectory extraction uses reverse modeling of 3D sensors to extract trajectory points. While it can extract trajectories, it suffers from poor real-time performance, high cost, and long debugging cycle when used in planar trajectory extraction scenarios. Summary of the Invention

[0006] The present invention aims to at least solve the technical problems existing in the prior art or related technologies, such as trajectory distortion, incompatibility between the physical coordinate system of trajectory points and the coordinate system of actual production equipment, and high extraction cost.

[0007] Therefore, a first aspect of the present invention is to provide a trajectory extraction method.

[0008] A second aspect of the present invention is that a trajectory extraction device is provided.

[0009] A third aspect of the invention is that it provides another trajectory extraction device.

[0010] A fourth aspect of the present invention is that a readable storage medium is provided.

[0011] A fifth aspect of the present invention is that a robot control system is provided.

[0012] In view of the above, according to a first aspect of the present invention, the present invention provides a trajectory extraction method, comprising: controlling a first camera to capture a first image, the first image including an image of a surface to be processed on a workpiece, the first camera being mounted on a robot; drawing a first trajectory based on the first image, the first trajectory including M first trajectory points, M being a positive integer greater than or equal to 1; extracting pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to the M first trajectory points; determining M first physical coordinates corresponding to the M first pixel coordinates according to the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, the first physical coordinates being the position coordinates of the robot in the second coordinate system; and determining a first running trajectory of the robot processing the surface to be processed based on the M first physical coordinates.

[0013] This invention proposes a trajectory extraction method. By running the above trajectory extraction method, the trajectory extraction can be performed. In this process, a first camera is used to capture an image of the surface to be processed on the workpiece to obtain a first image. A first trajectory is drawn based on the first image. By extracting M first trajectory points on the first trajectory, the first running trajectory of the robot when processing the surface to be processed can be obtained.

[0014] In this process, by utilizing the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, the coordinate system of the trajectory points can be unified with the coordinate system of the production equipment. Compared with the online trajectory extraction method, the trajectory extraction method proposed in this invention can reduce the cost of trajectory extraction while ensuring that the extracted trajectory is not distorted and that the physical coordinate system of the trajectory points is unified with the coordinate system of the real production equipment.

[0015] In addition, the trajectory extraction method proposed in this invention has the following additional technical features.

[0016] In some technical solutions, optionally, drawing a first trajectory based on a first image specifically includes: receiving a first input for drawing marker points; responding to the first input, displaying at least one marker point based on the first image; drawing a first trajectory based on at least one marker point; receiving a second input for entering a first number, where the first number is a value of M; and responding to the second input, dividing the first trajectory into equal distances to obtain M first trajectory points.

[0017] In this technical solution, considering that when the user manually draws the first trajectory, there may be jitter or stuttering, the first trajectory drawn will not be smooth. If the first running trajectory is generated directly according to the unsmooth first trajectory, the robot will stutter when running according to the generated first running trajectory, which will affect the robot's processing efficiency.

[0018] To reduce the occurrence of the above situation, after the first trajectory is drawn, the first input of the user-drawn marker points is received first. The marker points can be understood as the waypoints of the first trajectory. Before drawing the first trajectory, the marker points are drawn in advance, and the first trajectory can be automatically generated based on the marker points drawn by the user.

[0019] In this process, since the first trajectory is automatically generated, the smoothness of the first trajectory can be ensured, reducing the chance of the robot getting stuck when running according to the generated first trajectory, and improving the robot's processing efficiency.

[0020] In some technical solutions, optionally, before controlling the first camera to capture the first image, the method further includes: calibrating the first camera and the robot using an N-point calibration method or a calibration plate calibration method to obtain the coordinate system transformation relationship.

[0021] In the above technical solution, the corresponding calibration method can be selected according to the actual use needs to obtain the coordinate system transformation relationship.

[0022] In some technical solutions, the trajectory extraction method optionally further includes: when the position of the workpiece changes, controlling a first camera to capture a second image, the second image including an image of the surface to be processed, the second image including second pixel coordinates and third pixel coordinates; determining a coordinate system correction relationship based on the second pixel coordinates, third pixel coordinates, fourth pixel coordinates and fifth pixel coordinates, the fourth pixel coordinates and fifth pixel coordinates being two pixel coordinates in the first image, the second pixel coordinates and fourth pixel coordinates corresponding to a first position on the surface to be processed, the third pixel coordinates and fifth pixel coordinates corresponding to a second position on the surface to be processed, the first position and the second position being non-overlapping positions; and correcting the first running trajectory based on the coordinate system correction relationship.

[0023] In this technical solution, if the robot continues to operate using the previous initial trajectory when the workpiece's position changes, operational anomalies will occur. These anomalies include operation failure or inability to operate.

[0024] To reduce the occurrence of the above situation, in the technical solution of the present invention, when a change in the position of the workpiece is detected, the first camera is controlled to take another picture to obtain a second image including an image of the surface to be processed. The pixel coordinates corresponding to two different positions on the surface to be processed are found in the first image and the second image respectively to obtain the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate and the fifth pixel coordinate, so as to determine the position change of the workpiece, that is, the coordinate system correction relationship, based on the above pixel coordinates.

[0025] In this process, by determining the coordinate system correction relationship, the first running trajectory can be corrected using the coordinate system correction relationship, so that when the position of the workpiece changes, the robot adopts the corrected first running trajectory to operate, thus ensuring the reliability of the robot operation.

[0026] Wherein, the second pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the second graphic, the third pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the second graphic, the fourth pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the first graphic, and the fifth pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the first graphic.

[0027] In some technical solutions, optionally, a coordinate system correction relationship is determined based on the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate, and the fifth pixel coordinate. Specifically, this includes: determining the second physical coordinate corresponding to the second pixel coordinate, the third physical coordinate corresponding to the third pixel coordinate, the fourth physical coordinate corresponding to the fourth pixel coordinate, and the fifth physical coordinate corresponding to the fifth pixel coordinate based on the coordinate system transformation relationship; determining the workpiece offset based on the second physical coordinate, the third physical coordinate, the fourth physical coordinate, and the fifth physical coordinate; and determining the coordinate system correction relationship based on the workpiece offset.

[0028] In this technical solution, the coordinates of the second pixel, the third pixel, the fourth pixel, and the fifth pixel are transformed into the second coordinate system using coordinate system transformation relationships, so as to obtain the second physical coordinates, the third physical coordinates, the fourth physical coordinates, and the fifth physical coordinates, so as to describe the positional changes of the workpiece in the second coordinate system.

[0029] Specifically, the workpiece offset includes a first offset and a second offset, wherein the first offset is the coordinate deviation between the fourth physical coordinate and the second physical coordinate, and the second offset is the coordinate deviation between the fifth physical coordinate and the third physical coordinate.

[0030] In this technical solution, by determining the offset of the workpiece, the offset of the workpiece can be known, and then the corresponding coordinate system correction relationship can be generated. The coordinate system correction relationship can be understood as the correction relationship determined based on the offset of the workpiece.

[0031] In some technical solutions, optionally, the robot includes a body and an end effector, with a first camera located on the end effector. Controlling the first camera to capture a first image specifically includes: controlling the end effector to move relative to the body until the first camera faces the surface to be processed; controlling the first camera to capture the first image.

[0032] In this technical solution, the end effector is controlled to move relative to the body in order to adjust the orientation of the first camera, thereby making the first camera face the surface to be processed in order to obtain an image of the surface to be processed.

[0033] In this process, a planar image of the surface to be processed can be obtained directly. Compared with the method of taking a three-dimensional image containing the workpiece and then processing the three-dimensional image to obtain a planar image of the surface to be processed, no additional processing of the first image is required, which improves the efficiency of trajectory extraction.

[0034] In some technical solutions, the first camera is controlled to take pictures when its orientation is perpendicular to the surface to be processed.

[0035] In some technical solutions, optionally, when the first image does not include all the surfaces to be processed, controlling the first camera to capture the first image further includes: controlling the end effector to move relative to the body until the first camera faces other surfaces of the surface to be processed; controlling the first camera to capture a third image, both the third image and the first image containing a fourth image, the fourth image being an image of a portion of the surfaces to be processed; drawing a first trajectory based on the first image, further including: drawing a second trajectory based on the third image, the second trajectory including O second trajectory points, where O is a positive integer greater than or equal to 1; extracting pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to M first trajectory points, further including: extracting pixel coordinates from the drawn third image. Extracting and obtaining O sixth pixel coordinates corresponding to O second trajectory points; determining M first physical coordinates corresponding to M first pixel coordinates based on the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, further including: determining O sixth physical coordinates corresponding to O sixth pixel coordinates based on the coordinate system transformation relationship, where the sixth physical coordinates are the position coordinates of the robot in the second coordinate system; determining the first running trajectory of the robot when processing the surface to be processed based on the M first physical coordinates, further including: determining the second running trajectory of the robot when processing the surface to be processed based on the O sixth physical coordinates; and stitching the first running trajectory and the second running trajectory together using a mean fusion method to obtain the running trajectory of the robot when processing the surface to be processed.

[0036] In this technical solution, considering that the field of view of the first camera cannot completely cover the entire surface to be processed, in the case that the first image does not include the entire surface to be processed, the end effector is also controlled to move relative to the body in order to capture other surfaces of the surface to be processed, so as to obtain a third image.

[0037] Among them, the other surfaces of the surface to be processed can be understood as the surfaces of the surface to be processed that were not photographed.

[0038] This technical solution enables trajectory extraction even when the surface to be processed is too large.

[0039] In the above technical solution, by limiting both the third image and the first image to include a fourth image, which is an image of a portion of the surface to be processed, the obtained third image and the first image have the same image, so that the O second trajectory points and the M first trajectory points have the same trajectory points. Then, after determining the O sixth physical coordinates corresponding to the O second trajectory points, the M first physical coordinates corresponding to the M first trajectory points and the O sixth physical coordinates are stitched together based on the mean fusion method, thereby realizing the stitching of the second running trajectory and the first running trajectory, thus obtaining a complete running trajectory.

[0040] In this process, the mean fusion method can make the spliced ​​running trajectory smoother, making the robot run more smoothly according to the spliced ​​running trajectory.

[0041] In some technical solutions, the trajectory extraction method may optionally include: determining a trajectory program based on a first running trajectory; wherein, when the robot executes the trajectory program, the robot processes the surface to be processed according to the first running trajectory.

[0042] In this technical solution, a corresponding trajectory program can be generated based on the first running trajectory without user operation, thereby reducing the degree of user involvement and realizing the automatic generation of the trajectory program.

[0043] The trajectory program is the program that the robot runs.

[0044] According to a second aspect of the present invention, a trajectory extraction device is provided, comprising: a control unit for controlling a first camera to capture a first image, the first image including an image of a surface to be processed on a workpiece, the first camera being mounted on a robot; a drawing unit for drawing a first trajectory based on the first image, the first trajectory including M first trajectory points, M being a positive integer greater than or equal to 1; an extraction unit for extracting pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to the M first trajectory points; a transformation unit for determining M first physical coordinates corresponding to the M first pixel coordinates according to the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, the first physical coordinates being the position coordinates of the robot in the second coordinate system; and a determination unit for determining a first running trajectory of the robot processing the surface to be processed based on the M first physical coordinates.

[0045] This invention proposes a trajectory extraction device that can extract trajectories by using a first camera to capture an image of the surface to be processed on a workpiece, obtaining a first image, and drawing a first trajectory based on the first image. By extracting M first trajectory points on the first trajectory, the first running trajectory of the robot when processing the surface to be processed can be obtained.

[0046] In this process, by utilizing the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, the coordinate system of the trajectory points can be unified with the coordinate system of the production equipment. Compared with the online trajectory extraction method, the trajectory extraction method proposed in this invention can reduce the cost of trajectory extraction while ensuring that the extracted trajectory is not distorted and that the physical coordinate system of the trajectory points is unified with the coordinate system of the real production equipment.

[0047] In addition, the trajectory extraction device proposed in this invention has the following additional technical features.

[0048] In some technical solutions, optionally, the drawing unit is specifically used for: receiving a first input for drawing marker points; responding to the first input, displaying at least one marker point based on a first image; drawing a first trajectory based on at least one marker point; receiving a second input for entering a first number, wherein the first number is a value of M; and responding to the second input, dividing the first trajectory into equal distances to obtain M first trajectory points.

[0049] In this technical solution, considering that when the user manually draws the first trajectory, there may be jitter or stuttering, the first trajectory drawn will not be smooth. If the first running trajectory is generated directly according to the unsmooth first trajectory, the robot will stutter when running according to the generated first running trajectory, which will affect the robot's processing efficiency.

[0050] To reduce the occurrence of the above situation, after the first trajectory is drawn, the first input of the user-drawn marker points is received first. The marker points can be understood as the waypoints of the first trajectory. Before drawing the first trajectory, the marker points are drawn in advance, and the first trajectory can be automatically generated based on the marker points drawn by the user.

[0051] In this process, since the first trajectory is automatically generated, the smoothness of the first trajectory can be ensured, reducing the chance of the robot getting stuck when running according to the generated first trajectory, and improving the robot's processing efficiency.

[0052] In some technical solutions, optionally, before controlling the first camera to capture the first image, the control unit is also used to: calibrate the first camera and the robot using an N-point calibration method or a calibration board calibration method to obtain the coordinate system transformation relationship.

[0053] In the above technical solution, the corresponding calibration method can be selected according to the actual use needs to obtain the coordinate system transformation relationship.

[0054] In some technical solutions, optionally, when the position of the workpiece changes, the control unit is further configured to control the first camera to capture a second image, the second image including an image of the surface to be processed, the second image including second pixel coordinates and third pixel coordinates; the determining unit is further configured to determine a coordinate system correction relationship based on the second pixel coordinates, the third pixel coordinates, the fourth pixel coordinates and the fifth pixel coordinates, the fourth pixel coordinates and the fifth pixel coordinates being two pixel coordinates in the first image, the second pixel coordinates and the fourth pixel coordinates corresponding to a first position on the surface to be processed, the third pixel coordinates and the fifth pixel coordinates corresponding to a second position on the surface to be processed, the first position and the second position being non-overlapping positions; and to correct the first running trajectory based on the coordinate system correction relationship.

[0055] In this technical solution, if the robot continues to operate using the previous initial trajectory when the workpiece's position changes, operational anomalies will occur. These anomalies include operation failure or inability to operate.

[0056] To reduce the occurrence of the above situation, in the technical solution of the present invention, when a change in the position of the workpiece is detected, the first camera is controlled to take another picture to obtain a second image including an image of the surface to be processed. The pixel coordinates corresponding to two different positions on the surface to be processed are found in the first image and the second image respectively to obtain the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate and the fifth pixel coordinate, so as to determine the position change of the workpiece, that is, the coordinate system correction relationship, based on the above pixel coordinates.

[0057] In this process, by determining the coordinate system correction relationship, the first running trajectory can be corrected using the coordinate system correction relationship, so that when the position of the workpiece changes, the robot adopts the corrected first running trajectory to operate, thus ensuring the reliability of the robot operation.

[0058] Wherein, the second pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the second graphic, the third pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the second graphic, the fourth pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the first graphic, and the fifth pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the first graphic.

[0059] In some technical solutions, optionally, the determining unit is specifically used for: determining the second physical coordinate corresponding to the second pixel coordinate, the third physical coordinate corresponding to the third pixel coordinate, the fourth physical coordinate corresponding to the fourth pixel coordinate, and the fifth physical coordinate corresponding to the fifth pixel coordinate based on the coordinate system transformation relationship; determining the workpiece offset based on the second physical coordinate, the third physical coordinate, the fourth physical coordinate, and the fifth physical coordinate; and determining the coordinate system correction relationship based on the workpiece offset.

[0060] In this technical solution, the coordinates of the second pixel, the third pixel, the fourth pixel, and the fifth pixel are transformed into the second coordinate system using coordinate system transformation relationships, so as to obtain the second physical coordinates, the third physical coordinates, the fourth physical coordinates, and the fifth physical coordinates, so as to describe the positional changes of the workpiece in the second coordinate system.

[0061] Specifically, the workpiece offset includes a first offset and a second offset, wherein the first offset is the coordinate deviation between the fourth physical coordinate and the second physical coordinate, and the second offset is the coordinate deviation between the fifth physical coordinate and the third physical coordinate.

[0062] In this technical solution, by determining the offset of the workpiece, the offset of the workpiece can be known, and then the corresponding coordinate system correction relationship can be generated. The coordinate system correction relationship can be understood as the correction relationship determined based on the offset of the workpiece.

[0063] In some technical solutions, optionally, the robot includes a body and an end effector, a first camera is located on the end effector, and a control unit is specifically used to: control the end effector to move relative to the body until the first camera faces the surface to be processed; control the first camera to take a picture and obtain a first image.

[0064] In this technical solution, the end effector is controlled to move relative to the body in order to adjust the orientation of the first camera, thereby making the first camera face the surface to be processed in order to obtain an image of the surface to be processed.

[0065] In this process, a planar image of the surface to be processed can be obtained directly. Compared with the method of taking a three-dimensional image containing the workpiece and then processing the three-dimensional image to obtain a planar image of the surface to be processed, no additional processing of the first image is required, which improves the efficiency of trajectory extraction.

[0066] In some technical solutions, the first camera is controlled to take pictures when its orientation is perpendicular to the surface to be processed.

[0067] In some technical solutions, optionally, if the first image does not include the entire surface to be processed, the control unit is further configured to: control the end effector to move relative to the body until the first camera faces other surfaces of the surface to be processed; control the first camera to capture a third image, both the third image and the first image containing a fourth image, the fourth image being an image of a portion of the surface to be processed; the drawing unit is further configured to: draw a second trajectory based on the third image, the second trajectory including O second trajectory points, where O is a positive integer greater than or equal to 1; the extraction unit is further configured to: extract pixel coordinates from the drawn third image to obtain O sixth pixel coordinates corresponding to the O second trajectory points; the transformation unit is further configured to: determine O sixth physical coordinates corresponding to the O sixth pixel coordinates according to the coordinate system transformation relationship, the sixth physical coordinates being the robot's position coordinates in the second coordinate system; the determination unit is further configured to: determine the second running trajectory of the robot processing the surface to be processed based on the O sixth physical coordinates; and stitch the first running trajectory and the second running trajectory together using a mean fusion method to obtain the running trajectory of the robot processing the surface to be processed.

[0068] In this technical solution, considering that the field of view of the first camera cannot completely cover the entire surface to be processed, in the case that the first image does not include the entire surface to be processed, the end effector is also controlled to move relative to the body in order to capture other surfaces of the surface to be processed, so as to obtain a third image.

[0069] Among them, the other surfaces of the surface to be processed can be understood as the surfaces of the surface to be processed that were not photographed.

[0070] This technical solution enables trajectory extraction even when the surface to be processed is too large.

[0071] In the above technical solution, by limiting both the third image and the first image to include a fourth image, which is an image of a portion of the surface to be processed, the obtained third image and the first image have the same image, so that the O second trajectory points and the M first trajectory points have the same trajectory points. Then, after determining the O sixth physical coordinates corresponding to the O second trajectory points, the M first physical coordinates corresponding to the M first trajectory points and the O sixth physical coordinates are stitched together based on the mean fusion method, thereby realizing the stitching of the second running trajectory and the first running trajectory, thus obtaining a complete running trajectory.

[0072] In this process, the mean fusion method can make the spliced ​​running trajectory smoother, making the robot run more smoothly according to the spliced ​​running trajectory.

[0073] In some technical solutions, optionally, the determining unit is also used to: determine a trajectory program based on a first running trajectory; wherein, when the robot executes the trajectory program, the robot processes the surface to be processed according to the first running trajectory.

[0074] In this technical solution, a corresponding trajectory program can be generated based on the first running trajectory without user operation, thereby reducing the degree of user involvement and realizing the automatic generation of the trajectory program.

[0075] The trajectory program is the program that the robot runs.

[0076] According to a third aspect of the present invention, another trajectory extraction apparatus is provided, comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the trajectory extraction method as described above.

[0077] According to a fourth aspect of the present invention, the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the trajectory extraction method as described above.

[0078] According to a fifth aspect of the present invention, a robot control system is provided, comprising: a trajectory extraction device as described above; and / or a readable storage medium as described above.

[0079] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0080] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0081] Figure 1 This illustration shows one of the flowcharts of a trajectory extraction method according to an embodiment of the present invention;

[0082] Figure 2 The second schematic flowchart of a trajectory extraction method according to an embodiment of the present invention is shown;

[0083] Figure 3 A schematic diagram illustrating the principle of trajectory extraction in an embodiment of the present invention is shown;

[0084] Figure 4 A schematic diagram showing the connection of the body, end effector, and first camera in an embodiment of the present invention is shown;

[0085] Figure 5 A schematic block diagram of a trajectory extraction device according to an embodiment of the present invention is shown;

[0086] Figure 6 A schematic block diagram of another trajectory extraction device in an embodiment of the present invention is shown.

[0087] in, Figure 4 The correspondence between the reference numerals and component names in the attached drawings is as follows:

[0088] 402 Body, 404 End effector, 406 First camera. Detailed Implementation

[0089] To better understand the above aspects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0090] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0091] In one embodiment of this application, such as Figure 1 As shown, a trajectory extraction method is provided, including:

[0092] Step 102: Control the first camera to capture the first image. The first image includes an image of the surface to be processed on the workpiece. The first camera is located on the robot.

[0093] Step 104: Draw a first trajectory based on the first image. The first trajectory includes M first trajectory points, where M is a positive integer greater than or equal to 1.

[0094] Step 106: Extract pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to M first trajectory points;

[0095] Step 108: Determine the M first physical coordinates corresponding to the M first pixel coordinates based on the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot. The first physical coordinates are the position coordinates of the robot in the second coordinate system.

[0096] Step 110: Based on the M first physical coordinates, determine the first running trajectory of the robot when processing the surface to be processed.

[0097] This invention proposes a trajectory extraction method. By running the above trajectory extraction method, the trajectory extraction can be performed. In this process, a first camera is used to capture an image of the surface to be processed on the workpiece to obtain a first image. A first trajectory is drawn based on the first image. By extracting M first trajectory points on the first trajectory, the first running trajectory of the robot when processing the surface to be processed can be obtained.

[0098] In this process, by utilizing the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, the coordinate system of the trajectory points can be unified with the coordinate system of the production equipment. Compared with the online trajectory extraction method, the trajectory extraction method proposed in this invention can reduce the cost of trajectory extraction while ensuring that the extracted trajectory is not distorted and that the physical coordinate system of the trajectory points is unified with the coordinate system of the real production equipment.

[0099] In some embodiments, the first trajectory is optionally the processing path that the user wants the robot to follow when processing the surface to be processed.

[0100] In some embodiments, optionally, a first image captured by a first camera is displayed in a user interface (UI), and the user draws a first trajectory on the first image using drawing controls in the user interface.

[0101] Specifically, the first image captured by the first camera is stored locally on a first device, which is also the device that executes the trajectory extraction method proposed in this invention. Based on this, the user interface of the first device reads and displays the first image stored locally, so that the user can draw the first trajectory on the first image using the drawing controls in the user interface.

[0102] In some embodiments, the drawing control may optionally be a mouse, in which case the user can move the mouse to draw a first trajectory on a first image displayed in the user interface.

[0103] In some embodiments, the first trajectory drawn by the drawing control can be modified. In this case, if the first trajectory does not meet the user's needs, it can be redrawn.

[0104] In some embodiments, optionally, a first trajectory point can be understood as a point on a first trajectory. By determining M first trajectory points on the first trajectory, the first trajectory can be characterized using discrete M first trajectory points, thereby determining the first running trajectory.

[0105] In this process, compared with the trajectory extraction scheme that determines the first running trajectory based on all points in the first trajectory, the present invention uses M discrete first trajectory points to represent the first trajectory, which can reduce the amount of computation required in the process of determining the first running trajectory, thereby reducing the time cost and computational cost required for trajectory extraction.

[0106] By extracting the pixel coordinates of the first drawn image, the pixel coordinates corresponding to the M first trajectory points can be obtained, that is, the M first pixel coordinates can be obtained. The first pixel coordinates are also the pixel coordinates of the first trajectory points in the first image. By extracting the pixel coordinates, the planar trajectory points with pixel-level precision can be read.

[0107] In this process, the expression results of M first trajectory points in the first coordinate system can be obtained. Then, by using the coordinate system transformation relationship, the expression results of M first trajectory points in the first coordinate system can be transformed to the expression results in the second coordinate system, that is, the M first physical coordinates can be obtained. In this process, while realizing the unification between the physical coordinate system of the trajectory points and the coordinate system of the actual production equipment, the pixel-level standard is also achieved. Therefore, the accuracy of the determined first running estimate is improved.

[0108] In some embodiments, optionally, drawing a first trajectory based on a first image specifically includes: receiving a first input for drawing marker points; in response to the first input, displaying at least one marker point based on the first image; drawing a first trajectory based on at least one marker point; receiving a second input for entering a first number, wherein the first number is a value of M; and in response to the second input, dividing the first trajectory into equal distances to obtain M first trajectory points.

[0109] In this embodiment, considering that there may be jitter or stuttering when the user manually draws the first trajectory, the first trajectory drawn will not be smooth. If the first running trajectory is generated directly according to the unsmooth first trajectory, the robot will stutter when running according to the generated first running trajectory, which will affect the robot's processing efficiency.

[0110] To reduce the occurrence of the above situation, after the first trajectory is drawn, the first input of the user-drawn marker points is received first. The marker points can be understood as the waypoints of the first trajectory. Before drawing the first trajectory, the marker points are drawn in advance, and the first trajectory can be automatically generated based on the marker points drawn by the user.

[0111] In this process, since the first trajectory is automatically generated, the smoothness of the first trajectory can be ensured, reducing the chance of the robot getting stuck when running according to the generated first trajectory, and improving the robot's processing efficiency.

[0112] In the above embodiment, the user can enter a first number of second inputs to set the number of first trajectory points. During this process, the user can customize the first trajectory points on the first trajectory.

[0113] In this process, if the accuracy requirement of the generated first running trajectory is relatively high, the value of M can be increased, while if the accuracy requirement of the generated first running trajectory is relatively low, the value of M can be decreased, so as to meet the actual needs of different scenarios.

[0114] In the above embodiments, by dividing the first trajectory into equal parts, the smoothness of different positions on the generated first running trajectory is the same, ensuring the smoothness of the robot's processing operation.

[0115] In some embodiments, the first input may optionally be understood as a click operation on the first image using drawing controls in the user interface.

[0116] In some embodiments, optionally, before controlling the first camera to capture the first image, the method further includes: calibrating the first camera and the robot using an N-point calibration method or a calibration plate calibration method to obtain a coordinate system transformation relationship.

[0117] The N-point calibration is primarily used to calculate the transformation relationship between the image coordinate system of the first camera (i.e., the first coordinate system in this invention) and the physical coordinate system of the motion mechanism (i.e., the second coordinate system of the robot in this application). By inputting multiple one-to-one corresponding image pixels and physical points of the motion mechanism, the transformation relationship between the two coordinate systems (i.e., the coordinate system transformation relationship in this invention, also known as the calibration file) is calculated. This method is widely used in application scenarios where vision-guided motion mechanisms are required to perform actions such as grasping, correction, and alignment.

[0118] The calibration process involves inputting a calibration board image, calculating and outputting the transformation relationship (i.e., the calibration file) from the image coordinate system to the physical coordinate system based on the image coordinate position and the physical coordinate position. Once the transformation relationship is obtained, any pixel on the calibration board plane in the image can be transformed to the corresponding physical coordinate system.

[0119] In the above embodiments, the corresponding calibration method can be selected according to actual usage needs to obtain the coordinate system transformation relationship.

[0120] In some embodiments, the coordinate system transformation relation is optionally a 3×3 affine transformation matrix.

[0121] In some embodiments, the trajectory extraction method may optionally further include: when the position of the workpiece changes, controlling a first camera to capture a second image, the second image including an image of the surface to be processed, the second image including second pixel coordinates and third pixel coordinates; determining a coordinate system correction relationship based on the second pixel coordinates, third pixel coordinates, fourth pixel coordinates and fifth pixel coordinates, the fourth pixel coordinates and fifth pixel coordinates being two pixel coordinates in the first image, the second pixel coordinates and fourth pixel coordinates corresponding to a first position on the surface to be processed, the third pixel coordinates and fifth pixel coordinates corresponding to a second position on the surface to be processed, the first position and the second position being non-overlapping positions; and correcting the first running trajectory based on the coordinate system correction relationship.

[0122] In this embodiment, if the robot continues to operate using the previous first running trajectory when the position of the workpiece changes, an operational anomaly will occur. This anomaly includes operation failure or inability to operate.

[0123] To reduce the occurrence of the above situation, in an embodiment of the present invention, when a change in the position of the workpiece is detected, the first camera is controlled to take another picture to obtain a second image including an image of the surface to be processed. The pixel coordinates corresponding to two different positions on the surface to be processed are found in the first image and the second image respectively to obtain the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate and the fifth pixel coordinate, so as to determine the position change of the workpiece, that is, the coordinate system correction relationship, based on the above pixel coordinates.

[0124] In this process, by determining the coordinate system correction relationship, the first running trajectory can be corrected using the coordinate system correction relationship, so that when the position of the workpiece changes, the robot adopts the corrected first running trajectory to operate, thus ensuring the reliability of the robot operation.

[0125] Wherein, the second pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the second graphic, the third pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the second graphic, the fourth pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the first graphic, and the fifth pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the first graphic.

[0126] In some embodiments, optionally, a coordinate system correction relationship is determined based on the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate, and the fifth pixel coordinate. Specifically, this includes: determining the second physical coordinate corresponding to the second pixel coordinate, the third physical coordinate corresponding to the third pixel coordinate, the fourth physical coordinate corresponding to the fourth pixel coordinate, and the fifth physical coordinate corresponding to the fifth pixel coordinate based on the coordinate system transformation relationship; determining the workpiece offset based on the second physical coordinate, the third physical coordinate, the fourth physical coordinate, and the fifth physical coordinate; and determining the coordinate system correction relationship based on the workpiece offset.

[0127] In this embodiment, the second pixel coordinates, third pixel coordinates, fourth pixel coordinates, and fifth pixel coordinates are transformed into the second coordinate system using coordinate system transformation relationships to obtain the second physical coordinates, third physical coordinates, fourth physical coordinates, and fifth physical coordinates, so as to describe the positional changes of the workpiece in the second coordinate system.

[0128] Specifically, the workpiece offset includes a first offset and a second offset, wherein the first offset is the coordinate deviation between the fourth physical coordinate and the second physical coordinate, and the second offset is the coordinate deviation between the fifth physical coordinate and the third physical coordinate.

[0129] In this embodiment, by determining the offset of the workpiece, the offset of the workpiece can be known, and then a corresponding coordinate system correction relationship can be generated. The coordinate system correction relationship can be understood as a correction relationship determined based on the offset of the workpiece.

[0130] In some embodiments, the robot optionally includes a body and an end effector, with a first camera disposed on the end effector. Controlling the first camera to capture a first image specifically includes: controlling the end effector to move relative to the body until the first camera faces the surface to be processed; controlling the first camera to capture the first image.

[0131] In this embodiment, the end effector is controlled to move relative to the body to adjust the orientation of the first camera, thereby making the first camera face the surface to be processed in order to obtain an image of the surface to be processed.

[0132] In this process, a planar image of the surface to be processed can be obtained directly. Compared with the method of taking a three-dimensional image containing the workpiece and then processing the three-dimensional image to obtain a planar image of the surface to be processed, no additional processing of the first image is required, which improves the efficiency of trajectory extraction.

[0133] In some embodiments, the first camera is controlled to take pictures when it is oriented perpendicular to the surface to be processed.

[0134] In some embodiments, optionally, when the first image does not include all the surfaces to be processed, controlling the first camera to capture the first image further includes: controlling the end effector to move relative to the body until the first camera faces other surfaces of the surface to be processed; controlling the first camera to capture a third image, both the third image and the first image including a fourth image, the fourth image being an image of a portion of the surfaces to be processed; drawing a first trajectory based on the first image, further including: drawing a second trajectory based on the third image, the second trajectory including O second trajectory points, where O is a positive integer greater than or equal to 1; extracting pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to M first trajectory points, further including: extracting pixel coordinates from the drawn third image. The process involves: obtaining O sixth pixel coordinates corresponding to O second trajectory points; determining M first physical coordinates corresponding to M first pixel coordinates based on the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot; determining O sixth physical coordinates corresponding to O sixth pixel coordinates based on the coordinate system transformation relationship, where the sixth physical coordinates are the robot's position coordinates in the second coordinate system; determining a first running trajectory of the robot when processing the surface to be processed based on the M first physical coordinates; determining a second running trajectory of the robot when processing the surface to be processed based on the O sixth physical coordinates; and stitching the first running trajectory and the second running trajectory together using a mean fusion method to obtain the running trajectory of the robot when processing the surface to be processed.

[0135] In this embodiment, considering that the field of view of the first camera cannot completely cover the entire surface to be processed, if the first image does not include the entire surface to be processed, the end effector is also controlled to move relative to the body in order to capture other surfaces of the surface to be processed, thereby obtaining a third image.

[0136] Among them, the other surfaces of the surface to be processed can be understood as the surfaces of the surface to be processed that were not photographed.

[0137] In this embodiment, trajectory extraction can be achieved even when the surface to be processed is too large.

[0138] In the above embodiments, by limiting both the third image and the first image to include a fourth image, where the fourth image is an image of a portion of the surface to be processed, the obtained third image and the first image have the same image, so that the O second trajectory points and the M first trajectory points have the same trajectory points. Then, after determining the O sixth physical coordinates corresponding to the O second trajectory points, the M first physical coordinates corresponding to the M first trajectory points and the O sixth physical coordinates are stitched together based on the mean fusion method, thereby realizing the stitching of the second running trajectory and the first running trajectory, thus obtaining a complete running trajectory.

[0139] In this process, the mean fusion method can make the spliced ​​running trajectory smoother, making the robot run more smoothly according to the spliced ​​running trajectory.

[0140] In some embodiments, the trajectory extraction method may optionally further include: determining a trajectory program based on a first running trajectory; wherein, when the robot executes the trajectory program, the robot processes the surface to be processed according to the first running trajectory.

[0141] In this embodiment, a corresponding trajectory program can be generated based on the first running trajectory without user operation, thereby reducing the degree of user involvement and realizing the automatic generation of the trajectory program.

[0142] The trajectory program is the program that the robot runs.

[0143] In one embodiment, such as Figure 2 and Figure 3 As shown, trajectory extraction methods include:

[0144] Step 202: Set up the robot's 2D vision hardware environment.

[0145] The 2D vision hardware environment includes a first camera, which requires the lens of the first camera to be perpendicular to the surface of the workpiece where the trajectory to be extracted is located. Here, 2D means two-dimensional space.

[0146] Step 204, First camera calibration.

[0147] Step 206: Acquire images and extract the pixel coordinates of the trajectory points.

[0148] Step 208: Convert the pixel coordinates of the trajectory points to physical coordinates in the robot coordinate system.

[0149] Step 210: If a complete trajectory cannot be extracted in one image acquisition, then the trajectory is stitched together based on the mean value fusion.

[0150] Step 212: Assemble the communication data of the physical coordinates of the trajectory points and send it to the robot.

[0151] Step 214: The robot parses the communication data to obtain the coordinates of the trajectory points and programs the robot trajectory generation program.

[0152] Step 216: Calculate the offset of the workpiece to be processed.

[0153] Specifically, the image acquired in step 206 is named Image1, and the image of the workpiece to be processed is acquired and named Image2. Using vision software, two non-overlapping feature points with the largest possible distance between them are determined on the surface of the workpiece to be processed, namely the first position and the second position. The corresponding pixels on Image1 are {Px1, Px2}, and the corresponding pixels on Image2 are {Px3, Px4}. Based on the calibration results of the first camera, the pixel coordinates are converted into physical coordinates in the robot coordinate system, resulting in two feature physical points {Py1, Ph2} for Image1 and {Py3, Py4} for Image2. Then, the offset of the workpiece to be processed is calculated based on {Py1, Ph2} and {Py3, Py4}.

[0154] Step 218: Compensate the robot coordinate system based on the offset of the workpiece to be processed.

[0155] Specifically, the coordinate system correction relationship is determined based on the offset of the workpiece to be processed, so as to compensate the robot coordinate system.

[0156] Step 220: Complete the trajectory correction.

[0157] in, Figure 3 The feature surface image, which is also the image of the surface to be processed in this application, and the vision terminal, which is also the first camera in this application, after determining the physical coordinates of the complete trajectory points, assemble them into a string in a certain format, and then send the physical coordinates of the complete trajectory points to the robot end through communication between the first camera and the robot.

[0158] The string can be sent to the robot using either Transmission Control Protocol (TCP) or Internet Protocol (IP).

[0159] Transmission Control Protocol (TCP) is a connection-oriented, reliable, byte-stream-based transport layer communication protocol.

[0160] After determining the workpiece offset, the workpiece offset can be communicated and assembled, and transmitted to the robot via communication between the first camera and the robot, so that the running trajectory can be corrected on the robot.

[0161] In this embodiment, pixel-level precision planar trajectory points that the robot can directly execute are extracted, and the trajectory splicing function is realized based on mean fusion. When the workpiece position changes, the overall trajectory correction can be completed in 100 milliseconds or less.

[0162] The embodiments proposed in this application possess the characteristics of low cost and high flexibility, and are suitable for various planar trajectory processing scenarios. These planar trajectory processing scenarios include, but are not limited to, adhesive application and cutting scenarios.

[0163] For example, such as Figure 4 As shown, the end effector 404 in the robot is connected to the body 402. The end effector 404 is movable relative to the body 402. The first camera 406 is located on the end effector 404 and moves with the end effector 404. When the end effector 404 moves to the surface to be processed of the workpiece, it takes a picture of the surface to be processed of the workpiece.

[0164] In one embodiment, such as Figure 5 As shown, the present invention provides a trajectory extraction device 500, comprising: a control unit 502 for controlling a first camera to capture a first image, the first image including an image of the surface to be processed on a workpiece, the first camera being mounted on a robot; a drawing unit 504 for drawing a first trajectory based on the first image, the first trajectory including M first trajectory points, M being a positive integer greater than or equal to 1; an extraction unit 506 for extracting pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to the M first trajectory points; a transformation unit 508 for determining M first physical coordinates corresponding to the M first pixel coordinates according to the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, the first physical coordinates being the position coordinates of the robot in the second coordinate system; and a determination unit 510 for determining a first running trajectory of the robot processing the surface to be processed based on the M first physical coordinates.

[0165] This invention proposes a trajectory extraction device 500, which can extract trajectories by using a first camera to capture an image of the surface to be processed on the workpiece to obtain a first image, and drawing a first trajectory based on the first image. By extracting M first trajectory points on the first trajectory, the first running trajectory of the robot when processing the surface to be processed can be obtained.

[0166] In this process, by utilizing the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, the coordinate system of the trajectory points can be unified with the coordinate system of the production equipment. Compared with the online trajectory extraction method, the trajectory extraction method proposed in this invention can reduce the cost of trajectory extraction while ensuring that the extracted trajectory is not distorted and that the physical coordinate system of the trajectory points is unified with the coordinate system of the real production equipment.

[0167] In some embodiments, the first trajectory is optionally the processing path that the user wants the robot to follow when processing the surface to be processed.

[0168] In some embodiments, optionally, a first image captured by a first camera is displayed in a user interface (UI), and the user draws a first trajectory on the first image using drawing controls in the user interface.

[0169] Specifically, the first image captured by the first camera is stored locally on a first device, which is also the device that executes the trajectory extraction method proposed in this invention. Based on this, the user interface of the first device reads and displays the first image stored locally, so that the user can draw the first trajectory on the first image using the drawing controls in the user interface.

[0170] In some embodiments, the drawing control may optionally be a mouse, in which case the user can move the mouse to draw a first trajectory on a first image displayed in the user interface.

[0171] In some embodiments, the first trajectory drawn by the drawing control can be modified. In this case, if the first trajectory does not meet the user's needs, it can be redrawn.

[0172] In some embodiments, optionally, a first trajectory point can be understood as a point on a first trajectory. By determining M first trajectory points on the first trajectory, the first trajectory can be characterized using discrete M first trajectory points, thereby determining the first running trajectory.

[0173] In this process, compared with the trajectory extraction scheme that determines the first running trajectory based on all points in the first trajectory, the present invention uses M discrete first trajectory points to represent the first trajectory, which can reduce the amount of computation required in the process of determining the first running trajectory, thereby reducing the time cost and computational cost required for trajectory extraction.

[0174] By extracting the pixel coordinates of the first drawn image, the pixel coordinates corresponding to the M first trajectory points can be obtained, that is, the M first pixel coordinates can be obtained. The first pixel coordinates are also the pixel coordinates of the first trajectory points in the first image. By extracting the pixel coordinates, the planar trajectory points with pixel-level precision can be read.

[0175] In this process, the expression results of M first trajectory points in the first coordinate system can be obtained. Then, by using the coordinate system transformation relationship, the expression results of M first trajectory points in the first coordinate system can be transformed to the expression results in the second coordinate system, that is, the M first physical coordinates can be obtained. In this process, while realizing the unification between the physical coordinate system of the trajectory points and the coordinate system of the actual production equipment, the pixel-level standard is also achieved. Therefore, the accuracy of the determined first running estimate is improved.

[0176] In some embodiments, optionally, the drawing unit 504 is specifically configured to: receive a first input for drawing marker points; in response to the first input, display at least one marker point based on a first image; draw a first trajectory based on at least one marker point; receive a second input for entering a first number, wherein the first number is a value of M; and in response to the second input, divide the first trajectory into equal distances to obtain M first trajectory points.

[0177] In this embodiment, considering that there may be jitter or stuttering when the user manually draws the first trajectory, the first trajectory drawn will not be smooth. If the first running trajectory is generated directly according to the unsmooth first trajectory, the robot will stutter when running according to the generated first running trajectory, which will affect the robot's processing efficiency.

[0178] To reduce the occurrence of the above situation, after the first trajectory is drawn, the first input of the user-drawn marker points is received first. The marker points can be understood as the waypoints of the first trajectory. Before drawing the first trajectory, the marker points are drawn in advance, and the first trajectory can be automatically generated based on the marker points drawn by the user.

[0179] In this process, since the first trajectory is automatically generated, the smoothness of the first trajectory can be ensured, reducing the chance of the robot getting stuck when running according to the generated first trajectory, and improving the robot's processing efficiency.

[0180] In the above embodiment, the user can enter a first number of second inputs to set the number of first trajectory points. During this process, the user can customize the first trajectory points on the first trajectory.

[0181] In this process, if the accuracy requirement of the generated first running trajectory is relatively high, the value of M can be increased, while if the accuracy requirement of the generated first running trajectory is relatively low, the value of M can be decreased, so as to meet the actual needs of different scenarios.

[0182] In the above embodiments, by dividing the first trajectory into equal parts, the smoothness of different positions on the generated first running trajectory is the same, ensuring the smoothness of the robot's processing operation.

[0183] In some embodiments, the first input may optionally be understood as a click operation on the first image using drawing controls in the user interface.

[0184] In some embodiments, optionally, before controlling the first camera to capture the first image, the control unit 502 is further configured to: calibrate the first camera and the robot using an N-point calibration method or a calibration plate calibration method to obtain a coordinate system transformation relationship.

[0185] The N-point calibration is primarily used to calculate the transformation relationship between the image coordinate system of the first camera (i.e., the first coordinate system in this invention) and the physical coordinate system of the motion mechanism (i.e., the second coordinate system of the robot in this application). By inputting multiple one-to-one corresponding image pixels and physical points of the motion mechanism, the transformation relationship between the two coordinate systems (i.e., the coordinate system transformation relationship in this invention, also known as the calibration file) is calculated. This method is widely used in application scenarios where vision-guided motion mechanisms are required to perform actions such as grasping, correction, and alignment.

[0186] The calibration process involves inputting a calibration board image, calculating and outputting the transformation relationship (i.e., the calibration file) from the image coordinate system to the physical coordinate system based on the image coordinate position and the physical coordinate position. Once the transformation relationship is obtained, any pixel on the calibration board plane in the image can be transformed to the corresponding physical coordinate system.

[0187] In the above embodiments, the corresponding calibration method can be selected according to actual usage needs to obtain the coordinate system transformation relationship.

[0188] In some embodiments, the coordinate system transformation relation is optionally a 3×3 affine transformation matrix.

[0189] In some embodiments, optionally, when the position of the workpiece changes, the control unit 502 is further configured to control the first camera to capture a second image, the second image including an image of the surface to be processed, the second image including second pixel coordinates and third pixel coordinates; the determining unit 510 is further configured to determine a coordinate system correction relationship based on the second pixel coordinates, the third pixel coordinates, the fourth pixel coordinates and the fifth pixel coordinates, the fourth pixel coordinates and the fifth pixel coordinates being two pixel coordinates in the first image, the second pixel coordinates and the fourth pixel coordinates corresponding to a first position on the surface to be processed, the third pixel coordinates and the fifth pixel coordinates corresponding to a second position on the surface to be processed, the first position and the second position being non-overlapping positions; and to correct the first running trajectory based on the coordinate system correction relationship.

[0190] In this embodiment, if the robot continues to operate using the previous first running trajectory when the position of the workpiece changes, an operational anomaly will occur. This anomaly includes operation failure or inability to operate.

[0191] To reduce the occurrence of the above situation, in an embodiment of the present invention, when a change in the position of the workpiece is detected, the first camera is controlled to take another picture to obtain a second image including an image of the surface to be processed. The pixel coordinates corresponding to two different positions on the surface to be processed are found in the first image and the second image respectively to obtain the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate and the fifth pixel coordinate, so as to determine the position change of the workpiece, that is, the coordinate system correction relationship, based on the above pixel coordinates.

[0192] In this process, by determining the coordinate system correction relationship, the first running trajectory can be corrected using the coordinate system correction relationship, so that when the position of the workpiece changes, the robot adopts the corrected first running trajectory to operate, thus ensuring the reliability of the robot operation.

[0193] Wherein, the second pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the second graphic, the third pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the second graphic, the fourth pixel coordinate is the pixel coordinate of the first position of the surface to be processed in the first graphic, and the fifth pixel coordinate is the pixel coordinate of the second position of the surface to be processed in the first graphic.

[0194] In some embodiments, optionally, the determining unit 510 is specifically configured to: determine the second physical coordinate corresponding to the second pixel coordinate, the third physical coordinate corresponding to the third pixel coordinate, the fourth physical coordinate corresponding to the fourth pixel coordinate, and the fifth physical coordinate corresponding to the fifth pixel coordinate based on the coordinate system transformation relationship; determine the offset of the workpiece based on the second physical coordinate, the third physical coordinate, the fourth physical coordinate, and the fifth physical coordinate; and determine the coordinate system correction relationship based on the offset of the workpiece.

[0195] In this embodiment, the second pixel coordinates, third pixel coordinates, fourth pixel coordinates, and fifth pixel coordinates are transformed into the second coordinate system using coordinate system transformation relationships to obtain the second physical coordinates, third physical coordinates, fourth physical coordinates, and fifth physical coordinates, so as to describe the positional changes of the workpiece in the second coordinate system.

[0196] Specifically, the workpiece offset includes a first offset and a second offset, wherein the first offset is the coordinate deviation between the fourth physical coordinate and the second physical coordinate, and the second offset is the coordinate deviation between the fifth physical coordinate and the third physical coordinate.

[0197] In this embodiment, by determining the offset of the workpiece, the offset of the workpiece can be known, and then a corresponding coordinate system correction relationship can be generated. The coordinate system correction relationship can be understood as a correction relationship determined based on the offset of the workpiece.

[0198] In some embodiments, the robot may optionally include a body and an end effector, with a first camera disposed on the end effector. The control unit 502 is specifically configured to: control the end effector to move relative to the body until the first camera faces the surface to be processed; and control the first camera to capture an image to obtain a first image.

[0199] In this embodiment, the end effector is controlled to move relative to the body to adjust the orientation of the first camera, thereby making the first camera face the surface to be processed in order to obtain an image of the surface to be processed.

[0200] In this process, a planar image of the surface to be processed can be obtained directly. Compared with the method of taking a three-dimensional image containing the workpiece and then processing the three-dimensional image to obtain a planar image of the surface to be processed, no additional processing of the first image is required, which improves the efficiency of trajectory extraction.

[0201] In some embodiments, the first camera is controlled to take pictures when it is oriented perpendicular to the surface to be processed.

[0202] In some embodiments, optionally, if the first image does not include all the surfaces to be processed, the control unit 502 is further configured to: control the end effector to move relative to the body until the first camera faces other surfaces of the surface to be processed; control the first camera to capture a third image, the third image and the first image both containing a fourth image, the fourth image being an image of a portion of the surface to be processed; the drawing unit 504 is further configured to: draw a second trajectory based on the third image, the second trajectory including O second trajectory points, where O is a positive integer greater than or equal to 1; the extraction unit 506 is further configured to: extract pixel coordinates from the drawn third image to obtain O sixth pixel coordinates corresponding to the O second trajectory points; the transformation unit 508 is further configured to: determine O sixth physical coordinates corresponding to the O sixth pixel coordinates according to the coordinate system transformation relationship, the sixth physical coordinates being the robot's position coordinates in the second coordinate system; the determination unit 510 is further configured to: determine a second running trajectory of the robot processing the surface to be processed based on the O sixth physical coordinates; and stitch the first running trajectory and the second running trajectory together using a mean fusion method to obtain the running trajectory of the robot processing the surface to be processed.

[0203] In this embodiment, considering that the field of view of the first camera cannot completely cover the entire surface to be processed, if the first image does not include the entire surface to be processed, the end effector is also controlled to move relative to the body in order to capture other surfaces of the surface to be processed, thereby obtaining a third image.

[0204] Among them, the other surfaces of the surface to be processed can be understood as the surfaces of the surface to be processed that were not photographed.

[0205] In this embodiment, trajectory extraction can be achieved even when the surface to be processed is too large.

[0206] In the above embodiments, by limiting both the third image and the first image to include a fourth image, where the fourth image is an image of a portion of the surface to be processed, the obtained third image and the first image have the same image, so that the O second trajectory points and the M first trajectory points have the same trajectory points. Then, after determining the O sixth physical coordinates corresponding to the O second trajectory points, the M first physical coordinates corresponding to the M first trajectory points and the O sixth physical coordinates are stitched together based on the mean fusion method, thereby realizing the stitching of the second running trajectory and the first running trajectory, thus obtaining a complete running trajectory.

[0207] In this process, the mean fusion method can make the spliced ​​running trajectory smoother, making the robot run more smoothly according to the spliced ​​running trajectory.

[0208] In some embodiments, optionally, the determining unit 510 is further configured to: determine a trajectory program based on a first running trajectory; wherein, when the robot executes the trajectory program, the robot processes the surface to be processed according to the first running trajectory.

[0209] In this embodiment, a corresponding trajectory program can be generated based on the first running trajectory without user operation, thereby reducing the degree of user involvement and realizing the automatic generation of the trajectory program.

[0210] The trajectory program is the program that the robot runs.

[0211] In one embodiment, such as Figure 6 As shown, the present invention provides another trajectory extraction device 600, including a processor 602 and a memory 604. The memory 604 stores a program or instructions that can run on the processor 602. When the program or instructions are executed by the processor, they implement the steps of any of the trajectory extraction methods described above.

[0212] The memory 604 can be used to store software programs and various data. The memory 604 mainly includes a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 604 can include volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0213] In one embodiment, the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described above.

[0214] In one embodiment, the present invention provides a robot control system, comprising: a trajectory extraction device as described above; and / or a readable storage medium as described above.

[0215] The terms "first" and "second" in the specification and claims of this application may explicitly or implicitly include one or more of the features. In the textual description of this invention, unless otherwise stated, "a plurality of" means two or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0216] In the textual description of this invention, it is understood that, unless explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0217] In the claims, description, and accompanying drawings of this invention, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In the claims, description, and accompanying drawings of this invention, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0218] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A trajectory extraction method, characterized in that, include: The robot controls a first camera to capture a first image, the first image including an image of the surface to be processed on the workpiece, the first camera being mounted on the robot; A first trajectory is drawn based on the first image, and the first trajectory includes M first trajectory points, where M is a positive integer greater than or equal to 1; Pixel coordinates are extracted from the drawn first image to obtain M first pixel coordinates corresponding to M first trajectory points; Based on the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot, M first physical coordinates are determined corresponding to the M first pixel coordinates, and the first physical coordinates are the position coordinates of the robot in the second coordinate system; Based on M of the first physical coordinates, the first running trajectory of the robot when processing the surface to be processed is determined.

2. The trajectory extraction method according to claim 1, characterized in that, The step of drawing the first trajectory based on the first image specifically includes: Receive the first input for drawing the marker points; In response to the first input, at least one marker point is displayed based on the first image; Draw a first trajectory based on at least one of the aforementioned marker points; Receive a second input containing a first quantity, where the first quantity is a value of M; In response to the second input, the first trajectory is divided into equal distances to obtain M points on the first trajectory.

3. The trajectory extraction method according to claim 1, characterized in that, Before controlling the first camera to capture the first image, the method further includes: The first camera and the robot are calibrated using an N-point calibration method or a calibration board to obtain the coordinate system transformation relationship.

4. The trajectory extraction method according to claim 1, characterized in that, The trajectory extraction method further includes: When the position of the workpiece changes, the first camera is controlled to capture a second image, the second image including an image of the surface to be processed, and the second image including second pixel coordinates and third pixel coordinates; Based on the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate, and the fifth pixel coordinate, a coordinate system correction relationship is determined. The fourth pixel coordinate and the fifth pixel coordinate are two pixel coordinates in the first image. The second pixel coordinate and the fourth pixel coordinate correspond to the first position of the surface to be processed. The third pixel coordinate and the fifth pixel coordinate correspond to the second position of the surface to be processed. The first position and the second position are non-overlapping positions. The first running trajectory is corrected based on the coordinate system correction relationship.

5. The trajectory extraction method according to claim 4, characterized in that, The determination of the coordinate system correction relationship based on the second pixel coordinate, the third pixel coordinate, the fourth pixel coordinate, and the fifth pixel coordinate specifically includes: Based on the coordinate system transformation relationship, determine the second physical coordinate corresponding to the second pixel coordinate, the third physical coordinate corresponding to the third pixel coordinate, the fourth physical coordinate corresponding to the fourth pixel coordinate, and the fifth physical coordinate corresponding to the fifth pixel coordinate; The offset of the workpiece is determined based on the second physical coordinate, the third physical coordinate, the fourth physical coordinate, and the fifth physical coordinate; The coordinate system correction relationship is determined based on the offset of the workpiece.

6. The trajectory extraction method according to any one of claims 1 to 5, characterized in that, The robot includes a body and an end effector. The first camera is mounted on the end effector. Controlling the first camera to capture a first image specifically includes: Control the end effector to move relative to the body until the first camera faces the surface to be processed; Control the first camera to take a picture and obtain the first image.

7. The trajectory extraction method according to claim 6, characterized in that, If the first image does not include all of the surface to be processed, controlling the first camera to capture the first image further includes: Control the end effector to move relative to the body until the first camera is facing other surfaces of the surface to be processed; The first camera is controlled to capture a third image. Both the third image and the first image contain a fourth image, which is an image of a portion of the surface to be processed. The step of drawing the first trajectory based on the first image further includes: A second trajectory is drawn based on the third image, the second trajectory including O second trajectory points, where O is a positive integer greater than or equal to 1; The step of extracting pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to M first trajectory points further includes: Pixel coordinates are extracted from the drawn third image to obtain O sixth pixel coordinates corresponding to O second trajectory points; The step of determining the M first physical coordinates corresponding to the M first pixel coordinates based on the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot further includes: Based on the coordinate system transformation relationship, determine O sixth physical coordinates corresponding to the O sixth pixel coordinates, where the sixth physical coordinates are the position coordinates of the robot in the second coordinate system; The step of determining the first running trajectory of the robot when processing the surface to be processed based on M first physical coordinates further includes: Based on the O sixth physical coordinates, the second running trajectory of the robot when processing the surface to be processed is determined; The first running trajectory and the second running trajectory are spliced ​​together using the mean fusion method to obtain the running trajectory of the robot when processing the surface to be processed.

8. The trajectory extraction method according to any one of claims 1 to 5, characterized in that, The trajectory extraction method further includes: The trajectory determination program is based on the first running trajectory; When the robot executes the trajectory program, the robot processes the surface to be processed according to the first running trajectory.

9. A trajectory extraction device, characterized in that, include: A control unit is used to control a first camera to capture a first image, the first image including an image of the surface to be processed on the workpiece, the first camera being mounted on the robot; A drawing unit is used to draw a first trajectory based on the first image, the first trajectory including M first trajectory points, where M is a positive integer greater than or equal to 1; The extraction unit is used to extract pixel coordinates from the drawn first image to obtain M first pixel coordinates corresponding to M first trajectory points; The transformation unit is used to determine M first physical coordinates corresponding to M first pixel coordinates according to the coordinate system transformation relationship between the first coordinate system of the first camera and the second coordinate system of the robot. The first physical coordinates are the position coordinates of the robot in the second coordinate system. The determining unit is used to determine the first running trajectory of the robot when processing the surface to be processed based on M of the first physical coordinates.

10. A trajectory extraction device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the trajectory extraction method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the trajectory extraction method as described in any one of claims 1 to 8.

12. A robot control system, characterized in that, include: The trajectory extraction device as described in claim 9 or 10; and / or The readable storage medium as described in claim 11.

Images