Method for generating robot operation program, processing device and processing system

Abstract

The present application discloses a method for generating a robot operation program, a processing device and a processing system, the method comprising: receiving a first model of a robot's operating area sent by a handheld device; receiving a first position of a position to be operated in the first model sent by the handheld device; obtaining a target posture of the handheld device when photographing the operating area; receiving a second model of the operating area sent by a robot imaging device; determining a second position of the position to be operated in a first three-dimensional point cloud model according to the first position of the position to be operated in the first model; generating an operation program according to the second position of the position to be operated in the first three-dimensional point cloud model, and sending the operation program to the robot so that the robot operates according to the operation program. The method of the present application can achieve programming-free on the one hand, and can improve the convenience of operation and reduce costs on the other hand.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method for generating a robot operation program, a processing device, and a processing system. Background Art

[0002] With the development of economy, the demand for robots is increasing. In recent years, with the development of product customization, the homogeneous mass production mode has been gradually broken, and the demand for using robots for small batches or even single piece production has increased. However, before controlling the robot to perform tasks, it is necessary to pre-program the robot in large quantities, which is time-consuming and requires high quality of programmers. It is not suitable for small batch and single piece production mode. Summary of the invention

[0003] The present application provides a method for generating a robot operation program, a processing device and a processing system, which can achieve programming-free operation on the one hand, and improve the convenience of operation and reduce costs on the other hand.

[0004] According to a first aspect of an embodiment of the present application, there is provided a method for generating a robot operation program, the method comprising: receiving a first model of a robot's area to be operated sent by a handheld device, wherein the area to be operated includes a position to be operated, the position to be operated is located on a workpiece to be operated, the handheld device photographs the area to be operated, and establishes the first model based on the photographed data; receiving a first position of the position to be operated in the first model sent by the handheld device; acquiring a target posture of the handheld device when photographing the area to be operated; receiving a second model of the area to be operated sent by a robot imaging device, wherein the robot imaging device photographs the area to be operated with the target posture, and establishes the second model based on the photographed data, the second model at least including a first three-dimensional point cloud model; determining a second position of the position to be operated in the first three-dimensional point cloud model according to the first position of the position to be operated in the first model; generating an operation program according to the second position of the position to be operated in the first three-dimensional point cloud model, and sending the operation program to the robot so that the robot operates according to the operation program.

[0005] According to a second aspect of an embodiment of the present application, there is provided a method for generating a robot operation program, the method comprising: a handheld device photographs an area to be operated on a workpiece to be operated, and establishes a first model based on the photographed data, wherein the area to be operated includes a position to be operated; the handheld device determines a first position of the position to be operated in the first model; the handheld device sends the first model and the first position to a processing device; the processing device obtains a target posture of the handheld device when photographing the area to be operated; the processing device receives a second model of the area to be operated sent by a robot imaging device, wherein the robot imaging device photographs the area to be operated with the target posture, and establishes the second model based on the photographed data, the second model including at least a first three-dimensional point cloud model; the processing device determines a second position of the position to be operated in the first three-dimensional point cloud model according to the first position of the position to be operated in the first model; the processing device generates an operation program according to the second position of the position to be operated in the first three-dimensional point cloud model, and sends the operation program to the robot so that the robot operates according to the operation program.

[0006] A third aspect of an embodiment of the present application provides a processing device, which includes a processor, a memory, and a communication circuit, wherein the processor is coupled to the memory and the communication circuit, respectively, and program data is stored in the memory. The processor implements the steps in any one of the above methods by executing the program data in the memory.

[0007] A fourth aspect of an embodiment of the present application provides a processing system, including a handheld device and any one of the processing devices described above.

[0008] The beneficial effect is: the present application first uses a handheld device to shoot the work area to generate a first model, and according to the posture of the handheld device when shooting the work area, controls the robot imaging device to shoot the work area with the same posture, thereby establishing a high-precision second model, the second model at least includes a first three-dimensional point cloud model, and according to the position of the position to be operated in the first model, determines the position of the position to be operated in the first three-dimensional point cloud model, and finally generates an operation program according to the position of the position to be operated in the first three-dimensional point cloud model, so that the robot can operate according to the operation program, thereby completing the task. This process can achieve programming-free on the one hand, and on the other hand, does not require the establishment of an accurate 3D model of the entire scene, nor does it require manual operation of the robot. The process is simple, which improves convenience and reduces the cost of manual operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0010] Figure 1 It is a flowchart of an implementation method of a method for generating a robot operation program of the present application;

[0011] Figure 2 This is a schematic diagram of the connection structure of an embodiment of the robot workstation, processing equipment and handheld device of the present application;

[0012] Figure 3 It is a flowchart of another implementation method of the method for generating a robot operation program of the present application;

[0013] Figure 4 It is a structural schematic diagram of an implementation method of a processing device of the present application;

[0014] Figure 5 It is a flowchart of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] It should be noted that the terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0017] See also Figure 1 In one embodiment of the present application, a method for generating a robot operation program includes:

[0018] S110: Receive a first model of the robot's working area sent by a handheld device, wherein the working area includes a position to be operated, and the position to be operated is located on the workpiece to be operated. The handheld device photographs the working area and establishes a first model based on the photographed data.

[0019] Specifically, see Figure 2 The system involved in this application includes a robot workstation, a processing device 120 and a handheld device 130, wherein the robot workstation includes a robot 101 and a control device (not shown in the figure, the control device can be a computer or a robot control cabinet) that controls the movement of the robot 101, a robot image device 102 is installed on the robot 101 (usually, the shooting accuracy of the robot image device 102 is greater than the shooting accuracy of the handheld device 130), and an end effector 103 is installed at the end of the robot 101; the handheld device 130 can be any device with information collection function such as a camera, a mobile phone, a laser radar, etc. The processing device 120 can be a network edge terminal or a cloud server. In one embodiment, the handheld device 130 transmits information with the processing device 120 through a wireless communication network, and the processing device 120 communicates with the robot workstation through a wireless communication network or a wired communication network (specifically, communicates with the control device in the robot workstation).

[0020] In one implementation, steps S110 - S160 of the present application are executed by the processing device 120 .

[0021] Continue reading Figure 2 , wherein the end effector 103 installed at the end of the robot can be a welding gun, a spray gun, a clamp or a cutting tool, etc., and this application does not make any limitation. It can be understood that when the end effector 103 is a welding gun, the robot is a welding robot, which can perform welding tasks, and the position to be operated is the weld; when the end effector 103 is a spray gun, the robot is a spraying robot, which can perform spraying tasks, and the position to be operated is the spraying line.

[0022] Before step S110, the user takes a handheld device 130 to photograph the area to be operated on the workpiece including the position to be operated, and then the handheld device 130 establishes a first model based on the photographed result. The first model can be a two-dimensional model or a three-dimensional model. It is understood that compared with the two-dimensional model, the three-dimensional model contains depth information. It is understood that the first model is a local model with the workpiece.

[0023] Continue reading Figure 2In an application scenario, the workpieces to be operated include a first workpiece to be welded 111 and a second workpiece to be welded 112, and a weld 113 is formed between the first workpiece to be welded 111 and the second workpiece to be welded 112, that is, the weld 113 is the position to be operated. In this application scenario, the user holds the handheld device 130 to photograph the weld 113 between the first workpiece to be welded 111 and the second workpiece to be welded 112, and then the handheld device 130 establishes a first model based on the photographed result, and sends the first model to the processing device 120.

[0024] S120: Receive a first position of the position to be operated in the first model sent by the handheld device 130.

[0025] Specifically, after the handheld device 130 has established the first model, the handheld device 130 can automatically identify the first position of the position to be operated in the first model through an algorithm. For example, when the first model is a two-dimensional model, the handheld device 130 can automatically identify the position to be operated in the first model using algorithms such as edge detection, thereby obtaining the first position of the position to be operated in the first model.

[0026] In another embodiment, the handheld device 130 determines the position specified by the user in the first model as the first position. Specifically, the user can specify the position on the first model through the visual interface of the handheld device 130, so that the handheld device 130 uses the position as the first position, that is, the user specifies the position to be operated on the first model. When the handheld device 130 has a touch screen, the user can specify the position to be operated by manually drawing a line on the visual interface of the handheld device 130, or can also specify the position to be operated on the visual interface with a stylus, or can also specify the position to be operated in the first model by sliding the mouse.

[0027] In an application scenario, when there are multiple positions to be operated in the first model, through the above method, the user can select only one of the positions to be operated, so that subsequent steps are only performed for the selected position to be operated, that is, finally only the motion program corresponding to the position to be operated selected by the user is generated.

[0028] S130: Obtain the target posture of the handheld device 130 when photographing the work area.

[0029] The target posture includes the position and posture of the handheld device 130 during the shooting process.

[0030] In one embodiment, the handheld device 130 is integrated with a posture sensor. During the shooting process of the handheld device 130, the posture sensor automatically collects the posture data of the handheld device 130, thereby obtaining the target posture. It should be noted that in this embodiment, it is necessary to ensure that the coordinate system of the posture sensor is the same as the coordinate system of the robot 101. For example, the handheld device 130 is placed on the base of the robot 101, and the position of the handheld device 130 at this time is determined as the zero point position of the handheld device 130, thereby ensuring that the coordinate system of the posture sensor is the same as the base coordinate system of the robot 101.

[0031] In another embodiment, step S130 includes:

[0032] S131: receiving a third model of the workpiece to be worked on sent by the robot imaging device 102, wherein the robot imaging device 102 photographs the workpiece to be worked on and establishes the third model based on the photographed data.

[0033] S132: Determine the target posture of the handheld device 130 when photographing the work area based on the first model and the third model.

[0034] Specifically, when the robot 101 is in the initial position, the robot imaging device 102 installed on the robot 101 is in a high-position overlooking posture, and at this time, the robot 101 is controlled to move so that during the movement of the robot 101, the robot imaging device 102 takes a picture of the entire workpiece to be worked on, and then a third model of the workpiece to be worked on is established based on the results of the shooting by the robot imaging device 102. The third model can be a two-dimensional model or a three-dimensional model.

[0035] Since the first model is obtained by photographing the area to be worked on the workpiece to be worked, and the third model is obtained by photographing the entire workpiece to be worked, the first model is a local model and the third model is an overall model. Therefore, according to the relative position and posture of the first model in the third model, the target posture of the handheld device 130 when photographing the area to be worked can be determined. In one embodiment, the target posture of the handheld device 130 when photographing the area to be worked can be determined based on the first model and the third model, and using indoor positioning technologies such as UWB. Alternatively, the coordinates of the feature points in the first model and the coordinates of the feature points in the third model can also be extracted, and combined with, for example, the ICP (Iterative Closest Point) algorithm, the target posture of the handheld device 130 when photographing the area to be worked can be determined.

[0036] Among them, the specific process of determining the target posture when the handheld device 130 is photographed according to the first model and the third model belongs to the existing technology and is not specifically limited here.

[0037] In other embodiments, when the handheld device 130 is integrated with a posture sensor, the posture data collected by the posture sensor and the position and posture of the first model relative to the third model can be combined to determine the target posture of the handheld device 130 when shooting the work area.

[0038] S140: Receive a second model of the area to be operated sent by the robot imaging device 102, wherein the robot imaging device 102 photographs the area to be operated in a target posture and establishes a second model based on the photographed data, wherein the second model at least includes a first three-dimensional point cloud model.

[0039] Specifically, after obtaining the target posture, the robot 101 is controlled to move so that the robot imaging device 102 shoots the working area in the target posture, and a second model is established based on the results of the shooting by the robot imaging device 102. The second model includes at least one three-dimensional model, that is, a first three-dimensional point cloud model, and the first three-dimensional point cloud model shows spatial information.

[0040] It can be understood that both the first model and the third model are local models of the workpiece to be processed, but because the accuracy of the robot imaging device 102 is greater than the accuracy of the handheld device 130, the accuracy of the second model is higher than that of the first model.

[0041] It can be understood that, since the posture of the robot imaging device 102 when photographing the working area is the same as the posture of the handheld device 130 when photographing the working area, the content displayed by the first model and the second model is highly similar.

[0042] S150: Determine a second position of the position to be operated in the first three-dimensional point cloud model according to the first position of the position to be operated in the first model.

[0043] Specifically, since the content displayed by the first model and the second model is highly similar, the second position of the position to be operated in the first three-dimensional point cloud model can be determined according to the first position. Among them, since the first three-dimensional point cloud model displays spatial information, the second position of the position to be operated in the first three-dimensional point cloud model is also the position of the position to be operated in the three-dimensional space.

[0044] The specific process of step S150 can be found in the following description.

[0045] S160: Generate an operation program according to the second position of the position to be operated in the first three-dimensional point cloud model, and send the operation program to the robot 101 so that the robot 101 operates according to the operation program.

[0046] Specifically, since the second position of the position to be operated in the first three-dimensional point cloud model is the position of the position to be operated in the three-dimensional space, the operation program of the robot 101 can be generated based on the second position, and the operation program is sent to the robot 101, so that the robot 101 moves according to the operation program, thereby operating the position to be operated. In one embodiment, the processing device 120 can specifically send the operation program to the control device in the robot workstation, and then the control device controls the robot 101 to operate according to the operation program.

[0047] From the above content, it can be seen that the present application first uses the handheld device 130 to shoot the work area to be treated, thereby generating a first model, and according to the posture of the handheld device 130 when shooting the work area, controls the robot imaging device 102 installed on the robot 101 to shoot the work area with the posture, thereby establishing a high-precision second model, the second model includes a first three-dimensional point cloud model, and according to the first position of the position to be operated in the first model, determines the second position of the position to be operated in the first three-dimensional point cloud model, and finally generates an operation program according to the second position, so that the robot 101 can operate according to the operation program, thereby completing the task. On the one hand, this process does not require programming of the robot 101, and on the other hand, does not require the establishment of an accurate 3D model of the entire scene, nor does it require manual operation of the robot 101. The process is simple, which improves convenience and reduces the cost of manual operation.

[0048] In one embodiment, the first model includes a first image, the first position is the position of the position to be operated in the first image, the second model further includes a second image, the first three-dimensional point cloud model is collected by the first sensor in the robot imaging device 102, and the second image is collected by the second sensor in the robot imaging device 102. Specifically, in this embodiment, the first model collected by the handheld device 130 is a two-dimensional model, that is, the handheld device 130 shoots the area to be operated to obtain the first image; at the same time, the robot imaging device 102 includes a first sensor and a second sensor, the first sensor collects a two-dimensional image, and the second sensor collects a three-dimensional image. When shooting the area to be operated, the first sensor collects the second image, and the second sensor collects the first three-dimensional point cloud model. In one embodiment, the first sensor can be a camera, and the second sensor can be a radar.

[0049] In this embodiment, the step S150 of determining the second position of the position to be operated in the first three-dimensional point cloud model includes:

[0050] S151: extracting features from the first image to obtain first features, and extracting features from the second image to obtain second features.

[0051] Specifically, algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), neural networks, etc. may be used to extract features from the first image and the second image respectively.

[0052] S152: Determine a first transformation matrix between the coordinate system of the handheld device 130 and the coordinate system of the second sensor according to the first feature and the second feature.

[0053] Specifically, it can be seen from the above content that the content displayed by the first model and the second model is highly similar, so after obtaining the first feature and the second feature, the first feature and the second feature can be feature matched to obtain a transformation matrix between the coordinate system of the handheld device 130 and the coordinate system of the second sensor, and the transformation matrix is ​​defined as a first transformation matrix. In one embodiment, the first transformation matrix can be obtained based on the coordinates of the feature points in the first feature and the coordinates of the feature points in the second feature, combined with a geometric registration algorithm such as ICP (Iterative Closest Point).

[0054] Among them, the specific process of generating the first transformation matrix according to the first feature and the second feature belongs to the existing technology and will not be described in detail here.

[0055] S153: Obtain a third position of the position to be operated in the second image according to the first position and the first transformation matrix.

[0056] Specifically, after obtaining the first transformation matrix, the position to be operated can be mapped from the first image to the second image to obtain the position of the position to be operated in the second image, and the position is recorded as the third position.

[0057] In one embodiment, the first transformation matrix is ​​recorded as , is a matrix of four rows and four columns. In this embodiment, the first transformation matrix Specifically, it is the pose matrix of the coordinate system of the handheld device 130 in the coordinate system of the second sensor. At this time, the third position of the position to be operated in the second image can be determined by the following formula:

[0058]

[0059] in, is a matrix of three rows and one column corresponding to the first position, is a matrix with three rows and one column corresponding to the third position.

[0060] It should be noted that, in other implementations, in the specific process of obtaining the third position in step S153, algorithms such as scaling and perspective projection may also be utilized, but these are mature technologies and will not be described in detail in this application.

[0061] S154: Obtaining a second position of the position to be operated in the first three-dimensional point cloud model according to the third position and the second transformation matrix, where the second transformation matrix is ​​a transformation matrix between a coordinate system of the second sensor and a coordinate system of the first sensor.

[0062] Specifically, the second transformation matrix is ​​an internal parameter of the robot imaging device 102 and is a known parameter. Therefore, based on the second transformation matrix, the position to be operated can be mapped from the second image to the first three-dimensional point cloud model to obtain the second position of the position to be operated in the first three-dimensional point cloud model.

[0063] In one embodiment, the second transformation matrix is ​​recorded as , is a matrix of four rows and four columns, and in this embodiment, the second transformation matrix is the pose matrix of the coordinate system of the second sensor in the coordinate system of the first sensor. At this time, the second position of the position to be operated in the first three-dimensional point cloud model can be determined by the following formula:

[0064]

[0065] in, is a matrix of three rows and one column corresponding to the second position, is a matrix with three rows and one column corresponding to the third position.

[0066] In another embodiment, the first model includes a second three-dimensional point cloud model, and the first position is the position of the position to be operated in the second three-dimensional point cloud model. Specifically, after the handheld device 130 shoots the area to be operated, a three-dimensional model is established, that is, the second three-dimensional point cloud model. At this time, step S150 specifically includes:

[0067] S156: Perform feature extraction on the second three-dimensional point cloud model to obtain a third feature.

[0068] Specifically, algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), neural networks, etc. may be used to extract features from the second three-dimensional point cloud model to obtain the third feature.

[0069] S157: Perform feature extraction on the first three-dimensional point cloud model to obtain a fourth feature.

[0070] Specifically, algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), neural networks, etc. may be used to extract features from the first three-dimensional point cloud model to obtain the fourth feature.

[0071] S158: Obtain a third transformation matrix according to the third feature and the fourth feature.

[0072] Specifically, feature matching is performed on the third feature and the fourth feature to obtain a transformation matrix between the coordinate system of the handheld device 130 and the coordinate system of the robot imaging device 102 (specifically, the coordinate system of the first sensor), and the matrix is ​​defined as a third transformation matrix.

[0073] Among them, the specific process of obtaining the third transformation matrix according to the third feature and the fourth feature belongs to the existing technology and will not be introduced in detail here.

[0074] S159: Obtaining a second position of the position to be operated in the first three-dimensional point cloud model according to the first position and the third transformation matrix.

[0075] Specifically, based on the third transformation matrix, the position to be operated can be converted from the second three-dimensional point cloud model to the first three-dimensional point cloud model, thereby obtaining the position of the position to be operated in the first three-dimensional point cloud model, that is, the second position.

[0076] In one embodiment, the third transformation matrix is ​​recorded as , is a matrix of four rows and four columns, and in this embodiment, is the pose matrix of the coordinate system of the handheld device 130 in the coordinate system of the robot imaging device 102 (specifically, the coordinate system of the first sensor). At this time, the second position of the device to be operated in the first three-dimensional point cloud model can be determined using the following formula:

[0077]

[0078] in, is a matrix of three rows and one column corresponding to the first position, is a matrix with three rows and one column corresponding to the second position.

[0079] In another embodiment, the method for generating the operation program of the robot 101 includes:

[0080] S210: The handheld device 130 photographs the area to be operated on the workpiece, and establishes a first model based on the photographed data, wherein the area to be operated includes a position to be operated.

[0081] S220: The handheld device 130 determines a first position of the position to be operated in the first model.

[0082] S230 : The handheld device 130 sends the first model and the first position to the processing device 120 .

[0083] S240: The processing device 120 obtains the target posture of the handheld device 130 when photographing the work area.

[0084] S250: The processing device 120 receives the second model of the area to be operated sent by the robot imaging device 102, wherein the robot imaging device 102 photographs the area to be operated in a target posture and establishes a second model based on the photographed data, and the second model at least includes a first three-dimensional point cloud model.

[0085] S260: The processing device 120 determines a second position of the position to be operated in the first three-dimensional point cloud model according to the first position of the position to be operated in the first model.

[0086] S270: The processing device 120 generates an operation program according to the second position of the position to be operated in the first three-dimensional point cloud model, and sends the operation program to the robot 101, so that the robot 101 operates according to the operation program.

[0087] In one implementation, step S220 specifically includes: the handheld device 130 determines the position specified by the user in the first model as the first position.

[0088] In order to better understand the solution of this application, please refer to the following Figure 3 , the solution of the present application is specifically introduced in combination with a specific example. In this example, the robot 101 is a welding robot, and the position to be operated is a weld:

[0089] First, in the initial state, the robot imaging device 102 installed on the robot 101 looks down at the workpiece to be worked on. At this time, the control device controls the robot imaging device 102 to photograph the workpiece to be worked on, and establishes a third model of the workpiece to be worked on according to the data photographed by the robot imaging device 102, and then sends the third model to the processing device 120.

[0090] Next, the user holds the handheld device 130 and aims at the area to be worked on where the weld is located on the workpiece to be worked on to take a photo, and then the handheld device 130 establishes a first model of the area to be worked on according to the photographed data.

[0091] Then, the user specifies the first position of the weld in the first model on the handheld device 130. For example, when the handheld device 130 has a touch function, the user can specify the position of the weld by manually drawing a line on the handheld device 130. In other examples, the handheld device 130 can also automatically identify the weld in the first model through a recognition algorithm to obtain the first position of the weld in the first model. Afterwards, the handheld device 130 sends the first model and the first position to the processing device 120.

[0092] After receiving the third model and the first model, the processing device 120 determines the target posture of the handheld device 130 when photographing the work area based on the first model and the third model, and then the handheld device 130 sends the target posture to the control device in the robot 101 workstation.

[0093] After receiving the target posture, the control device controls the robot 101 to move, so that the robot imaging device 102 installed on the robot 101 takes a picture of the area to be operated in the target posture during the movement of the robot 101, and then the control device establishes a second model of the area to be operated based on the data taken by the robot imaging device 102, the second model at least including the first three-dimensional point cloud model, and sends the second model to the processing device 120. Since the accuracy of the robot imaging device 102 is higher than that of the handheld device 130, the accuracy of the second model is higher than that of the first model.

[0094] After receiving the second model, the processing device 120 determines a second position of the weld in the first three-dimensional point cloud model according to the first position of the weld in the first model.

[0095] Finally, the processing device 120 generates a welding program for the robot 101 according to the second position of the weld in the first three-dimensional point cloud model, and sends the welding program to the control device in the workstation of the robot 101, so that the control device controls the robot 101 to move according to the welding program, thereby performing the welding task.

[0096] During the movement of the robot 101, the control device can also control the robot imaging device 102 to take pictures, and correct the movement of the robot 101 according to the results of the shooting of the robot imaging device 102 to reduce the movement error of the robot 101 until the welding task is completed.

[0097] See also Figure 4 , Figure 4 1 is a schematic diagram of the structure of an embodiment of a processing device of the present application. The processing device 200 includes a processor 210, a memory 220 and a communication circuit 230. The processor 210 is coupled to the memory 220 and the communication circuit 230 respectively. The memory 220 stores program data. The processor 210 executes the program data in the memory 220 to implement the steps performed by the processing device 120 in any of the above-mentioned implementation methods. The detailed steps can be found in the above-mentioned implementations and will not be repeated here.

[0098] The processing device 200 may be any device with algorithm processing capabilities, such as a computer, a cloud server, or a network edge terminal, and is not limited here.

[0099] In addition, the present application also protects a processing system, including the above-mentioned processing device 200 and a handheld device. The handheld device can be any device with information collection capability such as a camera, a mobile phone or a laser radar, which is not limited here.

[0100] See also Figure 5 , Figure 5 The computer-readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps in any of the above methods.

[0101] The computer-readable storage medium 400 may specifically be a device that can store the computer program 410, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer program 410. The server may send the stored computer program 410 to other devices for execution, or may also execute the stored computer program 410 by itself.

[0102] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a robot operation program, characterized in that: The method comprises: Receiving a first model of a robot's area to be operated sent by a handheld device, wherein the area to be operated includes a position to be operated, and the position to be operated is located on a workpiece to be operated, and the handheld device photographs the area to be operated, and establishes the first model based on the photographed data; receiving a first position of the position to be operated in the first model sent by the handheld device; Acquiring a target posture of the handheld device when photographing the area to be operated; receiving a second model of the area to be operated sent by a robot imaging device, wherein the robot imaging device photographs the area to be operated in the target posture and establishes the second model based on the photographed data, wherein the second model at least includes a first three-dimensional point cloud model; Determining a second position of the position to be operated in the first three-dimensional point cloud model according to the first position of the position to be operated in the first model; An operation program is generated according to the second position of the position to be operated in the first three-dimensional point cloud model, and the operation program is sent to the robot so that the robot operates according to the operation program.

2. The method according to claim 1, characterized in that The step of obtaining the target posture of the handheld device when photographing the area to be operated includes: receiving a third model of the workpiece to be processed sent by the robot imaging device, wherein the robot imaging device photographs the workpiece to be processed and establishes the third model based on the photographed data; The target posture of the handheld device when photographing the area to be operated is determined according to the first model and the third model.

3. The method according to claim 1, characterized in that The handheld device is integrated with a posture sensor; The step of obtaining the target posture of the handheld device when photographing the area to be operated includes: Receiving data collected by the posture sensor when the handheld device photographs the area to be operated; The target posture is determined according to the data collected by the posture sensor.

4. The method according to claim 1, characterized in that: The first model includes a first image, the first position is the position of the position to be operated in the first image, the second model further includes a second image, the first three-dimensional point cloud model is collected by a first sensor in the robot imaging device, and the second image is collected by a second sensor in the robot imaging device; The step of determining a second position of the position to be operated in the first three-dimensional point cloud model according to the first position of the position to be operated in the first model comprises: Performing feature extraction on the first image to obtain a first feature, and performing feature extraction on the second image to obtain a second feature; Determine a first transformation matrix between a coordinate system of the handheld device and a coordinate system of the second sensor according to the first feature and the second feature; Obtaining a third position of the position to be operated in the second image according to the first position and the first transformation matrix; The second position of the position to be operated in the first three-dimensional point cloud model is obtained according to the third position and a second transformation matrix, wherein the second transformation matrix is ​​a transformation matrix between a coordinate system of the second sensor and a coordinate system of the first sensor.

5. The method according to claim 1, characterized in that The first model includes a second three-dimensional point cloud model, and the first position is the position of the position to be operated in the second three-dimensional point cloud model; The step of determining a second position of the position to be operated in the first three-dimensional point cloud model according to the first position of the position to be operated in the first model comprises: Performing feature extraction on the second three-dimensional point cloud model to obtain a third feature; Performing feature extraction on the first three-dimensional point cloud model to obtain a fourth feature; Obtaining a third transformation matrix according to the third feature and the fourth feature; The second position of the position to be operated in the first three-dimensional point cloud model is obtained according to the first position and the third transformation matrix.

6. A method for generating a robot operation program, characterized in that: The method comprises: The handheld device photographs the area to be operated on the workpiece to be operated, and establishes a first model based on the photographed data, wherein the area to be operated includes a position to be operated; The handheld device determines a first position of the position to be operated in the first model; The handheld device sends the first model and the first position to a processing device; The processing device obtains the target posture of the handheld device when photographing the area to be operated; The processing device receives a second model of the area to be operated sent by a robot imaging device, wherein the robot imaging device photographs the area to be operated in the target posture and establishes the second model based on the photographed data, wherein the second model at least includes a first three-dimensional point cloud model; The processing device determines, according to the first position of the position to be operated in the first model, a second position of the position to be operated in the first three-dimensional point cloud model; The processing device generates an operation program according to the second position of the position to be operated in the first three-dimensional point cloud model, and sends the operation program to the robot so that the robot operates according to the operation program.

7. The method according to claim 6, characterized in that The step of determining, by the handheld device, a first position of the position to be operated in the first model comprises: The handheld device determines a position specified by a user in the first model as the first position.

8. A processing device, characterized in that: The processing device includes a processor, a memory and a communication circuit, the processor is coupled to the memory and the communication circuit respectively, the memory stores program data, and the processor implements the steps in the method as described in any one of claims 1 to 5 by executing the program data in the memory.

9. A processing system, characterized in that: The invention comprises a handheld device and a processing device as claimed in claim 8.

10. The processing system according to claim 9, characterized in that The handheld device is a camera, a mobile phone or a laser radar.

Images