Welding teaching method, device, terminal device and computer-readable storage medium

Abstract

The present application is applicable to the field of welding technology, and provides a welding teaching method, device, terminal device and computer-readable storage medium. The method includes: acquiring a two-dimensional image and a three-dimensional point cloud of a workpiece to be welded; inputting the two-dimensional image into a weld seam target detection model to obtain a weld seam detection result; and obtaining a weld seam teaching path of the workpiece to be welded according to the weld seam detection result and the three-dimensional point cloud. The present application can improve the accuracy of automatic welding teaching.

Description

Technical Field

[0001] This application belongs to the field of welding technology, and particularly relates to a welding teaching method, device, terminal device, and computer-readable storage medium. Background Art

[0002] With the development of science and technology, welding technology, as an indispensable processing method, has been widely used in various industries (such as shipbuilding, vehicle manufacturing, construction, electronics, and machinery manufacturing, etc.). Currently, automated welding mainly relies on manual teaching, where workers need to operate the robotic arm to guide the robotic arm to record the welding seam teaching path so that the robotic arm can move according to the welding seam teaching path and use the carried sensing device to perform real-time guidance and deviation correction for welding at the precise coordinate points of the welding seam.

[0003] Welding teaching, as the first step of robotic welding, is still difficult to achieve full automation in actual application scenarios. Because during the application of automatic teaching, the following problems will occur: Oxidation slag, paint, water, rust, etc. on the welded workpiece introduce a large amount of image noise and non-welding seam interference into the image. Traditional image preprocessing methods cannot effectively eliminate image noise and non-welding seam interference, resulting in incorrect identification of the welding seam and unsatisfactory accuracy of welding automatic teaching. Summary of the Invention

[0004] Embodiments of this application provide a welding teaching method, device, terminal device, and computer-readable storage medium, which can solve the problem of unsatisfactory accuracy of welding automatic teaching.

[0005] In a first aspect, embodiments of this application provide a welding teaching method, including:

[0006] Obtain a two-dimensional image and a three-dimensional point cloud of the workpiece to be welded;

[0007] Input the two-dimensional image into a welding seam target detection model to obtain a welding seam detection result;

[0008] According to the welding seam detection result and the three-dimensional point cloud, obtain the welding seam teaching path of the workpiece to be welded.

[0009] Optionally, the welding seam teaching path includes welding seam feature points and welding seam teaching poses. According to the welding seam detection result and the three-dimensional point cloud, obtaining the welding seam teaching path of the workpiece to be welded includes:

[0010] Determine the welding seam feature points according to the welding seam detection result and the three-dimensional point cloud;

[0011] Determine the welding seam teaching poses according to the coordinates of the welding seam feature points.

[0012] Optionally, the welding seam detection result includes the type of the welding seam of the workpiece to be welded and the predicted region of the welding seam in the two-dimensional image;

[0013] Determine weld feature points based on the weld inspection results and the three-dimensional point cloud, including:

[0014] Determine the first target point cloud data corresponding to the prediction area in the three-dimensional point cloud according to the position of the prediction area in the two-dimensional image;

[0015] Determine weld feature points according to the type of the weld of the workpiece to be welded and the first target point cloud data.

[0016] Optionally, determine weld feature points according to the type of the weld of the workpiece to be welded and the first target point cloud data, including:

[0017] When the weld of the workpiece to be welded is a fillet weld, obtain a bounding box enclosing the first target point cloud data;

[0018] Insert a plurality of slicing planes into the bounding box; the slicing planes are perpendicular to the first coordinate axis of the bounding box coordinate system, the bounding box coordinate system takes a vertex of the bounding box as the origin, and the first coordinate axis is parallel to the longest side of the bounding box;

[0019] Determine weld feature points according to the plurality of slicing planes and the bounding box.

[0020] Optionally, determine weld feature points according to the plurality of slicing planes and the bounding box, including:

[0021] For each slicing plane respectively, perform the following steps:

[0022] Determine the point cloud in the first target point cloud data that lies on the slicing plane;

[0023] Perform linear fitting on the point cloud lying on the slicing plane to obtain the point cloud on the first straight line and the point cloud on the second straight line;

[0024] Determine the first point cloud and the second point cloud with the maximum distance on the first straight line, and the third point cloud and the fourth point cloud with the maximum distance on the second straight line;

[0025] Remove the two point clouds with the maximum mutual distance among the first point cloud, the second point cloud, the third point cloud and the fourth point cloud, and use the point cloud with the larger coordinate value on the second coordinate axis of the remaining two point clouds in the bounding box coordinate system as the weld feature point; the second coordinate axis is perpendicular to the imaging plane of the acquisition device for acquiring the two-dimensional image.

[0026] Optionally, determine weld feature points according to the type of the weld of the workpiece to be welded and the first target point cloud data, including:

[0027] When the weld of the workpiece to be welded is a butt weld, determine the center line of the butt weld in the two-dimensional image;

[0028] Determine a first groove region and a second groove region in the two-dimensional image according to the position of the center line in the two-dimensional image; the first groove region and the second groove region are respectively located on both sides of the center line;

[0029] Determine the weld feature points according to the first groove region, the second groove region and the first target point cloud data.

[0030] Optionally, determining the weld feature points according to the first groove region, the second groove region and the first target point cloud data includes:

[0031] Determine the second target point cloud data corresponding to the first groove region in the three-dimensional point cloud according to the position of the first groove region in the two-dimensional image, and determine the third target point cloud data corresponding to the second groove region in the three-dimensional point cloud according to the position of the second groove region in the two-dimensional image;

[0032] Fit the second target point cloud data into a first plane and fit the third target point cloud data into a second plane;

[0033] Determine the intersection line of the first plane and the second plane;

[0034] Take the points in the first target point cloud data whose distance from the intersection line is less than the preset distance value as the weld feature points.

[0035] In a second aspect, an embodiment of the present application provides a welding teaching device, including:

[0036] An acquisition module, configured to acquire a two-dimensional image and a three-dimensional point cloud of a workpiece to be welded;

[0037] A detection module, configured to input the two-dimensional image into a weld target detection model to obtain a weld detection result;

[0038] A determination module, configured to obtain a weld teaching path of the workpiece to be welded according to the weld detection result and the three-dimensional point cloud.

[0039] Optionally, the weld teaching path includes weld feature points and weld teaching poses. The determination module is specifically configured to determine the weld feature points according to the weld detection result and the three-dimensional point cloud, and determine the weld teaching poses according to the coordinates of the weld feature points.

[0040] Optionally, the weld detection result includes the type of the weld of the workpiece to be welded and the predicted region of the weld in the two-dimensional image; the determination module includes:

[0041] A first determination sub-module, configured to determine the first target point cloud data corresponding to the predicted region in the three-dimensional point cloud according to the position of the predicted region in the two-dimensional image;

[0042] A second determination sub-module, configured to determine weld feature points according to the type of the weld seam of the workpiece to be welded and the first target point cloud data.

[0043] Optionally, the second determination sub-module includes:

[0044] An acquisition unit, configured to acquire a bounding box enclosing the first target point cloud data when the weld seam of the workpiece to be welded is a fillet weld;

[0045] An insertion unit, configured to insert a plurality of slicing planes into the bounding box; the slicing planes are perpendicular to the first coordinate axis of the bounding box coordinate system, the bounding box coordinate system takes a vertex of the bounding box as the origin, and the first coordinate axis is parallel to the longest side of the bounding box;

[0046] A first determination unit, configured to determine weld feature points according to the plurality of slicing planes and the bounding box.

[0047] Optionally, the first determination unit is specifically configured to perform the following steps for each slicing plane: determine the point cloud in the first target point cloud data that lies on the slicing plane; perform linear fitting on the point cloud that lies on the slicing plane to obtain the point cloud that lies on the first line and the point cloud that lies on the second line; determine the first point cloud and the second point cloud with the largest distance on the first line, and the third point cloud and the fourth point cloud with the largest distance on the second line; remove the two point clouds with the largest mutual distance among the first point cloud, the second point cloud, the third point cloud, and the fourth point cloud, and use the point cloud with the larger coordinate value on the second coordinate axis of the remaining two point clouds in the bounding box coordinate system as the weld feature point; the second coordinate axis is perpendicular to the imaging plane of the acquisition device for acquiring the two-dimensional image.

[0048] Optionally, the second determination sub-module includes:

[0049] A second determination unit, configured to determine the center line of the butt weld in the two-dimensional image when the weld seam of the workpiece to be welded is a butt weld;

[0050] A third determination unit, configured to determine a first groove region and a second groove region in the two-dimensional image according to the position of the center line in the two-dimensional image; the first groove region and the second groove region are respectively located on both sides of the center line;

[0051] A fourth determination unit, configured to determine weld feature points according to the first groove region, the second groove region, and the first target point cloud data.

[0052] Optionally, the fourth determination unit includes:

[0053] A first determination subunit, configured to determine, according to the position of a first groove region in a two-dimensional image, second target point cloud data in the three-dimensional point cloud corresponding to the first groove region, and determine, according to the position of a second groove region in the two-dimensional image, third target point cloud data in the three-dimensional point cloud corresponding to the second groove region;

[0054] A fitting subunit, configured to fit the second target point cloud data into a first plane, and fit the third target point cloud data into a second plane;

[0055] A second determination subunit, configured to determine the intersection line between the first plane and the second plane;

[0056] A third determination subunit, configured to use the point cloud in the first target point cloud data whose distance from the intersection line is less than a preset distance value as the weld feature points.

[0057] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the welding teaching method described above is implemented.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the welding teaching method described above is implemented.

[0059] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device is enabled to execute the welding teaching method according to any one of the first aspects above.

[0060] The beneficial effects of the embodiments of the present application compared with the prior art are as follows:

[0061] In the embodiments of the present application, by acquiring a two-dimensional image and a three-dimensional point cloud of a workpiece to be welded, then using a weld target detection model to perform rough visual positioning on the two-dimensional image to obtain a weld detection result, and finally combining the three-dimensional point cloud of the workpiece to be welded and the weld detection result of the rough visual positioning to obtain a weld teaching path. Since after obtaining the weld detection result of the rough visual positioning, determining the weld teaching path in combination with the three-dimensional point cloud can accurately retain the three-dimensional information, thus effectively solving the problem of incorrect weld recognition caused by image noise and non-weld interference, and improving the accuracy of automatic welding teaching. Description of the Drawings

[0062] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0063] Figure 1 is a flowchart of a welding teaching method provided by an embodiment of the present application;

[0064] Figure 2 is a schematic diagram of a prediction area in an example of the present application;

[0065] Figure 3 is a flowchart of determining weld feature points provided by an embodiment of the present application;

[0066] Figure 4 is a flowchart of the first specific implementation manner of step 32 provided by an embodiment of the present application;

[0067] Figure 5 is a schematic diagram of a bounding box coordinate system provided by an embodiment of the present application;

[0068] Figure 6 is a schematic diagram of the first target point cloud data and a slicing plane in an example of the present application;

[0069] Figure 7 is a flowchart of the second specific implementation manner of step 32 provided by an embodiment of the present application;

[0070] Figure 8 is a schematic diagram of a groove area in an example of the present application;

[0071] Figure 9 is a schematic structural diagram of a welding teaching device provided by an embodiment of the present application;

[0072] Figure 10 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0073] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from hindering the description of the present application.

[0074] It should be understood that, as used in the specification of this application and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.

[0075] It should also be understood that the term "and / or" as used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0076] As used in the specification of this application and the appended claims, the term "if" may be construed, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrases "if determined" or "if [the described condition or event] is detected" may be construed, depending on the context, as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0077] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0078] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0079] Currently, during automatic welding teaching, a large amount of image noise and non-weld interference are introduced into the image by oxidation slag, paint, water, rust, etc. on the welding workpiece, and traditional image preprocessing methods cannot effectively eliminate the image noise and non-weld interference, resulting in incorrect weld recognition and unsatisfactory accuracy of automatic welding teaching.

[0080] In view of the above problems, in the embodiments of the present application, a two-dimensional image and a three-dimensional point cloud of the workpiece to be welded are obtained, and then a weld target detection model is used to perform rough visual positioning on the two-dimensional image to obtain a weld detection result. Finally, the three-dimensional point cloud of the workpiece to be welded and the weld detection result of the rough visual positioning are combined to obtain a weld teaching path. Since after obtaining the weld detection result of the rough visual positioning, combining the three-dimensional point cloud to determine the weld teaching path can accurately retain the three-dimensional information, effectively solving the problem of misidentification of welds caused by image noise and non-weld interference, and improving the accuracy of automatic welding teaching.

[0081] The welding teaching method provided by the present application will be exemplarily described below in conjunction with specific embodiments.

[0082] The embodiments of the present application provide a welding teaching method. This method can be executed by a terminal device or by a device (such as a chip) applied to the terminal device. The following embodiments take the method being executed by the terminal device as an example. As an example, the terminal device can be a tablet, a server, a laptop computer, etc., and the embodiments of the present application do not limit this.

[0083] As Figure 1 shown, the welding teaching method provided by the embodiments of the present application includes the following steps:

[0084] Step 11, obtain a two-dimensional image and a three-dimensional point cloud of the workpiece to be welded.

[0085] In some embodiments of the present application, the acquisition process of the above two-dimensional image and three-dimensional point cloud can be as follows: Place the workpiece to be welded on the workbench, install the acquisition device for acquiring the above two-dimensional image and three-dimensional point cloud above the workpiece to be welded, and then use the acquisition device to acquire the two-dimensional image and three-dimensional point cloud of the workpiece to be welded. The device parameters and installation height of the acquisition device can be adjusted according to the size of the workpiece to be welded.

[0086] Exemplarily, in some embodiments of the present application, the terminal device can obtain the two-dimensional image and three-dimensional point cloud of the workpiece to be welded from the above acquisition device, or can also use its own image acquisition function to acquire the two-dimensional image and three-dimensional point cloud of the workpiece to be welded.

[0087] Step 12, input the two-dimensional image into the weld target detection model to obtain a weld detection result.

[0088] In some embodiments of the present application, the above weld detection result includes the type of the weld of the workpiece to be welded and the predicted region of the weld in the two-dimensional image; the above weld target detection model is mainly used to detect the two-dimensional image to obtain the type of the weld and the predicted region of the weld in the two-dimensional image.

[0089] For example, assume that the workpiece to be welded is a T-shaped workpiece. After detecting the two-dimensional image of the workpiece using the weld seam target detection model, the weld seam detection results are as follows: There are 7 fillet welds and 1 butt weld on the workpiece. The predicted regions corresponding to the 7 fillet welds are respectively Figure 2 the dashed boxes A, B, C, D, E, F, G in Figure 2 and the predicted region corresponding to the 1 butt weld is

[0090] the dashed box H in. In some embodiments of the present application, the above weld seam target detection model can be a Faster RCNN recognition model. It should be noted that in order to improve the network feature extraction ability, the VGG network in the Faster RCNN recognition model is replaced by a ResNet network.

[0091] It should be further noted that since weld seam targets are generally small-sized and slender-sized targets, the Faster RCNN recognition model needs to be improved to adapt to the actual application scenario. Specifically, the size of the generated bounding box (i.e., the above predicted region) can be modified as follows: modify the area to 32 2 , 64 2 , 128 2 , and modify the aspect ratio to 1:1, 1:3, 3:1. In addition, to improve the recognition accuracy of small-sized targets, the output size of the pooling layer can also be reduced, and the output of the fourth convolutional layer (Conv-4) can be used to replace the output of the fifth convolutional layer (Conv-5).

[0092] It can be understood that in some embodiments of the present application, in order to improve the accuracy of the weld seam detection results, before detecting the two-dimensional image using the Faster RCNN recognition model, the Faster RCNN recognition model needs to be first. Among them, the training process of the Faster RCNN recognition model is as follows: First, collect a large number of sample pictures (the sample pictures are two-dimensional images) containing butt welds, fillet welds, and noise interference (such as non-weld interference like workbenches), and label the butt welds and fillet welds in the sample pictures; then use the labeled sample pictures to train the Faster RCNN recognition model to obtain the trained Faster RCNN recognition model.

[0093] Step 13, obtain the weld seam teaching path of the workpiece to be welded according to the weld seam detection results and the three-dimensional point cloud.

[0094] It is worth mentioning that in some embodiments of the present application, by acquiring the two-dimensional image and three-dimensional point cloud of the workpiece to be welded, then using the weld seam target detection model to perform rough visual positioning on the two-dimensional image to obtain the weld seam detection result, and finally combining the three-dimensional point cloud of the workpiece to be welded and the weld seam detection result of the rough visual positioning to obtain the weld seam teaching path. Among them, since after obtaining the weld seam detection result of the rough visual positioning, determining the weld seam teaching path by combining the three-dimensional point cloud can accurately retain the three-dimensional information, thus effectively solving the problem of misidentification of the weld seam caused by image noise and non-weld seam interference, and improving the accuracy of welding automatic teaching.

[0095] The following will exemplarily illustrate the specific implementation process of obtaining the weld seam teaching path of the workpiece to be welded according to the weld seam detection result and the three-dimensional point cloud in combination with specific embodiments.

[0096] In some embodiments of the present application, the above-mentioned weld seam teaching path includes weld seam feature points and weld seam teaching poses. Correspondingly, the specific implementation manner of step 13, obtaining the weld seam teaching path of the workpiece to be welded according to the weld seam detection result and the three-dimensional point cloud, can be: determining the weld seam feature points according to the weld seam detection result and the three-dimensional point cloud, and determining the weld seam teaching poses according to the coordinates of the weld seam feature points.

[0097] The number of the above-mentioned weld seam feature points is multiple. Correspondingly, in some embodiments of the present application, after determining the weld seam feature points, for each weld seam feature point, the normal vector of the weld seam feature point can be determined according to the coordinates of the weld seam feature point; then the average value of each normal vector is used as the weld seam teaching pose.

[0098] In some embodiments of the present application, as Figure 3 shown, the specific implementation manner of determining the weld seam feature points according to the weld seam detection result and the three-dimensional point cloud may include the following steps:

[0099] Step 31, determine the first target point cloud data corresponding to the prediction area in the three-dimensional point cloud according to the position of the prediction area in the two-dimensional image.

[0100] In some embodiments of the present application, the first target point cloud data corresponding to the prediction area can be determined in the three-dimensional point cloud by the projection method.

[0101] Step 32, determine the weld seam feature points according to the type of the weld seam of the workpiece to be welded and the first target point cloud data.

[0102] It should be noted that when the weld seam detection result indicates that there are multiple weld seams on the workpiece to be welded, the above steps 31 to 32 need to be executed respectively for the prediction area corresponding to each weld seam to determine the weld seam teaching poses of all the weld seams of the workpiece to be welded.

[0103] In some embodiments of the present application, the types of weld seams of the workpiece to be welded described above include fillet welds and butt welds. Correspondingly, the above step 32 includes two specific implementation manners.

[0104] As Figure 4 shown, for the above step 32, a first specific implementation manner for determining the weld seam feature points according to the type of the weld seam of the workpiece to be welded and the first target point cloud data includes the following steps:

[0105] Step 41, when the weld seam of the workpiece to be welded is a fillet weld, obtain a bounding box enclosing the first target point cloud data.

[0106] In the related art, the bounding box is also called the minimum circumscribed cuboid, and can be specifically obtained by an algorithm for solving the optimal bounding space of the discrete point set. That is, in some embodiments of the present application, all the point clouds in the above first target point cloud data are within the bounding box in step 41.

[0107] Step 42, insert a plurality of slicing planes into the bounding box.

[0108] The above slicing planes are perpendicular to the first coordinate axis of the bounding box coordinate system. The bounding box coordinate system takes a vertex of the bounding box as the origin, and the first coordinate axis is parallel to the longest side of the bounding box.

[0109] In some embodiments of the present application, as Figure 5 shown, the above bounding box coordinate system can be a space rectangular coordinate system XYZ constructed with any vertex (such as Figure 5 the vertex O in Figure 5 shown) of the bounding box (such as the cuboid V shown) as the origin, the first coordinate axis is the coordinate axis Y, and a plurality of slicing planes can be a set of slicing planes inserted at equal intervals in the bounding box, Figure 5 only one slicing plane U is shown.

[0110] Step 43, determine the weld seam feature points according to the plurality of slicing planes and the bounding box.

[0111] In some embodiments of the present application, all the weld seam feature points of the weld seam can be determined by determining the weld seam feature points in each slicing plane.

[0112] Specifically, to determine the weld seam feature points in each slicing plane, the following steps can be respectively executed for each slicing plane:

[0113] Step 1, determine the point clouds in the first target point cloud data that are located on the slicing plane.

[0114] For ease of understanding, a specific example is used here to explain the prediction area, slice plane, etc. of the fillet weld. In this example, it is assumed that the fillet weld is the weld between the first welding surface and the second welding surface of the workpiece to be welded. The prediction area corresponding to this fillet weld includes: the part on the first welding surface adjacent to the second welding surface, and the part on the second welding surface adjacent to the first welding surface; a part of the point cloud in the first target point cloud data corresponding to this prediction area (this part of the point cloud is the point cloud of the part on the first welding surface adjacent to the second welding surface) is the point cloud within the solid line frame X shown in Figure 6 (not shown in the figure), and another part of the point cloud (this part of the point cloud is the point cloud of the part on the second welding surface adjacent to the first welding surface) is the point cloud within the solid line frame Y shown in Figure 6 (not shown in the figure). Figure 6 The plane represented by the dashed line frame Z in Figure 6 is the inserted slice plane, and the intersection point cloud of the slice plane and the first target point cloud data (such as the point cloud represented by the dashed line M within the solid line frame X and the point cloud represented by the dashed line N within the solid line frame Y in

[0115] Step two, perform linear fitting on the point cloud located on the slice plane to obtain the point cloud located on the first straight line and the point cloud located on the second straight line.

[0116] In some embodiments of the present application, the random sample consensus (RANSAC) algorithm can be used to perform linear fitting on the point cloud located on the slice plane to obtain the first straight line and the second straight line. It can be understood that during the process of using the RANSAC algorithm to determine the first straight line and the second straight line, the point cloud located on the slice plane that is located on the first straight line and the point cloud located on the second straight line can be determined.

[0117] Step three, determine the first point cloud and the second point cloud with the maximum distance on the first straight line, and the third point cloud and the fourth point cloud with the maximum distance on the second straight line.

[0118] In some embodiments of the present application, for all the point clouds located on the first straight line, the distance between each two point clouds can be calculated, and the two point clouds corresponding to the maximum distance among the calculated distances are used as the first point cloud and the second point cloud.

[0119] Similarly, for all the point clouds located on the second straight line, the distance between each two point clouds can be calculated, and the two point clouds corresponding to the maximum distance among the calculated distances are used as the third point cloud and the fourth point cloud.

[0120] Step 4: Remove the two point clouds with the largest mutual distance among the first point cloud, the second point cloud, the third point cloud, and the fourth point cloud, and use the point cloud with the larger coordinate value on the second coordinate axis among the remaining two point clouds as the weld feature point.

[0121] The above-mentioned second coordinate axis is the second coordinate axis in the above-mentioned bounding box coordinate system (such as Figure 5 the coordinate axis Z in), and this second coordinate axis is perpendicular to the imaging plane of the acquisition device for acquiring the two-dimensional image.

[0122] In some embodiments of the present application, the specific implementation process of the above Step 4 is as follows: First, calculate the distance between every two point clouds among the first point cloud, the second point cloud, the third point cloud, and the fourth point cloud; then determine the two point clouds corresponding to the maximum distance among the calculated distances, remove these two point clouds, and use the point cloud with the larger coordinate value on the second coordinate axis among the remaining two point clouds as the weld feature point.

[0123] As Figure 7 shown, the above Step 32, the second specific implementation manner for determining the weld feature point according to the type of the weld of the workpiece to be welded and the first target point cloud data includes the following steps:

[0124] Step 71: When the weld of the workpiece to be welded is a butt weld, determine the center line of the butt weld in the two-dimensional image.

[0125] In some embodiments of the present application, the center line of the butt weld can be determined based on the prediction region corresponding to the butt weld in the two-dimensional image.

[0126] As a preferred example, to improve the determination speed of the center line, the prediction region can be reduced according to a preset reduction ratio (such as 60%).

[0127] In some embodiments of the present application, since the gray value change of the center line of the butt weld is relatively obvious, the center line of the butt weld can be extracted from the part of the two-dimensional image corresponding to the prediction region through the Canny edge detection algorithm.

[0128] In a possible embodiment of the present application, since the prediction region is a long and straight region, the center line perpendicular to the short side of the prediction region in the prediction region can be directly used as the center line of the butt weld.

[0129] Step 72: Determine the first groove region and the second groove region in the two-dimensional image according to the position of the center line in the two-dimensional image.

[0130] The above-mentioned first groove region and second groove region are respectively located on both sides of the center line. In some embodiments of the present application, in a two-dimensional image, a region with a preset area on one side of the center line can be used as the first groove region, and a region with a preset area on the other side of the center line can be used as the second groove region. Among them, the above-mentioned preset area can be the area corresponding to a number of (such as 8) pixel points.

[0131] Step 73, determine the weld feature points according to the first groove region, the second groove region, and the first target point cloud data.

[0132] In some embodiments of the present application, the specific implementation manner of the above step 73 includes the following steps:

[0133] The first step is to determine the second target point cloud data corresponding to the first groove region in the three-dimensional point cloud according to the position of the first groove region in the two-dimensional image, and determine the third target point cloud data corresponding to the second groove region in the three-dimensional point cloud according to the position of the second groove region in the two-dimensional image.

[0134] In some embodiments of the present application, the second target point cloud data corresponding to the first groove region in the three-dimensional point cloud and the third target point cloud data corresponding to the second groove region can be determined by the projection method.

[0135] The second step is to fit the second target point cloud data into a first plane and fit the third target point cloud data into a second plane.

[0136] In some embodiments of the present application, the point cloud of the second target point cloud data can be fitted by a common point cloud plane fitting algorithm to obtain the first plane. Similarly, the point cloud of the third target point cloud data can also be fitted by a common point cloud plane fitting algorithm to obtain the second plane.

[0137] The third step is to determine the intersection line of the first plane and the second plane.

[0138] For ease of understanding, a specific example is used here to explain the first groove region, the second groove region, the first plane, the second plane, etc. In this example, it is assumed that the butt weld is the weld of the third welding surface and the fourth welding surface of the workpiece to be welded, and the groove of the butt weld includes the groove of the third welding surface (such as Figure 8 the square region P in) and the groove of the fourth welding surface (such as Figure 8 the square region Q in), the second target point cloud data corresponding to the first groove region is the point cloud (not shown in the figure) within the solid line frame I as shown in Figure 8 , the third target point cloud data corresponding to the second groove region is the point cloud (not shown in the figure) within the solid line frame J as shown in Figure 8 , and the first plane obtained by fitting the second target point cloud data is Figure 8The plane S in it, and the second plane obtained by fitting the third target point cloud data is Figure 8 The plane T in it, and the intersection line of the first plane and the second plane is Figure 8 The straight line L in it.

[0139] Fourthly, the point clouds in the first target point cloud data whose distances from the intersection line are less than the preset distance value are used as weld feature points.

[0140] In some embodiments of the present application, the specific implementation manner of the above fourth step may be: for each point cloud in the first target point cloud data, calculate the distance between the point cloud and the intersection line, and determine whether the distance is less than the preset distance value. If so, use the point cloud as a weld feature point; otherwise, determine that the point cloud is not a weld feature point.

[0141] It can be seen that in some embodiments of the present application, during the automatic teaching of the weld, whether it is a fillet weld or a butt weld, the weld teaching path is determined based on the predicted region of the weld in the two-dimensional image and the three-dimensional point cloud data corresponding to the predicted region. It should be noted that since the three-dimensional point cloud can accurately retain the three-dimensional information, the welding teaching method provided by the embodiments of the present application can automatically, quickly and accurately complete the weld teaching while effectively eliminating image noise and non-weld interference.

[0142] In summary, the welding teaching method provided by the embodiments of the present application has the following effects:

[0143] First, during the automatic teaching of the weld, the weld teaching is realized by combining the two-dimensional image and the three-dimensional point cloud of the workpiece to be welded, which can effectively eliminate image noise and non-weld interference and improve the accuracy of automatic weld teaching;

[0144] Second, the weld teaching process is a full-automatic teaching process. Therefore, the welding teaching method provided by the embodiments of the present application has the advantages of strong universality, high speed, strong robustness and wide application scenarios.

[0145] Next, the welding teaching device provided by the present application will be exemplarily described with specific embodiments.

[0146] As Figure 9 shown, an embodiment of the present application provides a welding teaching device, and the welding teaching device 900 includes:

[0147] An acquisition module 901, configured to acquire a two-dimensional image and a three-dimensional point cloud of the workpiece to be welded;

[0148] A detection module 902, configured to input the two-dimensional image into a weld target detection model to obtain a weld detection result;

[0149] A determination module 903, configured to obtain a weld teaching path of a workpiece to be welded according to a weld detection result and a three-dimensional point cloud.

[0150] Optionally, the weld teaching path includes weld feature points and weld teaching poses. The determination module 903 is specifically configured to determine the weld feature points according to the weld detection result and the three-dimensional point cloud, and determine the weld teaching poses according to the coordinates of the weld feature points.

[0151] Optionally, the weld detection result includes the type of the weld of the workpiece to be welded and the predicted region of the weld in a two-dimensional image. The determination module 903 includes:

[0152] A first determination sub-module, configured to determine first target point cloud data corresponding to the predicted region in the three-dimensional point cloud according to the position of the predicted region in the two-dimensional image;

[0153] A second determination sub-module, configured to determine the weld feature points according to the type of the weld of the workpiece to be welded and the first target point cloud data.

[0154] Optionally, the second determination sub-module includes:

[0155] An acquisition unit, configured to obtain a bounding box enclosing the first target point cloud data when the weld of the workpiece to be welded is a fillet weld;

[0156] An insertion unit, configured to insert a plurality of slicing planes into the bounding box; the slicing planes are perpendicular to a first coordinate axis of the bounding box coordinate system, the bounding box coordinate system takes a vertex of the bounding box as the origin, and the first coordinate axis is parallel to the longest side of the bounding box;

[0157] A first determination unit, configured to determine the weld feature points according to the plurality of slicing planes and the bounding box.

[0158] Optionally, the first determination unit is specifically configured to perform the following steps for each slicing plane: determine the point cloud in the first target point cloud data that lies on the slicing plane; perform linear fitting on the point cloud that lies on the slicing plane to obtain the point cloud on a first straight line and the point cloud on a second straight line; determine the first point cloud and the second point cloud with the maximum distance on the first straight line, and the third point cloud and the fourth point cloud with the maximum distance on the second straight line; remove two point clouds with the maximum mutual distance among the first point cloud, the second point cloud, the third point cloud, and the fourth point cloud, and use the point cloud with the larger coordinate value on a second coordinate axis of the bounding box coordinate system among the remaining two point clouds as the weld feature point; the second coordinate axis is perpendicular to the imaging plane of the acquisition device for acquiring the two-dimensional image.

[0159] Optionally, the second determination sub-module includes:

[0160] A second determination unit, configured to determine the center line of the butt weld in the two-dimensional image when the weld of the workpiece to be welded is a butt weld;

[0161] A third determination unit, configured to determine a first groove area and a second groove area in the two-dimensional image according to the position of the center line in the two-dimensional image; the first groove area and the second groove area are respectively located on both sides of the center line;

[0162] A fourth determination unit, configured to determine weld feature points according to the first groove area, the second groove area, and the first target point cloud data.

[0163] Optionally, the fourth determination unit includes:

[0164] A first determination subunit, configured to determine second target point cloud data corresponding to the first groove area in the three-dimensional point cloud according to the position of the first groove area in the two-dimensional image, and determine third target point cloud data corresponding to the second groove area in the three-dimensional point cloud according to the position of the second groove area in the two-dimensional image;

[0165] A fitting subunit, configured to fit the second target point cloud data into a first plane and fit the third target point cloud data into a second plane;

[0166] A second determination subunit, configured to determine the intersection line of the first plane and the second plane;

[0167] A third determination subunit, configured to use the points in the first target point cloud data whose distance from the intersection line is less than a preset distance value as weld feature points.

[0168] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiment of the present application, their specific functions and the technical effects brought thereby can be specifically referred to in the method embodiment part, and will not be elaborated here.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments and will not be elaborated herein.

[0170] As Figure 10 shown, an embodiment of the present application provides a terminal device. As Figure 10 shown, the terminal device D10 in this embodiment includes: at least one processor D100 ( Figure 10 only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, the steps in any of the foregoing method embodiments are implemented.

[0171] The so-called processor D100 may be a central processing unit (CPU, Central Processing Unit), and this processor D100 may also be other general-purpose processors, digital signal processors (DSP, Digital Signal Processor), application specific integrated circuits (ASIC, Application Specific Integrated Circuit), field-programmable gate arrays (FPGA, Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0172] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk equipped on the terminal device D10, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory D101 may also include both the internal storage unit and the external storage device of the terminal device D10. The memory D101 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as program codes of the computer program. The memory D101 may also be used to temporarily store data that has been output or will be output.

[0173] It should be noted that, for the content such as information interaction and execution process between the above-mentioned device / units, since it is based on the same concept as the method embodiment of the present application, for its specific functions and the technical effects brought, reference may be specifically made to the method embodiment part, and details are not described herein again.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example for illustration. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

[0175] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0176] The embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device is enabled to execute the steps in the above-mentioned method embodiments.

[0177] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to the welding teaching device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0178] In the above embodiments, the descriptions of each embodiment have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0179] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0180] In the embodiments provided in this application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical, or other form.

[0181] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A welding teaching method, characterized in that, Including: Obtain a two-dimensional image and a three-dimensional point cloud of the workpiece to be welded; Input the two-dimensional image into a weld seam target detection model to obtain a weld seam detection result; Obtain a weld seam teaching path of the workpiece to be welded according to the weld seam detection result and the three-dimensional point cloud; The weld seam teaching path includes weld seam feature points and weld seam teaching poses. The step of obtaining the weld seam teaching path of the workpiece to be welded according to the weld seam detection result and the three-dimensional point cloud includes: Determine the weld seam feature points according to the weld seam detection result and the three-dimensional point cloud; Determine the weld seam teaching poses according to the coordinates of the weld seam feature points; The weld seam detection result includes the type of the weld seam of the workpiece to be welded and the predicted region of the weld seam in the two-dimensional image; The step of determining the weld seam feature points according to the weld seam detection result and the three-dimensional point cloud includes: Determine first target point cloud data corresponding to the predicted region in the three-dimensional point cloud according to the position of the predicted region in the two-dimensional image; Determine the weld seam feature points according to the type of the weld seam of the workpiece to be welded and the first target point cloud data; The step of determining the weld seam feature points according to the type of the weld seam of the workpiece to be welded and the first target point cloud data includes: When the weld seam of the workpiece to be welded is a fillet weld, obtain a bounding box enclosing the first target point cloud data; Insert a plurality of slicing planes into the bounding box; the slicing planes are perpendicular to a first coordinate axis of the bounding box coordinate system, the bounding box coordinate system takes a vertex of the bounding box as the origin, and the first coordinate axis is parallel to the longest side of the bounding box; Determine the weld seam feature points according to the plurality of slicing planes and the bounding box.

2. The method according to claim 1, characterized in that, The step of determining the weld seam feature points according to the plurality of slicing planes and the bounding box includes: For each slicing plane respectively, perform the following steps: Determine the point cloud located on the slicing plane in the first target point cloud data; Perform linear fitting on the point cloud located on the slicing plane to obtain the point cloud located on a first straight line and the point cloud located on a second straight line; Determine the first point cloud and the second point cloud with the largest distance on the first straight line, and the third point cloud and the fourth point cloud with the largest distance on the second straight line; Remove two point clouds with the largest mutual distance among the first point cloud, the second point cloud, the third point cloud, and the fourth point cloud, and use the point cloud with a larger coordinate value on a second coordinate axis of the bounding box coordinate system among the remaining two point clouds as the weld seam feature point; the second coordinate axis is perpendicular to the imaging plane of the acquisition device for acquiring the two-dimensional image.

3. The method according to claim 1, characterized in that, The step of determining the weld seam feature points according to the type of the weld seam of the workpiece to be welded and the first target point cloud data includes: When the weld seam of the workpiece to be welded is a butt weld, determine the center line of the butt weld in the two-dimensional image; Determine a first groove region and a second groove region in the two-dimensional image according to the position of the center line in the two-dimensional image; the first groove region and the second groove region are respectively located on both sides of the center line; Determine the weld feature points according to the first groove region, the second groove region, and the first target point cloud data.

4. The method according to claim 3, wherein The determining the weld feature points according to the first groove region, the second groove region, and the first target point cloud data includes: Determine the second target point cloud data corresponding to the first groove region in the three-dimensional point cloud according to the position of the first groove region in the two-dimensional image, and determine the third target point cloud data corresponding to the second groove region in the three-dimensional point cloud according to the position of the second groove region in the two-dimensional image; Fit the second target point cloud data into a first plane, and fit the third target point cloud data into a second plane; Determine the intersection line of the first plane and the second plane; Use the points in the first target point cloud data whose distance from the intersection line is less than a preset distance value as the weld feature points.

5. A welding teaching device, characterized in that, Includes: An acquisition module for acquiring a two-dimensional image and a three-dimensional point cloud of the workpiece to be welded; A detection module for inputting the two-dimensional image into a weld target detection model to obtain a weld detection result; A determination module for obtaining the weld teaching path of the workpiece to be welded according to the weld detection result and the three-dimensional point cloud; The weld teaching path includes weld feature points and weld teaching poses. The determination module is specifically configured to determine the weld feature points according to the weld detection result and the three-dimensional point cloud, and determine the weld teaching poses according to the coordinates of the weld feature points; The weld detection result includes the type of the weld of the workpiece to be welded and the predicted region of the weld in the two-dimensional image; The determination module includes: A first determination sub-module for determining the first target point cloud data corresponding to the predicted region in the three-dimensional point cloud according to the position of the predicted region in the two-dimensional image; A second determination sub-module for determining the weld feature points according to the type of the weld of the workpiece to be welded and the first target point cloud data; The second determination sub-module includes: An acquisition unit for acquiring a bounding box enclosing the first target point cloud data when the weld of the workpiece to be welded is a fillet weld; An insertion unit for inserting a plurality of slicing planes into the bounding box; the slicing planes are perpendicular to the first coordinate axis of the bounding box coordinate system, the bounding box coordinate system takes a vertex of the bounding box as the origin, and the first coordinate axis is parallel to the longest side of the bounding box; A first determination unit for determining the weld feature points according to the plurality of slicing planes and the bounding box.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the welding teaching method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the welding teaching method according to any one of claims 1 to 4.

Images