Welding track positioning method, device and equipment based on trinocular vision system

Abstract

The invention provides a welding track positioning method, device and equipment based on a trinocular vision system.The method comprises the steps that a first image pair and a second image pair of shooting positions are obtained according to the shooting positions, the first image pair comprises a first image shot by a first camera and a second image shot by a second camera, and the second image pair comprises a second image shot by a second camera; the second image pair comprises a third image shot by the first camera and a fourth image shot by the third camera; determining a target image pair based on the average orientation angle of the welding instance in the first image; determining a second edge line matched with the first edge line of the main image welding instance in the auxiliary image of the target image pair; determining second pixel points matched with the first pixel points in the first edge line based on the second edge line; based on the respective coordinates of the first pixel point and the second pixel point, determining a homogeneous coordinate of the first pixel point in a robot base coordinate system; and positioning a welding track based on the homogeneous coordinates of all the first pixel points. The positioning accuracy of the welding track can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a welding track positioning method, device and equipment based on a trinocular vision system. Background Art

[0002] With the rapid development of industrial automation, automated equipment such as welding robots have become widely used in welding production sites, significantly improving welding production efficiency and automation levels. However, during the actual welding process, due to inevitable dimensional, geometric, and installation errors in the weldment, coupled with the influence of welding thermal deformation, the actual welding trajectory may deviate from the preset trajectory. To ensure welding quality, welding robots need to be able to automatically sense the welding trajectory and correct the welding path in real time.

[0003] In existing technology, welding robots primarily use structured light vision sensing devices to automatically identify and locate welding paths. The basic principle of this method is to project a one-dimensional linear structured light stripe or a two-dimensional surface structured light pattern onto the weldment using a structured light projection device. A vision sensor captures the image and analyzes the distorted stripes or pattern to identify the welding path. This is because the distortion of the stripes or pattern reflects the geometric characteristics of the weld, enabling precise tracking of the welding path.

[0004] However, when the projected stripes or patterns have difficulty producing recognizable distortion on the weld, the geometric features of the weld will become unclear. In this case, structured light vision sensing will be unable to reliably identify and locate the welding track. In addition, for welds with highly reflective surfaces such as aluminum alloys and stainless steel, the projected structured light is easily reflected, resulting in most of the reflected light not being received by the visual sensing device. This will cause the stripes in the captured image to be greatly weakened or even lost, making it impossible to complete the identification of the welding track. Therefore, when using existing methods to identify welding tracks, the reliability of the located welding tracks will be low in scenarios such as multi-layer and multi-pass welding, narrow gap welding, or welding of workpieces with highly reflective surfaces. Summary of the Invention

[0005] The present invention provides a welding track positioning method, device and equipment based on a trinocular vision system, which is used to solve the defect of low reliability of the positioned welding track in scenarios such as multi-layer and multi-pass welding, narrow gap welding or welding of workpieces with strongly reflective surfaces in the prior art. The method can improve the positioning reliability of the welding track in the above-mentioned specific scenarios and reliably position the welding track in other welding scenarios.

[0006] The present invention provides a welding track positioning method based on a trinocular vision system, comprising: For each preset shooting position, a first image pair and a second image pair are acquired at the shooting position, where the first image pair includes a first image captured by a first camera and a second image captured by a second camera, and the second image pair includes a third image captured by the first camera and a fourth image captured by a third camera. The first camera, the second camera, and the third camera are fixed to a welding gun of a welding robot, optical centers of lenses of the first camera, the second camera, and the third camera form a target plane, and an angle between a straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and a straight line formed by the optical centers of the lenses of the first camera and the third camera is within a first preset angle range. determining a target image pair from the first image pair and the second image pair based on an average orientation angle of all welding instances in the first image; Determining, in a binary mask image of the welding instance corresponding to the secondary image of the target image pair, a second edge line that matches a first edge line of the welding instance included in a primary image of the target image pair; the primary image being an image captured by the first camera; Determining, based on the second edge line, second pixel points that match each first pixel point in the first edge line; Determining homogeneous coordinates of the first pixel point in a robot base coordinate system based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point; The welding trajectory is located based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system.

[0007] According to a welding trajectory positioning method based on a trinocular vision system provided by the present invention, determining a target image pair from the first image pair and the second image pair based on the average orientation angle of all welding instances in the first image includes: Determine the average orientation angle of all welding instances in the first image based on the following formula (1); (1) in, represents the average orientation angle of all welding instances, N represents the number of welding instances, 、 and represents the second-order moment of the image of the i-th welding instance, , when p is 1, q is 1, when p is 2, q is 0, when p is 0, q is 2, represents the geometric center coordinates of the i-th welding instance, represents the binary mask of the i-th welding instance, Represents pixel coordinates; determining the second image pair as the target image pair when the angle between the average orientation angle and the horizontal coordinate axis of the image is within a second preset angle range, wherein the second preset angle range is used to indicate that the orientation of the welding instance in the image coordinate system is a horizontal orientation; When the included angle between the average orientation angle and the horizontal coordinate axis of the image is not within the second preset angle range, the first image pair is determined as the target image pair.

[0008] According to a welding trajectory positioning method based on a trinocular vision system provided by the present invention, determining a second edge line that matches a first edge line of the welding instance included in a primary image of the target image pair in a binary mask image of the welding instance corresponding to a secondary image of the target image pair includes: Determining, by a forward difference algorithm, a first edge line of each welding instance in the welding instance binary mask image corresponding to the primary image, and determining a third edge line of each welding instance in the welding instance binary mask image corresponding to the secondary image; For each of the first edge lines, based on the position of the first edge line in the welding instance binary mask image corresponding to the main image and the position of each of the third edge lines in the welding instance binary mask image corresponding to the secondary image, determine the second edge line that matches the first edge line among all the third edge lines.

[0009] According to a welding trajectory positioning method based on a trinocular vision system provided by the present invention, determining a second pixel point matching each first pixel point in the first edge line based on the second edge line includes: For each of the first pixel points, intercepting a local image block with the first pixel point as the center; Taking the third pixel point corresponding to the first pixel point in the second edge line as the midpoint, intercepting a search window including a preset number of target pixel points; For each target pixel point in the search window, intercepting a target image block having the same size as the local image block with the target pixel point as the center; The similarities between each of the target image blocks and the local image blocks are calculated respectively, and the target pixel corresponding to the target image block with the maximum similarity is determined as the second pixel matching the first pixel.

[0010] According to a welding trajectory positioning method based on a trinocular vision system provided by the present invention, determining the homogeneous coordinates of the first pixel point in the robot base coordinate system based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point includes: The homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera are determined based on the following formula (2): (2) in, represents the homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera, represents the coordinates of the first pixel point on the first edge line in the main image, represents the pixel coordinates of the second pixel point, b represents the baseline of the first camera, f represents the focal length of the first camera, represents the coordinates of the principal point of the first camera, represents the coordinates of the principal point of the second camera; The homogeneous coordinates of the first pixel point in the robot base coordinate system are determined based on the following formula (3): (3) in, represents the homogeneous coordinates of the first pixel point in the robot base coordinate system, represents the posture of the welding gun, represents the homogeneous transformation matrix for hand-eye calibration, Represents a rotation matrix.

[0011] According to a welding trajectory positioning method based on a trinocular vision system provided by the present invention, the method further includes: Inputting the first image into an instance segmentation model, and outputting a binary mask image of the welding instance including the weldment base material when the instance segmentation model identifies that the first image includes a single-pass weld; and outputting a binary mask image of the welding instance including the weldment base material and each formed weld when the instance segmentation model identifies that the first image includes multiple-pass welds; The instance segmentation model is obtained by training an initial instance segmentation model using sample weldment images and annotation labels of each sample welding instance in the sample weldment images.

[0012] According to a welding trajectory positioning method based on a trinocular vision system provided by the present invention, obtaining the first image pair and the second image pair at the shooting position includes: Acquire a first initial image captured by the first camera at the shooting position, a second initial image captured by the second camera at the shooting position, and a third initial image captured by the third camera at the shooting position; Based on camera parameters of the first camera and camera parameters of the second camera, correcting the first initial image and the second initial image respectively using a binocular stereo correction algorithm to obtain a first image pair including the first image and the second image; Based on camera parameters of the first camera and camera parameters of the third camera, the first initial image and the third initial image are respectively corrected by a binocular stereo correction algorithm to obtain the second image pair including the third image and the fourth image.

[0013] The present invention also provides a welding track positioning device based on a trinocular vision system, comprising: an acquisition module configured to acquire, for each preset shooting position, a first image pair and a second image pair at the shooting position, the first image pair comprising a first image captured by a first camera and a second image captured by a second camera, and the second image pair comprising a third image captured by the first camera and a fourth image captured by a third camera, wherein the optical centers of the lenses of the first camera, the second camera, and the third camera form a target plane, and an angle between a straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and a straight line formed by the optical centers of the lenses of the first camera and the third camera is within a first preset angle range; a determination module, configured to determine a target image pair from the first image pair and the second image pair based on an average orientation angle of all welding instances in the first image; The determining module is further configured to determine, in a binary mask image of the welding instance corresponding to a secondary image of the target image pair, a second edge line that matches a first edge line of the welding instance included in a primary image of the target image pair; the primary image being an image captured by the first camera; The determining module is further configured to determine, based on the second edge line, second pixel points that match each first pixel point in the first edge line; The determining module is further configured to determine the homogeneous coordinates of the first pixel point in the robot base coordinate system based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point; A positioning module is used to locate the welding track based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the above-described welding trajectory positioning methods based on a trinocular vision system.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the welding trajectory positioning method based on the trinocular vision system as described above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described welding trajectory positioning methods based on a trinocular vision system.

[0017] The present invention provides a welding trajectory positioning method, device and equipment based on a trinocular vision system. For each pre-set shooting position, a first image pair and a second image pair are obtained at the shooting position, wherein the first image pair includes a first image taken by a first camera and a second image taken by a second camera, and the second image pair includes a third image taken by the first camera and a fourth image taken by the third camera. Based on the average orientation angle of all welding instances in the first image, a target image pair is determined from the first image pair and the second image pair. In the welding instance binary mask image corresponding to the secondary image of the target image pair, a second edge line matching the first edge line of the welding instance included in the main image in the target image pair is determined, and the main image is an image taken by the first camera. Based on the second edge line, a second pixel point matching each first pixel point in the first edge line is determined. Based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the robot base coordinate system are determined, and based on the homogeneous coordinates of all first pixel points in the robot base coordinate system, the welding trajectory is identified. Since the welding instance segmentation is performed on the image captured by the trinocular vision system, the edge lines of the welding instances in the binary mask images of the welding instances corresponding to the main image and the auxiliary image in the target image are identified and matched, and then the pixel points on the edge lines are accurately matched, the position of each pixel point on each edge line in the main image used to characterize the welding trajectory can be determined, thereby realizing the recognition of the welding trajectory. Since there is no need to rely on the structured light projection pattern, the welding trajectory recognition can be realized by only processing the image captured by the trinocular vision system. Therefore, it is not only possible to improve the positioning reliability of the welding trajectory in specific scenarios such as multi-layer and multi-pass welding, narrow gap welding or welding of workpieces with strongly reflective surfaces, but also to reliably locate the welding trajectory in other welding scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic structural diagram of a welding trajectory recognition system based on a trinocular vision system provided in an embodiment of the present invention.

[0020] Figure 2 A schematic diagram of the structure of a trinocular vision system provided in an embodiment of the present invention.

[0021] Figure 3 This is one of the flow charts of the welding trajectory positioning method based on the trinocular vision system provided in an embodiment of the present invention.

[0022] Figure 4 A schematic diagram of an example segmentation of a single weld image provided by an embodiment of the present invention.

[0023] Figure 5 A schematic diagram of an example segmentation of a multi-layer and multi-pass weld image provided by an embodiment of the present invention.

[0024] Figure 6 The second flow chart of the welding trajectory positioning method based on the trinocular vision system provided in an embodiment of the present invention.

[0025] Figure 7 A schematic structural diagram of a welding trajectory positioning device based on a trinocular vision system provided in an embodiment of the present invention.

[0026] Figure 8 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0028] In the prior art, structured light vision sensing equipment is incorporated into welding robots for identifying and locating welding tracks. A structured light projection device projects one-dimensional linear structured light stripes or two-dimensional surface structured light patterns onto the weldment. The image projected onto the weldment is then captured by a vision sensor. Because the distortion of the stripes or patterns can reflect the geometric characteristics of the weld, the weld track is identified by analyzing the distorted stripes or patterns in the image, achieving accurate tracking of the weld track. However, when the projected stripes or patterns have difficulty producing recognizable distortion on the weld, the weld's geometric features are unclear, making the structured light vision sensing device unable to reliably identify and locate the weld track. For example, in tracking fine-gap welds, industrial laser line structured light sensors often fail to produce deformed stripes on ultra-narrow gap welds. In multi-layer, multi-pass welding, high-level welds suffer from feature degradation, and the geometric differences between welds and between welds and the base material are minimal, making them difficult to distinguish based on stripe shape. Furthermore, weldments with highly reflective surfaces, such as aluminum alloys and stainless steel, can easily cause the projected structured light to reflect, preventing most of the reflected light from being received by the visual sensing device. This can significantly weaken or even eliminate the fringes in the captured image, making it impossible to identify the weld track. In summary, the existing approach of implementing weld track recognition by integrating structured light visual sensing devices into welding robots has low reliability in identifying and locating weld tracks in specific scenarios, such as multi-layer, multi-pass welding, narrow gap welding, or welding workpieces with highly reflective surfaces.

[0029] In view of the above problems, an embodiment of the present invention provides a welding track positioning method based on a trinocular vision system, in which the welding track can be accurately positioned based on the images captured by the trinocular vision system.

[0030] The following combination Figures 1 to 6 The welding trajectory positioning method based on the three-eye vision system provided in an embodiment of the present invention is described. The welding trajectory positioning method based on the three-eye vision system provided in an embodiment of the present invention can be applied to scenarios where welding trajectory identification is required during automatic welding, thereby performing welding path correction and automatic welding path planning. The execution subject of this method can be an electronic device such as a welding robot control cabinet, a welding robot control device, an industrial computer, a terminal device, a server, a server cluster, or a specially designed welding trajectory recognition device based on a three-eye vision system, or a welding trajectory positioning device based on a three-eye vision system provided in the electronic device. The welding trajectory positioning device based on the three-eye vision system can be implemented by software, hardware, or a combination of both.

[0031] Figure 1 A structural diagram of a welding trajectory recognition system based on a three-eye vision system provided in an embodiment of the present invention is shown in FIG. Figure 1As shown, the system includes a welding robot, a trinocular vision system, a robot control cabinet, and an industrial computer. The welding robot is used to weld the workpiece, the robot control cabinet is used to control the welding robot, and the industrial computer communicates with the robot control cabinet and the trinocular vision system via Ethernet to identify the welding trajectory. The trinocular vision system includes a first camera, a second camera, and a third camera for capturing images of the weld points of the workpiece. The first, second, and third cameras can be depth cameras or other cameras capable of capturing color images.

[0032] Figure 2 A schematic diagram of the structure of a three-eye vision system provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the first camera, the second camera and the third camera included in the three-camera vision system are of the same model and focal length, are rigidly connected by a bracket, and are fixed to the welding gun at the end of the welding robot. A rigid spatial relationship is maintained between the three cameras and between each camera and the welding gun. In addition, the orientation of the welding gun is roughly perpendicular to the plane formed by the optical centers of the lenses of the three cameras. The optical centers of the lenses of the first camera, the second camera and the third camera constitute a target plane, and the straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and the angle between the straight line formed by the optical centers of the lenses of the first camera and the third camera are within a first preset angle range, such as between +75° and +90°, and the distance between the first camera and the second camera is roughly the same as the distance between the first camera and the third camera. Specifically, the first camera and the second camera, and the first camera and the third camera respectively constitute a binocular vision system. The optical center of the first camera , the optical center of the second camera and the optical center of the third camera The determined plane is perpendicular to the direction of the welding gun, and the angles between the optical axes of the three camera lenses and the plane normal are all less than 5°. The two binocular vision systems are arranged in space to meet the following relationship: Optical center He Guangxin The rays formed are relative to the optical center He Guangxin , the angle range of the rays is between +75° and +90°. The spatial angles between the pixel plane coordinate axes of the images taken by the first camera and the second camera, and the first camera and the third camera are all less than 10°. and, and The ratio of the determined line segment lengths ranges from 0.8 to 1.25.

[0033] Figure 3 One of the flow charts of the welding trajectory positioning method based on the three-eye vision system provided in the embodiment of the present invention is as follows: Figure 3 As shown, the method includes: Step 301: For each pre-set shooting position, obtain a first image pair and a second image pair at the shooting position, the first image pair includes a first image taken by the first camera and a second image taken by the second camera, and the second image pair includes a third image taken by the first camera and a fourth image taken by the third camera.

[0034] for Figure 1 and Figure 2 The three-eye vision system shown in the figure can set the control point of the welding robot at the tungsten electrode of the welding gun or the tip of the welding wire through the tool center point (TCP) calibration. The first camera, the second camera, and the third camera independently complete the monocular camera calibration to determine the camera internal parameters. The first camera and the second camera, as well as the first camera and the third camera, respectively perform binocular calibration to determine the rotation matrix and displacement vector of the second camera and the third camera relative to the first camera. Finally, the calibration of the eye-to-hand conversion relationship between the first camera and the welding gun is completed to obtain the calibration result. ,Should Indicates the position of the welding gun.

[0035] In addition, a series of camera shooting positions can be pre-set for the welding robot, denoted as the first shooting position, the second shooting position, ..., and the Nth shooting position. When the welding robot is at the first shooting position, the fields of view of the first, second, and third cameras must fully cover the starting weld point of the weldment; when the welding robot is at the Nth shooting position, the fields of view of the first, second, and third cameras must fully cover the ending weld point of the weldment; and the combined fields of view of the first, second, and third cameras at the N shooting positions must fully cover the entire weld to be welded. When performing welding trajectory recognition, the welding robot can first be controlled to move to the first shooting position to capture images using the first, second, and third cameras. After the welding trajectory corresponding to the first shooting position is recognized, the welding robot is controlled to move to the second shooting position, and so on. After the welding trajectory corresponding to the Nth shooting position is recognized, the welding trajectory of the entire weld to be welded can be obtained by integrating the point cloud data of all generated welding trajectories through downsampling and spline curve interpolation. Where N is an integer greater than or equal to 1.

[0036] For any of the aforementioned shooting positions, the first, second, and third cameras can be controlled to synchronously capture weldment images at that shooting position, and the captured images can be corrected based on camera parameters, thereby obtaining a corrected first image and a corrected third image captured by the first camera, a corrected second image captured by the second camera, and a corrected fourth image captured by the third camera. The first and third images are different images obtained by correcting the same original image captured by the first camera using different camera parameters. Furthermore, the first and second images constitute a first image pair, and the third and fourth images constitute a second image pair.

[0037] Step 302: Determine a target image pair from the first image pair and the second image pair based on the average orientation angle of all welding instances in the first image.

[0038] In this step, image segmentation with instance-level pixel precision is performed to identify the regions of all weld instances in the first image that include the base material, or the weld and base material. After segmenting the weld instances, the average orientation angle of all weld instances is determined using the image's second-order moment. This average orientation angle is then used to identify a target image pair from the first and second image pairs. Welding trajectories are subsequently identified based on this target image pair. The weld instances contained in the images of the target image pair are oriented horizontally in the image coordinate system.

[0039] Step 303: Determine, in the binary mask image of the welding instance corresponding to the secondary image of the target image pair, a second edge line that matches the first edge line of the welding instance included in the primary image of the target image pair; the primary image is an image captured by the first camera.

[0040] In this step, the image captured by the first camera in the target image pair can be used as the primary image, and the other image can be used as the secondary image. For example, if the target image pair is the first image pair, the image captured by the second camera can be used as the secondary image; if the target image pair is the second image pair, the image captured by the third camera can be used as the secondary image.

[0041] For the primary and secondary images in the target image pair, single-pixel edge lines (i.e., individual weld trajectories) can be extracted from the regions of all weld instances segmented in the respective binary mask images of the weld instances in the primary and secondary images. A one-to-one correspondence can be established based on the left-to-right order of the identified edge lines within their respective binary mask images. Thus, for each first edge line in the binary mask image of the weld instance corresponding to the primary image, a corresponding second edge line can be matched in the binary mask image of the weld instance corresponding to the secondary image.

[0042] Step 304: Based on the second edge line, determine second pixel points that match each first pixel point in the first edge line.

[0043] In this step, the matching edge lines in the welding example binary mask corresponding to the primary image and the secondary image must be precisely matched. For each first pixel on the first edge line in the welding example binary mask corresponding to the primary image, a second pixel that precisely matches the first pixel can be found within a limited search range to the left and right of the matching second edge line, based on horizontal epipolar constraints.

[0044] Step 305: Based on the pixel coordinates of the first pixel and the pixel coordinates of the second pixel, determine the homogeneous coordinates of the first pixel in the robot base coordinate system.

[0045] In this step, binocular triangulation is used to determine the homogeneous coordinates of the first pixel on the first edge line in the binary mask image of the welding instance corresponding to the primary image and the corresponding second pixel in the binary mask image of the welding instance corresponding to the secondary image. Homogeneous coordinates are a coordinate representation method widely used in mathematics and computer graphics to extend the Euclidean coordinate system. They represent points in n-dimensional space by adding an additional dimension (usually 1), thereby expressing n-dimensional coordinates as (n+1)-dimensional vectors.

[0046] Step 306: Positioning the welding trajectory based on the homogeneous coordinates of all first pixel points in the robot base coordinate system.

[0047] In this step, the homogeneous coordinates of all first pixel points in the robot base coordinate system can be further transformed into the robot base coordinate system, thereby obtaining the homogeneous coordinates of each first pixel point in the robot base coordinate system. Since each first pixel point is a point on an edge line identified in the welding example binary mask image corresponding to the main image, and each edge line in the welding example binary mask image corresponding to the main image is also a welding trajectory, the position of each point cloud on the welding trajectory identified and located at the current shooting position can be obtained through the homogeneous coordinates of each first pixel point in the robot base coordinate system.

[0048] Furthermore, the welding robot is moved to the next shooting position, and the method of the present invention is executed to obtain the position of each point cloud on the welding trajectory located at the next shooting position. By repeating the above process, the position of each point cloud on the welding trajectory at each preset shooting position can be obtained. By integrating the point clouds on all generated welding trajectories, using downsampling and spline curve interpolation, the entire welding trajectory of the workpiece to be welded can be obtained.

[0049] An embodiment of the present invention provides a welding trajectory positioning method based on a trinocular vision system. For each pre-set shooting position, a first image pair and a second image pair are obtained at the shooting position, wherein the first image pair includes a first image taken by a first camera and a second image taken by a second camera, and the second image pair includes a third image taken by the first camera and a fourth image taken by the third camera. Based on the average orientation angle of all welding instances in the first image, a target image pair is determined from the first image pair and the second image pair. In a welding instance binary mask image corresponding to a secondary image of the target image pair, a second edge line matching the first edge line of the welding instance included in a main image in the target image pair is determined, where the main image is an image taken by the first camera. Based on the second edge line, a second pixel point matching each first pixel point in the first edge line is determined. Based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the robot base coordinate system are determined, and based on the homogeneous coordinates of all first pixel points in the robot base coordinate system, the welding trajectory is positioned. Since the welding instance segmentation is performed on the image captured by the trinocular vision system, the edge lines of the welding instances in the binary mask images of the welding instances corresponding to the main image and the auxiliary image in the target image are identified and matched, and then the pixel points on the edge lines are accurately matched, the position of each pixel point on each edge line in the main image used to characterize the welding trajectory can be determined, thereby realizing the recognition of the welding trajectory. Since there is no need to rely on the structured light projection pattern, the welding trajectory recognition can be realized by only processing the image captured by the trinocular vision system. Therefore, it is not only possible to improve the positioning reliability of the welding trajectory in specific scenarios such as multi-layer and multi-pass welding, narrow gap welding or welding of workpieces with strongly reflective surfaces, but also to reliably locate the welding trajectory in other welding scenarios.

[0050] In addition, the present invention uses passive stereo vision and deep learning to achieve reliable and robust processing and recognition of multi-view images taken directly from welds, thereby extracting the welding path. This overcomes the problem of inaccurate welding trajectory recognition in current structured light sensing methods when weld geometric features are unclear or welds have highly reflective surfaces, while maintaining the advantage of low cost. Moreover, the three-eye stereo vision system in the present invention constructs two orthogonal binocular vision system layouts, and through multi-view complementarity, the positioning method of the present invention can identify and locate welding trajectories in any orientation.

[0051] For example, based on the above embodiment, when determining the welding instance in the first image, it can be performed in the following manner: The first image is input into the instance segmentation model. When the instance segmentation model identifies that the first image includes a single-pass weld, a binary mask image of the welding instance including the weldment base material is output. When the instance segmentation model identifies that the first image includes multiple-pass welds, a binary mask image of the welding instance including the weldment base material and each formed weld is output. The instance segmentation model is obtained by training an initial instance segmentation model using sample weldment images and annotated labels of each sample welding instance in the sample weldment images.

[0052] Specifically, the instance segmentation model is implemented using a deep instance segmentation convolutional neural network. The instance segmentation network structure used can be, for example, SparseInst, which mainly includes a backbone network for feature extraction, an encoder for feature fusion, and a decoder. The instance branch and mask branch in the decoder jointly implement instance prediction. The characteristic of SparseInst is that it can adaptively represent instance objects using class instance activation maps. Before using SparseInst to perform image segmentation tasks, it is necessary to collect sample weld images and manually annotate the sample weld instances in the sample weld images with labels to train the initial instance segmentation model so that it can learn the regularities and patterns in the sample weld images. For example, the sample weld images can be input into the initial instance segmentation model to obtain the predicted weld instance segmentation results. Based on the predicted weld instance segmentation results and the labels, loss information is constructed. The network parameters of the initial instance segmentation model are adjusted using this loss information until the loss information is minimized or the network converges. The resulting model is then determined as the instance segmentation model.

[0053] Figure 4 This is a schematic diagram of an example segmentation of a single weld image provided by an embodiment of the present invention. Figure 5 Schematic diagram of the example segmentation of multi-layer and multi-pass weld images provided by the embodiment of the present invention, as shown in FIG. Figure 4 As shown in FIG, when the first image is input into the trained instance segmentation model, and the instance segmentation model recognizes that the first image includes a single weld, the welding instance in the segmentation result output by the instance segmentation model only includes the weldment base material. Figure 5 As shown, when the first image is input into the trained instance segmentation model, the instance segmentation model recognizes that the first image includes multiple welds, and in the segmentation result output by the instance segmentation model, the welding instance includes the weldment base material and each formed weld.

[0054] The instance segmentation model outputs segmentation results in the form of multiple binary mask images, which correspond one-to-one to all welding instances in the first image, and the foreground pixels in the binary mask images represent the instance areas.

[0055] It should be noted that the segmentation of welding instances in the second image, the third image and the fourth image can also be performed in a similar manner to that of the first image. For example, the second image, the third image or the fourth image can be input into the instance segmentation model to obtain the segmentation results of the welding instances in the second image, the third image or the fourth image.

[0056] In this embodiment, by training an instance segmentation model and implementing pixel-level segmentation of welding instances in the first image, the instance segmentation model not only improves instance segmentation accuracy but also enhances segmentation efficiency. Furthermore, the segmented first image is a welding image captured directly by a camera, without the need for structured light illumination. This avoids the problem of unclear or incomplete projected stripes or patterns caused by structured light illumination, thereby improving the accuracy of the subsequently determined welding trajectory.

[0057] Furthermore, this embodiment uses deep learning to extract weld instances with pixel-level segmentation accuracy at the object scale, improving the accuracy of weld trajectory extraction. Furthermore, the segmentation results provide prior constraints for stereo matching of binocular image pairs, improving matching efficiency and reliability.

[0058] For example, based on the above embodiment, when determining the target image pair from the first image pair and the second image pair based on the average orientation angle of all welding instances in the first image, the method may be as follows: Determine the average orientation angle of all welding instances in the first image based on the following formula (1); (1) in, represents the average orientation angle of all welding instances, N represents the number of welding instances, 、 and represents the second-order moment of the image of the i-th welding instance, , when p is 1, q is 1, when p is 2, q is 0, when p is 0, q is 2, represents the geometric center coordinates of the i-th welding instance, represents the binary mask of the i-th welding instance, Represents pixel coordinates.

[0059] When the angle between the average orientation angle and the horizontal coordinate axis of the image is within a second preset angle range, the second image pair is determined as the target image pair. The second preset angle range is used to characterize that the orientation of the welding instance in the image coordinate system is a horizontal orientation. When the angle between the average orientation angle and the horizontal coordinate axis of the image is not within the second preset angle range, the first image pair is determined as the target image pair.

[0060] Specifically, based on the segmentation results of the welding instances in the first image, the average orientation angle of all welding instances is calculated using formula (1), and then the average orientation angle is compared with a second preset angle range, where the second preset angle range is used to indicate that the orientation of the welding instance in the image coordinate system is horizontal. The second preset angle range can be, for example, -30° to 30°.

[0061] When the angle between the average orientation angle and the horizontal coordinate axis of the image is within a second preset angle range, the weld instance in the first image pair is horizontally oriented. However, subsequent edge line recognition and matching requires the weld instance to be vertically oriented. Therefore, the first image pair is excluded and the second image pair is selected as the target image pair. Due to the positioning of the three cameras in the trinocular vision system, if the image in the first image pair does not meet the vertical orientation requirement, the image in the second image pair will meet it. Therefore, the average orientation angle of the weld instance in the second image pair can be eliminated, and the second image pair can be directly used as the target image pair.

[0062] When the angle between the average orientation angle of the first image and the horizontal coordinate axis of the image is not within the second preset angle range, it means that the welding instance of the image in the first image pair is oriented vertically, and therefore, the first image pair is used as the target image pair.

[0063] In this embodiment, by calculating the average orientation angle of all welding instances in the first image and determining the target image pair from the first image pair and the second image pair based on the average orientation angle, the determined target image pair is easier to identify subsequent welding trajectories, thereby improving the efficiency of welding trajectory identification.

[0064] Exemplarily, when determining the second edge line that matches the first edge line of the welding instance included in the main image of the target image pair in the welding instance binary mask image corresponding to the secondary image of the target image pair, a forward difference algorithm can be used to determine the first edge line of each welding instance in the welding instance binary mask image corresponding to the main image, and determine the third edge line of each welding instance in the welding instance binary mask image corresponding to the secondary image. For each first edge line, based on the position of the first edge line in the welding instance binary mask image corresponding to the main image and the position of each third edge line in the welding instance binary mask image corresponding to the secondary image, a second edge line that matches the first edge line among all the third edge lines is determined.

[0065] Specifically, taking the binary mask image of the welding instance corresponding to the main image as an example, for the binary mask image representing each welding instance in the main image, a forward difference is calculated pixel by pixel on the horizontal coordinate axis of the mask image to obtain the left and right contour lines of the welding instance and their coordinates. The coordinate average of all contour lines shared by multiple welding instances is calculated as the pixel coordinates of the single-pixel edge line, that is, the coordinates of the welding trajectory. For example, the value of the background pixel point in the binary mask image is 0, and the value of the foreground pixel point is 1. For each pixel point, the difference between its value and the value of the previous pixel point is calculated. If the difference is 0, it means that the previous pixel point is also a background pixel point. If the absolute value of the difference is 1, it means that the previous pixel point is a foreground pixel point. The previous pixel point is then regarded as a point on the first edge line. In this way, the first edge line of each welding instance in the binary mask image of the welding instance corresponding to the main image, that is, the welding trajectory, can be identified. Similarly, the third edge line of each welding instance in the binary mask image of the welding instance corresponding to the secondary image can also be identified.

[0066] For each first edge line identified in the welding example binary mask image corresponding to the main image, the first edge lines will be sorted from left to right in the welding example binary mask image corresponding to the main image. Similarly, each third edge line identified in the welding example binary mask image corresponding to the secondary image can also be sorted from left to right in the welding example binary mask image corresponding to the secondary image, so as to match the sorted first edge lines and third edge lines. For example, the leftmost first edge line in the welding example binary mask image corresponding to the main image is matched with the leftmost third edge line in the welding example binary mask image corresponding to the secondary image, and the second left first edge line in the welding example binary mask image corresponding to the main image is matched with the second left third edge line in the welding example binary mask image corresponding to the secondary image, etc. In this way, the third edge line that successfully matches each first edge line can be determined.

[0067] In this embodiment, the forward difference algorithm can be used to accurately identify the first edge line of each welding instance in the welding instance binary mask image corresponding to the main image and the third edge line of each welding instance in the welding instance binary mask image corresponding to the secondary image. Based on the position of each first edge line in the welding instance binary mask image corresponding to the main image and the position of each third edge line in the welding instance binary mask image corresponding to the secondary image, edge line matching is achieved, making the matching method simpler and improving the matching efficiency.

[0068] For example, based on the above embodiments, when determining the second pixel points that match the first pixel points in the first edge line based on the second edge line, the following method can be used: For each first pixel point, a local image block is intercepted with the first pixel point as the center, and a search window including a preset number of target pixel points is intercepted with the third pixel point corresponding to the first pixel point in the second edge line as the midpoint. For each target pixel point in the search window, a target image block of the same size as the local image block is intercepted with the target pixel point as the center, and the similarity between each target image block and the local image block is calculated respectively, and the target pixel point corresponding to the target image block with the maximum similarity is determined as the second pixel point matching the first pixel point.

[0069] Specifically, for each first pixel point in the first edge line, it is necessary to determine the second pixel point that matches it on the second edge line by precise matching. For example, a local image block can be intercepted with the first pixel point on the first edge line as the center as a matching template. In addition, it is also necessary to determine the third pixel point corresponding to the first pixel point in the second edge line that matches the first edge line. The third pixel point can be determined based on the position of each pixel point in the edge line, or it can be a pixel point corresponding to the first pixel point preliminarily determined by other methods. For example, if the first pixel point is the second pixel point on the first edge line, the third pixel point is also the second pixel point on the second edge line.

[0070] With the third pixel point as the midpoint, a search window including a preset number of target pixels can be captured. For example, with the third pixel point as the midpoint, a search range of 5 target pixels on its left and right can be set as a search window.

[0071] Centered on each target pixel in the search window, target image blocks of the same size as the local image blocks corresponding to the matching template are intercepted from the secondary image, and similarities between the local image blocks and each target image block are calculated. The similarity evaluation metric may be a structural similarity index. The target pixel corresponding to the target image block with the highest structural similarity index is determined as the second pixel that matches the first pixel corresponding to the local image block.

[0072] For each first pixel point in each first edge line, a matching second pixel point can be determined in the above manner, and these second pixel points can constitute an edge line that precisely matches the first edge line.

[0073] In this embodiment, by intercepting the image blocks and calculating the similarity as described above, a second pixel point that accurately matches each first pixel point can be determined, which further improves the accuracy of the second pixel point, making the edge line that is ultimately determined to accurately match the first edge line higher in accuracy.

[0074] For example, based on the above embodiments, when determining the homogeneous coordinates of the first pixel point in the robot base coordinate system based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the following method can be used: The homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera are determined based on the following formula (2): (2) in, represents the homogeneous coordinates of the first pixel in the reference coordinate system of the first camera, represents the coordinates of the first pixel on the first edge line in the main image, represents the pixel coordinates of the second pixel point, b represents the baseline of the first camera, and f represents the focal length of the first camera. represents the coordinates of the principal point of the first camera, represents the coordinates of the principal point of the second camera; The homogeneous coordinates of the first pixel point in the robot base coordinate system are determined based on the following formula (3): (3) in, represents the homogeneous coordinates of the first pixel in the robot base coordinate system, Indicates the position of the welding gun. represents the homogeneous transformation matrix for hand-eye calibration, Represents the rotation matrix that describes the rotation of the first camera in stereo rectification.

[0075] In this embodiment, based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera are determined, and then further converted to obtain the homogeneous coordinates of the first pixel point in the robot base coordinate system, thereby providing a basis for the subsequent welding robot to correct the welding trajectory or plan the path of the welding trajectory.

[0076] Exemplarily, based on the above embodiments, when obtaining the first image pair and the second image pair at the shooting position, a first initial image taken by the first camera at the shooting position, a second initial image taken by the second camera at the shooting position, and a third initial image taken by the third camera at the shooting position can be obtained. Based on the camera parameters of the first camera and the camera parameters of the second camera, the first initial image and the second initial image are corrected respectively by a binocular stereo correction algorithm to obtain a first image pair including the first image and the second image. Based on the camera parameters of the first camera and the camera parameters of the third camera, the first initial image and the third initial image are corrected respectively by a binocular stereo correction algorithm to obtain a second image pair including the third image and the fourth image.

[0077] Specifically, after the welding robot is moved to a shooting position, a first initial image is captured by the first camera of the trinocular vision system, a second initial image is captured by the second camera, and a third initial image is captured by the third camera. Based on the calibration results of the first and second cameras and the camera parameters of the first and second cameras, a binocular stereo correction algorithm is performed on the first and second initial images to eliminate lens distortion and ensure that the image planes are coplanar and aligned in rows, thereby obtaining a first image and a second image. The first image and the second image can constitute a first image pair. Based on the calibration results of the first and third cameras and the camera parameters of the first and third cameras, a binocular stereo correction algorithm is performed on the first and third initial images to eliminate lens distortion and ensure that the image planes are coplanar and aligned in rows, thereby obtaining a third image and a fourth image. The third image and the fourth image can constitute a first image pair. The algorithm for performing the binocular stereo correction can be the Bougeut algorithm in the OpenCV library.

[0078] In this embodiment, the first initial image, the second initial image, and the third initial image are corrected by a binocular stereo correction algorithm, thereby not only eliminating lens distortion and improving image quality, but also making the image planes coplanar and the rows aligned.

[0079] Figure 6 The second flow chart of the welding trajectory positioning method based on the three-eye vision system provided in the embodiment of the present invention is as follows: Figure 6 As shown in the figure, after the welding robot is calibrated through TCP, the control point of the welding robot is set at the tungsten electrode of the welding gun or the tip of the welding wire. In addition, cameras 1-3 independently complete monocular camera calibration to determine the camera intrinsic parameters. Cameras 1 and 2, as well as cameras 1 and 3, perform binocular calibration to determine the rotation matrix and displacement vector of cameras 2 and 3 relative to camera 1. Finally, the calibration of the eye-to-hand conversion relationship between camera 1 and the welding gun is completed.

[0080] A series of camera positions are pre-set for the welding robot. First, after the welding robot moves to the first position, cameras 1-3 capture images. The initial images are then calibrated to produce a first image pair and a second image pair. The first image pair consists of the first image captured by camera 1 and the second image captured by camera 2. The second image pair consists of the third image captured by camera 1 and the fourth image captured by camera 3.

[0081] Each corrected image is fed into the SparseInst instance segmentation network for image segmentation, outputting all weld instances in each image as binary images. The average orientation angle of all weld instances in the first image is calculated. If the absolute value of the angle between the average orientation angle and the horizontal axis of the image is less than 30°, the second image pair is used as the target image pair; otherwise, the first image pair is used as the target image pair.

[0082] Next, the edge lines in the main image and the secondary image of the target image pair will be extracted, and the initial matching will be completed based on the sequential relationship, that is, the first edge line in the welding instance binary mask corresponding to the main image and the third edge line in the welding instance binary mask corresponding to the secondary image will be matched one by one. Furthermore, for the first edge line and the second edge line that matches it, a fine matching of the welding path is required. Specifically, a pixel point on the first edge line on the welding instance binary mask corresponding to the main image will be selected, and a local image block will be extracted on the welding instance binary mask corresponding to the main image with it as the center. The pixel corresponding to the pixel point on the second edge line of the welding instance binary mask corresponding to the secondary image will be used as the midpoint to set a search window, and then a series of target image blocks with the same size as the local image block will be extracted with each target pixel point in the search window as the center. Taking structural similarity consistency as the evaluation index, the similarity between all target image blocks and local image blocks is calculated. The target pixel points corresponding to the target image blocks with the highest similarity are used as points on the precise matching edge line. After determining that all first pixel points on the first edge line of the welding instance binary mask image corresponding to the main image have been finely matched, the coordinates of each first pixel point are calculated, and the point cloud coordinates are converted to obtain the homogeneous coordinates in the robot base coordinate system.

[0083] When it is determined that the current first shooting position is not the last shooting position, the welding robot will be controlled to move to the next shooting position to continue shooting. If it is the last shooting position, all the point clouds obtained will be integrated, and the complete welding trajectory will be obtained through downsampling and spline curve interpolation.

[0084] An embodiment of the present invention provides a welding trajectory positioning method based on a trinocular vision system. For each pre-set shooting position, a first image pair and a second image pair are obtained at the shooting position, wherein the first image pair includes a first image taken by a first camera and a second image taken by a second camera, and the second image pair includes a third image taken by the first camera and a fourth image taken by the third camera. Based on the average orientation angle of all welding instances in the first image, a target image pair is determined from the first image pair and the second image pair. In a binary mask image of the welding instance corresponding to the secondary image of the target image pair, a second edge line matching the first edge line of the welding instance included in the main image of the target image pair is determined, where the main image is an image taken by the first camera. Based on the second edge line, a second pixel point matching each first pixel point in the first edge line is determined. Based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the robot base coordinate system are determined, and based on the homogeneous coordinates of all first pixel points in the robot base coordinate system, the welding trajectory is identified. Since the welding instance segmentation is performed on the image captured by the trinocular vision system, the edge lines of the welding instances in the binary mask images of the welding instances corresponding to the main image and the auxiliary image in the target image are identified and matched, and then the pixel points on the edge lines are accurately matched, the position of each pixel point on each edge line in the main image used to characterize the welding trajectory can be determined, thereby realizing the recognition of the welding trajectory. Since there is no need to rely on the structured light projection pattern, the welding trajectory recognition can be realized by only processing the image captured by the trinocular vision system. Therefore, it is not only possible to improve the positioning reliability of the welding trajectory in specific scenarios such as multi-layer and multi-pass welding, narrow gap welding or welding of workpieces with strongly reflective surfaces, but also to reliably locate the welding trajectory in other welding scenarios.

[0085] In addition, the present invention uses passive stereo vision and deep learning to achieve reliable and robust processing and recognition of multi-view images taken directly from welds, thereby extracting the welding path. This overcomes the problem of inaccurate welding trajectory recognition in current structured light sensing methods when weld geometric features are unclear or welds have highly reflective surfaces, while maintaining the advantage of low cost. Moreover, the three-eye stereo vision system in the present invention constructs two orthogonal binocular vision system layouts, and through multi-view complementarity, the positioning method of the present invention can identify and locate welding trajectories in any orientation.

[0086] The welding track positioning device based on the trinocular vision system provided by the present invention is described below. The welding track positioning device based on the trinocular vision system described below and the welding track positioning method based on the trinocular vision system described above can be referred to each other.

[0087] Figure 7 A schematic diagram of the structure of a welding trajectory positioning device based on a three-eye vision system provided in an embodiment of the present invention is shown in FIG. Figure 7 As shown, the welding trajectory positioning device 700 based on the trinocular vision system includes: An acquisition module 11 is configured to acquire, for each preset shooting position, a first image pair and a second image pair at the shooting position, wherein the first image pair includes a first image captured by a first camera and a second image captured by a second camera, and the second image pair includes a first image captured by the first camera and a fourth image captured by a third camera, wherein the first camera, the second camera, and the third camera are fixed to a welding gun of a welding robot, wherein a plane on which the first camera and the second camera are located and optical centers of lenses of the first camera, the second camera, and the third camera constitute a target plane, and an angle between a straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and a straight line formed by the optical centers of the lenses of the first camera and the third camera is within a first preset angle range; a determination module 12, configured to determine a target image pair from the first image pair and the second image pair based on an average orientation angle of all welding instances in the first image; The determining module 12 is further configured to determine, in the binary mask image of the welding instance corresponding to the secondary image of the target image pair, a second edge line that matches a first edge line of the welding instance included in the primary image of the target image pair; the primary image being an image captured by the first camera; The determining module 12 is further configured to determine, based on the second edge line, second pixel points that match each first pixel point in the first edge line; The determining module 12 is further configured to determine the homogeneous coordinates of the first pixel point in the robot base coordinate system based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point; The positioning module 13 is used to locate the welding track based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system.

[0088] In an exemplary embodiment, the determining module 12 is specifically configured to: Determine the average orientation angle of all welding instances in the first image based on the following formula (1); (1) in, represents the average orientation angle of all welding instances, N represents the number of welding instances, 、 and represents the second-order moment of the image of the i-th welding instance, , when p is 1, q is 1, when p is 2, q is 0, when p is 0, q is 2, represents the geometric center coordinates of the i-th welding instance, represents the binary mask of the i-th welding instance, Represents pixel coordinates; determining the second image pair as the target image pair when the angle between the average orientation angle and the horizontal coordinate axis of the image is within a second preset angle range, wherein the second preset angle range is used to indicate that the orientation of the welding instance in the image coordinate system is a horizontal orientation; When the included angle between the average orientation angle and the horizontal coordinate axis of the image is not within the second preset angle range, the first image pair is determined as the target image pair.

[0089] In an exemplary embodiment, the determining module 12 is specifically configured to: Determining, by a forward difference algorithm, a first edge line of each welding instance in the welding instance binary mask image corresponding to the primary image, and determining a third edge line of each welding instance in the welding instance binary mask image corresponding to the secondary image; For each of the first edge lines, based on the position of the first edge line in the welding instance binary mask image corresponding to the main image and the position of each of the third edge lines in the welding instance binary mask image corresponding to the secondary image, determine the second edge line that matches the first edge line among all the third edge lines.

[0090] In an exemplary embodiment, the determining module 12 is specifically configured to: For each of the first pixel points, intercepting a local image block with the first pixel point as the center; Taking the third pixel point corresponding to the first pixel point in the second edge line as the midpoint, intercepting a search window including a preset number of target pixel points; For each target pixel point in the search window, intercepting a target image block having the same size as the local image block with the target pixel point as the center; The similarities between each of the target image blocks and the local image blocks are calculated respectively, and the target pixel corresponding to the target image block with the maximum similarity is determined as the second pixel matching the first pixel.

[0091] In an exemplary embodiment, the determining module 12 is specifically configured to: The homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera are determined based on the following formula (2): (2) in, represents the homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera, represents the coordinates of the first pixel point on the first edge line in the main image, represents the pixel coordinates of the second pixel point, b represents the baseline of the first camera, f represents the focal length of the first camera, represents the coordinates of the principal point of the first camera, represents the coordinates of the principal point of the second camera; The homogeneous coordinates of the first pixel point in the robot base coordinate system are determined based on the following formula (3): (3) in, represents the homogeneous coordinates of the first pixel point in the robot base coordinate system, represents the posture of the welding gun, represents the homogeneous transformation matrix for hand-eye calibration, Represents a rotation matrix.

[0092] In an exemplary embodiment, the apparatus further comprises: an input module, wherein: The input module is configured to input the first image into an instance segmentation model, and output a binary mask image of the welding instance including the weldment base material when the instance segmentation model identifies that the first image includes a single-pass weld; and output a binary mask image of the welding instance including the weldment base material and each formed weld when the instance segmentation model identifies that the first image includes multiple-pass welds; The instance segmentation model is obtained by training an initial instance segmentation model using sample weldment images and annotation labels of each sample welding instance in the sample weldment images.

[0093] In an exemplary embodiment, the acquisition module 11 is specifically configured to: Acquire a first initial image captured by the first camera at the shooting position, a second initial image captured by the second camera at the shooting position, and a third initial image captured by the third camera at the shooting position; Based on camera parameters of the first camera and camera parameters of the second camera, correcting the first initial image and the second initial image respectively using a binocular stereo correction algorithm to obtain a first image pair including the first image and the second image; Based on camera parameters of the first camera and camera parameters of the third camera, the first initial image and the third initial image are respectively corrected by a binocular stereo correction algorithm to obtain the second image pair including the third image and the fourth image.

[0094] The device of this embodiment can be used to execute the method of any embodiment in the embodiment of the welding trajectory positioning method based on the three-eye vision system. Its specific implementation process and technical effects are similar to those in the embodiment of the welding trajectory positioning method based on the three-eye vision system. For details, please refer to the detailed description in the embodiment of the welding trajectory positioning method based on the three-eye vision system, which will not be repeated here.

[0095] Figure 8 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communications interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a welding trajectory positioning method based on a trinocular vision system, the method comprising: for each pre-set shooting position, obtaining a first image pair and a second image pair at the shooting position, the first image pair comprising a first image taken by a first camera and a second image taken by a second camera, the second image pair comprising a third image taken by the first camera and a fourth image taken by a third camera, the first camera, the second camera and the third camera being fixed on a welding gun of a welding robot, the optical centers of the lenses of the first camera, the second camera and the third camera forming a target plane, and the straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and the straight line formed by the optical centers of the lenses of the first camera and the third camera The angle between the straight lines is within a first preset angle range; based on the average orientation angle of all welding instances in the first image, a target image pair is determined from the first image pair and the second image pair; in the welding instance binary mask image corresponding to the secondary image of the target image pair, a second edge line that matches the first edge line of the welding instance included in the main image of the target image pair is determined; the main image is an image taken by the first camera; based on the second edge line, a second pixel point that matches each first pixel point in the first edge line is determined; based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the robot base coordinate system are determined; based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system, the welding trajectory is located.

[0096] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the welding trajectory positioning method based on the trinocular vision system provided by the above methods, the method including: for each pre-set shooting position, obtaining a first image pair and a second image pair at the shooting position, the first image pair including a first image taken by a first camera and a second image taken by a second camera, the second image pair including a third image taken by the first camera and a fourth image taken by a third camera, the first camera, the second camera and the third camera are fixed on the welding gun of the welding robot, the optical center of the lens of the first camera, the second camera and the third camera constitute a target plane, and the first camera and the second camera on the target plane The angle between the straight line formed by the optical center of the lens and the straight line formed by the optical centers of the first camera and the third camera lenses is within a first preset angle range; based on the average orientation angle of all welding instances in the first image, a target image pair is determined from the first image pair and the second image pair; in the welding instance binary mask image corresponding to the secondary image of the target image pair, a second edge line that matches the first edge line of the welding instance included in the main image of the target image pair is determined; the main image is an image taken by the first camera; based on the second edge line, a second pixel point that matches each first pixel point in the first edge line is determined; based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the robot base coordinate system are determined; based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system, the welding trajectory is located.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the welding trajectory positioning method based on the trinocular vision system provided by the above-mentioned methods, the method comprising: for each pre-set shooting position, obtaining a first image pair and a second image pair at the shooting position, the first image pair comprising a first image taken by a first camera and a second image taken by a second camera, the second image pair comprising a third image taken by the first camera and a fourth image taken by a third camera, the first camera, the second camera and the third camera being fixed on a welding gun of a welding robot, the optical centers of the lenses of the first camera, the second camera and the third camera forming a target plane, and the straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and the straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane are aligned with the straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane. The angle between the straight lines formed by the optical centers of the first camera and the third camera lenses is within a first preset angle range; based on the average orientation angle of all welding instances in the first image, a target image pair is determined from the first image pair and the second image pair; in the welding instance binary mask image corresponding to the secondary image of the target image pair, a second edge line that matches the first edge line of the welding instance included in the main image in the target image pair is determined; the main image is an image taken by the first camera; based on the second edge line, a second pixel point that matches each first pixel point in the first edge line is determined; based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the robot base coordinate system are determined; based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system, the welding trajectory is located.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0100] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A welding track positioning method based on a three-eye vision system, characterized in that: include: For each preset shooting position, a first image pair and a second image pair are acquired at the shooting position, where the first image pair includes a first image captured by a first camera and a second image captured by a second camera, and the second image pair includes a third image captured by the first camera and a fourth image captured by a third camera. The first camera, the second camera, and the third camera are fixed to a welding gun of a welding robot, optical centers of lenses of the first camera, the second camera, and the third camera form a target plane, and an angle between a straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and a straight line formed by the optical centers of the lenses of the first camera and the third camera is within a first preset angle range. determining a target image pair from the first image pair and the second image pair based on an average orientation angle of all welding instances in the first image; Determining, in a binary mask image of the welding instance corresponding to the secondary image of the target image pair, a second edge line that matches a first edge line of the welding instance included in a primary image of the target image pair; the primary image being an image captured by the first camera; Determining, based on the second edge line, second pixel points that match each first pixel point in the first edge line; Determining homogeneous coordinates of the first pixel point in a robot base coordinate system based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point; The welding trajectory is located based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system.

2. The welding trajectory positioning method based on the trinocular vision system according to claim 1 is characterized in that: The determining of a target image pair from the first image pair and the second image pair based on an average orientation angle of all welding instances in the first image comprises: Determine the average orientation angle of all welding instances in the first image based on the following formula (1); （1） in, represents the average orientation angle of all welding instances, N represents the number of welding instances, 、 and represents the second-order moment of the image of the i-th welding instance, , when p is 1, q is 1, when p is 2, q is 0, when p is 0, q is 2, represents the geometric center coordinates of the i-th welding instance, represents the binary mask of the i-th welding instance, Represents pixel coordinates; determining the second image pair as the target image pair when the angle between the average orientation angle and the horizontal coordinate axis of the image is within a second preset angle range, wherein the second preset angle range is used to indicate that the orientation of the welding instance in the image coordinate system is a horizontal orientation; When the included angle between the average orientation angle and the horizontal coordinate axis of the image is not within the second preset angle range, the first image pair is determined as the target image pair.

3. The welding trajectory positioning method based on the trinocular vision system according to claim 1 is characterized in that: The step of determining, in the welding instance binary mask corresponding to the secondary image of the target image pair, a second edge line that matches a first edge line of the welding instance included in the primary image of the target image pair, comprises: Determining, by a forward difference algorithm, a first edge line of each welding instance in the welding instance binary mask image corresponding to the primary image, and determining a third edge line of each welding instance in the welding instance binary mask image corresponding to the secondary image; For each of the first edge lines, based on the position of the first edge line in the welding instance binary mask image corresponding to the main image and the position of each of the third edge lines in the welding instance binary mask image corresponding to the secondary image, determine the second edge line that matches the first edge line among all the third edge lines.

4. The welding trajectory positioning method based on the trinocular vision system according to claim 1 is characterized in that: Determining, based on the second edge line, second pixel points that match each first pixel point in the first edge line includes: For each of the first pixel points, intercepting a local image block with the first pixel point as the center; Taking the third pixel point corresponding to the first pixel point in the second edge line as the midpoint, intercepting a search window including a preset number of target pixel points; For each target pixel point in the search window, intercepting a target image block having the same size as the local image block with the target pixel point as the center; The similarities between each of the target image blocks and the local image blocks are calculated respectively, and the target pixel corresponding to the target image block with the maximum similarity is determined as the second pixel matching the first pixel.

5. The welding trajectory positioning method based on the trinocular vision system according to claim 1 is characterized in that: The determining, based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point, the homogeneous coordinates of the first pixel point in the robot base coordinate system includes: The homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera are determined based on the following formula (2): （2） in, represents the homogeneous coordinates of the first pixel point in the reference coordinate system of the first camera, represents the coordinates of the first pixel point on the first edge line in the main image, represents the pixel coordinates of the second pixel point, b represents the baseline of the first camera, f represents the focal length of the first camera, represents the coordinates of the principal point of the first camera, represents the coordinates of the principal point of the second camera; The homogeneous coordinates of the first pixel point in the robot base coordinate system are determined based on the following formula (3): （3） in, represents the homogeneous coordinates of the first pixel point in the robot base coordinate system, represents the posture of the welding gun, represents the homogeneous transformation matrix for hand-eye calibration, Represents a rotation matrix.

6. The welding trajectory positioning method based on a trinocular vision system according to any one of claims 1 to 5, characterized in that: The method further comprises: Inputting the first image into an instance segmentation model, and outputting a binary mask image of the welding instance including the weldment base material when the instance segmentation model identifies that the first image includes a single-pass weld; and outputting a binary mask image of the welding instance including the weldment base material and each formed weld when the instance segmentation model identifies that the first image includes multiple-pass welds; The instance segmentation model is obtained by training an initial instance segmentation model using sample weldment images and annotation labels of each sample welding instance in the sample weldment images.

7. The welding trajectory positioning method based on a trinocular vision system according to any one of claims 1 to 5, characterized in that: The acquiring of the first image pair and the second image pair at the shooting position includes: Acquire a first initial image captured by the first camera at the shooting position, a second initial image captured by the second camera at the shooting position, and a third initial image captured by the third camera at the shooting position; Based on camera parameters of the first camera and camera parameters of the second camera, correcting the first initial image and the second initial image respectively using a binocular stereo correction algorithm to obtain a first image pair including the first image and the second image; Based on camera parameters of the first camera and camera parameters of the third camera, the first initial image and the third initial image are respectively corrected by a binocular stereo correction algorithm to obtain the second image pair including the third image and the fourth image.

8. A welding track positioning device based on a three-eye vision system, characterized in that: include: an acquisition module, configured to acquire, for each preset shooting position, a first image pair and a second image pair at the shooting position, the first image pair comprising a first image captured by a first camera and a second image captured by a second camera, and the second image pair comprising a third image captured by the first camera and a fourth image captured by a third camera, the first camera, the second camera, and the third camera being fixed to a welding gun of a welding robot, the optical centers of the lenses of the first camera, the second camera, and the third camera forming a target plane, and an angle between a straight line formed by the optical centers of the lenses of the first camera and the second camera on the target plane and a straight line formed by the optical centers of the lenses of the first camera and the third camera being within a first preset angle range; a determination module, configured to determine a target image pair from the first image pair and the second image pair based on an average orientation angle of all welding instances in the first image; The determining module is further configured to determine, in a binary mask image of the welding instance corresponding to a secondary image of the target image pair, a second edge line that matches a first edge line of the welding instance included in a primary image of the target image pair; the primary image being an image captured by the first camera; The determining module is further configured to determine, based on the second edge line, second pixel points that match each first pixel point in the first edge line; The determining module is further configured to determine the homogeneous coordinates of the first pixel point in the robot base coordinate system based on the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point; A positioning module is used to locate the welding track based on the homogeneous coordinates of all the first pixel points in the robot base coordinate system.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the welding trajectory positioning method based on the trinocular vision system as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the welding trajectory positioning method based on the trinocular vision system as described in any one of claims 1 to 7 is implemented.

Images