Automatic welding method, device and equipment, storage medium and product

Abstract

The invention discloses an automatic welding method, device and equipment, a storage medium and a product, and relates to the technical field of welding automation. According to the method, the three-dimensional point cloud data and the two-dimensional image data of the welding area are collected, the three-dimensional point cloud data and the two-dimensional image data are fused, and the fused image is obtained. The rough welding area is determined through semantic segmentation based on the two-dimensional information contained in the fused image, and the target welding seam is determined in the rough welding area based on the depth information contained in the fused image, so that the precision and robustness of welding seam positioning are remarkably improved by utilizing the characteristics of the fused image; the system can stably operate in a complex factory environment with uneven illumination, strong reflection or partial shielding; and the dependence of the algorithm on environmental conditions is reduced, so that the algorithm is more suitable for the application scene of an actual manufacturing industry factory. And on the basis of more accurate positioning, the welding gun is controlled to weld the target welding seam, and the accuracy of automatic welding can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of welding automation technology, and in particular to an automatic welding method, device, equipment, storage medium and product. Background Art

[0002] In the manufacturing industry, welding is a critical process. The positioning of the weld to be welded and the tracking accuracy of the welding gun directly affect the quality of automated welding and the performance of the welded product.

[0003] Traditional weld seam location methods rely primarily on manual operation or simple sensor technology, resulting in low accuracy and a low degree of automation. Therefore, achieving higher-precision automated welding is an urgent issue that needs to be addressed. Summary of the Invention

[0004] The main purpose of this application is to provide an automatic welding method, device, equipment, storage medium and product, aiming to solve the technical problem of how to achieve higher-precision automated welding.

[0005] To achieve the above objectives, the present application proposes an automatic welding method, which includes: Collect 3D point cloud data and 2D image data of the welding area; Fusing three-dimensional point cloud data and two-dimensional image data to obtain a fused image; Based on the two-dimensional information contained in the fused image, the rough welding area is determined through semantic segmentation; Determine the target weld in the rough weld area based on the depth information contained in the fused image; Control the welding gun to weld the target weld.

[0006] In some embodiments, fusing the three-dimensional point cloud data and the two-dimensional image data to obtain a fused image includes: Unify the 3D point cloud data and 2D image data acquired at the same sampling time to the same size ratio; The three-dimensional point cloud data and the corresponding two-dimensional image data are input into the cross-modal fusion framework neural network to obtain a fused image; wherein the fused image includes two-dimensional information and depth information, the two-dimensional information includes at least one of color information, texture information and pixel information corresponding to each pixel block, and the depth information includes a depth value.

[0007] In some embodiments, determining a rough welding area by semantic segmentation based on pixel information contained in the fused image includes: Performing image enhancement on the fused image to obtain an enhanced image; Inputting the enhanced image into a semantic segmentation model to obtain a classification result of the enhanced image; wherein the semantic segmentation model is used to classify each pixel block based on the two-dimensional information of each pixel block, and different types of pixel blocks are marked with different label values; According to the classification results, pixel blocks with preset marking values ​​are intercepted in the enhanced image as rough welding areas.

[0008] In some embodiments, determining a target weld in a rough weld region based on depth information contained in the fused image includes: Divide the rough welding area into multiple sub-areas according to the specified step size; For each sub-region, a corresponding function change curve is determined based on the depth value corresponding to each pixel block in the sub-region. The function change curve is used to represent the change relationship between the horizontal coordinate of the fused image and the depth value; The valley point in the function change curve is determined as the weld point corresponding to the sub-region; The set of weld points in each sub-area is determined as the target weld.

[0009] In some embodiments, after controlling the welding gun to weld the target weld, the automatic welding method further includes: Continuously obtain real-time positioning information of the welding gun; When the difference between the real-time positioning information corresponding to the current sampling moment and the previous sampling moment is greater than the specified offset, determining the offset direction and offset distance; Generate corresponding control signals according to the offset direction and offset distance; The welding gun is moved based on the control signal.

[0010] In some embodiments, collecting three-dimensional point cloud data of the welding area includes: Scan the welding area with a laser scanner to obtain a point cloud dataset; Each point in the point cloud data set is mapped to a two-dimensional plane to obtain a depth map, which includes three-dimensional point cloud data.

[0011] In addition, to achieve the above-mentioned purpose, the present application also proposes an automatic welding device, which includes: Data acquisition module, used to collect 3D point cloud data and 2D image data of the welding area; A data fusion module is used to fuse three-dimensional point cloud data and two-dimensional image data to obtain a fused image; A first positioning module is used to determine a rough welding area through semantic segmentation based on the two-dimensional information contained in the fused image; a second positioning module for determining a target weld in the rough welding area based on depth information contained in the fused image; The control module is used to control the welding gun to weld the target weld.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes an automatic welding device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the automatic welding method described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the automatic welding method described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the automatic welding method described above are implemented.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: By collecting 3D point cloud data and 2D image data of the welding area, the 3D point cloud data and 2D image data are fused to produce a fused image. Based on the 2D information contained in the fused image, the rough welding area is determined through semantic segmentation. Based on the depth information contained in the fused image, the target weld is located within the rough welding area. This utilizes the characteristics of the fused image to significantly improve the accuracy and robustness of weld positioning. It can operate stably in complex factory environments with uneven lighting, strong reflections, or partial occlusion. It also reduces the algorithm's dependence on environmental conditions, making it more suitable for application scenarios in actual manufacturing plants. Based on more precise positioning, controlling the welding gun to weld the target weld can significantly improve the accuracy of automatic welding. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A schematic diagram of a process of an automatic welding method provided in one embodiment of the present application is shown; Figure 2A schematic diagram of a point cloud processing process according to an exemplary embodiment of the present application is shown; Figure 3 A simplified flowchart of step S120 provided by an exemplary embodiment of the present application is shown; Figure 4 FIG2 shows a simplified flowchart of step S130 provided by an exemplary embodiment of the present application; Figure 5 A simplified flowchart of step S140 provided by an exemplary embodiment of the present application is shown; Figure 6 A schematic diagram of a smoothed function change curve provided by an exemplary embodiment of the present application is shown; Figure 7 FIG1 shows a simplified flowchart of step S150 provided by an exemplary embodiment of the present application; Figure 8 A schematic structural diagram of an automatic welding device provided in one embodiment of the present application is shown; Figure 9 A structural schematic diagram of an automatic welding device provided in one embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0020] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0021] In the related technologies, welding is a key process in the manufacturing industry. The positioning of the weld to be welded and the tracking accuracy of the welding gun directly affect the quality of automated welding and the performance of the welded product.

[0022] Traditional weld seam location methods rely primarily on manual operation or simple sensor technology, resulting in low accuracy and limited automation. The development of optical and laser sensor technologies, combined with image processing techniques and deep learning algorithms, has enabled even higher-precision weld seam location and welding gun tracking.

[0023] In addition, related technologies use image technology to automatically locate welds, but: ① A single two-dimensional image combined with digital image processing technology cannot obtain spatial information, and has high requirements for the shooting environment and image quality; ② Although the combination of two-dimensional images and deep learning improves the ability to understand image features, it lacks three-dimensional information and has poor accuracy; ③ Although single three-dimensional laser scanning provides spatial information, it is not combined with weld color and texture information, making it difficult to locate complex welds, and has high requirements for weld morphology and laser scanning methods.

[0024] In summary, how to achieve higher-precision automated welding is a problem that needs to be solved urgently.

[0025] Based on this, the present application provides an automatic welding method, which, after collecting three-dimensional point cloud data and two-dimensional image data of the welding area, fuses the collected data to obtain a fused image; then, based on the two-dimensional information contained in the fused image, determines the rough welding area through semantic segmentation; based on the depth information contained in the fused image, determines the target weld in the rough welding area; controls the welding gun to weld the target weld, which can achieve automatic welding with higher precision and solve problems such as weak robustness of existing weld positioning.

[0026] The execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an automatic welding device capable of performing the above functions. The following uses the automatic welding device as an example to illustrate this embodiment and the following embodiments.

[0027] Reference Figure 1 , Figure 1 The flowchart of the automatic welding method provided by an embodiment of the present application is shown. The automatic welding method can be applied to automatic welding equipment, including the following steps S110 to S150: Step S110 , collecting three-dimensional point cloud data and two-dimensional image data of the welding area.

[0028] The welding area refers to the area before welding, including the base metal to be welded and the gap between them. The purpose of welding is to connect two or more independent metal (or non-metal) parts (i.e. the base metal to be welded) into a whole.

[0029] A weld is the metallic bond formed during or after welding, consisting of the solidified molten base metal and / or filler metal. Welding is essentially the process of forming a weld.

[0030] In this embodiment, a camera can be used to capture the weld area, thereby obtaining two-dimensional image data corresponding to the weld area. It is understood that the camera can be a color line array camera. A color line array camera scans line by line using a single or multiple rows of photosensitive cells, then synthesizes a complete two-dimensional image. It has excellent color reproduction and detail capture capabilities. Other types of cameras can also be used, and this embodiment is not limited thereto.

[0031] In this embodiment, the weld area can also be scanned using a laser scanner to obtain a point cloud dataset corresponding to the weld area. The laser scanner can be a line laser scanner, which is a three-dimensional contour measurement device based on the principle of triangulation. A line laser scanner projects a high-precision laser line onto the surface of an object (e.g., a base material), captures the deformation of the laser line with a camera, and reconstructs the three-dimensional coordinates of the object surface in real time to obtain a point cloud dataset.

[0032] like Figure 2 As shown, in some embodiments, due to different product specifications, the collected point cloud data may contain invalid edge data. The point cloud height extraction algorithm can be used to filter out the invalid data, that is, Figure 2 Edge filtering step in the process; Since the base material may be metal, there may be abnormal reflection when the laser is irradiated. At this time, a sliding window can be used to remove the more obvious abnormal points in the point cloud, that is, Figure 2 The step of removing abnormal points in the point cloud may have certain holes after removing abnormal points, so the point cloud can be filtered. For example, the hole area can be interpolated and filled using the point cloud data in a certain neighborhood around the hole, that is, Figure 2 The step of point cloud filtering; Due to the difference between the camera coordinate system and the physical coordinate system, the coordinate system of the point cloud data can also be transformed according to the internal and external parameters of the camera, and the point cloud can be calibrated and unified into the same coordinate system, that is, Figure 2 Steps for point cloud calibration.

[0033] It is understandable that a large amount of point cloud data may occupy a large amount of storage space and increase the complexity of the algorithm, resulting in a decrease in the efficiency of subsequent image processing. In order to compress the storage capacity and reduce the complexity of the algorithm, in this embodiment, the point cloud data after point cloud calibration can be mapped to a two-dimensional image plane (i.e. Figure 2 ), and obtain the corresponding depth map.

[0034] A depth map is a special type of two-dimensional image in which each pixel represents the distance (depth) from the corresponding point in the scene to the camera or sensor. Depth values ​​are typically used to represent distance. Mapping point cloud data onto a two-dimensional image plane to form a depth map is similar to storing three-dimensional data in a two-dimensional format. In other words, a depth map contains three-dimensional point cloud data.

[0035] Specifically, for any point in the 3D point cloud data, set its coordinates in the world coordinate system (i.e., the coordinates collected by the line laser scanner) to (X, Y, Z), set the pre-calibrated focal lengths of the camera (color line array camera) sensor to fx and fy, and set the optical center of the camera to cx and cy, then: Among them, u and v are the pixel positions of the point in the depth map, and the pixel value of the pixel is Z.

[0036] In some embodiments, the depth map obtained after mapping may have some pixel missing, for example, due to high reflectivity, the reflected signal received by the laser scanner may be too strong, data transmission and processing errors, etc. Therefore, it is possible to use Figure 2 The grayscale filling method shown is used to fill these missing data or invalid data areas (ie, hole areas).

[0037] For example, for the missing parts in the depth map, commonly used grayscale filling methods may include interpolation methods (nearest neighbor interpolation, bilinear interpolation, etc.), neighborhood statistics-based methods (calculating the mean or median of the neighboring pixels around the missing area and filling the missing area with this value), and image restoration algorithms (based on texture synthesis).

[0038] In addition, in some embodiments, the point cloud data collected by the line laser scanner may have a "transmission" problem, which may affect the subsequent weld positioning accuracy after mapping. Therefore, in this embodiment, morphological filtering can be used for processing (i.e. Figure 2 Grayscale morphology step in .

[0039] Specifically, it can be processed by dilation and erosion operations. The dilation operation can be expressed as , whose purpose is to fill the small holes in the depth map caused by abnormal missing. The corrosion operation can be expressed as , its purpose is to eliminate abnormal protrusions appearing in the depth map.

[0040] For example, assuming that the depth value in a 3*3 window of a depth map is (15, none, 16, 14, 200, 15, 16, 17, 18), then the maximum value of the remaining values ​​​​except the outlier can be taken for filling, that is, max(15, 16, 14, 15, 16, 17, 18) = 18, replacing none with 18. This operation is called dilation operation; the value 200 is obviously abnormally large, so the minimum value of the remaining values ​​can be taken for elimination, that is, min(15, 18, 16, 14, 15, 16, 17, 18) = 14, replacing 200 with 14. This operation is called erosion operation.

[0041] It can be understood that morphological filtering can greatly improve the scattering / absorption problem of laser on the weld surface (ie, the aforementioned transmission).

[0042] Step S120: Fusing the 3D point cloud data and the 2D image data to obtain a fused image. The simplified process of step S120 can be as follows: Figure 3 As shown. First, the depth map and 2D image data are aligned, and then the aligned images are grouped and paired to achieve dataset segmentation. The dataset segmentation results can be input into the CMX neural network, and the final fused image output is obtained through CM-FRM feature correction and FFM feature fusion. Figure 3 Each step and module in the

[0043] Specifically, the 3D point cloud data and 2D image data acquired at the same sampling time can be unified to the same size ratio. Specifically, the depth image is scaled based on the size of the 2D image data, so that the basic image features such as the size ratio and number of pixels of the two images remain consistent, allowing for smooth subsequent image fusion.

[0044] In this embodiment, the 2D image data and 3D point cloud data can be stored in different folders, and then paired based on the order in which the images were acquired. It is understood that image data acquisition is a continuous process, and over a period of time, both 2D image data and 3D point cloud data are acquired simultaneously at each sampling moment. The interval between sampling moments can be set based on actual needs and is not limited in this embodiment.

[0045] After the pairing is completed, for a set of two-dimensional image data and three-dimensional point cloud data, the three-dimensional point cloud data and the corresponding two-dimensional image data can be input into the cross-modal fusion framework neural network to obtain a fused image.

[0046] The Cross-Modal Fusion Framework (CMX) neural network consists of two main modules: the Cross-Modal Feature Rectification Module (CM-FRM) and the Feature Fusion Module (FFM). The CM-FRM module rectifies bimodal features, using one modality (3D point cloud data, i.e., depth map) to correct the features of the other modality (2D image data), thereby enhancing complementary information and reducing noise and uncertainty. The rectified features are input into the FFM module, which uses a cross-attention mechanism and mixed channel embedding to achieve full information exchange and fusion of bimodal features.

[0047] The fused image obtained after fusion includes two-dimensional information and depth information. The two-dimensional information includes at least one of color information, texture information and pixel information corresponding to each pixel block, and the depth information includes a depth value.

[0048] Step S130 : determining a rough welding area through semantic segmentation based on the two-dimensional information contained in the fused image.

[0049] The simplified process of step S130 can be as follows: Figure 4 As shown. After obtaining the input of the fused image, the fused image can be enhanced first, and then the enhanced image can be filtered, and then the morphological processing as described above can be performed to enhance the features in the image. Semantic segmentation is performed on the enhanced image after feature enhancement to extract the result area. Figure 4 Each step is described in detail.

[0050] In some implementations, image enhancement may be performed on the fused image to obtain an enhanced image.

[0051] Specifically, traditional digital image processing techniques can be used for processing, such as filling invalid points, contrast enhancement, sharpening, outlier filtering, grayscale morphological operations, etc. on the fused image. These image processing techniques can be used to enhance the features of the image, thereby obtaining an enhanced image, which helps to improve the subsequent recognition accuracy.

[0052] Among them, invalid point filling refers to filling the areas in the depth map where the depth information is missing or invalid, and neighborhood interpolation and other methods can be used. Contrast enhancement refers to adjusting the contrast of the image to make the boundaries and details of objects in the image clearer. For example, the depth map can be equalized to make the grayscale value distribution of the image more uniform, thereby enhancing the contrast; sharpening refers to highlighting the edges and details of the image to make the image clearer. Gradient-based sharpening or high-pass filtering sharpening and other methods can be used. The specific steps of outlier filtering and grayscale morphology can be similar to the relevant implementation methods in step S110, and this embodiment will not be repeated here.

[0053] After obtaining the enhanced image, the following operations can be performed on the enhanced image: Figure 2 The image filtering process shown is as follows. Image filtering refers to a preprocessing step for smoothing an image or removing noise, and generally, methods such as Gaussian filtering, median filtering, and mean filtering can be used, which are not limited in this embodiment.

[0054] After the image filtering is completed, morphological processing may be performed, wherein the morphological processing may be similar to the related implementation in step S110, and will not be described in detail in this embodiment.

[0055] Then, the enhanced image after the above processing can be input into the semantic segmentation model to obtain the classification result of the enhanced image.

[0056] Specifically, a semantic segmentation model can be pre-trained using a large amount of sample data. The semantic segmentation model can perform pixel-by-pixel classification operations, classifying each pixel in the image into predefined categories, such as weld, base material, background, etc.

[0057] After the enhanced image is input into the semantic segmentation model, the semantic segmentation model can classify each pixel block based on the two-dimensional information of each pixel block, marking pixels with similar features such as color and texture as the same type, and different types of pixel blocks are marked with different label values.

[0058] Ultimately, the semantic segmentation model outputs an image whose pixel values ​​represent the corresponding classification results. Based on the classification results, pixel blocks with preset label values ​​can be intercepted from the enhanced image to serve as the rough welding area.

[0059] Step S140 : determining a target weld in the rough welding area based on the depth information contained in the fused image.

[0060] The simplified process of step S140 can be as follows: Figure 5 As shown. You can first execute Figure 5 Extract the depth of the ROI area in the rough welding area, determine the ROI area (i.e. the area corresponding to the target weld) from the rough welding area, perform function fitting on the depth information of the extracted ROI area, perform smoothing on the fitted function image, and then calculate the valley point of the function, determine it as the position of the target weld, and output the weld position. Figure 5 Each step is described in detail.

[0061] In some embodiments, the rough welding area may be traversed with a specified step size (eg, 20 rows of pixels) and the rough welding area may be divided into multiple sub-areas according to the specified step size.

[0062] For each sub-region, the image's horizontal coordinate can be used as the function's horizontal coordinate, and the depth information can be used as the function's vertical coordinate to fit the functional relationship of the depth information in the region with the horizontal coordinate (corresponding to a variation curve). Since the functional relationship may contain noise, a Gaussian function can also be used to smooth it. As an example, the smoothed image of the functional relationship can be as follows Figure 6 As shown, Figure 6 The white part in the figure is the smoothed function curve.

[0063] Based on the fitted function relationship, the valley point values ​​in the horizontal axis can be calculated. It's understandable that welds are generally recessed into the parent material on either side, and this is reflected in the function curve as the location of the valley point. Therefore, the valley points in the function curve can be identified as the weld points corresponding to the subregions, and the collection of weld points in each subregion is identified as the target weld.

[0064] Step S150: Control the welding gun to weld the target weld.

[0065] The simplified process of step S150 can be as follows: Figure 7 As shown. First, obtain the positioning result, that is, the target weld. Then read the position of the welding gun, calculate the deviation between adjacent positions during the movement of the welding gun, and judge whether the deviation is out of tolerance. If it is not out of tolerance, it means that the movement route of the welding gun is accurate, and return to the step of obtaining the positioning result to continue welding; if it is out of tolerance, it means that there is an error in the movement route of the welding gun, so it is necessary to output an offset signal. After knowing the deviation amount according to the offset signal, control the welding gun offset through PLC to return the welding gun to the correct position, and then return to the step of obtaining the positioning result to continue welding. The following is for Figure 7 Each step is described in detail.

[0066] After the target weld is determined, the welding gun can be controlled to perform welding at the location of the target weld.

[0067] During the welding process, the real-time positioning information of the welding gun can be continuously obtained; for example, the world coordinates of the welding gun tip can be obtained. In some embodiments, a sampling interval can be set, and the world coordinates of the welding gun tip are obtained once every sampling interval and at a sampling time.

[0068] In this embodiment, when the difference between the real-time positioning information corresponding to the current sampling moment and the previous sampling moment is greater than the specified offset, it can be determined that the offset of the welding gun position must be corrected. At this time, the offset direction and offset distance can be determined.

[0069] After determining the offset direction and offset distance, a corresponding control signal can be generated according to the offset direction and offset distance to move the welding gun back based on the control signal so that the welding gun tip can be aligned with the target weld to achieve more accurate automatic welding.

[0070] This embodiment provides an automatic welding method, which collects three-dimensional point cloud data and two-dimensional image data of the welding area, fuses the three-dimensional point cloud data and the two-dimensional image data, and obtains a fused image. Based on the two-dimensional information contained in the fused image, the rough welding area is determined through semantic segmentation, and based on the depth information contained in the fused image, the target weld is determined in the rough welding area, thereby utilizing the characteristics of the fused image, significantly improving the accuracy and robustness of weld positioning, and being able to operate stably in complex factory environments with uneven lighting, strong reflections, or partial occlusion; reducing the algorithm's dependence on environmental conditions, making it more suitable for application scenarios in actual manufacturing factories. Based on more precise positioning, controlling the welding gun to weld the target weld can significantly improve the accuracy of automatic welding.

[0071] This application also provides an automatic welding device, please refer to Figure 8, the automatic welding device 100 includes: The data acquisition module 110 is used to acquire three-dimensional point cloud data and two-dimensional image data of the welding area.

[0072] The data fusion module 120 is configured to fuse the three-dimensional point cloud data and the two-dimensional image data to obtain a fused image.

[0073] The first positioning module 130 is configured to determine a rough welding area through semantic segmentation based on the two-dimensional information contained in the fused image.

[0074] The second positioning module 140 is configured to determine a target weld in the rough welding area based on depth information contained in the fused image.

[0075] The control module 150 is used to control the welding gun to weld the target weld.

[0076] The automatic welding device 100 provided in this application, employing the automatic welding method of the aforementioned embodiment, can solve the technical problem of achieving higher-precision automated welding. Compared to the prior art, the beneficial effects of the automatic welding device 100 provided in this application are the same as those of the automatic welding method provided in the aforementioned embodiment. Other technical features of the automatic welding device 100 are the same as those disclosed in the aforementioned embodiment and are not further described here.

[0077] The present application provides an automatic welding device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the automatic welding method in the above-mentioned embodiment 1.

[0078] Reference below Figure 9 , which shows a schematic structural diagram of automatic welding equipment suitable for implementing the embodiments of the present application. The automatic welding equipment in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 9 The automatic welding equipment shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0079] like Figure 9 As shown, the automatic welding equipment 200 may include a processing device 210 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 220 or programs loaded from a storage device 230 into a random access memory (RAM) 240. RAM 240 also stores various programs and data required for the operation of the automatic welding equipment. Processing device 210, ROM 220, and RAM 240 are interconnected via a bus 250. An input / output (I / O) interface 260 is also connected to the bus. Typically, the following systems may be connected to I / O interface 260: input devices 270, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 280, such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 230, such as a magnetic tape, hard disk, etc.; and communication device 290. The communication device 290 can allow the automatic welding device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an automatic welding device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0080] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 230, or installed from a ROM 220. When the computer program is executed by the processing device 210, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0081] The automatic welding equipment provided in this application, utilizing the automatic welding method described in the aforementioned embodiment, can solve the technical problem of achieving higher-precision automated welding. Compared to the prior art, the beneficial effects of the automatic welding equipment provided in this application are the same as those of the automatic welding method described in the aforementioned embodiment. Other technical features of the automatic welding equipment are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0082] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0083] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0084] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the automatic welding method in the above-mentioned embodiment.

[0085] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0086] The computer-readable storage medium may be included in the automatic welding equipment, or may exist independently without being assembled into the automatic welding equipment.

[0087] The computer-readable storage medium carries one or more programs that, when executed by the automatic welding device, enable the automatic welding device to write computer program code for performing the operations of the present application in one or more programming languages, or a combination thereof. The programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0089] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0090] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned automatic welding method. This computer-readable storage medium can address the technical problem of achieving higher-precision automated welding. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the automatic welding method provided in the aforementioned embodiments and are not further elaborated here.

[0091] The present application also provides a computer program product, comprising a computer program, which implements the steps of the automatic welding method as described above when executed by a processor.

[0092] The computer program product provided in this application can solve the technical problem of how to achieve higher-precision automated welding. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the automated welding method provided in the above embodiment, and will not be elaborated here.

[0093] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An automatic welding method, characterized in that: The automatic welding method comprises: Collect 3D point cloud data and 2D image data of the welding area; fusing the three-dimensional point cloud data and the two-dimensional image data to obtain a fused image; determining a rough welding area through semantic segmentation based on the two-dimensional information contained in the fused image; determining a target weld in the rough weld area based on depth information contained in the fused image; The welding gun is controlled to weld the target weld.

2. The automatic welding method according to claim 1, wherein: The fusing the three-dimensional point cloud data and the two-dimensional image data to obtain a fused image includes: Unify the 3D point cloud data and 2D image data acquired at the same sampling time to the same size ratio; The three-dimensional point cloud data and the corresponding two-dimensional image data are input into a cross-modal fusion framework neural network to obtain the fused image; wherein the fused image includes the two-dimensional information and the depth information, the two-dimensional information includes at least one of the color information, texture information and pixel information corresponding to each pixel block, and the depth information includes a depth value.

3. The automatic welding method according to claim 1 or 2, characterized in that: The determining of a rough welding area by semantic segmentation based on pixel information contained in the fused image includes: performing image enhancement on the fused image to obtain an enhanced image; Inputting the enhanced image into a semantic segmentation model to obtain a classification result of the enhanced image; wherein the semantic segmentation model is used to classify each pixel block based on the two-dimensional information of each pixel block, and different types of pixel blocks are marked with different label values; According to the classification result, a pixel block with a mark value of a preset mark value is intercepted in the enhanced image as the rough welding area.

4. The automatic welding method according to claim 1 or 2, characterized in that: Determining a target weld in the rough welding area based on depth information contained in the fused image includes: Dividing the rough welding area into a plurality of sub-areas according to a specified step size; For each of the sub-regions, determining a corresponding function change curve based on the depth value corresponding to each pixel block in the sub-region, wherein the function change curve is used to represent a change relationship between the horizontal coordinate of the fused image and the depth value; Determine the valley point in the function change curve as the weld point corresponding to the sub-region; A set of weld points in each of the sub-areas is determined as the target weld.

5. The automatic welding method according to claim 2, wherein: After the controlled welding gun welds the target weld, the automatic welding method further comprises: Continuously obtaining real-time positioning information of the welding gun; When the difference between the real-time positioning information corresponding to the current sampling moment and the previous sampling moment is greater than the specified offset, determining the offset direction and offset distance; generating a corresponding control signal according to the offset direction and offset distance; The welding gun is moved based on the control signal.

6. The automatic welding method according to any one of claims 1, 2 and 5, characterized in that: The collecting of three-dimensional point cloud data of the welding area includes: Scanning the welding area with a laser scanner to obtain a point cloud data set; Each point in the point cloud data set is mapped to a two-dimensional plane to obtain a depth map, where the depth map includes the three-dimensional point cloud data.

7. An automatic welding device, characterized in that: The automatic welding device comprises: Data acquisition module, used to collect 3D point cloud data and 2D image data of the welding area; A data fusion module, configured to fuse the three-dimensional point cloud data and the two-dimensional image data to obtain a fused image; a first positioning module, configured to determine a rough welding area by semantic segmentation based on the two-dimensional information contained in the fused image; a second positioning module, configured to determine a target weld in the rough welding area based on depth information contained in the fused image; The control module is used to control the welding gun to weld the target weld.

8. An automatic welding device, characterized in that: The automatic welding device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the automatic welding method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the automatic welding method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the automatic welding method according to any one of claims 1 to 6 are implemented.

Images