A welding image determination method, device, equipment and storage medium

Abstract

The application discloses a kind of welding image determination method, device, equipment and storage medium.The method includes receiving the laser stripe image fed back by laser tracking device, and determining target area image from the laser stripe image;Through given information extraction model, extract the stripe pixel set constituting laser stripe from the target area image;From the stripe pixel set, filter target pixel point meeting set condition, and according to each target pixel point, the welding image applied to welding scene is formed.The multi-scale feature fusion network of U-shaped structure is introduced, which can solve the problem of accurate and stable acquisition of laser stripe in the image in the process of determining the welding trajectory in multi-layer multi-pass welding, such as serious arc light, splash interference and other situations, which can effectively avoid the interference of splash, reflection and ambient light on laser stripe extraction, realize the stable and rapid extraction of laser stripe, to improve the accuracy and efficiency of determining the welding trajectory.

Description

Technical Field

[0001] This invention relates to the field of multi-layer, multi-pass welding technology, and in particular to a welding image determination method, apparatus, equipment, and storage medium. Background Technology

[0002] In multi-layer, multi-pass welding processes, due to machining and butt joint deviations in the bevel, manual intervention is often required to meet welding requirements, especially in automated welding of long bevels. By using laser tracking sensors to guide automated welding equipment, manual intervention in the welding process can be eliminated, achieving fully automated welding in multi-layer, multi-pass welding.

[0003] In fully automated welding processes, precise weld seam positioning and bevel size measurement are required before the automated welding equipment can begin. In multi-layer, multi-pass welding, the automated welding equipment needs to determine the welding trajectory before welding. However, interference from light reflection, spatter, and smoke at the bevel in multi-layer, multi-pass welding makes stable extraction of laser stripes a major challenge for laser tracking, making it difficult to guarantee accurate determination of the welding trajectory during fully automated welding. Traditional image processing methods are often affected by various factors, including lighting conditions, environmental interference, and variations in welding parameters, which can lead to instability and inaccuracy in the calculation results. Summary of the Invention

[0004] This invention provides a welding image determination method, apparatus, device, and storage medium to solve the problem of difficulty in ensuring accurate determination of the welding trajectory during fully automated welding processes.

[0005] According to a first aspect of the present invention, a method for determining a welding image is provided, comprising:

[0006] Receive laser stripe images fed back by a laser tracking device, and determine the target area image from the laser stripe images;

[0007] Using a given information extraction model, the set of stripe pixels constituting the laser stripes is extracted from the target area image. The information extraction model is a U-shaped multi-scale feature fusion network, which is pre-trained using a constructed sample dataset. Each sample tuple in the sample dataset includes a sample laser stripe image and a corresponding set of labeled pixels.

[0008] Target pixels that meet the set conditions are selected from the stripe pixel set, and a welding image for use in the welding scene is constructed based on each target pixel.

[0009] According to a second aspect of the present invention, a welding image determination apparatus is provided, comprising:

[0010] The image acquisition module is used to receive the laser stripe image fed back by the laser tracking device and determine the target area image from the laser stripe image;

[0011] The extraction module is used to extract the set of stripe pixels constituting the laser stripes from the target area image using a given information extraction model. The information extraction model is a U-shaped multi-scale feature fusion network, which is pre-trained using a constructed sample dataset. Each sample tuple in the sample dataset includes a sample laser stripe image and a corresponding set of labeled pixels.

[0012] The image processing module is used to select target pixels that meet set conditions from the stripe pixel set, and to construct a welding image for use in the welding scene based on each target pixel.

[0013] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and a memory communicatively connected to said at least one processor;

[0015] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the welding image determination method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the welding image determination method according to any embodiment of the present invention.

[0017] The technical solution of this invention first receives a laser stripe image fed back by a laser tracking device and determines a target region image from the laser stripe image. Then, using a given information extraction model, it extracts a set of stripe pixels constituting the laser stripes from the target region image. The information extraction model is a U-shaped multi-scale feature fusion network, pre-trained using a constructed sample dataset. Each sample tuple in the sample dataset includes a sample laser stripe image and a corresponding set of labeled pixels. Target pixels that meet set conditions are selected from the stripe pixel set, and a welding image applied to a welding scenario is constructed based on these target pixels. This embodiment introduces a U-shaped multi-scale feature fusion network to select target pixels that meet set conditions and constructs a welding image applied to a welding scenario based on these target pixels. This method solves the technical problem that, in multi-layer, multi-pass welding processes, severe arc light and spatter interference make it difficult to accurately and stably acquire laser stripes in images obtained by laser tracking sensors when determining the welding trajectory. In the implementation of this scheme, by introducing a U-shaped multi-scale feature fusion network, the interference of spatter, reflection and ambient light on laser stripe extraction can be effectively avoided, and stable and rapid extraction of laser stripes can be achieved. This enables precise positioning of welds and measurement of groove dimensions in multi-layer and multi-pass welding processes, thereby improving the accuracy and efficiency of determining the welding trajectory in multi-layer and multi-pass welding processes.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a welding image determination method provided in Embodiment 1 of the present invention;

[0021] Figure 2 This is a flowchart of a welding image determination method provided according to Embodiment 2 of the present invention;

[0022] Figure 3 This is a schematic diagram of a welding image determination device according to Embodiment 3 of the present invention;

[0023] Figure 4This is a schematic diagram of the structure of an electronic device that implements a welding image determination method according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "target," "history," "initial," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] Example 1

[0027] Figure 1 This is a flowchart of a welding image determination method according to Embodiment 1 of the present invention. This embodiment is applicable to the extraction of laser stripe images during multi-layer, multi-pass welding. The method can be executed by a welding image determination device, which can be implemented in hardware and / or software and can be configured in a computer device. Figure 1 As shown, the method includes:

[0028] S101. Receive the laser stripe image fed back by the laser tracking device, and determine the target area image from the laser stripe image.

[0029] It should be noted that the laser tracking device mentioned in this embodiment may include an image acquisition device and a laser projector. The laser tracking device can be installed on an automatic welding equipment and perform the operation of acquiring laser stripe images according to instructions sent by a computer.

[0030] In this embodiment, the laser stripe image can be an image captured by the image acquisition device when the laser projector is in laser projection mode. For example, a line laser is projected onto the workpiece surface by the laser projector, and the image acquisition device acquires the image of the laser stripes, thereby measuring the surface morphology of the workpiece to achieve precise positioning of the weld and measurement of the bevel dimensions.

[0031] In this embodiment, to quickly and stably extract the laser stripe image, the target area image should be a region that includes the complete laser stripe image. This ensures that the target area image accurately reflects the weld's characteristic information, such as width and height, thereby enabling more accurate positioning of the laser stripe boundary in subsequent operations. The target area image can be an image obtained after preprocessing the laser stripe image (such as cropping the complete laser stripe region).

[0032] S102. Using a given information extraction model, extract the set of stripe pixels constituting the laser stripes from the target area image. The information extraction model is a U-shaped multi-scale feature fusion network, which is pre-trained using a constructed sample dataset. Each sample tuple in the sample dataset includes a sample laser stripe image and a corresponding set of labeled pixels.

[0033] In this embodiment, the information extraction model can be used to extract the set of stripe pixels constituting the laser stripes from the target region image. The stripe pixel set includes the pixels of the complete laser stripe image. The information extraction model is a U-shaped multi-scale feature fusion network model. By using a pre-trained U-shaped multi-scale feature fusion network model to extract laser stripe image pixels, stable and rapid extraction of laser stripes can be achieved. In fact, the output of the information extraction model is a set of laser stripe image pixels, which includes the pixels of the complete laser stripe image.

[0034] The U-shaped multi-scale feature fusion network consists of an encoder and a decoder connected by skip connections. The encoder is responsible for feature extraction and downsampling of the input laser stripe image, gradually reducing the spatial size of the image while increasing the number of feature channels to capture the high-level semantic information of the image. The decoder restores the feature map to the size of the original image through upsampling operations and uses skip connections to fuse features at the corresponding levels in the encoder, thereby restoring image details and performing accurate pixel extraction by combining high-level semantic information.

[0035] Furthermore, a multi-scale feature fusion module is designed within the U-shaped multi-scale feature fusion network. At different levels of the encoder, convolutional kernels or pooling operations of different sizes are used to acquire features at different scales. For example, during information extraction, for the target region image, small-scale features can identify the subtle edges and breaks in the laser stripes, while large-scale features can determine the position of the laser stripes within the entire welding area, thus avoiding misjudgments due to local noise or background interference. Then, the fusion module fuses these features at different scales, enabling the network to simultaneously utilize both detailed and global information from the image, improving the accuracy of laser stripe pixel extraction. Specific fusion methods can include concatenating feature maps of different scales along the channel dimension, or fusing them through weighted summation, etc.

[0036] Optionally, the information extraction model is pre-trained. Specifically, a large number of laser stripe images can be collected as a training dataset and labeled. These laser stripe images should cover different scenes, lighting conditions, laser stripe shapes, and widths, such as laser stripe images in an idle state and laser stripe images in a welding state. The information extraction model is then pre-trained. Pre-training tasks can include image classification, object detection, and other related tasks to learn general feature representations of images, such as the edges, textures, and colors of the object to be welded. Pre-training the information extraction model allows for the initialization of weights, bringing them closer to the optimal solution, thereby accelerating the training convergence speed on the laser stripe image pixel extraction task and improving the performance of the information extraction model.

[0037] S103. Select target pixels that meet the set conditions from the stripe pixel set, and construct a welding image for the welding scene based on each target pixel.

[0038] In this embodiment, considering the expected trajectory of the laser stripes during the welding process and their typically high grayscale value in the image, while the background area has a relatively low grayscale value, the set condition can be the pixel with a grayscale value of 255 and the smallest pixel coordinates.

[0039] In this embodiment, the stripe pixel set is traversed column by column and point by point, checking whether each pixel meets the set conditions. During the traversal, a loop structure can be used to access each element in the pixel set and call the corresponding function to determine whether the pixel meets conditions such as geometric position and grayscale value. Only when a pixel meets all of the above conditions is it determined as the target pixel.

[0040] In this embodiment, a blank welding image is created based on the size and pixel format of the original laser stripe image. The background color of this blank image can be set to black or other suitable colors to highlight the laser stripes formed by the target pixels. The selected target pixels are then filled into the blank welding image one by one. Specific color values ​​(such as white) can be set at the corresponding pixel positions in the welding image according to the coordinates of the target pixels to indicate that the position is a laser stripe pixel. During the filling process, array operations or image processing library functions can be used to assign pixel values.

[0041] Optionally, post-processing operations can be performed on the filled welding image to improve image quality and readability. For example, the image can be smoothed to remove possible noise points; edge detection can be performed to enhance the edge clarity of laser stripes; and geometric transformations such as scaling and rotation can be applied to the image as needed to make it conform to the display requirements of the welding equipment or the input format of subsequent image processing algorithms.

[0042] Furthermore, to facilitate operators' analysis and understanding of welding images during the welding process, annotations can be added to the welding images. For example, these annotations can mark the boundaries of the welding area, the center line position of the laser stripes, and key welding parameters. These annotations can be represented using graphic elements such as text, lines, and arrows, and should be arranged appropriately according to the image layout and display effect.

[0043] The technical solution described in this embodiment introduces a U-shaped multi-scale feature fusion network to filter target pixels that meet set conditions, and constructs a welding image for use in welding scenarios based on these target pixels. This method solves the technical problem of difficulty in accurately and stably acquiring laser stripes in images obtained by laser tracking sensors due to severe arc light and spatter interference when determining the welding trajectory in multi-layer, multi-pass welding processes. In this implementation, by introducing a U-shaped multi-scale feature fusion network, interference from spatter, reflection, and ambient light on laser stripe extraction can be effectively avoided, achieving stable and rapid extraction of laser stripes. This enables precise positioning of welds and measurement of bevel dimensions in multi-layer, multi-pass welding processes, thereby improving the accuracy and efficiency of determining the welding trajectory in such processes.

[0044] As an optional embodiment of this example, the welding image determination method provided in this example may further include the following steps:

[0045] a1) Analyze the welding image to determine the welding position of the multi-layer, multi-pass welding object.

[0046] In this embodiment, to display the welding image more clearly, image preprocessing operations can be performed on the welding image. For example, Gaussian filtering, median filtering, and other methods can be used to remove noise; histogram equalization or adaptive histogram equalization (CLAHE) can be used to enhance the image contrast and make features such as laser stripes more obvious; and a preset threshold can be used to convert the image into a binary image to facilitate subsequent feature extraction.

[0047] Furthermore, edge detection algorithms are used to detect the edges of the laser stripes, and then morphological operations (such as dilation and erosion) are applied to connect and refine the edges to obtain the centerline of the laser stripes. The position and shape of the centerline reflect the surface contour of the object being welded. Based on the distribution and characteristics of the laser stripes, the layers and number of passes in a multi-layer, multi-pass weld are identified. Different layers and passes can be determined by analyzing features such as stripe spacing, angle, and continuity. For example, the stripe spacing between adjacent layers may differ, and stripes in different passes within the same layer may have similar angles and continuity. Combining the welding process requirements and the identification results of the multi-layer, multi-pass welds, the welding location is determined. For each layer and each pass, the boundary of the area to be welded is calculated based on the position of the centerline of the laser stripes and the required welding width. Simultaneously, the start and end points of the weld, as well as the transition zone during the welding process, are considered.

[0048] b1) Determine the welding trajectory of the welding equipment according to each of the described welding positions.

[0049] In this embodiment, considering that the welding trajectory should simultaneously satisfy the conditions of continuity, safety, and improved welding efficiency, a fitting operation can be performed on the position to be welded. For example, straight line fitting or curve fitting. After generating the initial welding trajectory, a path optimization algorithm is used to optimize the trajectory to further improve welding efficiency. For example, genetic algorithms or ant colony algorithms.

[0050] c1) Control the welding equipment to perform welding operations according to the welding trajectory.

[0051] In this embodiment, a suitable hardware interface, such as serial communication, Ethernet communication, or CAN bus communication, is selected based on the type of welding equipment and control requirements. The corresponding communication protocol is used to achieve communication with the welding equipment, ensuring that control commands are transmitted to the welding equipment accurately. Furthermore, based on the coordinate information of the welding trajectory, the moving parts of the welding equipment are controlled to precisely move to the welding position, and the welding speed of the welding equipment is controlled according to the welding process requirements.

[0052] Example 2

[0053] Figure 2This is a flowchart of a welding image determination method according to Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is that this embodiment further refines the specific implementation of extracting the set of stripe pixels constituting laser stripes from the target area image using a given information extraction model. Furthermore, based on the above embodiments, this embodiment further refines the specific implementation of selecting target pixels that meet set conditions from the stripe pixel set, and constructing a welding image applied to a welding scene based on each target pixel. Figure 2 As shown, the welding image determination method provided in this embodiment may specifically include:

[0054] S201. Receive the laser stripe image fed back by the laser tracking device, and determine the target area image from the laser stripe image.

[0055] S202. Using the information extraction model, retrieve the pre-stored weight file to configure network parameters.

[0056] In this embodiment, the sample dataset is input into the information extraction model, a preset learning rate, a preset number of samples input to the model for each training session, and a preset number of training iterations are set. An appropriate loss function is selected for training, and a weight file is finally output. Furthermore, storing the weight file in memory in Open Neural Network Exchange (ONNX) format simplifies the model deployment process and improves real-time performance.

[0057] In this embodiment, the integrity and compatibility of the pre-stored weight file can be verified when retrieving it. The information extraction model is trained using a progressive weight loading method. After retrieving the pre-stored weight file, the parameter values ​​are assigned to the corresponding parameters of the information extraction model. After parameter assignment, the information extraction model is configured based on the parameters in the weight file. At this point, the information extraction model can perform forward propagation calculations based on these parameters to extract features from the input target region image. After configuration, the information extraction model is verified to ensure it can operate normally and output reasonable results.

[0058] S203. Using the configured information extraction model, feature extraction is performed on the target region image to obtain image contour features, image edge and texture features.

[0059] In this embodiment, the configured information extraction model can accurately identify laser stripes through multi-scale feature fusion, without being affected by interference factors such as surrounding welding spatter and changes in lighting.

[0060] Furthermore, during the information extraction process, for the target area image, small-scale features can identify the subtle edges and breaks of the laser stripes, while large-scale features can determine the position of the laser stripes in the entire welding area, thereby avoiding misjudgment due to local noise or background interference.

[0061] In this embodiment, the image's contour features include the basic shape and boundaries of the object to be welded, which helps distinguish the categories of objects to be welded. Edge features include areas of dramatic grayscale changes in the image, which helps determine the specific location and shape details of the object to be welded. Texture features include the structure and patterns of the image surface, which helps identify the material and surface properties of the object to be welded. By extracting image contour features, as well as image edge and texture features, the laser stripe image in the target area can be highlighted, and irrelevant background noise and redundant information can be removed, making subsequent analysis and processing more accurate and efficient.

[0062] In this embodiment, the extracted image contour features, edge features, and texture features are enhanced. Specifically, methods such as histogram equalization or wavelet transform can be used.

[0063] S204. Based on the image contour features and the image edge and texture features, extract the pixels that constitute the laser stripe contour and texture in the target area image to obtain multiple stripe pixels and form a stripe pixel set.

[0064] In this embodiment, when extracting image contour features, thresholds can be set based on parameters such as the perimeter and area of ​​the contour to filter out contour pixels that conform to the laser stripe characteristics. When extracting image edge and texture features, thresholds are set based on the intensity and direction of the edges to retain only edge pixels with high intensity and directions that conform to the laser stripe characteristics. Furthermore, thresholds are set based on feature values ​​such as contrast and correlation to filter out pixels with specific texture features. Each pixel in the target region image is traversed, and pixels that meet the thresholds for contour, edge, and texture features are marked as pixels that may belong to laser stripes. Then, the marked pixels are further filtered to remove isolated pixels that do not conform to the continuity and shape characteristics of laser stripes.

[0065] In this embodiment, connected component analysis is performed on the filtered pixels to divide interconnected pixels into different connected regions. Each connected region may represent a laser stripe or a portion thereof. By analyzing the area, shape, and other features of the connected regions, connected regions that conform to the characteristics of laser stripes are further filtered out. The pixels within the filtered connected regions that conform to the characteristics of laser stripes are integrated to form a stripe pixel set. During the integration process, the pixels can be sorted, for example, according to their position in the image, for subsequent processing and analysis.

[0066] S205. Convert each stripe pixel in the stripe pixel set into a grayscale pixel with a set grayscale value, and construct a stripe binary image with a set image size based on each grayscale pixel, wherein the set image size is the image size of the laser stripe image.

[0067] In this embodiment, each stripe pixel in the stripe pixel set is converted into a grayscale pixel with a grayscale value of 255, and a stripe binary image of the same size as the laser stripe image is constructed based on the grayscale pixels. The stripe binary image is a single-channel image generated using the OpenCV image processing library, where the grayscale value of the background pixels is 0 and the grayscale value of the grayscale pixels is 255.

[0068] S206. Traverse the striped binary image, and for each column of pixels in the striped binary image, determine the pixel with the smallest pixel value and the smallest pixel coordinate as the target pixel.

[0069] In this embodiment, the image is typically stored in computer memory as a matrix. For a binary image, each element in the matrix corresponds to a pixel with a value of 0 or 255, representing black and white pixels respectively. When processing images, it is necessary to correctly read and store image data according to the characteristics of the programming language and related libraries. For example, when reading an image using the OpenCV library in Python, the image is stored as a NumPy array, and we can access and modify pixel values ​​through the array index. By traversing each column of pixels in the striped binary image, the pixel with a grayscale value of 255 and the smallest y-coordinate value in each column is selected and determined as the target pixel. The specific formula is as follows:

[0070] S(x,y min )=255 where y min =min{y|I(x,y)=255} (1)

[0071] S(x,y)=0 for all other y S(x,y)=0 for all other y (2)

[0072] Here, S represents a new binary image obtained after specific processing. x represents the column coordinate in the image, and y represents the row coordinate. A coordinate system is established with the top-left corner of the image as the origin. The x-axis points horizontally to the right, and its value ranges from 0 to an integer from the image width minus 1, used to identify different columns in the image. The y-axis points vertically downwards, and its value ranges from 0 to an integer from the image width minus 1, used to identify different rows in the image. The position of a pixel in the image can be uniquely determined by the combination of x and y coordinates (x, y).

[0073] Optionally, formula (1) indicates that for each column x in the image, the y-coordinates of all pixels with a pixel value of 255 in that column are found, and the minimum value among these y-coordinates is selected and denoted as y. min In the new image S, the coordinates (x, y) min Set the pixel value at () to 255.

[0074] Formula (2) shows that for each column x, except for y = y min Apart from this position, the pixel values ​​at all other positions (x, y) in this column of the new image S are set to 0.

[0075] For example, suppose we have a (3*3) binary image I with the following pixel value distribution:

[0076]

[0077] Process image I according to formulas ① and ② to obtain a new image S:

[0078] For the first column (x = 0): start traversing from the first row (y = 0), I(0,0) = 0; continue to the second row, I(1,0) = 255, y min =1, in the new image S, set S(0,1) to 255, and set the pixel values ​​of other positions in this column to 0.

[0079] For the second column (x = 1): start traversing from the first row, I(0,1) = 0, continue to the second row, I(1,1) = 255, and get y. min =1. In the new image S, set S(1,1) to 255, and set the pixel values ​​at other positions in this column to 0.

[0080] For the third column (x = 2): start traversing from the first row, I(0,2) = 255; y min =0. In the new image S, set S(2,0) to 255, and set the pixel values ​​at other positions in this column to 0.

[0081] After the above processing, the new image S is as follows:

[0082]

[0083] In this embodiment, the pixels in the new image S obtained after the above processing are the target pixels.

[0084] S207. Summarize all the target pixels, determine the region formed by each target pixel as the welding region, and obtain a welding image containing each welding region.

[0085] In this embodiment, connected component marking is performed on the target pixels. An algorithm is used to mark interconnected pixels as the same connected component, where each connected component represents a welding area. By marking the target pixels as connected components, different welding areas can be clearly identified, facilitating subsequent processing.

[0086] The above technical solution in this embodiment provides a specific implementation for obtaining welding images containing each of the welding regions. In this implementation, the information extraction model retrieves a pre-stored weight file for network parameter configuration. When the weight file is used to configure the information extraction model, the information extraction model does not need to be retrained, which greatly reduces the training time and number of iterations of the model, speeds up the convergence speed, saves a lot of computing time and hardware resources, and reduces training costs.

[0087] Meanwhile, the above-mentioned technical solution in this embodiment, through the configured information extraction model, performs feature extraction on the target area image to obtain image contour features, image edge and texture features. Based on the image contour features, image edge and texture features, the pixels constituting the laser stripe contour and texture in the target area image are extracted to obtain multiple stripe pixels and form a stripe pixel set. Each stripe pixel in the stripe pixel set is converted into a grayscale pixel with a set grayscale value, and a stripe binary image with a set image size is constructed based on each grayscale pixel. Traversing the stripe binary image, for each column of pixels in the stripe binary image, the pixel with the set grayscale value and the smallest pixel coordinate is determined as the target pixel. All target pixels are summarized, and the area formed by each target pixel is determined as the welding area, and a welding image containing each welding area is obtained. This can improve computational efficiency, effectively reduce measurement errors, improve measurement accuracy, and reduce misjudgment and incorrect extraction caused by redundant information. On the other hand, it can effectively avoid the interference of spatter, reflection and ambient light on laser stripe extraction, and achieve stable and rapid extraction of laser stripes. This enables precise positioning of welds and measurement of groove dimensions in multi-layer and multi-pass welding processes, thereby improving the accuracy and efficiency of determining the welding trajectory in multi-layer and multi-pass welding processes.

[0088] Based on the above embodiments, this embodiment also proposes an optional embodiment. This optional embodiment can further perform the following after obtaining the welding image applied to the welding scenario: analyze the welding image to determine the welding position of the multi-layer, multi-pass welding object; determine the welding trajectory of the welding equipment according to each welding position; and control the welding equipment to perform welding operations according to the welding trajectory.

[0089] In this embodiment, the welding trajectory of the welding equipment can be determined using a path planning algorithm based on information about the position to be welded. Furthermore, after determining the welding trajectory, the welding equipment is controlled to perform welding operations according to the welding trajectory based on real-time detected welding parameters and welding quality feedback information.

[0090] As an optional embodiment of this example, a further implementation of determining the sample dataset is provided. Specifically, the steps for determining the sample dataset may include:

[0091] a2) Acquire multiple historical laser stripe images captured by a laser tracking device over a historical period of time for a relatively multi-layer, multi-pass welding object.

[0092] In this embodiment, the multi-layer, multi-pass welding object can be a thick plate structure, pressure vessel, piping system, or other welding object with high requirements for welding quality, strength, and performance, or with thick weld joints and complex structures. This embodiment does not specifically limit the multi-layer, multi-pass welding object.

[0093] In this embodiment, the historical time can be a continuous period of time in the past, which can be several hours or several days. This embodiment does not specifically limit the historical time.

[0094] b2) For each historical laser stripe image, the historical laser stripe image is segmented according to the set reference image segmentation model to obtain at least one laser stripe region.

[0095] In this embodiment, a baseline image segmentation model is used to segment historical laser stripe images to obtain at least one laser stripe region, which includes the complete laser stripe image. The baseline image segmentation model is a general image segmentation model such as the Segment Anything Model (SMA).

[0096] In this embodiment, before obtaining at least one laser stripe region, image preprocessing techniques are used, such as median filtering to remove noise, and edge detection algorithms are used to accurately identify the edges of the laser stripes, thereby determining the region where the laser stripes are located, ensuring that the located region includes a complete and accurate portion of the laser stripes.

[0097] c2) Summarize the pixels in each of the laser stripe regions to form a pixel set, use the historical laser stripes as sample laser stripe images, and determine the pixel set as the label pixel set.

[0098] In this embodiment, when summarizing the pixels in each laser stripe region, the pixels are filtered and classified according to their grayscale value, color features and other attributes to form a pixel set. The historical laser stripe image is used as the sample laser stripe image, and the pixel set is determined as the label pixel set.

[0099] d2) Take the sample laser stripe image and the set of label pixels as a sample tuple, and summarize the sample tuples to form the sample dataset.

[0100] In this embodiment, when constructing sample binary pairs, detailed metadata information is added to each sample laser stripe image, such as the shooting time, environmental parameters at the time of shooting (e.g., light intensity, temperature), the specific type of welding object, and related parameters (e.g., material, thickness). For the labeled pixel set, in addition to recording basic information such as pixel coordinates, the welding process parameters corresponding to the laser stripe (e.g., welding current, voltage, welding speed) can also be labeled, providing more information for subsequent data analysis and model training.

[0101] Based on the above optional embodiments, further optimizations can be made including: performing information generalization processing on each of the sample pairs to obtain corresponding enhanced sample pairs, and adding each of the enhanced sample pairs to the sample dataset.

[0102] In this embodiment, to enhance the quantity and diversity of the sample laser stripe images and label pixel sets, information generalization processing is performed on each sample tuple to obtain corresponding enhanced sample tuples, and each enhanced sample tuple is added to the sample dataset. For example, rotation, scaling, color transformation, and grayscale adjustment are applied. This can improve the model's generalization ability, enabling it to perform better when faced with real images of different poses, lighting conditions, and noise levels.

[0103] As another optional embodiment of this example, one implementation of the information extraction model determination can be described as follows:

[0104] a3) Input the sample laser stripe image of each sample binary in the sample dataset into the pre-constructed initial information extraction model to obtain the actual pixel set output by the initial information extraction model.

[0105] In this embodiment, each sample tuple in the sample dataset includes a sample laser stripe image and a set of label pixels. The pre-built initial information extraction model is used to provide a reference pixel set for subsequent model training.

[0106] b3) Based on the actual pixel set and the label pixel set in the corresponding sample tuple, and in conjunction with the given loss function, determine the loss function value.

[0107] In this embodiment, the loss function may include the mean squared error loss function, the Huber loss function, the Dice coefficient, the focal loss function, etc. For example, if the coordinates of the laser stripe pixel set are continuous values ​​and all errors are required to be treated equally, the mean squared error loss function can be selected; if there are some abnormal pixels in the coordinates of the laser stripe pixel set, the Huber loss function can be selected. This embodiment does not specifically limit the loss function. Furthermore, the calculated loss function value is verified to check whether it is within a preset range.

[0108] c3) Update the initial weight file corresponding to the initial information extraction model based on the loss function value.

[0109] In this embodiment, gradient descent and its variants are used to update the weights of the initial information extraction model. The weights of the initial information extraction model are updated along the negative gradient direction of the loss function to gradually reduce the value of the loss function. After updating the weights of the initial information extraction model, the updated weight file is stored in memory.

[0110] d3) Construct the initial information extraction model using the updated initial weight file, and return to re-execute the determination operation of the actual pixel point set until the training iteration condition is met, and obtain the weight file after the iteration is completed.

[0111] In the embodiment, during the initial information extraction model construction stage, the initial information extraction model is instantiated according to the architecture definition of the initial information extraction model. The parameters in the updated weight file are accurately loaded into the corresponding parameter positions of the model. The parameters in the weight file are matched and assigned with the parameters of the initial information extraction model to ensure that the initial state of the initial information extraction model is based on the updated weight file.

[0112] In this embodiment, to improve the stability and convergence speed of the information extraction model training, the training iteration condition can be reaching a pre-set threshold for the number of iterations, such as 100 iterations. A convergence condition for the loss function value can also be set; for example, when the loss function value no longer decreases for several consecutive iterations (e.g., 10 consecutive iterations), the model is considered to have converged, and training is stopped.

[0113] In this embodiment, the weight file after the iteration ends contains all the parameters learned by the information extraction model during training, which can be used to extract information from new laser stripe images.

[0114] e3) The initial information extraction model with the weighted file is determined as the information extraction model.

[0115] In this embodiment, before determining the initial information extraction model with the weighted file as the final information extraction model, the integrity of the weighted file must first be checked. This is done by calculating the hash value of the weighted file (e.g., MD5, SHA-256) and comparing it with the hash value recorded during the original storage or transmission. If the hash values ​​match, it indicates that the weighted file has not been damaged or tampered with during storage or transmission, and the file is intact. After the weighted file passes verification, the initial information extraction model of that specific weighted file is determined as the information extraction model and used for real-time laser stripe extraction.

[0116] Example 3

[0117] Figure 3 This is a schematic diagram of a welding image determination device according to Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0118] Image acquisition module 31 is used to receive the laser stripe image fed back by the laser tracking device and determine the target area image from the laser stripe image;

[0119] Extraction module 32 is used to extract the set of stripe pixels constituting the laser stripe from the target area image through a given information extraction model. The information extraction model is a U-shaped multi-scale feature fusion network, which is pre-trained through a constructed sample dataset. Each sample tuple in the sample dataset includes a sample laser stripe image and a corresponding set of labeled pixels.

[0120] The image processing module 33 is used to select target pixels that meet set conditions from the stripe pixel set, and to construct a welding image for use in the welding scene based on each target pixel.

[0121] This embodiment provides a welding image determination device that introduces a U-shaped multi-scale feature fusion network to filter target pixels that meet set conditions, and constructs a welding image for application in a welding scenario based on these target pixels. This method solves the technical problem of difficulty in accurately and stably acquiring laser stripes in images obtained by laser tracking sensors due to severe arc light and spatter interference when determining the welding trajectory in multi-layer, multi-pass welding processes. In this implementation, by introducing a U-shaped multi-scale feature fusion network, interference from spatter, reflections, and ambient light on laser stripe extraction can be effectively avoided, achieving stable and rapid extraction of laser stripes. This enables precise positioning of the weld and measurement of bevel dimensions in multi-layer, multi-pass welding processes, thereby improving the accuracy and efficiency of determining the welding trajectory in such processes.

[0122] Optionally, the image acquisition module 31 can be specifically used for:

[0123] The network parameters are configured by retrieving the pre-stored weight file using the information extraction model.

[0124] The configured information extraction model is used to extract features from the target region image to obtain image contour features, as well as image edge and texture features.

[0125] Based on the image contour features and the image edge and texture features, the pixels that constitute the laser stripe contour and texture in the target area image are extracted to obtain multiple stripe pixels and form a stripe pixel set.

[0126] Optionally, the image processing module 31 may specifically be used for:

[0127] Each stripe pixel in the stripe pixel set is converted into a grayscale pixel with a set grayscale value, and a stripe binary image with a set image size is constructed based on each grayscale pixel, wherein the set image size is the image size of the laser stripe image;

[0128] Traverse the striped binary image, and for each column of pixels in the striped binary image, determine the pixel with the smallest pixel value and the smallest pixel coordinate as the target pixel;

[0129] Traverse the striped binary image, and for each column of pixels in the striped binary image, determine the pixel with the smallest pixel value and the smallest pixel coordinate as the target pixel;

[0130] By summing up all the target pixels, the region formed by each target pixel is determined as the welding region, and a welding image containing each welding region is obtained.

[0131] Optionally, the laser tracking device includes an image acquisition device and a laser projector, wherein the laser stripe image is an image captured by the image acquisition device when the laser projector is in laser projection mode.

[0132] The device also includes a sample determination module, which can be specifically used for:

[0133] Acquire multiple historical laser stripe images captured by a laser tracking device over a historical period of time for a relatively multi-layer, multi-pass welding object;

[0134] For each historical laser stripe image, the historical laser stripe image is segmented according to the set benchmark image segmentation model to obtain at least one laser stripe region;

[0135] The pixels in each of the laser stripe regions are aggregated to form a pixel set. The historical laser stripes are used as sample laser stripe images, and the pixel set is determined as the label pixel set.

[0136] The sample laser stripe image and the set of label pixels are taken as a sample tuple, and all the sample tuples are combined to form the sample dataset.

[0137] Optionally, the sample determination module can also be used for:

[0138] Information generalization processing is performed on each of the sample pairs to obtain corresponding augmented sample pairs, and each of the augmented sample pairs is added to the sample dataset.

[0139] Optionally, the device may further include:

[0140] The acquisition module is used to input the sample laser stripe image of each sample binary in the sample dataset into the pre-constructed initial information extraction model to obtain the actual pixel set output by the initial information extraction model;

[0141] The determination module is used to determine the loss function value based on the actual pixel set and the label pixel set in the corresponding sample binary, combined with a given loss function.

[0142] The update module is used to update the initial weight file corresponding to the initial information extraction model based on the loss function value;

[0143] The model training module is used to construct the initial information extraction model using the updated initial weight file, and return to re-execute the determination operation of the actual pixel point set until the training iteration condition is met, and obtain the weight file after the iteration ends.

[0144] The model determination unit is used to determine the initial information extraction model with the weight file as the information extraction model.

[0145] The device may further include:

[0146] The analysis module is used to analyze the welding image and determine the welding position of the multi-layer, multi-pass welding object;

[0147] The trajectory determination module is used to determine the welding trajectory of the welding equipment based on each of the positions to be welded.

[0148] The operation module is used to control the welding equipment to perform welding operations according to the welding trajectory.

[0149] The automatic operation and maintenance device provided in this embodiment of the invention can execute an automatic operation and maintenance method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0150] Example 4

[0151] Figure 4 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0152] like Figure 4 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 41. An input / output (I / O) interface 45 is also connected to the bus 41.

[0153] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0154] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as the welding image determination method.

[0155] In some embodiments, the welding image determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the welding image determination method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the welding image determination method by any other suitable means (e.g., by means of firmware).

[0156] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0157] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0158] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0159] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0160] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0161] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0162] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0163] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining welding images, characterized in that, include: Receive laser stripe images fed back by a laser tracking device, and determine the target area image from the laser stripe images; Using a given information extraction model, the set of stripe pixels constituting the laser stripes is extracted from the target area image. The information extraction model is a U-shaped multi-scale feature fusion network, which is pre-trained using a constructed sample dataset. Each sample tuple in the sample dataset includes a sample laser stripe image and a corresponding set of labeled pixels. Target pixels that meet the set conditions are selected from the stripe pixel set, and a welding image for use in the welding scene is constructed based on each target pixel.

2. The method according to claim 1, characterized in that, Using a given information extraction model, the set of stripe pixels constituting the laser stripes is extracted from the target region image, including: The network parameters are configured by retrieving the pre-stored weight file using the information extraction model. The configured information extraction model is used to extract features from the target region image to obtain image contour features, as well as image edge and texture features. Based on the image contour features and the image edge and texture features, the pixels that constitute the laser stripe contour and texture in the target area image are extracted to obtain multiple stripe pixels and form a stripe pixel set.

3. The method according to claim 1, characterized in that, The step of selecting target pixels that meet set conditions from the stripe pixel set and constructing a welding image for the welding scene based on each target pixel includes: Each stripe pixel in the stripe pixel set is converted into a grayscale pixel with a set grayscale value, and a stripe binary image with a set image size is constructed based on each grayscale pixel, wherein the set image size is the image size of the laser stripe image; Traverse the striped binary image, and for each column of pixels in the striped binary image, determine the pixel with the smallest pixel value and the smallest pixel coordinate as the target pixel; By summing up all the target pixels, the region formed by each target pixel is determined as the welding region, and a welding image containing each welding region is obtained.

4. The method according to claim 1, characterized in that, The laser tracking device includes an image acquisition device and a laser projector; The laser stripe image is an image captured by the image acquisition device when the laser projector is in laser projection mode.

5. The method according to claim 1, characterized in that, The steps for determining the sample dataset include: Acquire multiple historical laser stripe images captured by a laser tracking device over a historical period of time for a relatively multi-layer, multi-pass welding object; For each historical laser stripe image, the historical laser stripe image is segmented according to the set benchmark image segmentation model to obtain at least one laser stripe region; The pixels in each of the laser stripe regions are aggregated to form a pixel set. The historical laser stripes are used as sample laser stripe images, and the pixel set is determined as the label pixel set. The sample laser stripe image and the set of label pixels are taken as a sample tuple, and all the sample tuples are combined to form the sample dataset.

6. The method according to claim 5, characterized in that, Also includes: Information generalization processing is performed on each of the sample pairs to obtain corresponding augmented sample pairs, and each of the augmented sample pairs is added to the sample dataset.

7. The method according to any one of claims 1-6, characterized in that, The steps for determining the information extraction model include: The sample laser stripe image of each sample binary in the sample dataset is input into the pre-constructed initial information extraction model to obtain the actual pixel set output by the initial information extraction model; Based on the actual pixel set and the label pixel set in the corresponding sample binary, and in conjunction with the given loss function, the loss function value is determined. The initial weight file corresponding to the initial information extraction model is updated based on the loss function value; The initial information extraction model is constructed using the updated initial weight file, and the determination operation of the actual pixel set is re-executed until the training iteration condition is met, and the weight file after the iteration is completed is obtained. The initial information extraction model with the weighted file is determined as the information extraction model.

8. The method according to any one of claims 1-6, characterized in that, Also includes: Analyze the welding images to determine the welding positions of multi-layer, multi-pass welding objects; The welding trajectory of the welding equipment is determined based on each of the described welding positions; Control the welding equipment to perform welding operations according to the stated welding trajectory.

9. A welding image determining device, characterized in that, include: The image acquisition module is used to receive the laser stripe image fed back by the laser tracking device and determine the target area image from the laser stripe image; The extraction module is used to extract the set of stripe pixels constituting the laser stripes from the target area image using a given information extraction model. The information extraction model is a U-shaped multi-scale feature fusion network, which is pre-trained using a constructed sample dataset. Each sample tuple in the sample dataset includes a sample laser stripe image and a corresponding set of labeled pixels. The image processing module is used to select target pixels that meet set conditions from the stripe pixel set, and to construct a welding image for use in the welding scene based on each target pixel.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the welding image determination method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the welding image determination method according to any one of claims 1-8.

Images