A data augmentation method for a weld seam tracking model and a weld seam tracking method

Abstract

The present invention relates to the field of computer vision technology, and provides a data enhancement method for a weld tracking model and a weld tracking method, including: the arc light noise simulation algorithm randomly designates a point at the bottom of the weld image as the arc light center, designates the arc light center brightness, and generates an arc light noise image based on the arc light center brightness and the distance between any point in the weld image and the arc light center; the welding spatter simulation algorithm randomly selects a center point and an initial point within the weld image, determines an end point at a certain distance from the initial point on the extension line of the connection line between the center point and the initial point, randomly generates the brightness and the light width, draws a spatter light ray segment, and generates multiple spatter light rays by adjusting the center point, the initial point, the end point, the brightness and the light width to form a welding spatter image; both the arc light noise image and the welding spatter image are superimposed on the weld image. The training effect of the weld tracking neural network model can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a data augmentation method for a weld tracking model and a weld tracking method. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Welding is a key process in modern manufacturing and has wide applications in fields such as machinery manufacturing, petrochemical industry, and shipbuilding. With the continuous development of computer technology, many researchers have applied advanced vision sensor technology to welding robots to achieve the positioning and real-time correction of welding trajectories. The line structured light sensor based on active light vision is favored by the automatic welding robot industry due to its non-contact, high robustness, high precision, and low cost. Among them, how to quickly and accurately locate the weld feature points from laser images in a noisy environment is the key to realizing weld tracking.

[0004] In previous studies, most scholars used traditional image processing methods to extract and locate the feature positions of welds. Although they could maintain good operation speed and accuracy in a weak noise environment, they were prone to failure when facing a complex strong noise environment. The neural network algorithm shows powerful capabilities in automatically extracting image features and is especially suitable for the field of automatic welding that requires feature extraction under complex noise conditions. However, its processing effect highly depends on the quality of the training dataset and is extremely prone to failure when there are scenes not covered by the dataset.

[0005] Therefore, a single weld dataset often cannot cover all welding scenarios. Facing different welding noise interferences, different weld shapes and sizes, and different weld positions, etc., it cannot adapt in time and is prone to misjudgment. To better improve the generalization ability of the neural network model and reduce image overfitting, in addition to collecting high-quality datasets, data augmentation is also required during training to achieve the purpose of expanding the dataset. Data augmentation for weld datasets can be divided into two types: one is traditional image processing operations, and the other is the simulation of welding noise for images.

[0006] However, the spatter, arc light, etc. noises generated during the welding process have great randomness and are almost impossible to predict comprehensively. The smoke noise in welding images has more complex morphological features compared to spatter, arc light, etc. noises, and traditional data augmentation methods are difficult to simulate the spatter, arc light, and smoke noises in the weld tracking process. Summary of the Invention

[0007] To solve the technical problems existing in the above-mentioned background art, the present invention provides a data augmentation method for a weld seam tracking model and a weld seam tracking method. The spatter noise in welding noise is approximately characterized as discrete line segments emitted outward from a point area under the weld image plane, and the arc light noise is represented as a brightness change image diffusing outward from a point at the bottom of the weld image, realizing the simulation of arc light noise and welding spatter. By superimposing them on the weld image for data augmentation, the training effect of the weld seam tracking neural network model can be improved.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] The first aspect of the present invention provides a data augmentation method for a weld seam tracking model, which includes:

[0010] Obtain a set of weld images;

[0011] For each weld image in the set of weld images, generate an arc light noise image through an arc light noise simulation algorithm, generate a welding spatter image through a welding spatter simulation algorithm, and superimpose both the arc light noise image and the welding spatter image on the weld image;

[0012] Among them, the arc light noise simulation algorithm randomly designates a point at the bottom of the weld image as the arc light center, designates the arc light center brightness, and generates an arc light noise image based on the arc light center brightness and the distance between any point in the weld image and the arc light center; the welding spatter simulation algorithm randomly selects a center point and an initial point within the weld image, determines an end point at a certain distance from the initial point on the extension line of the connection line between the center point and the initial point, randomly generates the brightness and the light width, draws a spatter light ray segment, and generates multiple spatter light rays by adjusting the center point, the initial point, the end point, the brightness, and the light width to form a welding spatter image.

[0013] Further, the generation step of the arc light noise image includes:

[0014] Calculate the pixel value of any point in the arc light noise image based on the arc light center brightness and the distance between any point in the weld image and the arc light center;

[0015] Perform threshold segmentation on the arc light noise image, adjust the pixel values greater than the threshold to the threshold size, and then apply Gaussian blur to the arc light noise image.

[0016] Further, the pixel value of any point in the arc light noise image is: , where \(p\) represents the position of any point in the weld image, \(p_0\) represents the position of the arc center, \(dis(p, p_0)\) represents the distance from any point in the weld image to the arc center, \(value0\) represents the brightness of the arc center, and \(max(dis(p, p_0))\) represents the distance between the farthest point in the weld image and the arc center. Further, it also includes: Gaussian blurring the welding spatter image and then superimposing it on the weld image.

[0017] Further, it also includes: for each weld image in the weld image set, generating a smoke noise image through a smoke noise simulation algorithm, and superimposing the smoke noise image on the weld image.

[0018] Further, the step of superimposing the smoke noise image on the weld image includes: arbitrarily intercepting a section of the area in the smoke noise image, scaling it to the size of the weld image, and then superimposing it on the original weld image.

[0019] Further, the smoke noise simulation algorithm includes: setting grids of different sizes, based on each size of the grid, generating a Perlin noise image through the Perlin noise algorithm, and after superimposing all the Perlin noise images corresponding to the grids of all sizes, performing normalization to obtain the smoke noise image.

[0020] Further, the superimposition of the Perlin noise images uses a fractal function: ; where represents the number of superimposed pictures, represents taking the absolute value of the variable, represents the th pixel value of the Perlin noise image.

[0021] Further, it also includes: performing perspective, scaling, rotation, shearing, translation, flipping, and HSV channel transformation on the weld image in sequence.

[0022] The second aspect of the present invention provides a weld tracking method, including:

[0023] Obtaining a weld image;

[0024] For the weld image, obtaining the weld position through a weld tracking model;

[0025] Among them, during the training process of the weld tracking model, the weld image set is data - enhanced through a data enhancement method for the weld tracking model described in the first aspect.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] In the present invention, the spatter noise in welding noise is approximately characterized as discrete line segments emitted outward from a point area under the weld image plane, and the arc light noise is represented as an image of brightness change diffusing outward from a point at the bottom of the weld image. The simulation of arc light noise and welding spatter is realized, and they are superimposed on the weld image for data enhancement, which can improve the training effect of the weld tracking neural network model.

[0028] Aiming at the problem that the natural smoke effect cannot be obtained from a single generated Perlin noise image, by setting different grid sizes, generating noise pictures multiple times and superimposing them, the blurring effect of smoke on the image can be simulated.

[0029] In the process of superimposing Perlin noise images in the present invention, a fractal function is adopted to obtain the turbulent image of welding smoke, realizing diverse natural texture effects. Brief Description of the Drawings

[0030] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0031] Figure 1 is a flowchart of a data enhancement method for a weld tracking model in Embodiment 1 of the present invention;

[0032] Figure 2 is a schematic diagram of welding-related noise in Embodiment 1 of the present invention;

[0033] Figure 3 is a schematic diagram of smoke noise in an image in Embodiment 1 of the present invention;

[0034] Figure 4 is a schematic diagram of the Perlin noise algorithm in Embodiment 1 of the present invention;

[0035] Figure 5 is a schematic diagram of the application effect of Perlin noise and fractal noise in Embodiment 1 of the present invention;

[0036] Figure 6 is a schematic diagram of spatter and arc light noise in Embodiment 1 of the present invention;

[0037] Figure 7 is an algorithm effect diagram of spatter and arc light noise generation in Embodiment 1 of the present invention. Detailed Embodiments

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention.

[0039] It should be noted that the following detailed description is illustrative and aims to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention pertains.

[0040] Example 1

[0041] This example provides a data augmentation method for a weld seam tracking model.

[0042] The data augmentation method for a weld seam tracking model provided in this example performs data augmentation operations on the images collected by an intelligent welding robot based on a line laser sensor during the welding process to improve the training effect of the weld seam tracking neural network model.

[0043] The data augmentation method for a weld seam tracking model provided in this example, as Figure 1 shown, includes the following steps:

[0044] Step 1, Original data acquisition, obtaining a weld seam image set.

[0045] Step 101, Turn on the line laser sensor and the robot device, and connect them to the host computer terminal.

[0046] Step 102, Take a welding piece, fix the welding piece, and use the robot to teach a weld seam path. The taught path should ensure that the weld seam is in the center position in the image of the line laser sensor.

[0047] Step 103, Select whether to start arc welding, control the robotic arm to execute the teaching program, and at the same time record the weld seam images taken by the sensor in real time and save them as a video.

[0048] Step 104, Perform frame extraction on the video, export it in picture format and save it.

[0049] Step 105, Repeat steps 102 to 104 until all types of welding pieces required are basically covered to obtain a weld seam image set.

[0050] Step 2, A smoke noise simulation algorithm based on Perlin noise and fractal functions.

[0051] During the actual arc welding process, due to the welding process principle, noises such as spatter, arc light, smoke, and blurring are very likely to appear in the captured weld seam images, as Figure 2As shown. When the surfaces of the welding wire and the welded part are heated to a certain degree, they melt and form a molten pool. Under the influence of factors such as the energy of the electric arc and the surface tension of the molten metal, the surface of the molten pool is prone to instability, resulting in splashing of liquid metal. At the same time, due to the excessively high local temperature between the welding wire and the surface of the welded part during welding, some substances between them are prone to volatilization, forming gaseous or particulate smoke. When performing arc welding, a very large current is formed between the welding wire and the welded part. The current passing through the air will cause the air molecules to be ionized, forming a plasma and generating arc light. When the structured light sensor captures the weld seam, due to factors such as vibration and movement, the captured image may be blurred. Thus, it can be seen that the noise generated during the welding process has great randomness and is almost impossible to predict comprehensively. Therefore, designing a simulation algorithm for welding noise generation is of great significance for enhancing the robust performance of the prediction network.

[0052] The smoke noise in the welding image has more complex morphological features compared to noises such as splashing and arc light, such as Figure 3 shown. Due to the influence of air flow, the smoke forms textures similar to turbulence, making it inconvenient to generate its trajectory in the image through traditional geometric methods. In the image, since the central area of the smoke often has a higher brightness, along with its irregular movement, it is prone to situations such as confusion with the laser stripes and covering the laser stripes on the image. At the same time, the smoke also blurs the edges of the laser stripes, greatly increasing the difficulty of feature point prediction.

[0053] Perlin noise is a random number generation algorithm that can generate continuous and smooth patterns and is often used in scenarios such as simulating natural textures, terrain generation, and dynamic smoke. Its main idea is to divide the image into grids, take the random values of the grid corner points, and use interpolation algorithms in the grids to ensure continuous changes.

[0054] The process of the Perlin noise algorithm can be described by Figure 4 As follows. First, divide the image into a certain number of grids and set a random gradient vector for each corner point of each grid; for any point calculated in the image, perform a dot product of the gradient vector of each corner point within the grid where it is located and the distance vector of the inner point to obtain the gradient value of the current inner point in the direction of each corner point; then, use the smoothing function and interpolation algorithm in sequence to interpolate the inner points and obtain the mapped pixel value of the current inner point; repeat the above operations and traverse each point in the image to obtain a continuous and smooth Perlin noise image, as Figure 5 shown.

[0055] Among them, the calculation formula for the gradient value of the corner point corresponding to the inner point is: ; in the formula, grad (p,p0) represents the gradient value calculated between the inner point p and a certain corner point p0, x grad and y grad represent the random vectors corresponding to the corner points, and x and y represent the coordinates of the inner point.

[0056] Among them, the mathematical formula of the smoothing function is: ; in the formula, fade ( ) represents the smoothing function, and t represents the coordinate value of a certain axis at the current point.

[0057] Among them, the mathematical formula of the single interpolation function is: ; in the formula, lerp ( ) represents the interpolation function, t fade represents the coordinate value mapped by the smoothing function in a certain axis direction at the current point, grad a and grad b respectively represent the gradient values of the current internal point to be interpolated relative to the adjacent grid corner points.

[0058] The single-generated Perlin noise image cannot obtain a natural smoke effect. By setting different grid sizes, generating noise pictures multiple times and performing linear superposition, the effect of (b) in Figure 5 can be obtained. As shown in Figure 5 Although the smoke effect of (b) in is already relatively natural, when superimposed on the welding image, it can simulate the blurring effect of the smoke on the image, but still cannot obtain the turbulent image of the welding smoke. In the process of superimposing the Perlin noise image, applying a more complex fractal function can effectively improve this problem and achieve diverse natural texture effects.

[0059] In this embodiment, pictures with different grid sizes are generated, and the following superposition formula is used to superimpose different pictures: ; among them, represents the number of superimposed pictures, represents taking the absolute value of the variable, represents the th pixel value of the picture, is a grayscale image with a pixel value of 0.

[0060] After superimposing by the above method, the image is normalized, and the image value is recalculated using , and finally mapped to the range of 0~255 to obtain a smoke noise image; the final imaging effect is as shown in (c) in Figure 5 ; the image generates irregular light stripes and blurred edge astigmatism; any section of the area is intercepted in the smoke noise image, scaled to the size of the original weld image, and superimposed on the original weld image to form a smoke effect, as shown in (d) in Figure 5 .

[0061] Step 3, Welding spatter and arc light noise simulation algorithm based on random transformation sampling.

[0062] As Figure 6 shown, the spatter noise in the welding noise can be approximately characterized as discrete line segments emitted outward within a point area under the image plane, and the arc light noise can be represented as an image of the brightness change diffusing outward from a point at the bottom of the image. Based on this, in this embodiment, an analog generation algorithm for welding spatter and arc light noise based on random transformation sampling is designed.

[0063] Step 301: Simulation of arc light noise.

[0064] (1) Randomly specify a point 10 - 50 pixels away from the center position of the bottom as the arc light center below the bottom of the picture, and specify the arc light center brightness. Calculate the cube value of the Euclidean distance of each point in the image relative to the center point, and map it to the range of 0 to the maximum brightness.

[0065] The transformation formula it contains is as follows: ; ; where p represents the position of any point in the weld image, p0 represents the position of the arc light center, dis(p, p0) represents the distance from any point in the weld image to the arc light center, value0 represents the pixel value of the arc light center (i.e., the specified arc light center brightness), and max(dis(p, p0)) represents the distance between the farthest point in the weld image and the arc light center.

[0066] (2) Subsequently, perform threshold segmentation, and adjust the pixel values greater than the threshold to the threshold size.

[0067] (3) Finally, apply Gaussian blur to reduce pixel mutations and obtain the arc light noise image.

[0068] Step 302: Simulation of spatter noise. As shown in Algorithm 1 in Table 1, first, for each spatter ray, randomly generate a center point within the neighborhood of a fixed point outside the image, and specify an initial point and the line segment length in the image. Determine the position of the end point at a certain distance from the initial point on the extension line of the connection between the center point and the initial point. Then randomly generate the brightness and the light width, and draw the spatter ray line segment; iterate multiple times to generate multiple spatter rays, and perform random Gaussian blur processing on the spatter ray image; by changing the length, width, etc. of the rays multiple times and adjusting the corresponding random factors, diverse welding spatter images are obtained.

[0069] Table 1. Spatter noise generation algorithm

[0070]

[0071] In the above spatter noise generation algorithm, different spatter noises with different widths, lengths, and brightnesses can be simulated by adjusting different noise parameters.

[0072] The algorithm effect is as Figure 7As shown, the welding noise content in the original image is well enhanced.

[0073] Step 4: A general data augmentation method based on image transformation.

[0074] An image is usually represented in the form of pixels in a computer. If a fixed corner point of the image is selected to establish a pixel coordinate system, the image can be matrix-transformed mathematically. As Figure 2 shown, by performing different matrix transformation operations on the image, operations such as translation, rotation, scaling, flipping, shearing, perspective, and HSV channel adjustment can be achieved, so as to imitate different shapes and sizes of the weld seam and the visual effects of photographing the weld seam from different angles.

[0075] Among them, the functions are briefly described as follows:

[0076] Translation: Move the image along the X-axis and Y-axis directions, and the moving distance is a random number;

[0077] Rotation: Rotate the image around the center point of the image by a certain angle, and the rotation angle is a random number;

[0078] Scaling: Adjust the length and width of the image to a certain multiple of the original size of the image respectively, and keep the center position unchanged. The scaling multiples in different directions are different random numbers;

[0079] Flipping: Flip the image along the X-axis and Y-axis respectively, and the probability of whether to flip along each axis is a random number;

[0080] Shearing: Perform shearing on the image along the X-axis and Y-axis directions respectively, and the shearing degree is a random number;

[0081] Perspective: Deform the image from different perspectives to simulate the perspective effect, and the degree and direction of perspective are determined by random numbers;

[0082] HSV (hue, saturation, and value) channel adjustment: By adjusting the values of different HSV channels of the image, the saturation, brightness, and color of the image can be adjusted to simulate the visual effects of different lighting conditions and different camera filters. The change range of the values of each channel is a random number.

[0083] Among them, the above operations are used to process each batch of input images during the neural network training process.

[0084] Among them, the execution order of the above operations is: perspective, scaling, rotation, shearing, translation, flipping, HSV channel transformation. The edges of the transformed image are filled with black, and the final output image size is the same as the original image size.

[0085] A data augmentation method for a weld seam tracking model provided in this embodiment approximates the spatter noise in welding noise as discrete line segments emitted outward from a point area under the weld image plane, represents the arc light noise as an image of the brightness change diffusing outward from a point at the bottom of the weld image, realizes the simulation of arc light noise and welding spatter, superimposes them on the weld image, and performs data augmentation, which can improve the training effect of the weld seam tracking neural network model.

[0086] A data augmentation method for a weld seam tracking model provided in this embodiment, aiming at the problem that a natural smoke effect cannot be obtained from a single generated Perlin noise image, generates noise pictures multiple times and superimposes them by setting different grid sizes, and can simulate the blurring effect of smoke on the image.

[0087] A data augmentation method for a weld seam tracking model provided in this embodiment adopts a fractal function during the superposition of Perlin noise images to obtain the turbulent image of welding smoke and realize diverse natural texture effects.

[0088] Embodiment 2

[0089] This embodiment provides a weld seam tracking method, which specifically includes:

[0090] Obtain a weld image;

[0091] For the weld image, obtain the weld position through the weld seam tracking model;

[0092] Among them, during the training process of the weld seam tracking model, a data augmentation method for a weld seam tracking model described in Embodiment 1 is used to perform data augmentation on the weld image set (that is, the training set).

[0093] It should be noted here that each module in this embodiment corresponds to each step in Embodiment 1, and the specific implementation process is the same, so it will not be repeated here.

[0094] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data augmentation method for a weld seam tracking model, characterized in that Including: Obtain a set of weld images; For each weld image in the set of weld images, generate an arc light noise image through an arc light noise simulation algorithm, generate a welding spatter image through a welding spatter simulation algorithm, and superimpose both the arc light noise image and the welding spatter image onto the weld image; Among them, the arc light noise simulation algorithm randomly designates a point at the bottom of the weld image as the arc light center, designates the arc light center brightness, and generates an arc light noise image based on the arc light center brightness and the distance of any point in the weld image relative to the arc light center; The welding spatter simulation algorithm randomly selects a center point and an initial point within the weld image, determines an end point at a certain distance from the initial point on the extension line of the connection line between the center point and the initial point, randomly generates brightness and light width, draws a spatter light ray segment, and generates multiple spatter light rays by adjusting the center point, the initial point, the end point, the brightness, and the light width to form a welding spatter image.

2. The data enhancement method for a weld seam tracking model according to claim 1, characterized in that The generation steps of the arc light noise image include: Based on the arc light center brightness and the distance of any point in the weld image relative to the arc light center, calculate the pixel value of any point in the arc light noise image; Perform threshold segmentation on the arc light noise image, adjust the pixel values greater than the threshold to the threshold size, and then apply Gaussian blur to the arc light noise image.

3. A data augmentation method for a weld seam tracking model according to claim 2, characterized in that The pixel value of any point in the arc light noise image is as follows: , where p represents the position of any point in the weld image, p0 represents the position of the arc light center, dis(p, p0) represents the distance from any point in the weld image to the arc light center, value0 represents the brightness of the arc light center, and max(dis(p, p0)) represents the distance between the farthest point in the weld image and the arc light center.

4. A data augmentation method for a weld seam tracking model according to claim 1, characterized in that, Also including: Apply Gaussian blur to the welding spatter image and then superimpose it onto the weld image.

5. A data augmentation method for a weld seam tracking model according to claim 1, characterized in that, Also including: For each weld image in the set of weld images, generate a smoke noise image through a smoke noise simulation algorithm, and superimpose the smoke noise image onto the weld image.

6. A data augmentation method for a weld seam tracking model according to claim 5, characterized in that The step of superimposing the smoke noise image onto the weld image includes: arbitrarily intercepting a section of the area in the smoke noise image, scaling it to the size of the weld image, and then superimposing it onto the original weld image.

7. A data augmentation method for a weld seam tracking model according to claim 5, characterized in that, The smoke noise simulation algorithm includes: setting grids of different sizes, generating a Perlin noise image based on each size of grid through the Perlin noise algorithm, and after superimposing the Perlin noise images corresponding to all sizes of grids, performing normalization to obtain the smoke noise image.

8. A data augmentation method for a weld seam tracking model according to claim 7, characterized in that The superposition of the Berlin noise images uses a fractal function: ; where represents the number of superimposed pictures, represents taking the absolute value of the variable, represents the th pixel value of the Berlin noise image.

9. A data augmentation method for a weld seam tracking model according to claim 1, characterized in that, Also including: Perform perspective, scaling, rotation, shearing, translation, flipping, and HSV channel transformation on the weld image in sequence.

10. A weld seam tracking method, characterized in that, Including: Obtain a weld image; For the weld image, obtain the weld position through a weld tracking model; Among them, during the training process of the weld tracking model, the set of weld images is data-augmented through a data augmentation method for the weld tracking model described in any one of claims 1-9.

Images