Data enhancement method for welding seam tracking model and welding seam tracking method

Abstract

The invention relates to the technical field of computer vision, and provides a data enhancement method for a weld seam tracking model and a weld seam tracking method.The method comprises the steps that an arc light noise simulation algorithm randomly appoints a point at the bottom of a weld seam image as an arc light center, and appoints the brightness of the arc light center; generating an arc light noise image based on the brightness of the arc light center and the distance between any point in the welding seam image and the arc light center; according to the welding spatter simulation algorithm, a center point and an initial point are randomly selected in a welding seam image, a tail end point with a certain distance from the initial point is determined on an extension line of a connecting line of the center point and the initial point, brightness and light width are randomly generated, a spatter light line segment is drawn, and the welding spatter is simulated by adjusting the center point, the initial point, the tail end point, the brightness and the light width. Generating a plurality of spattering light rays to form a welding spattering image; and overlapping the arc light noise image and the welding spatter image to the welding seam image. The training effect of the weld tracking neural network model can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a data enhancement method for a weld tracking model and a weld tracking method. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Welding is a key process in modern manufacturing and is widely used in machinery manufacturing, petrochemicals, shipbuilding and other fields. With the continuous development of computer technology, many researchers have applied advanced visual sensor technology to welding robots to achieve the positioning of welding trajectories and real-time deviation correction. Line structured light sensors based on active optical vision have been favored by the automatic welding robot industry because of their non-contact, high robustness, high precision and low cost. Among them, how to quickly and accurately locate weld feature points from laser images in a noisy environment is the key to achieving weld tracking.

[0004] In previous studies, most scholars used traditional image processing methods to extract and locate the characteristic positions of welds. Although they can maintain good computing speed and accuracy in weak noise environments, they are prone to failure in complex strong noise environments. Neural network algorithms have shown strong capabilities in automatically extracting image features, and are particularly suitable for the field of automatic welding that requires feature extraction under complex noise conditions. However, their processing effects are highly dependent on the quality of the training data set, and are prone to failure when scenes that are not covered by the data set appear.

[0005] Therefore, a single weld dataset often cannot cover all welding scenarios. It cannot adapt to different welding noise interference, different weld shapes and sizes, and different weld positions in time, and is prone to misjudgment. In order to better improve the generalization ability of the neural network model and reduce image overfitting, in addition to collecting high-quality datasets, data enhancement is also required during training to achieve the purpose of expanding the dataset. Data enhancement for weld datasets can be divided into two types: one is traditional image processing operations, and the other is welding noise simulation for images.

[0006] However, the noises such as spatter and arc generated during the welding process are extremely random and almost impossible to predict comprehensively. The smoke noise in the welding image has more complex morphological characteristics than spatter, arc and other noises. Traditional data enhancement methods are difficult to simulate the spatter, arc and smoke noise in the weld tracking process. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a data enhancement method and a weld tracking method for a weld tracking model, which approximately characterizes the spatter noise in the welding noise as discrete line segments emitted outward from a point area under the weld image plane, and represents the arc noise as a brightness change image diffusing outward from a point at the bottom of the weld image, thereby realizing the simulation of arc noise and welding spatter, superimposing them on the weld image, and performing data enhancement, which can improve the training effect of the weld tracking neural network model.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: A first aspect of the present invention provides a data enhancement method for a weld tracking model, comprising: Obtaining a weld image set; For each weld image in the weld image set, an arc noise image is generated by an arc noise simulation algorithm, and a welding spatter image is generated by a welding spatter simulation algorithm, and both the arc noise image and the welding spatter image are superimposed on the weld image; Among them, the arc noise simulation algorithm randomly specifies a point at the bottom of the weld image as the arc center, specifies the arc center brightness, and generates an arc noise image based on the arc center brightness and the distance of any point in the weld image relative to the arc center; the welding spatter simulation algorithm randomly selects a center point and an initial point in the weld image, determines the end point at a certain distance from the initial point on the extension line of the center point and the initial point, randomly generates brightness and light width, draws a spatter light segment, and generates multiple spatter rays by adjusting the center point, initial point, end point, brightness and light width to form a welding spatter image.

[0009] Furthermore, the step of generating the arc noise image includes: Based on the arc center brightness and the distance of any point in the weld image relative to the arc center, the pixel value of any point in the arc noise image is calculated; Threshold segmentation is performed on the arc light noise image. After adjusting the pixel values ​​greater than the threshold to the threshold value, Gaussian blur is applied to the arc light noise image.

[0010] Furthermore, the pixel value of any point in the arc noise image is: , where p represents the position of any point in the weld image, p0 represents the position of the arc center, dis(p,p0) represents the distance from any point in the weld image to the arc center, value0 represents the arc center brightness, and max(dis(p,p0)) represents the distance from the farthest point in the weld image to the arc center. Furthermore, the method further includes: Gaussian blurring the weld spatter image and then superimposing it on the weld image.

[0011] Furthermore, it also includes: for each weld image in the weld image set, generating a smoke noise image through a smoke noise simulation algorithm, and superimposing the smoke noise image on the weld image.

[0012] Furthermore, the step of superimposing the smoke noise image onto the weld image includes: arbitrarily intercepting a section of the smoke noise image, scaling it to the size of the weld image, and superimposing it onto the original weld image.

[0013] Furthermore, the smoke noise simulation algorithm includes: setting grids of different sizes, generating a Perlin noise image based on each size of the grid through the Perlin noise algorithm, and superimposing the Perlin noise images corresponding to grids of all sizes and normalizing them to obtain a smoke noise image.

[0014] Furthermore, the superposition of the Perlin noise image adopts a fractal function: ;in, Indicates the number of superimposed images. Indicates taking the absolute value of a variable. Indicates Pixel values ​​of the Zhang Bailin noise image.

[0015] Furthermore, it also includes: performing perspective, scaling, rotation, shearing, translation, flipping and HSV channel transformation on the weld image in sequence.

[0016] A second aspect of the present invention provides a weld tracking method, comprising: Acquire weld images; For weld images, the weld position is obtained through the weld tracking model; During the training process of the weld tracking model, data enhancement is performed on the weld image set by using a data enhancement method for the weld tracking model described in the first aspect.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention approximately characterizes the spatter noise in the welding noise as discrete line segments radiating outward from a point area under the weld image plane, and represents the arc noise as a brightness change image diffusing outward from a point at the bottom of the weld image, thereby realizing the simulation of arc noise and welding spatter, superimposing them on the weld image, and performing data enhancement, which can improve the training effect of the weld tracking neural network model.

[0018] The present invention aims to solve the problem that a single generated Perlin noise image cannot obtain a natural smoke effect. By setting different grid sizes, the noise images are generated multiple times and superimposed, so as to simulate the blurring effect caused by smoke on the image.

[0019] The present invention adopts a fractal function in the process of superimposing Perlin noise images to obtain a turbulent image of welding smoke and achieve various natural texture effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0021] Figure 1 is a flow chart of a data enhancement method for a weld tracking model according to a first embodiment of the present invention; Figure 2 is a schematic diagram of welding-related noise according to the first embodiment of the present invention; Figure 3 is a schematic diagram of smoke noise in an image according to the first embodiment of the present invention; Figure 4 is a schematic diagram of the Perlin noise algorithm of the first embodiment of the present invention; Figure 5 Schematic diagram of the application effect of Perlin noise and fractal noise according to the first embodiment of the present invention; Figure 6 Schematic diagram of splashing and arc noise according to the first embodiment of the present invention; Figure 7 This is a diagram showing the effect of the splash and arc noise generation algorithm according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0022] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0023] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0024] Embodiment 1 This embodiment provides a data enhancement method for a weld tracking model.

[0025] This embodiment provides a data enhancement method for a weld tracking model, which performs data enhancement operations on images collected by an intelligent welding robot based on a line laser sensor during the welding process to improve the training effect of the weld tracking neural network model.

[0026] This embodiment provides a data enhancement method for a weld tracking model, such as Figure 1 As shown, the following steps are included: Step 1: Collect raw data and obtain weld image sets.

[0027] Step 101: Turn on the line laser sensor and the robot device and connect them to the host computer.

[0028] Step 102, take a welded part, fix the welded part, and use the robot to teach a weld path. The taught path must ensure that the weld is centered in the online laser sensor image.

[0029] Step 103, select whether to start arc welding, control the robot arm to execute the teaching program, and record the weld image taken by the sensor in real time and save it as a video.

[0030] Step 104: extract frames from the video, export it to a picture format and save it.

[0031] Step 105, repeat steps 102 to 104 until the required welding parts types are basically covered to obtain a weld image set.

[0032] Step 2: Smoke noise simulation algorithm based on Perlin noise and fractal function.

[0033] In the actual arc welding process, due to the welding process principle, the weld image is prone to splash, arc light, smoke, blur and other noises. Figure 2 As shown. When the surface of the welding wire and the weldment is heated to a certain degree, it melts and forms a molten pool. Under the factors of the energy of the arc and the surface tension of the molten metal, the surface of the molten pool is easily unstable, resulting in liquid metal splashing. At the same time, due to the excessively high local temperature of the welding wire and the weldment surface during welding, some substances between them are easy to volatilize, forming gaseous or particulate smoke. During arc welding, a large current is formed between the welding wire and the weldment. The current passing through the air will cause the air molecules to ionize, forming plasma and generating arc light. When the structured light sensor shoots the weld, the image may be blurred due to factors such as vibration and movement. It can be seen that the noise generated during the welding process is extremely random and almost impossible to be fully predicted. Therefore, designing a simulated welding noise generation algorithm is of great significance to enhance the robust performance of the prediction network.

[0034] Compared with splash, arc and other noises, smoke noise in welding images has more complex morphological features, such as Figure 3 As shown in the figure. Since smoke is affected by air flow, it forms a turbulent texture, which makes it difficult to generate its trajectory in the image using traditional geometric methods. In the image, the central area of ​​the smoke often has a higher brightness, and with its irregular movement, it is easy to confuse with the laser stripes and cover the laser stripes. At the same time, the smoke also blurs the edge of the laser stripes, greatly increasing the difficulty of feature point prediction.

[0035] Perlin noise is a random number generation algorithm that can generate continuous and smooth patterns. It is often used to simulate natural textures, terrain generation, dynamic smoke and other scenes. The main idea is to divide the image into a grid, take random values ​​at the corners of the grid, and use an interpolation algorithm in the grid to ensure continuous changes.

[0036] The Perlin noise algorithm process can be Figure 4 Description, first, divide the image into a certain number of grids, and set a random gradient vector for each corner point of each grid; for any point in the calculated image, multiply the gradient vector of each corner point in the grid with the distance vector of the inner point to obtain the gradient value of the current inner point relative to each corner point; then, use the smoothing function and interpolation algorithm in turn to interpolate the inner point and obtain the mapping pixel value of the current inner point; repeat the above operation and traverse each point in the image to obtain a continuous and smooth Perlin noise image, such as Figure 5 shown.

[0037] Among them, the calculation formula of the gradient value of the corner point corresponding to the inner point is: ; In the formula, grad (p, p0) represents the gradient value calculated between the interior point p and a corner point p0, x grad and grad Indicates the random vector corresponding to the corner point, and x and y represent the coordinates of the interior point.

[0038] Among them, the mathematical formula of the smoothing function is: ; In the formula, fade ( ) represents a smooth function, and t represents the coordinate value of a certain axis of the current point.

[0039] Among them, the mathematical formula of the single interpolation function is: ; In the formula, lerp ( ) represents the interpolation function, t fade Indicates the coordinate value of the current point in a certain axis direction after being mapped by a smooth function. grad a and grad b They respectively represent the gradient value of the current inner point that needs to be interpolated relative to the adjacent grid corner point.

[0040] A single generated Perlin noise image cannot produce a natural smoke effect. By setting different grid sizes, generating noise images multiple times and performing linear superposition, we can obtain Figure 5 The effect of (b) in Fig. Figure 5Although the smoke effect in (b) is relatively natural and can simulate the blurring effect of smoke on the image when superimposed on the welding image, it still cannot obtain the turbulent image of welding smoke. In the process of superimposing the Perlin noise image, applying more complex fractal functions can effectively improve this problem and achieve a variety of natural texture effects.

[0041] This embodiment generates pictures with different grid sizes and uses the following superposition formula to superimpose different pictures: ;in, Indicates the number of superimposed images. Indicates taking the absolute value of a variable. Indicates The pixel value of the image, A grayscale image with pixel value 0.

[0042] After superposition of the above methods, the image is normalized and used Recalculate the image value and finally map it to the range of 0~255 to obtain the smoke noise image; the final imaging effect is as follows Figure 5 As shown in (c) in the figure, the image generates irregular light stripes and blurred scattered light at the edges. By randomly cutting out a section of the smoke noise image, scaling it to the size of the original weld image, and superimposing it on the original weld image, a smoke effect can be formed, as shown in Figure 5 As shown in (d) in .

[0043] Step 3: Welding spatter and arc noise simulation algorithm based on random transformation sampling.

[0044] like Figure 6 As shown in FIG. 1 , the spatter noise in the welding noise can be approximately characterized as a discrete line segment radiating outward from a point area under the image plane, and the arc noise can be represented as a brightness change image spreading outward from a point at the bottom of the image. Based on this, this embodiment designs a simulation generation algorithm for welding spatter and arc noise based on random transformation sampling.

[0045] Step 301: Simulate arc noise.

[0046] (1) Randomly specify a point 10 to 50 pixels below the bottom of the image as the arc center and specify the arc center brightness. Calculate the cube of the Euclidean distance of each point in the image relative to the center point and map it to the range of 0 to maximum brightness.

[0047] The transformation formula is as follows: ; ; In the formula, p represents the position of any point in the weld image, p0 represents the position of the arc center, dis(p,p0) represents the distance from any point in the weld image to the arc center, value0 represents the arc center pixel value (that is, the specified arc center brightness), and max(dis(p,p0)) represents the distance from the farthest point in the weld image to the arc center.

[0048] (2) Threshold segmentation is then performed to adjust pixel values ​​greater than the threshold to the threshold size.

[0049] (3) Finally, Gaussian blur is applied to reduce pixel mutations and obtain an arc noise image.

[0050] Step 302, for the simulation of spatter noise, as shown in Algorithm 1 in Table 1, first, for each spatter ray, a center point is randomly generated in the neighborhood of a fixed point outside the image, and the initial point and the length of the line segment are specified in the image, and the position of the end point at a certain distance from the initial point is determined on the extension line of the line connecting the center point and the initial point, and then the brightness and light width are randomly generated to draw the spatter ray line segment; it is iterated multiple times to generate multiple spatter rays, and a random Gaussian blur process is performed on the spatter ray image; by repeatedly changing the length, width and other information of the light and adjusting the corresponding random factors, a variety of welding spatter images are obtained.

[0051] Table 1. Splash noise generation algorithm

[0052] In the splash noise generation algorithm, splash noises of different widths, lengths and brightness can be simulated by adjusting different noise parameters.

[0053] The algorithm effect is as follows Figure 7 As shown, the welding noise content in the original image is well enhanced.

[0054] Step 4: General data enhancement method based on image transformation.

[0055] Images are usually represented in the form of pixels in computers. If fixed corner points of the image are selected to establish a pixel coordinate system, the image can be transformed into a matrix in a mathematical way. Figure 2 As shown, by performing different matrix transformation operations on the image, operations such as translation, rotation, scaling, flipping, shearing, perspective and HSV channel adjustment can be achieved, so as to imitate the different shapes and sizes of welds and the visual effects of shooting welds from different angles.

[0056] Among them, each function is briefly described as follows: Translation: Move the image along the X-axis and Y-axis, and the moving distance is a random number; Rotation: Rotate the image around the center point of the image by a certain angle, and the rotation angle is a random number; Scaling: adjust the length and width of the image to a certain multiple of the original size of the image, and keep the center position unchanged. The scaling multiples in different directions are different random numbers; Flip: Flip the image along the X-axis and Y-axis respectively, where the probability of flipping along each axis is a random number; Shearing: Shear the image along the X-axis and Y-axis respectively, where the degree of shearing is a random number; Perspective: deform the image from different perspectives to simulate the perspective effect. The degree and direction of perspective are determined by random numbers. HSV (hue, saturation and value) channel adjustment: By adjusting the values ​​of different HSV channels of the image, the saturation, brightness and color of the image can be adjusted to simulate the visual effects of different lighting conditions and different camera filters. The range of the value of each channel is a random number.

[0057] The above operations are used to process each batch of input images during the neural network training process.

[0058] The operations are performed in the following order: perspective, scaling, rotation, shearing, translation, flipping, and HSV channel transformation. The edges of the transformed image are filled with black, and the final output image size is the same as the original image size.

[0059] This embodiment provides a data enhancement method for a weld tracking model, which approximately characterizes the spatter noise in the welding noise as discrete line segments radiating outward from a point area under the weld image plane, and represents the arc noise as a brightness change image diffusing outward from a point at the bottom of the weld image, thereby realizing the simulation of arc noise and welding spatter, superimposing them on the weld image, and performing data enhancement, which can improve the training effect of the weld tracking neural network model.

[0060] A data enhancement method for a weld tracking model provided in this embodiment addresses the problem that a single generated Perlin noise image cannot obtain a natural smoke effect. By setting different grid sizes, noise images are generated multiple times and superimposed, the blurring effect of smoke on the image can be simulated.

[0061] The present embodiment provides a data enhancement method for a weld tracking model, which uses a fractal function in the process of Perlin noise image superposition to obtain a turbulent image of welding smoke and achieve a variety of natural texture effects.

[0062] Embodiment 2 This embodiment provides a weld tracking method, which specifically includes: Acquire weld images; For weld images, the weld position is obtained through the weld tracking model; During the training process of the weld tracking model, data enhancement is performed on the weld image set (ie, the training set) by using a data enhancement method for the weld tracking model described in the first embodiment.

[0063] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data enhancement method for a weld tracking model, characterized in that: include: Obtaining a weld image set; For each weld image in the weld image set, an arc noise image is generated by an arc noise simulation algorithm, and a welding spatter image is generated by a welding spatter simulation algorithm, and both the arc noise image and the welding spatter image are superimposed on the weld image; The arc noise simulation algorithm randomly specifies a point at the bottom of the weld image as the arc center, specifies the arc center brightness, and generates an arc noise image based on the arc center brightness and the distance of any point in the weld image relative to the arc center. The welding spatter simulation algorithm randomly selects a center point and an initial point in the weld image, determines an end point at a certain distance from the initial point on the extension line of the center point and the initial point, randomly generates brightness and light width, draws a spatter light segment, and generates multiple spatter rays by adjusting the center point, initial point, end point, brightness and light width to form a welding spatter image.

2. A data enhancement method for a weld tracking model according to claim 1, characterized in that: The step of generating the arc noise image comprises: Based on the arc center brightness and the distance of any point in the weld image relative to the arc center, the pixel value of any point in the arc noise image is calculated; Threshold segmentation is performed on the arc light noise image. After adjusting the pixel values ​​greater than the threshold to the threshold value, Gaussian blur is applied to the arc light noise image.

3. A data enhancement method for a weld tracking model as claimed in claim 2, characterized in that: The pixel value of any point in the arc noise image is: , where p represents the position of any point in the weld image, p0 represents the position of the arc center, dis(p,p0) represents the distance from any point in the weld image to the arc center, value0 represents the brightness of the arc center, and max(dis(p,p0)) represents the distance from the farthest point in the weld image to the arc center.

4. A data enhancement method for a weld tracking model as claimed in claim 1, characterized in that: Also includes: The welding spatter image is Gaussian blurred and then superimposed on the weld image.

5. A data enhancement method for a weld tracking model as claimed in claim 1, characterized in that: Also includes: For each weld image in the weld image set, a smoke noise image is generated by using a smoke noise simulation algorithm, and the smoke noise image is superimposed on the weld image.

6. A data enhancement method for a weld tracking model as claimed in claim 5, characterized in that: The step of superimposing the smoke noise image on the weld image includes: randomly cutting out a section of the smoke noise image, scaling it to the size of the weld image, and superimposing it on the original weld image.

7. A data enhancement method for a weld tracking model as claimed in claim 5, characterized in that: The smoke noise simulation algorithm includes: setting grids of different sizes, generating a Perlin noise image based on each size of the grid through a Perlin noise algorithm, and superimposing the Perlin noise images corresponding to grids of all sizes and normalizing them to obtain a smoke noise image.

8. A data enhancement method for a weld tracking model as claimed in claim 7, characterized in that: The superposition of the Perlin noise image adopts a fractal function: ;in, Indicates the number of superimposed images. Indicates taking the absolute value of a variable. Indicates Pixel values ​​of the Zhang Bailin noise image.

9. A data enhancement method for a weld tracking model as claimed in claim 1, characterized in that: Also includes: The weld image is subjected to perspective, scaling, rotation, shearing, translation, flipping and HSV channel transformation in turn.

10. A weld tracking method, characterized in that: include: Acquire weld images; For weld images, the weld position is obtained through the weld tracking model; During the training process of the weld tracking model, data enhancement is performed on the weld image set by using a data enhancement method for a weld tracking model according to any one of claims 1 to 9.

Images