A lightweight image segmentation neural network construction method, a real-time robust weld tracking detection method and system based on the lightweight image segmentation neural network

Abstract

The application discloses a kind of real-time robust weld joint detection methods based on lightweight image segmentation neural network, including steps: S01, lightweight image segmentation neural network is transplanted to industrial control computer;S02, before welding, industrial control computer carries out initialization processing to weld image;S03, after welding, lightweight image segmentation neural network extracts complete laser stripe to input weld image, and obtains laser stripe binary image;S04, using digital image processing method, weld feature point is extracted to laser stripe binary image;S05, the three-dimensional coordinate value of converted weld feature point is subtracted from the current position of robot, and the deviation is controlled according to control cabinet to move welding gun, and automatic tracking of weld is realized.The application also discloses a kind of lightweight image segmentation neural network construction method.The application realizes under the condition of limited computing resources, solves the problem of weld tracking with strong noise interference.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of robot weld seam automatic tracking, and particularly relates to a lightweight image segmentation neural network construction method, a real-time robust weld seam tracking detection method and system based on the lightweight image segmentation neural network. BACKGROUND

[0002] With the development of industrial automation, robot automatic welding has been widely used in industrial production, solving the problems of high work intensity and poor working environment of manual welding. However, the current robot automatic welding is mainly in the teaching-reproduction mode, and the robot welds along the teaching trajectory. Due to the existence of welding thermal deformation and clamping error, the actual welding trajectory of the workpiece will deviate from the teaching trajectory, resulting in a decrease in welding precision.

[0003] With the development of machine vision technology, vision sensors based on laser structured light are introduced into robot welding. Due to the characteristics of laser vision sensor, such as no need to contact with the workpiece, high measurement accuracy and strong anti-interference ability, robot automatic welding based on laser vision sensor has been widely studied and applied. To make the robot based on laser vision realize high-precision automatic welding, the key is how to exclude the interference of arc light, spatter and other noises in the welding process and accurately detect the weld feature points. For this reason, scholars at home and abroad have proposed different methods to reduce the influence of noise on the welding process. Among them, the method based on traditional digital image processing mainly uses filtering, morphological processing and other methods to remove noise, which is easy to fail in the face of strong noise. The method based on deep learning uses neural network to denoise the welding image, which has high robustness and accuracy, but needs high-performance hardware for calculation support.

[0004] Semantic segmentation is one of the basic tasks in computer vision, which means dividing an image into different regions with semantic meaning according to the content, and obtaining a segmentation image with pixel-by-pixel semantic annotation. Ronneberge et al. proposed a U-Net network with excellent performance, which adopts a model structure called encoder-decoder. The encoder is responsible for feature extraction, and the decoder is responsible for restoring the feature map size. In addition, transverse connections are introduced between the encoder and the decoder to increase the segmentation accuracy by using low-level features.

[0005] Neural network compression compresses the neural network model, reduces its parameter quantity, computation quantity, memory usage during inference, and speeds up its inference speed under the condition of ensuring certain application accuracy. Common methods include direct design of lightweight model, model pruning, quantization and knowledge distillation, etc. SUMMARY

[0006] The first object of the present application is to overcome the shortcomings and deficiencies of the prior art, and provide a lightweight image segmentation neural network construction method, which can perform fast inference on a platform with limited computing resources and meet the real-time demand of weld tracking.

[0007] The second object of the present application is to provide a real-time robust weld target detection method based on a lightweight image segmentation neural network, which accurately segments complete laser stripes from strong noise weld images by using temporal context information, and solves the problem of interference of strong noise generated in the welding process on weld tracking.

[0008] The third object of the present application is to provide a real-time robust weld target detection system based on a lightweight image segmentation neural network.

[0009] The first object of the present application is achieved by the following technical scheme: a lightweight image segmentation neural network construction method, comprising the following steps:

[0010] S1, constructing an image segmentation neural network using temporal context information;

[0011] S2, collecting weld images with and without noise, and constructing a data set for training the image segmentation neural network;

[0012] S3, training the image segmentation neural network through the data set, pruning the redundant channels of the image segmentation neural network through a pruning algorithm, and obtaining a lightweight image segmentation neural network.

[0013] Preferably, in step S1, the image segmentation neural network using temporal context information comprises an image encoder, a mask encoder and a mask decoder;

[0014] The image encoder is used to extract image deep features from the input weld image, and the image encoder is built based on a ShufflNetv2 lightweight image classification network, comprising a first convolutional layer, a second stage of four Shuffle blocks, a third stage of eight Shuffle blocks and a fourth stage of four Shuffle blocks of the ShufflNetv2 lightweight image classification network;

[0015] The mask encoder is used to extract mask deep features from the predicted mask of the previous frame, and the mask encoder is composed of four convolutional layers, each of which contains a 3x3 depth separable convolutional layer with a step of 2, a batch normalization layer and a ReLU activation function;

[0016] The image deep feature and the mask deep feature are fused to obtain a fused feature map. The mask decoder is used to up-sample the fused feature map, and then the up-sampled feature map is channel spliced with a feature map of the same resolution output by the image encoder. Finally, convolution operation is performed on the spliced feature map to decode a predicted mask image of the same resolution as the input weld image. The mask decoder is composed of four up-sampling layers and one 1x1 convolution layer.

[0017] Preferably, the step S2 specifically comprises:

[0018] S21, let the welding robot collect weld images along the taught weld trajectory on the same workpiece, obtain a noise-free weld image sequence when there is no welding, and obtain a noisy weld image sequence when there is welding, and the weld images of the two weld image sequences correspond one by one;

[0019] S22, processing the weld images of the two weld image sequences, the data set is composed of input and label, and the specific method is as follows:

[0020] A vector is added to each corresponding weld feature point of the two weld image sequences, the direction of the vector is random, the value is less than 50 pixels, and each weld image with a resolution of 300x300 is cropped with the end of the corresponding vector as the center, respectively, to obtain a group of cropped noisy weld images and a group of cropped noise-free weld images, and the cropped noisy weld images are used as the input of the data set.

[0021] The cropped noise-free weld images are processed by a median filter algorithm and a threshold segmentation algorithm to obtain a laser stripe binary image, and the laser stripe binary image is used as the label of the data set.

[0022] Preferably, the step S3 specifically comprises:

[0023] S31, first, the image segmentation neural network is trained by the data set:

[0024] The batch normalization layer of the mask decoder of the image segmentation neural network performs normalization operation on the channels of the convolution feature map, and is expressed as follows:

[0025]

[0026] Wherein, x and y respectively represent the input feature map and the output feature map of a single channel, γ is the scaling coefficient of the batch normalization layer, β is the bias value, ε is the smoothing value, E[χ] is the mean function, and Var[χ] is the variance function.

[0027] By adding a penalty term for the scaling factor γ to the original loss function, the total loss function for training the network is expressed as follows:

[0028] l′=l(w,b)+λ∑|γ|

[0029] Where w and b are the learnable parameters of the image segmentation neural network, l(·) is the original loss function, and λ is the weight of the L1 regularization term;

[0030] S32. Next, the redundant feature channels of the mask decoder of the image segmentation neural network are pruned:

[0031] The scaling coefficients γ obtained after sparsification training in step S32 are sorted from largest to smallest. Different redundant feature channels are obtained by setting different clipping rates. The channel containing the scaling coefficients that are ranked lower according to the clipping rate is the redundant feature channel. After sparsification training, the redundant feature channels under the set clipping rate are clipped.

[0032] S33. Finally, the image segmentation neural network is fine-tuned. The fine-tuning involves training after removing the penalty term of the normalization layer of the mask encoder. The iteration terminates when the increase in the inference speed of the image segmentation neural network reaches the minimum threshold, or the decrease in the cross-union ratio accuracy predicted by the image segmentation neural network through the test set reaches the maximum threshold.

[0033] The second objective of this invention is achieved through the following technical solution: a real-time robust weld seam target detection method based on a lightweight image segmentation neural network, comprising the following steps:

[0034] S01. The lightweight image segmentation neural network constructed based on a lightweight image segmentation neural network construction method is ported to an industrial control computer.

[0035] S02. Before welding begins, the industrial camera of the laser vision sensor acquires an initial weld seam image and transmits it to the industrial control computer. The industrial control computer performs initialization processing on the initial weld seam image, including threshold segmentation, skeleton extraction, and line fitting, to obtain the initial pixel coordinate values ​​of the weld seam feature points.

[0036] S03. After welding begins, the industrial camera of the laser vision sensor continuously acquires weld seam images and sends them frame by frame to the industrial control computer. The complete laser stripes are extracted from the weld seam images through a lightweight image segmentation neural network to obtain a binary image of the laser stripes.

[0037] S04. Extract weld feature points from the binary image of the laser stripes in step S03 using the Zhang-Suen algorithm and the Ramer-Douglas-Peucker algorithm to obtain the pixel coordinate values ​​of the weld feature points.

[0038] S05, converting the pixel coordinate values of the weld feature points into three-dimensional coordinate values in the robot base coordinate system through a robot calibration algorithm, and differencing the three-dimensional coordinate values from the current position of the robot, and sending the deviation values to the control cabinet, the control cabinet controlling the welding torch to move along the weld trajectory to realize automatic tracking of the weld.

[0039] Preferably, the step S03 specifically comprises:

[0040] S031, the lightweight image segmentation neural network crops an image of a complete laser stripe with a resolution of 300x300 centered on the weld feature points of the previous frame of weld images, and inputs the image into an image encoder, the image encoder extracts deep features of the image, and outputs feature maps with a width and a height of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image;

[0041] S032, the lightweight image segmentation neural network predicts a laser stripe mask based on the previous frame of weld images, inputs the laser stripe mask into a mask encoder, the mask encoder extracts deep features of the mask, and outputs feature maps with a width and a height of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image;

[0042] S033, fusing the deep features of the image of step S031 and the deep features of the mask of step S032, and inputting the fused feature maps into a mask decoder, the mask decoder decodes and outputs a noise-free laser stripe binary image.

[0043] Preferably, in step S033, the inputting of the fused feature maps into the mask decoder and the decoding and outputting of the noise-free laser stripe binary image by the mask decoder specifically comprises:

[0044] S0331, each upsampling layer of the mask decoder upsamples the fused feature maps by a bilinear interpolation algorithm, so that the width and height of the fused feature maps are expanded to twice the original;

[0045] S0332, channel splicing the upsampled feature maps with the feature maps of the same resolution output by the image encoder;

[0046] S0333, performing 3x3 convolution operation on the spliced feature maps with a step size of 1 to obtain a feature map with the same resolution as the input weld image;

[0047] S0334, the 1x1 convolution layer of the mask decoder performs 1x1 convolution operation on the feature map of step S0333 with a step size of 1 to predict a noise-free laser stripe binary image.

[0048] Preferably, the step S04 specifically comprises:

[0049] S041, extracting a single-pixel width center skeleton of the laser stripe from the binary image of the laser stripe by a Zhang-Suen algorithm;

[0050] S042, performing linear fitting on the center skeleton by a Ramer-Douglas-Peucker algorithm, and selecting an intersection point with the smallest distance to the weld seam feature point of the previous frame weld seam image as the weld seam feature point of the current frame weld seam image from the intersection points of the fitted straight lines.

[0051] A third object of the present application is achieved by the following technical solution: a real-time robust weld seam target detection system based on a lightweight image segmentation neural network, comprising a matching welding device, a welding robot, a welding torch, a laser vision sensor, a welding workbench and a control cabinet, and further comprising: an industrial control computer embedded with a lightweight image segmentation neural network constructed based on a lightweight image segmentation neural network construction method; the laser vision sensor is installed on the welding torch, the welding torch is installed at the end of the welding robot, the matching welding device provides energy and materials for the welding torch, the industrial control computer is connected with the control cabinet and the laser vision sensor through Ethernet lines, the laser vision sensor sends the image to the industrial control computer after acquiring the weld seam image, the industrial control computer extracts the weld seam feature point and the predicted position according to the image, and transmits the predicted position to the control cabinet, the control cabinet outputs a signal to control the motion trail of the welding torch, and realizes automatic weld seam tracking of the welding workpiece on the welding workbench.

[0052] The present application has the following advantages and effects relative to the prior art:

[0053] (1) The present application is based on the classic encoding-decoding model idea in semantic segmentation, constructs an image segmentation neural network using temporal context information, uses a pruning algorithm to perform sparse training and redundant feature channel pruning on the image segmentation neural network, and constructs a lightweight image segmentation neural network, which has small calculation amount, high segmentation accuracy, and can perform fast inference on a platform with limited computing resources, meeting the real-time requirements of weld seam tracking.

[0054] (2) The present application constructs the structure of a specific mask encoder and mask decoder, combines the specific structure of the constructed image encoder, establishes an image segmentation neural network using temporal context information, and trains a lightweight image segmentation neural network based on the image segmentation neural network, which can utilize the continuity features of the weld seam image during welding.

[0055] (3) The lightweight image segmentation neural network of the present invention uses temporal context information to accurately segment complete laser stripes from noisy weld seam images, which solves the interference of strong noise generated during welding on weld seam tracking. It also utilizes the continuity feature of images acquired during welding to improve the segmentation accuracy.

[0056] (3) In the real-time robust weld target detection method of the present invention, the laser stripe binary image output by the lightweight image segmentation neural network does not contain noise. The weld feature points can be detected and located from it using only a simple digital image processing method. The present invention simplifies the weld feature detection process.

[0057] (4) The present invention provides a real-time robust weld target detection system based on a lightweight image segmentation neural network, which improves the robustness of weld tracking and meets the real-time requirements of welding, and has a relatively low cost. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall structure of a real-time robust weld target detection system based on a lightweight image segmentation neural network according to the present invention.

[0059] Figure 2 This is a flowchart of the real-time robust weld target detection method based on a lightweight image segmentation neural network according to the present invention.

[0060] In the diagram: 1-Welding equipment; 2-Welding robot; 3-Welding torch; 4-Laser vision sensor; 5-Workpiece; 6-Welding workbench; 7-Industrial control computer; 8-Control cabinet. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0062] Example 1

[0063] The present invention provides a lightweight image segmentation neural network construction method, comprising the following steps:

[0064] S1. Construct an image segmentation neural network that utilizes temporal context information;

[0065] S2. Acquire noisy and noiseless weld seam images and construct a dataset for training the image segmentation neural network;

[0066] S3. The image segmentation neural network is trained with sparsity using the dataset, and redundant channels are pruned using a pruning algorithm to obtain a lightweight image segmentation neural network.

[0067] Specifically, the application is based on a classic encoding-decoding model in semantic segmentation, and a lightweight image segmentation neural network is constructed by using a pruning algorithm to train and prune the model. The neural network has small calculation amount and high segmentation accuracy, and can perform fast inference on a platform with limited computing resources, meeting the real-time demand of weld tracking.

[0068] In step S1, the image segmentation neural network using the temporal context information comprises an image encoder, a mask encoder and a mask decoder;

[0069] The image encoder is used to extract image deep features from the input weld image, and the image encoder is built based on a ShufflNetv2 lightweight image classification network and comprises a first convolutional layer, a second stage of four Shuffle blocks, a third stage of eight Shuffle blocks and a fourth stage of four Shuffle blocks of the ShufflNetv2 lightweight image classification network;

[0070] The mask encoder is used to extract mask deep features from the predicted mask of the previous frame, and the mask encoder is composed of four convolutional layers, each of which comprises a 3x3 depth separable convolutional layer with a step of 2, a batch normalization layer and a ReLU activation function;

[0071] The image deep features and the mask deep features are fused to obtain a fused feature map, and the mask decoder is used to up-sample the fused feature map, then concatenate the up-sampled feature map with a feature map of the same resolution output by the image encoder, and finally perform convolution operation on the concatenated feature map to decode a predicted mask image of the same resolution as the input weld image, wherein the mask decoder is composed of four up-sampling layers and one 1x1 convolutional layer.

[0072] Specifically, the application constructs specific mask encoder and mask decoder structures, combines the specific structure of the constructed image encoder, establishes an image segmentation neural network for realizing the use of time sequence context information, and trains a lightweight image segmentation neural network based on the image segmentation neural network. The mask encoder learns the semantic feature representation of the previous frame of segmentation mask to guide the subsequent mask decoder to predict the laser stripe segmentation of the current frame of image; the mask encoder performs up-sampling on the deep features fused with the outputs of the image encoder and the mask encoder, and gradually fuses the feature maps with the same resolution from the image encoder in the up-sampling process, and finally decodes the predicted mask image with the same resolution as the input image. The lightweight image segmentation neural network of the application accurately segments the complete laser stripe from the strong noise weld seam image by using the time sequence context information, solves the interference of strong noise generated in the welding process on the weld seam tracking, and utilizes the continuity feature of the collected image during welding to improve the segmentation accuracy.

[0073] The step S2 specifically comprises:

[0074] S21, let the welding robot collect weld seam images along the taught weld seam trajectory on the same workpiece, obtain a noise-free weld seam image sequence in the absence of welding, and obtain a noisy weld seam image sequence in the presence of welding, and the weld seam images of the two weld seam image sequences correspond one by one;

[0075] S22, processing the weld seam images of the two weld seam image sequences, the data set is composed of input and label, and the specific method is as follows:

[0076] A vector is added to the weld seam feature point of each pair of corresponding weld seam images in the two weld seam image sequences, the direction of the vector is random, the value is less than 50 pixels, and each weld seam image with a resolution of 300x300 is cropped with the end of the corresponding vector as the center, a group of cropped noisy weld seam images and a group of cropped noise-free weld seam images are obtained, and the cropped noisy weld seam images are used as the input of the data set.

[0077] The cropped noise-free weld seam images are subjected to feature extraction by a median filter algorithm and a threshold segmentation algorithm, and a laser stripe binary image is obtained, which is used as the label of the data set.

[0078] Specifically, the step S22 adds a vector for each pair of corresponding weld feature points in the two weld image sequences, so as to increase the sample size of the data set; the cropped image only contains the region of interest of the weld feature information, the noisy weld image after cropping is used as the input of the data set, and the binary image of the noiseless laser stripe after cropping and processing is used as the label of the data set, and the two together constitute the data set for training the image segmentation neural network.

[0079] The step S3 specifically comprises:

[0080] S31, first, the image segmentation neural network is trained by the data set:

[0081] The batch normalization layer of the mask decoder of the image segmentation neural network performs a normalization operation on the channels of the feature map after convolution, and is expressed as follows:

[0082]

[0083] Wherein, x and y respectively represent the input feature map and the output feature map of a single channel, γ is the scaling coefficient of the batch normalization layer, β is the bias value, ε is the smoothing value, ε is used to prevent the denominator from being zero in the iteration process, E[χ] is the mean function, and Var[χ] is the variance function; a penalty term for the scaling coefficient γ is added to the original loss function, and the total loss function of the training network is expressed as follows:

[0084] l' = l(w, b) + λΣ|γ|

[0085] Wherein, w and b are the learnable parameters of the image segmentation neural network, l(·) is the original loss function, and λ is the weight of the L1 regularization term; the total loss function is iteratively solved, the scaling coefficient γ will approach 0, a set of scaling coefficients γ of different sizes are obtained, by setting the clipping rate, after the sparse training, the corresponding unimportant feature channels are pruned (the importance of the feature channels is determined by comparing the size of γ), so as to compress the neural network, reduce the calculation amount of the neural network, and achieve the purpose of accelerating the inference speed;

[0086] S32, the redundant feature channels of the mask decoder of the image segmentation neural network are pruned:

[0087] The scaling coefficients γ obtained after the sparse training of step S32 are sorted from large to small, different redundant feature channels are obtained by setting different clipping rates, and the channels where the scaling coefficients corresponding to the clipping rate are located at the back of the sorting are the redundant feature channels; after the sparse training, the redundant feature channels under the clipping rate are pruned; the present application selects a clipping rate of 20%, and after the sparse training, the feature channels with a γ value smaller and at the back 20% of the sorting in the mask decoder are pruned.

[0088] S33, finally, the image segmentation neural network is fine-tuned, the fine-tuning is training after removing the normalization layer of the mask encoder; when the increase of the inference speed of the image segmentation neural network reaches a preset minimum threshold, or the decrease of the intersection over union precision predicted by the image segmentation neural network on the test set reaches a preset maximum threshold, the iteration is terminated.

[0089] Specifically, the preset minimum threshold and the preset maximum threshold change with the change of the set clipping rate, and the termination condition of the iteration is set as that the increase of the inference speed of the image segmentation neural network is less than 10%, or the decrease of the intersection over union precision of the image segmentation neural network on the test set is greater than 5%, 10% is the preset minimum threshold of the increase of the inference speed, and 5% is the preset maximum threshold of the decrease of the intersection over union precision.

[0090] The present application adopts the iteration strategy of "sparse training-channel clipping-fine tuning", firstly performs sparse training on the image segmentation neural network in each iteration, then clips the unimportant feature channels in the mask decoder, and finally fine-tunes the network to restore the precision. When the image segmentation neural network obtained in a certain iteration is compared with the last iteration, the precision of the image segmentation neural network decreases too much or the inference speed increases too little, the iteration is stopped, and the image segmentation neural network is taken as the final lightweight image segmentation model.

[0091] When the image segmentation neural network is trained, the training hyperparameters and the specific training process are as follows:

[0092] a, the image encoder is loaded with the pre-trained weight parameters (including the convolution kernel parameters and the bias of the convolution layer) on ImageNet; the parameters of the mask encoder and the mask decoder (including the scaling coefficient and the bias of the normalization layer) are randomly initialized;

[0093] b, the weld image in the data set is used as the training input of the image encoder;

[0094] c, the corresponding mask label is randomly rotated at an angle within ±3° and randomly horizontally or vertically translated within 5 pixels to simulate the mask image predicted by the previous frame in the weld image sequence, which is used as the training input of the mask encoder;

[0095] d, the number of training times is set to 50, the batch size is set to 8, and the weight λ of the L1 regularization term of the total loss function is set to 0.002;

[0096] e, the stochastic gradient descent method is adopted, the initial learning rate is set to 0.01, the momentum is set to 0.9, and the weight decay coefficient is set to 1×10 -4 ;

[0097] f、The preheating strategy is used in the initial training stage to ensure the stability of the training, and the learning rate is increased by 0.01 per training iteration until the initial learning rate is reached at the 500th iteration; meanwhile, the learning rate decay strategy is used after preheating, and the learning rate is linearly reduced until it is reduced to 0 at the last iteration.

[0098] Embodiment 2

[0099] As Figure 2 A flowchart of a real-time robust weld seam target detection method based on a lightweight image segmentation neural network is shown, which includes the following steps:

[0100] S01, transplant the lightweight image segmentation neural network constructed by the lightweight image segmentation neural network construction method described in embodiment 1 to an industrial control computer;

[0101] S02, before welding starts, the industrial camera of the laser vision sensor collects an initial weld seam image and transmits it to the industrial control computer; the industrial control computer performs threshold segmentation, skeleton extraction and straight line fitting initialization processing on the initial weld seam image to obtain the pixel coordinate values of the initial weld seam feature points;

[0102] S03, after welding starts, the industrial camera of the laser vision sensor continuously collects weld seam images and sends them to the industrial control computer frame by frame, and extracts complete laser stripes from the weld seam images through the lightweight image segmentation neural network to obtain laser stripe binary images;

[0103] S04, extract the weld seam feature points from the laser stripe binary images of step S03 through the Zhang-Suen algorithm and the Ramer-Douglas-Peucker algorithm to obtain the pixel coordinate values of the weld seam feature points;

[0104] S05, through the robot calibration algorithm, the pixel coordinate values of the weld seam feature points are converted into three-dimensional coordinate values in the robot base coordinate system, and the three-dimensional coordinate values are subtracted from the current position of the robot, and the deviation values are sent to the control cabinet, the control cabinet controls the welding gun to move along the weld seam trajectory, and realizes automatic tracking of the weld seam.

[0105] Specifically, the lightweight image segmentation neural network of the present application uses the time sequence context information to accurately segment the complete laser stripes from the strong noise weld seam image, solves the interference of the strong noise generated during the welding process on the weld seam tracking, and uses the continuity feature of the collected image during welding to improve the segmentation accuracy.

[0106] The step S03 specifically includes:

[0107] S031、lightweight image segmentation neural network, the weld feature points of the last frame of weld image are taken as the center to crop the image of complete laser stripe with a resolution of 300*300, and the image is input into the image encoder; the image encoder extracts deep features of the image, and outputs feature maps with a width and a height of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image;

[0108] S032、lightweight image segmentation neural network, a laser stripe mask is obtained based on the prediction of the last frame of weld image; the laser stripe mask is input into the mask encoder; the mask encoder extracts deep features of the mask, and outputs feature maps with a width and a height of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image;

[0109] S033、the deep features of the image in step S031 and the deep features of the mask in step S032 are fused, and the fused feature maps are input into the mask decoder; the mask decoder decodes and outputs a noise-free laser stripe binary image.

[0110] Specifically, the lightweight image segmentation neural network combines the predicted mask of the last frame of image and the current frame of image, and segments the laser stripe binary image, thereby excluding the interference of welding noise.

[0111] In step S033, the fused feature maps are input into the mask decoder, and the mask decoder decodes and outputs a noise-free laser stripe binary image, which specifically includes:

[0112] S0331、each up-sampling layer of the mask decoder up-samples the fused feature maps by a bilinear interpolation algorithm, so that the width and the height of the fused feature maps are expanded to twice the original;

[0113] S0332、the up-sampled feature maps are channel-spliced with the feature maps of the same resolution output by the image encoder;

[0114] S0333、a 3*3 convolution operation with a step of 1 is performed on the spliced feature maps, so as to obtain a feature map with the same resolution as the input weld image;

[0115] S0334、a 1*1 convolution layer of the mask decoder performs a 1*1 convolution operation with a step of 1 on the feature maps in step S0333, so as to predict a noise-free laser stripe binary image.

[0116] The step S04 specifically includes:

[0117] S041、the single-pixel width center skeleton of the laser stripe is extracted from the binary image of the laser stripe by a Zhang-Suen algorithm;

[0118] S042. Then, the Ramer-Douglas-Peucker algorithm is used to fit a straight line to the central skeleton. Among the intersections of the fitted lines, the intersection point with the smallest distance to the weld feature point of the previous frame weld image is selected as the weld feature point of the current frame weld image.

[0119] Specifically, in the real-time robust weld target detection method of the present invention, the binary image of laser stripes output by the lightweight segmentation neural network does not contain noise, and only simple digital image processing methods are needed to detect and locate weld feature points from it. The present invention simplifies the weld feature detection process.

[0120] Example 3

[0121] like Figure 1 As shown, a real-time robust weld seam target detection system based on a lightweight image segmentation neural network includes a welding device 1, a welding robot 2, a welding torch 3, a laser vision sensor 4, a welding workbench 6, and a control cabinet 8. It also includes an industrial control computer 7, which embeds a lightweight image segmentation neural network constructed based on the lightweight image segmentation neural network construction method described in Embodiment 1. The laser vision sensor 4 is mounted on the welding torch 3 and includes an industrial camera, a laser generator, and a filter. The welding torch 3 is mounted at the end of the welding robot 2. The welding device 1 provides energy and materials to the welding torch 3. The welding equipment includes a welding machine and a protective gas cylinder. The welding machine is a Yaskawa MOTOWELD-RD350, used for wire feeding and retraction of the welding robot. The protective gas cylinder contains CO2 (20%) and N2 (80%). The industrial control computer 7 is connected to the control cabinet 8 and the laser vision sensor 4 via Ethernet cables. After acquiring the weld seam image, the laser vision sensor 4 sends the image to the industrial control computer 7. The industrial control computer 7 extracts the weld seam feature points and predicted positions based on the image and transmits the predicted positions to the control cabinet 8. The control cabinet outputs signals to control the movement trajectory of the welding torch 3, realizing automatic weld seam tracking of the workpiece on the welding workbench 6.

[0122] Specifically, the present invention provides a real-time robust weld seam target detection system based on a lightweight image segmentation neural network, which improves the robustness of weld seam tracking, meets the real-time requirements of welding, and has a relatively low cost.

[0123] The above embodiments are preferred embodiments of the present invention and are not intended to limit the present invention. Any changes or other equivalent substitutions made without departing from the technical solution of the present invention are included within the protection scope of the present invention.

Claims

1. A lightweight image segmentation neural network construction method, characterized in that, Includes the following steps: S1. Construct an image segmentation neural network that utilizes temporal context information; S2. Acquire noisy and noiseless weld seam images and construct a dataset for training the image segmentation neural network; S3. The image segmentation neural network is trained in a sparse manner using the dataset, and redundant channels of the image segmentation neural network are pruned using a pruning algorithm to obtain a lightweight image segmentation neural network. In step S1, the image segmentation neural network utilizing temporal context information includes an image encoder, a mask encoder, and a mask decoder. The image encoder is used to extract deep features from the input weld image. The image encoder is built on the ShuffleNetv2 lightweight image classification network, including the first convolutional layer of the ShuffleNetv2 lightweight image classification network, the second stage with 4 Shuffle blocks, the third stage with 8 Shuffle blocks, and the fourth stage with 4 Shuffle blocks. The mask encoder is used to extract deep features of the prediction mask of the previous frame. The mask encoder consists of four stacked convolutional layers. Each convolutional layer contains a 3×3 depth separable convolutional layer with a stride of 2, a batch normalization layer, and a ReLU activation function. The image deep features and mask deep features are fused to obtain a fused feature map. The mask decoder is used to upsample the fused feature map, and then the upsampled feature map is concatenated with the feature map of the same resolution output by the image encoder. Finally, the concatenated feature map is convolved to decode the predicted mask image of the same resolution as the input weld seam image. The mask decoder consists of 4 upsampling layers and 1 1×1 convolutional layer stacked together.

2. The lightweight image segmentation neural network construction method according to claim 1, characterized in that, Step S2 specifically includes: S21. Allow the welding robot to collect weld images of the same workpiece along the taught weld trajectory. Obtain a noise-free weld image sequence when there is no welding and a noisy weld image sequence when welding. The weld images of the two weld image sequences correspond one-to-one. S22. Weld image processing of the two weld image sequences, wherein the dataset consists of input and labels, and the specific method is as follows: A vector is added to the weld feature points of each pair of corresponding weld images in two weld image sequences. The direction of the vector is random and the value is less than 50 pixels. Weld images with a resolution of 300×300 are cropped with the end of the corresponding vector as the center. A set of noisy weld images after cropping and a set of noiseless weld images after cropping are obtained respectively. The noisy weld images after cropping are used as the input of the dataset. By using median filtering and threshold segmentation algorithms, feature extraction is performed on the cropped, noise-free weld seam image to obtain a binary image of laser stripes. This binary image of laser stripes is then used as the label for the dataset.

3. The lightweight image segmentation neural network construction method according to claim 1, characterized in that, Step S3 specifically includes: S31. First, perform sparsity training on the image segmentation neural network using the aforementioned dataset: The batch normalization layer of the mask decoder in the image segmentation neural network normalizes the channels of the convolutional feature map, as shown below: Where x and y represent the input and output feature maps of a single channel, respectively, γ is the scaling factor of the batch normalization layer, β is the bias value, ε is the smoothing value, E[x] is the mean function, and Var[x] is the variance function; By adding a penalty term for the scaling factor γ to the original loss function, the total loss function for training the network is expressed as follows: l′=l(w,b)+λΣ|γ| Where w and b are the learnable parameters of the image segmentation neural network, l(·) is the original loss function, and λ is the weight of the L1 regularization term; S32. Next, the redundant feature channels of the mask decoder of the image segmentation neural network are pruned: The scaling coefficients γ obtained after sparsification training in step S32 are sorted from largest to smallest. Different redundant feature channels are obtained by setting different clipping rates. The channel containing the scaling coefficients that are ranked lower according to the clipping rate is the redundant feature channel. After sparsification training, the redundant feature channels under the set clipping rate are clipped. S33. Finally, the image segmentation neural network is fine-tuned. The fine-tuning involves training after removing the penalty term of the normalization layer of the mask encoder. The iteration terminates when the increase in the inference speed of the image segmentation neural network reaches the minimum threshold, or the decrease in the cross-union ratio accuracy predicted by the image segmentation neural network through the test set reaches the maximum threshold.

4. A real-time robust weld seam target detection method based on a lightweight image segmentation neural network, characterized in that, Includes the following steps: S01. The lightweight image segmentation neural network constructed based on the lightweight image segmentation neural network construction method according to any one of claims 1-3 is ported to an industrial control computer. S02. Before welding begins, the industrial camera of the laser vision sensor acquires an initial weld seam image and transmits it to the industrial control computer. The industrial control computer performs initialization processing on the initial weld seam image, including threshold segmentation, skeleton extraction, and line fitting, to obtain the initial pixel coordinate values ​​of the weld seam feature points. S03. After welding begins, the industrial camera of the laser vision sensor continuously acquires weld seam images and sends them frame by frame to the industrial control computer. The complete laser stripes are extracted from the weld seam images through a lightweight image segmentation neural network to obtain a binary image of the laser stripes. S04. Extract weld feature points from the binary image of the laser stripes in step S03 using the Zhang-Suen algorithm and the Ramer-Douglas-Peucker algorithm to obtain the pixel coordinate values ​​of the weld feature points. S05. Through the robot calibration algorithm, the pixel coordinate values ​​of the weld feature points are converted into three-dimensional coordinate values ​​in the robot's base coordinate system. The difference between this three-dimensional coordinate value and the robot's current position is calculated, and the deviation value is sent to the control cabinet. The control cabinet controls the welding torch to move along the weld trajectory to achieve automatic weld tracking.

5. The real-time robust weld seam target detection method based on a lightweight image segmentation neural network according to claim 4, characterized in that, Step S03 specifically includes: S031. Lightweight image segmentation neural network: Using the weld feature points of the previous frame weld image as the center, crop out a complete laser stripe image with a resolution of 300×300, and input this image into the image encoder. The image encoder extracts deep features of the image and outputs feature maps with width and height of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image. S032. The lightweight image segmentation neural network predicts the laser stripe mask based on the weld seam image of the previous frame. This laser stripe mask is input into the mask encoder, which extracts the deep features of the mask and outputs feature maps with width and height of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image. S033. The deep features of the image in step S031 and the deep features of the mask in step S032 are fused together. The fused feature map is input into the mask decoder. The mask decoder decodes and outputs a noise-free binary image of laser stripes.

6. The real-time robust weld seam target detection method based on a lightweight image segmentation neural network according to claim 5, characterized in that, In step S033, the step of inputting the fused feature map into the mask decoder, and the mask decoder decoding and outputting a noise-free binary image of laser stripes specifically includes: S0331. Each upsampling layer of the mask decoder upsamples the fused feature map using a bilinear interpolation algorithm, thereby doubling the width and height of the fused feature map. S0332. Perform channel-by-channel stitching between the upsampled feature map and the feature map of the same resolution output by the image encoder. S0333. Perform a 3×3 convolution operation with a stride of 1 on the stitched feature map to obtain a feature map with the same resolution as the input weld image. S0334. The 1×1 convolutional layer of the mask decoder performs a 1×1 convolution operation with a stride of 1 on the feature map of step S0333 to predict a noise-free binary image of laser stripes.

7. The real-time robust weld seam target detection method based on a lightweight image segmentation neural network according to claim 4, characterized in that, Step S04 specifically includes: S041. Extract the single-pixel width center skeleton of the laser stripe from the binary image of the laser stripe using the Zhang-Suen algorithm; S042. Then, the Ramer-Douglas-Peucker algorithm is used to fit a straight line to the central skeleton. Among the intersections of the fitted lines, the intersection point with the smallest distance to the weld feature point of the previous frame weld image is selected as the weld feature point of the current frame weld image.

8. A real-time robust weld seam target detection system based on a lightweight image segmentation neural network model, comprising a welding device (1), a welding robot (2), a welding torch (3), a laser vision sensor (4), a welding workbench (6), and a control cabinet (8), characterized in that, Also includes: An industrial control computer (7) is embedded with a lightweight image segmentation neural network constructed based on a lightweight image segmentation neural network construction method according to any one of claims 1-3; the laser vision sensor (4) is installed on the welding torch (3), the welding torch (3) is installed at the end of the welding robot (2), the supporting welding equipment (1) provides energy and materials to the welding torch (3), the industrial control computer (7) is connected to the control cabinet (8) and the laser vision sensor (4) respectively via Ethernet cable, the laser vision sensor (4) acquires the weld seam image and sends the image to the industrial control computer (7), the industrial control computer (7) extracts the weld seam feature points and predicted positions according to the image, and transmits the predicted positions to the control cabinet (8), the control cabinet outputs signals to control the movement trajectory of the welding torch (3), and realizes automatic weld seam tracking of the welding workpiece on the welding workbench (6).

Images