Welding splash segmentation and identification system and method based on deep learning

Abstract

The invention discloses a welding splash segmentation and recognition system and method based on deep learning, and the system comprises the steps: setting an image collection module and an image preprocessing module, constructing a splash segmentation module through an improved deep learning network, carrying out the pixel-level segmentation of a processed image, recognizing a welding splash region, and outputting an initial segmentation result. Morphological processing and boundary optimization are carried out on the initial segmentation result, a segmentation result is generated, the segmentation result is converted into position information under an actual coordinate system, guidance is provided for automatic grinding equipment, a detection result is displayed on an original image in an overlapping mode, and a visual detection result is provided. The detection precision of the welding spatter of the body in white is remarkably improved, and the method has higher adaptability to the welding spatter in a scattering shape and an irregular shape; detection tasks under complex background and variable illumination conditions can be processed; the real-time performance is good, and the production line rhythm requirement is met; accurate area positioning information is provided for automatic grinding equipment, and the grinding efficiency and quality are improved.

Description

Technical Field

[0001] This invention relates to the field of automobile manufacturing technology, specifically to a deep learning-based system and method for segmenting and recognizing welding spatter. Background Technology

[0002] In automobile manufacturing welding workshops, welding spatter generated during the body-in-white welding process adheres to the vehicle's surface, creating a scattered pollution effect. If this spatter is not removed promptly, it directly impacts subsequent painting quality and the overall vehicle appearance. Currently, the detection and removal of welding spatter primarily relies on manual labor. Workers visually identify areas of welding spatter and then manually grind them away. This method has the following significant drawbacks: 1. Low efficiency of manual inspection, making it difficult to meet the demands of large-scale production; 2. Inspection results depend on worker experience, leading to subjective differences and the risk of missed detections; 3. High labor intensity and harsh working environment for manual grinding; 4. Unstable grinding quality, potentially causing secondary damage to the vehicle's surface.

[0003] With the development of computer vision and artificial intelligence technologies, some automated detection methods have been applied to welding spatter detection. For example, methods based on traditional image processing, such as edge detection and morphological operations, are used, but these methods have poor adaptability to complex backgrounds and lighting changes. In recent years, deep learning-based object detection algorithms such as YOLO (You Only LookOnce) have been attempted for welding spatter detection, but they suffer from the following problems: 1. Welding spatter is irregular in shape, varies in size, and is distributed in a scattering pattern, significantly different from the regular-shaped objects targeted by traditional object detection algorithms; 2. When using rectangular bounding boxes for object detection, the actual boundaries of the welding spatter cannot be accurately described, leading to inaccurate localization; 3. The contrast between the spatter and the vehicle body surface is sometimes low, especially at certain angles and under specific lighting conditions, increasing the difficulty of detection; 4. The large number and dense distribution of welding spatter place higher demands on the recognition capabilities of the detection algorithm.

[0004] Currently, there is no deep learning detection algorithm specifically designed for the characteristics of welding spatter in automotive body-in-white. Existing methods suffer from insufficient detection accuracy and inaccurate boundary positioning in practical applications, failing to meet the precision requirements of automated grinding. Therefore, there is an urgent need for an algorithm system capable of high-precision identification and segmentation of welding spatter, providing accurate positional information for automated grinding equipment, automating the detection and removal of welding spatter, and improving production efficiency and product quality. Summary of the Invention

[0005] To address the above technical problems, this invention provides a deep learning-based welding spatter segmentation and recognition system and method, aiming to solve the problem of insufficient recognition accuracy of scattered welding spatter in the prior art, and to provide accurate regional positioning information for automated grinding systems.

[0006] The technical solution is as follows: The first aspect of this invention provides a deep learning-based welding spatter segmentation and recognition system, the key point of which is that the system sequentially includes:

[0007] Image acquisition module: set at the welding fixture of the body-in-white, used to acquire the original image a of the surface of the body-in-white;

[0008] Image preprocessing module: Enhances, denoises, and normalizes the original image a to obtain a normalized image b;

[0009] Spatter segmentation module: Utilizes an improved deep learning segmentation network to perform pixel-level segmentation on the standardized image b, identify the welding spatter region, and output the initial segmentation result c;

[0010] Post-processing optimization module: Performs morphological processing and boundary optimization on the initial segmentation result c to generate segmentation result d;

[0011] Region positioning module: converts the segmentation result d into position information e in the actual coordinate system;

[0012] Results visualization module: Converts location information e into visualization results and location data.

[0013] Preferably, the image acquisition module includes an industrial camera, a light source system, and a control unit, wherein the industrial camera uses a CMOS sensor and a lens, the light source system uses a ring LED light source, and the control unit is used for camera parameter setting, trigger signal control, and image data transmission;

[0014] The image acquisition module is installed at the welding image acquisition station of the body-in-white and adopts at least four angle shooting methods.

[0015] Preferably, the image preprocessing module performs image denoising, illumination equalization, image normalization, and image size adjustment on the original image a;

[0016] The image denoising employs a bilateral filtering algorithm, which is expressed as follows:

[0017]

[0018] Where I represents the image, (x,y) represents the pixel coordinates, Ω represents the window region, f is the spatial domain kernel function, g is the pixel value domain kernel function, W is the normalization coefficient, and I filteredThis represents the output value of the pixel after filtering.

[0019] The illumination equalization uses an improved adaptive histogram equalization (CLAHE), which introduces a dynamic block size adaptive mechanism. Based on the local variation coefficient of the image, the CLAHE block size is automatically adjusted from 8x8 to 32x32, enabling the algorithm to adapt to welding spatter of different sizes and complex backgrounds, enhance image contrast, and highlight spatter areas.

[0020] The image normalization normalizes the pixel values ​​of the image to the range of [0, 1].

[0021] The image resizing involves adjusting the image to the input size required by the segmentation network while maintaining the aspect ratio. If the resized image does not meet the requirements of the segmentation network, a blank whiteboard is used for filling.

[0022] Preferably, the standardized image b is input into the splash segmentation module, which is an improved deep learning segmentation network architecture based on the U-Net architecture. The network architecture includes an image input device, an encoder, a decoder, and an output layer in sequence.

[0023] The encoder uses ResNext-50 as the backbone network to extract multi-level image features of the normalized image b. The multi-level image features include shallow feature maps, medium feature maps and deep feature maps. The medium feature map is further refined by multi-scale feature fusion module MFF, welding spatter feature enhancement module SFEM and scattering object boundary refinement mechanism SBFM.

[0024] The multi-level image features extracted by the encoder are input into the decoder for decoding. The decoder uses multi-stage progressive upsampling and gradually restores the feature map resolution through transposed convolution and skip connections to generate an initial segmentation mask.

[0025] The initial segmentation mask is input into the output layer and converted into a single-channel probability map through a 1×1 convolution. The improved standard Sigmoid activation function is used to output the probability that each pixel belongs to the splash, and probability correction is performed to obtain the final segmentation mask, which is the initial segmentation result c.

[0026] Preferably, the splash segmentation module also includes a network training strategy, which employs a combined loss function, including binary cross-entropy loss (BCE), Dice loss, and boundary-aware loss (Loundary). The total loss expression is as follows:

[0027] L total =α·L BCE +β·LDice +γ·L boundary

[0028] The weighting coefficients are set as follows: α = 0.5, β = 0.3, γ = 0.2;

[0029] The design of this combined loss function takes into account that BCE loss is suitable for pixel-level classification tasks, Dice loss, and boundary-aware loss Loundary.

[0030] Preferably: after the middle layer feature map is input to the multi-scale feature fusion module MFF, it is processed through four parallel branches, and the number of special map channels output by each branch is uniformly 256. The weight of each branch is calculated through the channel attention mechanism, the feature map of each branch is multiplied by the corresponding attention weight, the weighted feature maps are summed to obtain the fused feature, and feature map A is output.

[0031] The weight calculation for each branch includes global average pooling, dimensionality reduction fully connected layer, ReLU activation, dimensionality increase fully connected layer, and Sigmoid activation, generating channel attention weights;

[0032] The mathematical expression for the multi-scale feature fusion module (MFF) is:

[0033]

[0034] Where F i α represents the convolutional output features with different dilation rates. i These are dual-weighted coefficients generated through the channel attention mechanism, with i taking values ​​from 1 to 4, representing the relevant parameters for each branch.

[0035] The double-weighting coefficient α i The expression is:

[0036] α i =σ(W2·ReLU(W1·GAP(F) i )));

[0037] Where GAP is the global average pooling operation, W1 and W2 are the weights of the fully connected layer, and σ is the Signoid activation function.

[0038] Preferably: The obtained feature map A is input into the welding spatter feature enhancement module SFEM, and processed through parallel main branch and enhancement branch. After processing, attention weights are calculated through an adaptive fusion mechanism, and the features of the main branch and enhancement branch are weighted and fused to obtain fused feature B0. Then, the input feature map A is added to the fused feature B0 to achieve residual connection, and finally the feature map B is output.

[0039] The main branch keeps the input feature map A unchanged, and the operation of the enhancement branch includes, in sequence, feature extraction using an improved local binary pattern (LBP) feature extraction layer, extraction of the spatial relationship of LBP features using a 3×3 convolutional layer, batch normalization, and ReLU activation.

[0040] The mathematical expression for the welding spatter feature enhancement module SFEM is:

[0041] F SFEM =F in +λ·BN(ReLU(Conv 3×3 (LBP(F in ))))

[0042] Where F in It is feature map A, LBP is Local Binary Pattern Operation, Conv 3×3 It is a convolutional layer, where λ is the weight coefficient, BN is batch normalization, and ReLU is the activation function.

[0043] Preferably: The feature map B is input into the scattering object boundary refinement mechanism SBFM. After processing by the parallel feature processing branch and the boundary-aware feature extraction branch, the original features and gradient features are obtained respectively. The original features and gradient features are connected and fused by 1×1 convolution. Batch normalization and ReLU activation are performed. The boundary-aware loss function and boundary refinement inference are used to obtain the boundary enhancement features.

[0044] The expression for the boundary-aware loss function is:

[0045]

[0046] Among them, Ω B Show the boundary region, P (x,y) G (x,y) These are the predicted and true labels, respectively, and IoU is the intersection-union ratio.

[0047] Preferably, the multi-level image features extracted by the encoder are input into the decoder, which sequentially performs decoding operations through a multi-stage progressive upsampling module, an improved transposed convolution module, a skip connection module, a residual thinning module, a progressive channel reduction module, and a spatial attention module, so that the extracted multi-level image features are restored to the initial segmentation mask of the original resolution.

[0048] The number of channels in the progressive channel reduction module decreases in sequence from 512 to 256 to 128 to 64 to 32, and the final stage retains 32 channels in the final feature map.

[0049] The output layer adopts a feature pyramid structure, which fuses features from different stages of the decoder. It uses 1×1 convolution to map the final feature map into a single-channel probability map. It adopts an improved standard Sigmoid function to make it more sensitive to boundary regions, and performs local adjustments to the response near the segmentation boundary. It calculates the probability that each pixel belongs to the splash and sets a probability calibration layer. The final segmentation mask is obtained through the probability calibration function, and the initial segmentation result c is output.

[0050] The improved standard Sigmoid function expression is as follows:

[0051] σ boundary (x)=σ(x)·(1+α'·|G Sobel (F integrated )|) β'

[0052] Where α' = 0.2 and β' = 0.5 are adjustment parameters, G Sobel (F integrated ) represents the Sobel gradient operator of the input feature map, which can highlight the boundary region, and σ is the Sigmoid activation function;

[0053] The probability calibration layer reduces probability estimation bias, and its calibration function is based on Bayesian theory and takes into account the prior class distribution.

[0054] The calibration function expression for the probability calibration layer is:

[0055]

[0056] Where r is the proportion of positive samples (splashes) in the training set, and P represents the probability that each pixel in the image belongs to a splash.

[0057] Preferably, the initial segmentation result c is input to the post-processing optimization module for threshold processing, small region filtering, morphological operations, boundary refinement, and region merging operations;

[0058] The thresholding process converts the probability map output by the network into a binary mask, with the threshold set to 0.5; its expression is:

[0059]

[0060] The small region filtering removes regions with an area smaller than a preset threshold, eliminates false detection regions, and uses connected component analysis to calculate the area of ​​each candidate region, removing regions with an area smaller than 10 pixels.

[0061] The morphological operations apply opening and closing operations to optimize the shape of the segmented region, as expressed by the formula:

[0062] Maskopt =Close(Open(Mask,K1),K2)

[0063] Where Open and Close represent opening and closing operations respectively, and K1 and K2 are morphological kernels;

[0064] The boundary refinement uses a Conditional Random Field (CRF) algorithm to refine the boundaries of the segmentation results, as expressed by the formula:

[0065]

[0066] Where x is the pixel label assignment, ψ u It is a univariate potential energy term, based on the probability graph output by the network, ψ p These are paired potential energy terms, based on pixel space and color similarity;

[0067] The region merging process analyzes adjacent splash areas. If the distance is less than a preset threshold, they are merged into one region, and the final segmentation result d is obtained.

[0068] The region positioning module converts the segmentation result d into position information e in the actual coordinate system, providing accurate positioning for the automatic grinding equipment.

[0069] The result visualization module converts the location information e into visual results and location data, which are then overlaid on the original image a to provide intuitive detection results.

[0070] The second aspect of this invention provides a method for segmenting and recognizing welding spatter based on deep learning, the key point of which is that it includes the welding spatter segmentation and recognition system based on deep learning described in the first aspect of this invention, the steps of which are as follows:

[0071] S1 Image Acquisition: Acquire high-resolution images of the surface of the white body;

[0072] S2 Image Preprocessing: Denoising, illumination equalization, and standardization are performed on the acquired images;

[0073] S3 Splash Segmentation: Pixel-level segmentation of images using an improved deep learning segmentation network;

[0074] S4 post-processing optimization: Apply morphological operations and boundary refinement algorithms to the segmentation results;

[0075] S5 region localization: Maps the segmentation results to the actual coordinate system;

[0076] S6 Output: Generates visualization results and location data.

[0077] Compared with existing technologies, the beneficial effects of this invention are: significantly improved detection accuracy of welding spatter on body-in-white, with pixel-level segmentation accuracy exceeding 95%, far surpassing traditional target detection methods; stronger adaptability to scattering and irregularly shaped welding spatter; ability to handle detection tasks under complex backgrounds and changing lighting conditions; excellent real-time performance, meeting production line cycle requirements; and provision of precise area positioning information for automated grinding equipment, improving grinding efficiency and quality. Attached Figure Description

[0078] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0079] Figure 2 Diagram of the splash material segmentation module architecture;

[0080] Figure 3 This is a diagram of the architecture of the multi-scale feature fusion module.

[0081] Figure 4 Architecture diagram of the welding spatter feature enhancement module;

[0082] Figure 5 This is a diagram illustrating the architecture of a mechanism for refining the boundaries of scattering objects. Detailed Implementation

[0083] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0084] like Figure 1 As shown, a deep learning-based welding spatter segmentation and recognition system is presented. The system includes six modules, which are sequentially connected through data flow to form a complete processing architecture.

[0085] Image acquisition module: set at the welding fixture of the body-in-white, used to acquire the original image a of the surface of the body-in-white;

[0086] Image preprocessing module: Enhances, denoises, and normalizes the original image a to obtain a normalized image b;

[0087] Spatter segmentation module: Utilizes an improved deep learning segmentation network to perform pixel-level segmentation on the standardized image b, identify the welding spatter region, and output the initial segmentation result c;

[0088] Post-processing optimization module: Performs morphological processing and boundary optimization on the initial segmentation result c to generate segmentation result d;

[0089] Region positioning module: converts the segmentation result d into position information e in the actual coordinate system;

[0090] Results visualization module: Converts location information e into visual results and location data;

[0091] The image acquisition module includes an industrial camera, a light source system, and a control unit. The industrial camera uses a high-resolution CMOS sensor and is equipped with a lens suitable for close-up shooting to ensure clear images of fine splashes. The light source system uses a ring LED light source to provide uniform and stable illumination and reduce shadow interference. The control unit is responsible for camera parameter settings, trigger signal control, and image data transmission.

[0092] The image acquisition module is installed at the image acquisition station after the body-in-white welding to acquire the original image a of the body-in-white surface. It adopts a multi-angle shooting method to ensure coverage of all areas where welding spatter may exist. The industrial camera maintains a working distance of 800mm from the body-in-white surface to ensure a balance between image clarity and field of view.

[0093] The image preprocessing module processes the original image a acquired by the image acquisition module to obtain a standardized image b, thereby improving the accuracy of subsequent segmentation. This includes image denoising, illumination equalization, image standardization, and image size adjustment.

[0094] The image denoising employs a bilateral filtering algorithm to eliminate noise while preserving the edge details of the splashes. The expression for the bilateral filtering algorithm is as follows:

[0095]

[0096] Where I represents the image, (x,y) represents the pixel coordinates, Ω represents the window region, f is the spatial domain kernel function, g is the pixel value domain kernel function, W is the normalization coefficient, and I filtered This represents the output value of the pixel after filtering.

[0097] The illumination equalization uses an improved adaptive histogram equalization (CLAHE), which introduces a dynamic block size adaptive mechanism. Based on the local variation coefficient of the image, the CLAHE block size is automatically adjusted from 8×8 to 32×32, enabling the algorithm to adapt to welding spatter of different sizes and complex backgrounds, enhance image contrast, and highlight spatter areas.

[0098] The image standardization normalizes the pixel values ​​of the image to the range of [0, 1], further unifying the pixel value distribution and providing the segmentation network with a uniform and controllable input.

[0099] The image resizing involves adjusting the image to the input size required by the segmentation network while maintaining the aspect ratio. If the resized image does not meet the requirements of the segmentation network, a blank whiteboard is used for filling.

[0100] like Figure 2As shown, the spatter segmentation module is the core part of the system. It adopts an improved deep learning segmentation network architecture, which is based on the U-Net architecture with several improvements to adapt to the characteristics of welding spatter. It includes an image input device, an encoder, a decoder and an output layer. The standardized image b is input into the spatter segmentation module for processing to obtain the initial segmentation result c.

[0101] The encoder uses ResNext-50 as the backbone network to extract multi-level image features. Each level contains multiple convolutional blocks, and each convolutional block consists of two 3x3 convolutional layers. It also introduces residual connections and group convolution mechanisms to effectively extract spatter texture features.

[0102] The multi-layer image features include shallow feature maps, medium feature maps, and deep feature maps. The medium feature maps are further refined by extracting refined features through the multi-scale feature fusion module MFF, the welding spatter feature enhancement module SFEM, and the scattering object boundary refinement mechanism SBFM.

[0103] like Figure 3 As shown, the multi-scale feature fusion module (MFF) is used to process welding spatter of different sizes. After the mid-layer feature map is input into the multi-scale feature fusion module (MFF), it undergoes feature processing through four parallel branches. The four branches are as follows: Branch 1: 1×1 convolution, maintaining the original spatial resolution; Branch 2: 3×3 dilated convolution, with a dilation rate of 6; Branch 3: 3×3 dilated convolution, with a dilation rate of 12; Branch 4: 3×3 dilated convolution, with a dilation rate of 18.

[0104] The number of channels in the feature map output by each branch is uniformly 256. The weights of each branch are calculated through a channel attention mechanism, including global average pooling (GAP): compressing the feature map into a 1×1×C dimensional vector; dimensionality reduction fully connected layer: C→C / 16; ReLU activation; dimensionality increase fully connected layer: C / 16→C; Sigmoid activation, generating channel attention weights.

[0105] Multiply the feature map of each branch by the corresponding attention weight; sum the weighted feature maps to obtain the fused feature map, and output feature map A;

[0106] The mathematical expression for the multi-scale fusion module MFF is:

[0107]

[0108] Where F i α represents the convolutional output features with different dilation rates. i These are dual-weighted coefficients generated through a channel attention mechanism, where i takes values ​​from 1 to 4, representing the relevant parameters for each branch:

[0109] The double-weighting coefficient α i The expression is:

[0110] α i =σ(W2·ReLU(W1·GAP(F) i )))

[0111] GAP(F) i ) is the global average pooling operation, W1 and W2 are the weights of the fully connected layer, σ is the Signoid activation function, and ReLU is the activation function.

[0112] The advantages of using a multi-scale feature fusion module are: convolutions with different dilation rates can capture contextual information at different scales, adapting to welding spatter of various sizes; the channel attention mechanism dynamically adjusts the weights of features at each scale, further enhancing the model's adaptability; and the parallel structure design improves computational efficiency while maintaining real-time performance.

[0113] like Figure 4 As shown, the welding spatter feature enhancement module SFEM can enhance low-contrast spatter features. The obtained feature map A is input into the welding spatter feature enhancement module SFEM. It is processed through parallel main branch and enhancement branch. After processing, attention weights are calculated through an adaptive fusion mechanism, and the features of the main branch and enhancement branch are weighted and fused to obtain fused feature B. Then, the input feature map A and fused feature B are added to realize residual connection, and finally feature map B is output.

[0114] The main branch keeps the original feature map unchanged, while the enhancement branch extracts the unique texture and morphological features of welding spatter. The enhancement branch also includes an improved Local Local Binary Pattern (LBP) feature extraction layer; a 3×3 convolutional layer to extract the spatial relationship of LBP features; batch normalization; and ReLU activation.

[0115] The mathematical expression for the welding spatter feature enhancement module SFEM is:

[0116] F SFEM =F in +λ·BN(ReLU(Conv 3×3 (LBP(F in ))))

[0117] Where F in It is feature map A, LBP(F) in ) is a local binary mode operation, Conv 3×3 It is a convolutional layer, where λ is the weight coefficient, BN is batch normalization, and ReLU is the activation function.

[0118] The Local Binary Pattern (LBP) operator has significant advantages in extracting texture features, especially suitable for capturing the texture characteristics of welding spatter. LBP features are robust to changes in illumination and can maintain stable feature representations under different illumination conditions. The improved LBP feature extraction layer adjusts the neighborhood range, sampling distribution points, and comparison rules of LBP to make it more suitable for welding spatter. It needs to be used in conjunction with convolutional layers and incorporates feature weighting with an attention mechanism.

[0119] The advantages of the welding spatter feature enhancement module are: it effectively extracts and enhances the texture features of spatter, improving the detection capability of low-contrast spatter; the adaptive weighting mechanism dynamically adjusts the enhancement intensity according to the input features to avoid over-enhancement; and the residual connection design ensures that the original feature information is not lost, while increasing the gradient propagation path, which is beneficial to network training.

[0120] like Figure 5 As shown, the feature map B is input into the scattering object boundary refinement mechanism SBFM. After processing by the parallel feature processing branch and the boundary-aware feature extraction branch, the original features and gradient features are obtained respectively. The original features and gradient features are connected and fused by 1×1 convolution. Batch normalization and ReLU activation are performed. The boundary-aware loss function and boundary refinement inference are used to obtain the boundary enhancement features.

[0121] The extraction of boundary-aware features uses the Sobel operator to extract gradient information from the features, highlighting the boundaries of the object.

[0122] Boundary-aware loss is obtained through a boundary-aware loss function. First, a boundary mask is generated to identify the boundary regions of the true segmentation mask. Then, the LOU loss of the boundary regions is calculated. This boundary-aware loss function is:

[0123]

[0124] Among them, Ω B Show the boundary region, P (x,y) G (x,y) These are the predicted and true labels, respectively, and IoU is the intersection-union ratio.

[0125] Boundary refinement inference uses boundary enhancement features during the inference stage to generate more refined segmentation results, applies higher confidence thresholds to boundary regions, and reduces oversegmentation.

[0126] The advantages of the proposed scattering object boundary refinement mechanism are: it explicitly focuses on the object boundary region, improving boundary segmentation accuracy; the boundary-aware loss function guides the network to learn a more accurate boundary representation; it combines boundary gradient features and semantic features to generate more accurate segmentation results; and it is particularly suitable for irregularly shaped welding spatter with scattering boundaries.

[0127] The decoder is responsible for restoring the low-resolution, high-semantic features extracted by the encoder to the initial segmentation mask of the original resolution. In view of the characteristics of the scattering and blurred boundaries of welding spatter, the present invention has made several improvements to the traditional decoder, including multi-stage progressive upsampling, improved transposed convolution module, enhanced skip connection, residual refinement module, progressive feature channel reduction and spatial attention enhancement.

[0128] The decoder employs a five-stage progressive upsampling process, doubling the resolution of the feature map in each stage, while maintaining a strict 1:2 ratio between adjacent stages to avoid information loss.

[0129] The improved transposed convolution module adopts a parameterized transposed convolution with a 4×4 convolution kernel, a stride of 2, and padding of 1. The weights are initialized with a bilinear interpolation kernel, followed by a 3×3 regular convolution layer to adjust the feature channels and extract local correlations.

[0130] The enhanced skip connection uses an innovative feature fusion mechanism to fuse the encoder's feature map with the same resolution and the upsampled feature map. Before fusion, the encoder features are recalibrated, and a channel attention mechanism is used to adjust the importance of the skip connection features.

[0131] A residual refinement module is added after each decoding stage to optimize feature representation. It consists of two 3×3 convolutional layers and a residual connection, which effectively strengthens the checkerboard effect caused by upsampling and improves boundary smoothness.

[0132] A progressive feature channel reduction is adopted, with the number of channels decreasing in the sequence 512→256→128→64→32 to reduce computational complexity while retaining sufficient feature representation capability. In the final stage, 32 channels are retained as the final feature map, balancing accuracy and computational efficiency.

[0133] A spatial attention mechanism is introduced in the high-level decoding stage to enhance the feature response for irregular splash areas;

[0134] These improvements enable the decoder to more accurately recover the shape and boundaries of welding spatter, especially for areas with scattered spatter and blurred boundaries. Compared with traditional decoders, the improved decoder of this invention effectively reduces over-segmentation and under-segmentation.

[0135] The output layer is the last component of the splash segmentation module network architecture, including multi-scale feature integration, optimized 1×1 convolutional layers, boundary-aware activation functions, and probability calibration.

[0136] The multi-scale feature integration integrates features with different splitting rates before generating the final prediction, and adopts a feature pyramid structure to fuse features from different stages of the decoder.

[0137] The optimized 1×1 convolutional layer uses 1×1 convolution to map the final feature map (32 channels) into a single-channel probability map. The convolutional weights are initialized using the He initialization method to promote stable training.

[0138] The boundary-aware activation function employs a modified standard Sigmoid function to calculate the probability that each pixel belongs to a splash. This modified standard Sigmoid function makes it more sensitive to boundary regions and performs local adjustments to the response near the segmentation boundary.

[0139] The improved activation function is expressed as follows:

[0140] σ boundary (x)=σ(x)·(1+α'·|G Sobel (F integrated )|) β'

[0141] Where α' = 0.2 and β' = 0.5 are adjustment parameters, G Sobel (F integrated ) represents the Sobel gradient operator, which can highlight the boundary region, and σ is the Sigmoid activation function;

[0142] The probability calibration layer can reduce the bias of probability estimation. The calibration function is based on Bayesian theory and takes into account the prior class distribution. Its calibration function expression is as follows:

[0143]

[0144] Where r is the proportion of positive samples (splashes) in the training set, usually set to 0.15, and P represents the probability that each pixel in the image belongs to a splash.

[0145] The output layer mechanism significantly improves the accuracy of welding spatter segmentation, particularly in the boundary region. The optimized output layer of this invention effectively addresses the segmentation challenges posed by the blurred boundaries and scattering distribution of welding spatter.

[0146] The splash segmentation module also includes a network training strategy that employs a combined loss function, comprising binary cross-entropy loss (BCE), Dice loss, and boundary-aware loss (Loundary). The expression for the total loss (Total) is as follows:

[0147] L total =α·L BCE +β·L Dice +γ·L boundary

[0148] The weighting coefficients are set as follows: α = 0.5, β = 0.3, γ = 0.2;

[0149] The design of the combined loss function fully considers different aspects of the segmentation task: BCE loss is suitable for pixel-level classification tasks, Dice loss addresses the class imbalance problem, and boundary-aware loss focuses on boundary accuracy. This multi-objective optimization strategy significantly improves segmentation accuracy, especially for boundary regions.

[0150] The output layer is seamlessly integrated with the post-processing optimization module. The threshold selection of the probability map and the post-processing parameters are adaptively adjusted according to the confidence mapping, which further improves the quality of the final segmentation result.

[0151] The post-processing optimization module further optimizes the initial segmentation result output by the segmentation network, including the following steps: thresholding, small region filtering, morphological operations, boundary refinement, and region merging, to finally obtain the segmentation result d;

[0152] The threshold processing transforms the probability map output by the network into a binary mask, with the threshold set to 0.5. Its expression is as follows:

[0153]

[0154] The small region filtering removes regions with an area smaller than a preset threshold, eliminates false detection regions, and uses connected component analysis to calculate the area of ​​each candidate region, removing regions with an area smaller than 10 pixels.

[0155] The morphological operation refers to opening and closing operations to optimize the shape of the segmented region, and its expression is:

[0156] Mask opt =Close(Open(Mask,K1),K2)

[0157] Where Open and Close represent opening and closing operations respectively, and K1 and K2 are morphological kernels;

[0158] The boundary refinement uses a Conditional Random Field (CRF) algorithm to refine the boundaries of the segmentation results, as expressed by the formula:

[0159]

[0160] Where x is the pixel label assignment, ψ u It is a univariate potential energy term, based on the probability graph output by the network, ψ p It is a pairwise potential term based on pixel space and color similarity; the conditional random field considers the spatial relationship and color similarity between pixels, and can generate segmentation results that are more in line with physical boundaries, which is particularly suitable for optimizing the boundaries of scattering splashes;

[0161] The region merging process analyzes adjacent splatter areas. If the distance is less than a preset threshold (5 pixels), they are merged into one area to more effectively plan the polishing path.

[0162] The post-processing optimization module can effectively improve the quality and usability of the segmentation results, providing a more reliable foundation for subsequent region localization and polishing path planning.

[0163] The region positioning module converts the segmentation results in the image space into position information in the actual coordinate system, providing accurate positioning for the automatic grinding equipment.

[0164] The results visualization module presents the detection and location results to the operators in an intuitive way, facilitating monitoring and verification.

[0165] The implementation of this system includes data collection and annotation, model training, and system deployment and optimization.

[0166] Training a high-precision segmentation model requires a large amount of high-quality labeled data. The data collection and labeling steps are as follows:

[0167] Data Acquisition: Collect welding spatter images of different vehicle models and under different lighting conditions in the actual production environment to ensure data diversity. A total of no fewer than 2000 original images should be collected.

[0168] Data annotation: Each area of ​​welding spatter is precisely annotated by professional annotators using polygon tools. A quality control process is adopted, including a three-level annotation process: initial annotation → advanced review → expert verification;

[0169] A probabilistic labeling method is used for regions with ambiguous boundaries;

[0170] Welding process experts were brought in to verify the annotation results.

[0171] Data Augmentation: A data augmentation method targeting the characteristics of welding spatter was designed, including geometric transformation, illumination transformation, noise addition, and special enhancements. The geometric transformations include, for example, rotation (±30°), scaling (0.8-1.2x), translation (±10%), and horizontal / vertical flipping; illumination transformations include, for example, brightness adjustment (±20%), contrast adjustment (0.8-1.2x), and gamma correction (0.8-1.2); noise addition includes, for example, Gaussian noise (g = 0.01-0.03) and calibration noise (0.5-1.5%); and special enhancements include simulating light sources at different angles, adding reflective areas, and simulating changes in the shape of welding spatter.

[0172] Dataset partitioning: The dataset is divided into training, validation and test sets in a 7:2:1 ratio to ensure that the three datasets contain samples of different car models and different welding areas.

[0173] Data augmentation methods targeting the characteristics of welding spatter significantly improve the generalization ability of the model, enabling it to adapt to various complex situations in real production environments. In particular, augmentation methods that simulate different lighting conditions and changes in spatter morphology are especially important for improving the model's performance in real-world scenarios.

[0174] The model training employs the following strategies and hyperparameters:

[0175] The Adam optimizer is used, with an initial learning rate of 1e-4 and a cosine annealing scheduling strategy.

[0176] Batch size: Depends on hardware, for example, set to 16;

[0177] Training rounds: For example, a total of 100 training rounds are used, with an early stopping mechanism. If the performance on the validation set does not improve for 10 consecutive rounds, training is stopped.

[0178] Loss function: The combined loss function used in the network training described above is adopted;

[0179] Weight initialization: The encoder part uses ImageNet pre-trained weights, and the decoder part uses He for initialization;

[0180] Training strategy: Freeze the encoder for the first 10 rounds and train only the decoder part; unfreeze the entire network and train it together in subsequent rounds; fine-tune with a smaller learning rate (1e-5) in the last 10 rounds.

[0181] Data loading: Use a multi-threaded data loader with a prefetch factor of 4, depending on the hardware, for real-time data augmentation.

[0182] The carefully designed training strategy and hyperparameter tuning significantly improved the model's training efficiency and final performance. In particular, the staged training strategy and cosine annealing learning rate scheduling played a key role in obtaining a high-precision segmentation model.

[0183] When deploying and optimizing a system in a production environment, the following aspects need to be considered:

[0184] Hardware configuration: Computing unit: Industrial-grade GPU server (NVIDIA T4 or higher specification); Storage: High-speed SSD with a capacity of not less than 1TB; Memory: Not less than 32GB.

[0185] Real-time optimization: Model quantization: INT8 quantization technology is used to reduce computational complexity and improve inference speed; TensorRT acceleration: The trained model is converted to TensorRT format to improve inference speed; Parallel processing: Image preprocessing, model inference and post-processing adopt a pipelined parallel architecture to improve overall processing efficiency.

[0186] A deep learning-based method for welding spatter segmentation and recognition, comprising the aforementioned deep learning-based welding spatter segmentation and recognition system, includes the following steps:

[0187] S1 Image Acquisition: Acquire high-resolution images of the surface of the white body;

[0188] S2 Image Preprocessing: Denoising, illumination equalization, and standardization are performed on the acquired images;

[0189] S3 Splash Segmentation: Pixel-level segmentation of images using an improved deep learning segmentation network;

[0190] S4 post-processing optimization: Apply morphological operations and boundary refinement algorithms to the segmentation results;

[0191] S5 region localization: Maps the segmentation results to the actual coordinate system;

[0192] S6 Output: Generates visualization results and location data.

[0193] The above-mentioned system and method significantly improve the accuracy of welding spatter detection, achieving a pixel-level segmentation accuracy of over 95%, far exceeding traditional target detection methods. It also demonstrates greater adaptability to scattering and irregularly shaped welding spatter; it can handle detection tasks under complex backgrounds and varying lighting conditions; it exhibits excellent real-time performance, meeting production line cycle requirements; and it provides precise area positioning information for automated grinding equipment, improving grinding efficiency and quality.

[0194] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. Those skilled in the art, under the guidance of the present invention, can make various similar representations without departing from the spirit and claims of the present invention, and such modifications all fall within the protection scope of the present invention.

Claims

1. A deep learning-based welding spatter segmentation and recognition system, characterized in that: The system comprises, in sequence: Image acquisition module: set at the welding fixture of the body-in-white, used to acquire the original image a of the surface of the body-in-white; Image preprocessing module: Enhances, denoises, and normalizes the original image a to obtain a normalized image b; Spatter segmentation module: Utilizes an improved deep learning segmentation network to perform pixel-level segmentation on the standardized image b, identify the welding spatter region, and output the initial segmentation result c; Post-processing optimization module: Performs morphological processing and boundary optimization on the initial segmentation result c to generate segmentation result d; Region positioning module: converts the segmentation result d into position information e in the actual coordinate system; Results visualization module: Converts location information e into visualization results and location data.

2. The system according to claim 1, characterized in that: The image preprocessing module performs image denoising, illumination equalization, image standardization, and image size adjustment on the original image a. The image denoising employs a bilateral filtering algorithm to eliminate noise while preserving the edge details of the splashes. The bilateral filtering algorithm is expressed as follows: Where I represents the image, (x, y) represents the pixel coordinates, Ω represents the window region, f is the spatial domain kernel function, g is the pixel value domain kernel function, W is the normalization coefficient, and I filtered This represents the output value of the pixel after filtering. The illumination equalization uses an improved adaptive histogram equalization (CLAHE), which introduces a dynamic block size adaptive mechanism that automatically adjusts the CLAHE block size based on the local variation coefficient of the image, with a size ranging from 8×8 to 32×32. The image normalization normalizes the pixel values ​​of the image to the range of [0, 1]. The image resizing involves adjusting the image to the input size required by the segmentation network while maintaining the aspect ratio. If the resized image does not meet the requirements of the segmentation network, a blank whiteboard is used for filling.

3. The system according to claim 1, characterized in that: The splatter segmentation module is based on an improved deep learning segmentation network architecture built on the U-Net architecture, which includes an image input device, an encoder, a decoder, and an output layer in sequence. The encoder uses ResNext-50 as the backbone network to extract multi-level image features of the normalized image b. The multi-level image features include shallow feature maps, medium feature maps and deep feature maps. The medium feature map is further refined by multi-scale feature fusion module MFF, welding spatter feature enhancement module SFEM and scattering object boundary refinement mechanism SBFM. The multi-level image features extracted by the encoder are input into the decoder for decoding. The decoder adopts multi-stage progressive upsampling and gradually restores the feature map resolution through transposed convolution and skip connections to generate an initial segmentation mask. The initial segmentation mask is input into the output layer and converted into a single-channel probability map through a 1×1 convolution. The improved standard Sigmoid activation function is used to output the probability that each pixel belongs to the splash, and probability correction is performed to obtain the final segmentation mask, which is the initial segmentation result c.

4. The system according to claim 3, characterized in that: The splash segmentation module also includes a network training strategy that employs a combined loss function, comprising binary cross-entropy loss (BCE), Dice loss, and boundary-aware loss (Loundary). The expression for the total loss (Total) is as follows: L total =α·L BCE +β·L Dice +γ·L boundary The weighting coefficients are set as follows: α = 0.5, β = 0.3, γ = 0.

2.

5. The system according to claim 3, characterized in that: After the mid-layer feature map is input to the multi-scale feature fusion module MFF, it is processed through four parallel branches, and the number of special map channels output by each branch is uniformly 256. The weight of each branch is calculated through the channel attention mechanism, the feature map of each branch is multiplied by the corresponding attention weight, the weighted feature maps are summed to obtain the fused feature A, and feature map A is generated and output. The weight calculation for each branch includes global average pooling, dimensionality reduction fully connected layer, ReLU activation, dimensionality increase fully connected layer, and Sigmoid activation, generating channel attention weights; The mathematical expression for the multi-scale feature fusion module (MFF) is: Where F i α represents the convolutional output features with different dilation rates. i These are dual-weighted coefficients generated through the channel attention mechanism, with i taking values ​​from 1 to 4, representing the relevant parameters for each branch. The dual-weighting coefficient α i The expression is: a i =σ(W2·ReLU(W1·GAP(F i ))); GAP(F) i ) is the global average pooling operation, W1 and W2 are the weights of the fully connected layer, σ is the Signoid activation function, and ReLU is the activation function.

6. The system according to claim 5, characterized in that: Feature map A is input into the welding spatter feature enhancement module SFEM. It is processed through parallel main branch and enhancement branch. After processing, attention weights are calculated through an adaptive fusion mechanism, and the features of the main branch and enhancement branch are weighted and fused to obtain fused feature B0. Then, the input feature map A is added to the fused feature B0 to realize residual connection, and finally feature map B is output. The main branch keeps the input feature map A unchanged, and the operation of the enhancement branch includes, in sequence, feature extraction using an improved local binary pattern (LBP) feature extraction layer, extraction of the spatial relationship of LBP features using a 3×3 convolutional layer, batch normalization, and ReLU activation. The improved Local Binary Pattern (LBP) feature extraction layer adjusts the neighborhood range, sampling distribution points, and comparison rules of LBP. It needs to be used in conjunction with convolutional layers and incorporates feature weighting with an attention mechanism. The mathematical expression for the welding spatter feature enhancement module SFEM is: F SFEM =F in +λ·BN(ReLU(Conv 3×3 (LBP(F in )))) Where F in It is feature map A, LBP(F) in ) is a local binary mode operation, Conv 3×3 It is a convolutional layer operation, λ is the feature weighting coefficient of the attention mechanism, BN is batch normalization, and ReLU is the activation function.

7. The system according to claim 6, characterized in that: The feature map B is input into the scattering object boundary refinement mechanism SBFM. After processing by the parallel feature processing branch and the boundary-aware feature extraction branch, the original features and gradient features are obtained respectively. The original features and gradient features are connected and fused by 1×1 convolution. Batch normalization and ReLU activation are performed. The boundary-aware loss function and boundary refinement inference are used to obtain the boundary enhancement features. The expression for the boundary-aware loss function is: Among them, Ω B Show the boundary region, P (x,y) G (x,y) These are the predicted and true labels, respectively, and IoU is the intersection-union ratio.

8. The system according to claim 3, characterized in that: The multi-level image features extracted by the encoder are sequentially input into the decoder. The decoder performs decoding operations sequentially through a multi-stage progressive upsampling module, an improved transposed convolution module, a skip connection module, a residual thinning module, a progressive channel reduction module, and a spatial attention module, so that the extracted multi-level image features are restored to the initial segmentation mask of the original resolution. The number of channels in the progressive channel reduction module decreases in sequence from 512 to 256 to 128 to 64 to 32, and the final stage retains 32 channels in the final feature map. The output layer adopts a feature pyramid structure, which fuses features from different stages of the decoder. It uses 1×1 convolution to map the final feature map into a single-channel probability map. It adopts an improved standard Sigmoid function to make it more sensitive to boundary regions, and performs local adjustments to the response near the segmentation boundary. It calculates the probability that each pixel belongs to the splash and sets a probability calibration layer. The final segmentation mask is obtained through the probability calibration function, and the initial segmentation result c is output. The improved standard Sigmoid function expression is as follows: s boundary (x)=σ(x)·(1+α'·|G Sobel (F integrated )|) β' Where α' = 0.2 and β' = 0.5 are adjustment parameters, G Sobel (F integrated ) represents the Sobel gradient operator of the input feature map, which can highlight the boundary region, and σ is the Sigmoid activation function; The calibration function expression for the probability calibration layer is: Where r is the proportion of positive splashes in the training set, and P represents the probability that each pixel in the image belongs to a splash.

9. The system according to claim 3, characterized in that: The initial segmentation result c is input into the post-processing optimization module for threshold processing, small region filtering, morphological operations, boundary refinement and region merging operations. The thresholding process converts the probability map into a binary mask, with the threshold set to 0.

5. Its expression is: The small region filtering removes regions with an area smaller than a preset threshold, eliminates false detection regions, and uses connected component analysis to calculate the area of ​​each candidate region, removing regions with an area smaller than 10 pixels. The morphological operations apply opening and closing operations to optimize the shape of the segmented region, as expressed by the formula: Mask opt =Close(Open(Mask,K1),K2) Where Open and Close represent opening and closing operations respectively, and K1 and K2 are morphological kernels; The boundary refinement uses a Conditional Random Field (CRF) algorithm to refine the boundaries of the segmentation results, as expressed by the formula: Where x is the pixel label assignment, ψ u It is a univariate potential energy term, based on the probability graph output by the network, ψ p These are paired potential energy terms, based on pixel space and color similarity; The region merging process analyzes adjacent splash areas. If the distance is less than a preset threshold, they are merged into one region, resulting in the final segmentation result d.

10. A method for segmenting and recognizing welding spatter based on deep learning, characterized in that: The welding spatter segmentation and recognition system based on deep learning, as described in any one of claims 1-9, comprises the following steps: S1 Image Acquisition: Acquire high-resolution images of the surface of the white body; S2 Image Preprocessing: Denoising, illumination equalization, and standardization are performed on the acquired images; S3 Splash Segmentation: Pixel-level segmentation of images using an improved deep learning segmentation network; S4 post-processing optimization: Apply morphological operations and boundary refinement algorithms to the segmentation results; S5 region localization: Maps the segmentation results to the actual coordinate system; S6 Output: Generates visualization results and location data.

Images