A few-shot oriented welding defect recognition method and platform

Abstract

The application discloses a kind of welding defect identification methods and platform for few samples, it is related to welding defect identification technical field, including: welding original image acquisition and annotation, through visual system in real time acquisition filter pool gray image with optical filter in welding process, combine process parameter adjustment to generate multiclass defect image, and with actual welding defect as label;Welding original image pre-processing, including pool region clipping, data enhancement operation and standardization data division;Welding defect identification model's stage training, through main network extraction spatial feature, cross fusion horizontal and vertical direction features by multidimensional feature fusion module and enhance pool positioning, then generate high-order statistical features by Brown distance covariance matrix module;Defect identification in actual welding process, based on meta-learning strategy to construct prototype welding network space, generate the class prototype vector of each type of defect, realize the classification of query image by similarity measurement, and dynamically adjust process parameters.

Description

Technical Field

[0001] This invention belongs to the field of welding defect identification technology, and in particular relates to a welding defect identification method and platform for small sample sizes. Background Technology

[0002] Welding, as a fundamental process in modern manufacturing, plays an irreplaceable and crucial role in equipment manufacturing. However, due to factors such as low assembly precision and fluctuations in heat input, various defects often appear at weld joints, directly affecting the overall integrity and service reliability of the welded structure. Welding defect identification based on molten pool images is currently the most commonly used real-time defect identification method, mainly including traditional machine learning methods and deep learning methods. Machine learning methods rely on data preprocessing and feature engineering, resulting in low efficiency and difficulty in handling noise and lighting variations. Deep learning methods, on the other hand, significantly improve identification accuracy and robustness through automatic feature extraction and end-to-end learning. However, challenges remain, such as the complexity of the welding process, making it difficult to obtain high-quality images; the high cost and time-consuming nature of manual annotation, limiting the number of samples; and the imbalance in the number of samples per category, causing the model to favor the category with more samples.

[0003] To address the aforementioned issues, researchers have introduced few-shot learning methods, with metric-based methods receiving particular attention. These methods learn image representations through similarity metrics. The image features extracted by the network can be viewed as random vectors in a high-dimensional space, and their similarity is typically measured using statistical features such as mean and covariance combined with Euclidean distance or KL divergence. However, these methods primarily consider marginal distributions, neglecting joint distributions and nonlinear relationships between variables. Although some methods attempt to utilize joint distributions, their computational cost is high, making them unsuitable for real-time applications. Therefore, there is an urgent need to propose a welding defect recognition method and platform for few-shot applications. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a welding defect identification method and platform for small sample sizes. It introduces the Brownian distance covariance metric into the welding defect identification network, effectively modeling and extracting the nonlinear dependency between the input welding image and the prototype vector of the welding defect class, thereby achieving high-accuracy classification in complex scenarios.

[0005] To achieve the above objectives, this invention provides a welding defect identification method for small sample sizes, comprising:

[0006] Welding original image acquisition and annotation: The vision system acquires grayscale images of the molten pool with filters in real time during the welding process, and generates multiple types of defect images by adjusting process parameters, and uses actual welding defects as labels;

[0007] Preprocessing of the original welding images, including molten pool region cropping, data augmentation operations, and standardized data partitioning;

[0008] The phased training of the welding defect identification model involves extracting spatial features through the backbone network, enhancing the localization of the molten pool by cross-fusion of horizontal and vertical features through the multi-dimensional feature fusion module, and generating high-order statistical features through the Brownian distance covariance matrix module.

[0009] Defect identification in actual welding processes is achieved by constructing a prototype welding network space based on a meta-learning strategy, generating class prototype vectors for various defects, classifying query images through similarity measurement, and dynamically adjusting process parameters.

[0010] Optionally, in the acquisition and annotation of the original welding image, the vision system includes a camera integrating a neutral density filter and a bandpass filter, and a side-illumination diode laser for reducing plasma interference; the actual welding defect label is determined by observation of the weld surface, back side and cross-section after welding.

[0011] Optionally, in the preprocessing of the original welding image, the data augmentation operation includes flipping the molten pool image, adding Gaussian noise, and randomly adjusting the brightness and contrast; after the dataset is divided, a standard training dataset is constructed by standardizing the number of samples in each category.

[0012] Optionally, the backbone network adopts a four-group residual block stacked structure, each residual block containing a 3×3 convolutional kernel, a group normalization layer and residual connections, used to output the first welding feature vector.

[0013] Optionally, the operation of the multidimensional feature fusion module includes: performing horizontal and vertical pooling on the first welding feature vector respectively, fusing it through matrix multiplication to generate a first spatial weight feature vector, generating a second spatial weight feature vector through convolution, group normalization and sigmoid activation, and multiplying it with the first welding feature vector channel by channel to output the second welding feature vector.

[0014] Optionally, the operation of the Brownian distance covariance matrix module includes: calculating the Euclidean distance matrix of the input feature vector, and subtracting the row, column and global mean through centering to generate a welding feature vector representing the nonlinear dependency.

[0015] Optionally, in the phased training, the meta-training phase constructs the support set and query set according to the N-way K-shot task; the class prototype vector is generated by the mean of the feature vectors of the same support set samples; the similarity measure is implemented by calculating the inner product of the matrix trace of the query feature vector and the class prototype vector.

[0016] On the other hand, to achieve the above objectives, the present invention also provides a welding defect identification platform for small sample sizes, comprising:

[0017] Vision systems, industrial computers, edge controllers, motion mechanisms, laser welding heads, fiber lasers, protective gas devices, and I / O modules;

[0018] The vision system includes a camera and a diode laser for illumination. The camera is mounted on the coaxial optical port of the laser welding head to capture real-time images of the molten pool. The diode laser is mounted on the side of the laser welding head to illuminate the area to be welded. After preprocessing the acquired data, the industrial control computer transmits the processing results to the edge controller. The edge controller determines the current welding defect category based on the deployed model and feeds back the determination result to the industrial control computer. At the same time, the industrial control computer calculates and sends the position data of the motion mechanism according to the teaching trajectory, receives the acquired data from the camera, and sends control signals to the IO module to adjust the power of the fiber laser, the flow rate of the protective gas device, and the real-time triggering of the camera's acquisition signal.

[0019] Technical advantages of this invention: This invention discloses a welding defect recognition method and platform for limited sample sizes. It employs a meta-learning training strategy to construct a prototype welding network space, generating prototype vectors for each welding defect class. The category of the welding defect image is identified based on the similarity between the input welding image and the prototype vectors of the welding defect classes. Compared to traditional methods, this method performs even better with sparse samples, significantly reducing dependence on large-scale labeled data and improving the system's adaptability and robustness in complex welding environments. By introducing the Brownian distance covariance metric into the welding defect recognition network, the nonlinear dependency between the input welding image and the prototype vectors of the welding defect classes is effectively modeled and extracted, achieving high-accuracy classification in complex scenarios while ensuring a simple and efficient computation process, making it suitable for real-time scenarios. A multi-dimensional feature fusion module is designed, which effectively identifies and locates the molten pool region by cross-fusing horizontal and vertical features, significantly improving feature extraction capabilities under limited sample conditions. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 These are molten pool diagrams corresponding to various welding defects in embodiments of the present invention;

[0022] Figure 2 This is a flowchart illustrating a welding defect identification method for a small sample size according to an embodiment of the present invention.

[0023] Figure 3 This is a model structure diagram of an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the backbone network according to an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the multidimensional feature fusion (MFF) module according to an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the model phased training method according to an embodiment of the present invention;

[0027] Figure 7 The figure shows the test results of an embodiment of the present invention;

[0028] Figure 8 This is a schematic diagram of a welding defect identification platform according to an embodiment of the present invention;

[0029] Figure 9 This is a physical image of the welding defect identification platform according to an embodiment of the present invention. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0032] like Figures 1-7 As shown, this embodiment provides a welding defect identification method for small sample sizes, including:

[0033] Step S1: Refer to Figure 1 The original welding image acquisition and annotation involves acquiring the original welding image in real time during the welding process using a vision system. By changing parameters such as laser power and offset, the original welding images corresponding to six defect types are obtained, and the actual welding defects are used as model labels.

[0034] Specifically, the original welding image is a grayscale image captured by a vision system, which includes a camera and a diode laser for illumination. The camera also includes a neutral density filter and an 808nm bandpass filter, integrated into the camera lens. The camera captures images at 50 frames per second, with each original image measuring 384×340 pixels. The diode laser power is set to 3W, with a working wavelength of 808nm, capable of illuminating the welding area and reducing plasma interference generated during welding. Actual welding defects are accurately obtained by observing the weld surface, back side, and cross-section after welding, and used as labels for model training. Figure 1 As can be seen, the actual welding defects set here include incomplete penetration, good penetration, over-penetration, burn-through, weld misalignment, and weld displacement. Each set of process parameters was tested multiple times to obtain sufficient data.

[0035] Step S2: Preprocessing of the original welding image, including:

[0036] Step S21: Region of Interest (ROI) cropping, specifically cropping the original welding image to extract the relevant parts of the molten pool and reduce the image size to obtain the molten pool image;

[0037] Step S22: Data augmentation, specifically, flipping the molten pool image, adding Gaussian noise, and randomly adjusting the image's brightness and contrast;

[0038] Step S23: Label assignment, specifically using an integer sequence starting from 0 as labels for the molten pool images to represent the defect categories of these images: 0 indicates incomplete penetration, 1 indicates good penetration, 2 indicates over-penetration, 3 indicates burn-through, 4 indicates weld offset, and 5 indicates weld misalignment.

[0039] Step S24: Dataset partitioning. The dataset consists of a known subset and an unknown subset. The known subset includes 35,000 images of molten stainless steel 304 weld pools. These 35,000 labeled weld pool images are randomly partitioned into a preliminary training set, a validation set, and a test set in a ratio of 0.8:0.1:0.1. The unknown subset contains 21,133 images of molten stainless steel 304 weld pools. This unknown subset was collected under different welding process parameters than the known subset and is specifically used to test the performance of the proposed method. Subsequently, the number of weld pool images for each defect category in the preliminary training set is standardized to be the same, forming the standard training dataset D. L .

[0040] Step S3: Refer to Figures 3-6 The architecture design and phased training strategy of the welding defect identification model include the following steps:

[0041] Step S31: Refer to Figure 3 The architecture of the welding defect identification model includes a backbone network, a multidimensional feature fusion module (MFF), and a Brownian distance covariance matrix module (BDCM).

[0042] Step S311: Refer to Figure 4 The backbone network adopts a stacked structure of four residual blocks. Each residual block contains S 3×3 convolutional kernels, a group of normalization layers and residual connections, which are used to extract the spatial features of the molten pool image and obtain the first welding feature vector.

[0043] Step S312: Referring to step 5, the MFF module extracts spatial weight features through bidirectional pooling. The specific algorithm is as follows:

[0044] The first welding feature vector is pooled in both the horizontal and vertical directions to generate a horizontal feature vector and a vertical feature vector.

[0045] The horizontal and vertical eigenvectors are fused using matrix multiplication to generate the first spatial weighted eigenvector. The calculation formula is as follows:

[0046]

[0047] in, It is the first spatial weight eigenvector, z x It is a horizontal eigenvector, z y Vertical feature vectors;

[0048] The first spatial weight feature vector is subjected to 2D convolution, group normalization, and sigmoid activation to generate the second spatial weight feature vector. The calculation formula is as follows:

[0049]

[0050] Where f is the second space weight feature vector, σ is the sigmoid activation function, GN is the group normalization operation, and Conv is the 2D convolution operation;

[0051] The second spatial weight feature vector is multiplied channel by channel with the first welding feature vector to obtain the second welding feature vector M output by the MFF module. out The calculation formula is:

[0052]

[0053] Among them, M out It is the second welding feature vector. It is the first welding feature vector;

[0054] Step S313: The BDCM module generates high-order statistical features based on the Brownian distance covariance matrix. The specific algorithm is as follows:

[0055] For two molten pool images, vectors A∈R are obtained after extraction by the backbone network. p , B∈R q Their dimensions are p and q, respectively. in The Euclidean distance matrix is ​​calculated from the discrete observation pairs of vector A, where m is the number of observations. Similarly, the Euclidean distance matrix is ​​calculated... in n is the number of observations. In the following discussion, it is assumed that m is equal to n. The Euclidean distance matrix is ​​in the following form:

[0056]

[0057] Matrix X = (x kl This is called the welding feature vector. Specifically, x kl The definition is as follows:

[0058]

[0059] The last three items are matrices. The matrix Y can also be based on the mean of the l-th column, the k-th row, and all elements. Calculated in a similar manner, the input molten pool image is thus converted into a welding feature vector by the welding defect recognition model;

[0060] Step S32: Refer to Figure 6 The phased training strategy for the welding defect identification model includes the following steps:

[0061] Step S321: Pre-training phase, used to initialize the network parameters of the welding defect recognition model, using the standard training dataset D. L Input the data into the model in batches, and output the defect category prediction through a fully connected classifier;

[0062] Step S322: Meta-training phase. Based on the N-way K-shot task construction strategy, N defect categories are randomly selected for each training task. Each category contains K support samples and L query samples, which together form the support set. and query set Among them, y j It is a specific molten pool image z j Category tags.

[0063] Step S3221: Generate the welding feature vector of the query sample. In each training task, the query set D... que The N×L molten pool image is input into the welding defect recognition model to generate welding feature vectors;

[0064] Step S3222: Class prototype vector generation. In each training task, the support set D will be generated. sup Inputting K similar molten pool images into the welding defect recognition model generates class prototype vectors, including:

[0065] Specifically, it supports sample filtering from the support set D. sup Select a subset of melt pool images of a certain category;

[0066] Specifically, welding feature vector extraction involves using a welding defect recognition model to encode the features of each specific molten pool image in the subset, thereby obtaining the welding feature vector.

[0067] Specifically, the mean of the welding feature vectors corresponding to images of the same type of molten pool is calculated as the class prototype vector, and the calculation formula is as follows:

[0068]

[0069] Where v represents the support set D sup A certain category, M v It is a class prototype vector, K is the number of support samples, O v It is a subset, X θ (z j ) is the welding feature vector, and θ is the network parameter;

[0070] Step S3223: Similarity measurement, for query set D que Welding feature vectors are extracted from the molten pool image, and their similarity scores with the class prototype vectors of each class are calculated using the following formula:

[0071] s j，v =τ·tr(X) θ (z j ) T M v );

[0072] Among them, s j，v It is the similarity score between the welding feature vector of the molten pool image and the prototype vector of class v, where tr(·) represents the trace of the matrix, T represents the transpose of the matrix, and τ is a learnable scaling parameter. Here, we need to calculate the similarity score between it and the prototype vectors of six classes (incomplete penetration, good penetration, over-penetration, burn-through, weld offset, and weld misalignment).

[0073] Since the Brownian distance covariance matrix is ​​symmetric, the similarity score can be regarded as the inner product of the vectorized upper triangular parts of the two matrices, which simplifies the calculation.

[0074] Step S3224: Classification decision, including:

[0075] Specifically, the probability distribution is generated by converting the similarity scores into a category probability distribution using the softmax function. The calculation formula is as follows:

[0076]

[0077] Specifically, the welding defect classification decision is made by selecting the defect category with the highest probability as the identification result based on the category probability distribution of the query samples. The calculation formula is as follows:

[0078]

[0079] For example, the probability of the incompletely penetrated category. If the probability of other categories such as good, over-penetration, burn-through, weld misalignment and weld displacement is less than 0.8, then the current identification result is determined to be the "incomplete penetration" defect category.

[0080] Specifically, loss optimization uses the cross-entropy loss function to measure the error between the predicted probability and the true label, and updates the network parameters using the gradient descent algorithm. The calculation formula is as follows:

[0081]

[0082] Step S4: Defect identification in the actual welding process specifically includes:

[0083] The trained model is deployed in the edge controller. During the actual welding process, the camera continuously acquires original welding images. The acquired images are first transmitted to the industrial control computer for image preprocessing, and the preprocessed data is then transmitted to the edge controller. Combined with the deployed model, the system identifies welding defect types in real time and transmits the identification results to the industrial control computer for real-time recording and display. Once a defect is detected, the system automatically triggers an early warning feedback and dynamically adjusts the corresponding process parameters based on the defect type to optimize welding quality.

[0084] Reference Figure 7 This invention has achieved excellent welding defect identification results in various welding scenarios. It achieves high accuracy using only 5% of the labeled data and demonstrates outstanding performance and generalization ability in complex environments, showing broad application prospects for real-time quality monitoring of industrial welding defects.

[0085] This invention proposes a welding defect recognition method for sparse samples. A meta-learning training strategy is employed to construct a prototype welding network space, generating prototype vectors for each welding defect class. The category of a welding defect image is then identified based on the similarity between the input welding image and the prototype vectors of the welding defect classes. Compared to traditional methods, this method performs even better with sparse samples, significantly reducing the dependence on large-scale labeled data and improving the system's adaptability and robustness in complex welding environments.

[0086] In this invention, the Brownian distance covariance metric is introduced into the welding defect recognition network to effectively model and extract the nonlinear dependency between the input welding image and the prototype vector of the welding defect class, achieving high-accuracy classification in complex scenarios while ensuring the simplicity and efficiency of the calculation process, making it suitable for real-time scenarios.

[0087] In this invention, a multi-dimensional feature fusion module is designed. By cross-fusing horizontal and vertical features, the melt pool region can be effectively identified and located, significantly improving the feature extraction capability under limited sample conditions.

[0088] like Figures 8-9 As shown, this embodiment also provides a welding defect identification platform for small sample sizes, including:

[0089] Vision systems, industrial computers, edge controllers, motion mechanisms, laser welding heads, fiber lasers, protective gas devices, and I / O modules;

[0090] The vision system includes a camera and a diode laser for illumination. The camera is mounted on the coaxial optical port of the laser welding head to capture real-time images of the molten pool. The diode laser is mounted on the side of the laser welding head to illuminate the area to be welded, reducing plasma interference during the welding process. The industrial control computer preprocesses the acquired data and transmits the processing results to the edge controller. The edge controller determines the current welding defect category based on the deployed model and feeds back the determination result to the industrial control computer. Simultaneously, the industrial control computer calculates and sends the position data of the motion mechanism based on the taught trajectory, receives the acquired data from the camera, and sends control signals to the I / O module to adjust the power of the fiber laser, the flow rate of the protective gas device, and the real-time trigger signal of the camera's acquisition signal.

[0091] This embodiment proposes a welding defect recognition method and platform for limited sample sizes. It employs a meta-learning training strategy to construct a prototype welding network space, generating prototype vectors for each welding defect class. The category of the welding defect image is then identified based on the similarity between the input welding image and the prototype vectors of the welding defect classes. Compared to traditional methods, this method performs even better with sparse samples, significantly reducing the dependence on large-scale labeled data and improving the system's adaptability and robustness in complex welding environments. By introducing the Brownian distance covariance metric into the welding defect recognition network, the nonlinear dependency between the input welding image and the prototype vectors of the welding defect classes is effectively modeled and extracted, achieving high-accuracy classification in complex scenarios while maintaining a simple and efficient computational process, making it suitable for real-time scenarios. A multi-dimensional feature fusion module is designed, which effectively identifies and locates the molten pool region by cross-fusing horizontal and vertical features, significantly improving feature extraction capabilities under limited sample conditions.

[0092] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying welding defects with a small sample size, characterized in that, include: Welding original image acquisition and annotation: The vision system acquires grayscale images of the molten pool with filters in real time during the welding process, and generates multiple types of defect images by adjusting process parameters, and uses actual welding defects as labels; Preprocessing of the original welding images, including molten pool region cropping, data augmentation operations, and standardized data partitioning; The phased training of the welding defect identification model involves extracting spatial features through the backbone network, enhancing the localization of the molten pool by cross-fusion of horizontal and vertical features through the multi-dimensional feature fusion module, and generating high-order statistical features through the Brownian distance covariance matrix module. Defect identification in actual welding process: Based on meta-learning strategy, a prototype welding network space is constructed to generate class prototype vectors of various defects. The similarity measure is used to classify query images and dynamically adjust process parameters. The backbone network adopts a four-group residual block stacked structure. Each residual block contains a 3×3 convolutional kernel, a group normalization layer and residual connections, which are used to output the first welding feature vector. The operation of the multidimensional feature fusion module includes: performing horizontal and vertical pooling on the first welding feature vector respectively, fusing it through matrix multiplication to generate a first spatial weight feature vector, generating a second spatial weight feature vector through convolution, group normalization and Sigmoid activation, and multiplying it with the first welding feature vector channel by channel to output the second welding feature vector; The operation of the Brownian distance covariance matrix module includes: calculating the Euclidean distance matrix of the input feature vector, and subtracting the row, column and global mean through centering to generate a welding feature vector representing the nonlinear dependency relationship.

2. The welding defect identification method for small sample sizes as described in claim 1, characterized in that, In the acquisition and annotation of the original welding images, the vision system includes a camera integrating a neutral density filter and a bandpass filter, as well as a side-illumination diode laser for reducing plasma interference; the actual welding defect labels are determined by observing the surface, back, and cross-section of the weld after welding.

3. The welding defect identification method for small sample sizes as described in claim 1, characterized in that, In the preprocessing of the original welding image, the data augmentation operations include flipping the molten pool image, adding Gaussian noise, and randomly adjusting the brightness and contrast; after the dataset is divided, a standard training dataset is constructed by standardizing the number of samples in each category.

4. The welding defect identification method for small sample sizes as described in claim 1, characterized in that, In the phased training, the meta-training phase constructs support sets and query sets according to the N-way K-shot task; class prototype vectors are generated by the mean of feature vectors of similar support set samples; similarity measurement is achieved by calculating the inner product of the matrix trace of the query feature vector and the class prototype vector.

5. A platform for a welding defect identification method for small sample sizes according to any one of claims 1-4, characterized in that, include: Vision systems, industrial computers, edge controllers, motion mechanisms, laser welding heads, fiber lasers, protective gas devices, and I / O modules; The vision system includes a camera and a diode laser for illumination. The camera is mounted on the coaxial optical port of the laser welding head to capture real-time images of the molten pool. The diode laser is mounted on the side of the laser welding head to illuminate the area to be welded. After preprocessing the acquired data, the industrial control computer transmits the processing results to the edge controller. The edge controller determines the current welding defect category based on the deployed model and feeds back the determination result to the industrial control computer. At the same time, the industrial control computer calculates and sends the position data of the motion mechanism according to the teaching trajectory, receives the acquired data from the camera, and sends control signals to the IO module to adjust the power of the fiber laser, the flow rate of the protective gas device, and the real-time triggering of the camera's acquisition signal.

Images