A method for detecting anomalies in fusion welding additive manufacturing based on unsupervised learning

By adopting unsupervised learning multi-student-teacher network model in welding manufacturing, combining forward and reverse knowledge distillation processes, the problems of poor robustness and low detection accuracy in welding manufacturing are solved, and the welding is achieved High efficiency and accuracy of abnormal detection during additive processes.

CN118365976BInactive Publication Date: 2024-11-12NANJING UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410799380.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-20
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In welding manufacturing, traditional unsupervised image anomaly detection methods are poorly robust, slow detection speed and low accuracy. The supervised learning-based methods are difficult to adapt to diversified anomalies, and they cannot effectively detect potential anomalies in the welding additive process.

Method used

Using an unsupervised learning-based fusion welding additive anomaly detection method, using industrial cameras to capture melt pool images, train multi-student-teacher network models including forward and reverse knowledge distillation processes, knowledge transfer through teacher network and student network network and Feature fusion realizes implicit modeling and abnormal detection of normal data.

Benefits of technology

Accurate detection of various morphological and mass abnormalities during welding, such as cracks, offsets, holes, air holes, slag inclusions, etc., improve detection accuracy and efficiency, and can detect potential abnormalities without labeled data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118365976B_ABST
    Figure CN118365976B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for detecting anomalies in fusion welding additives based on unsupervised learning, comprising the following steps: in the process of fusion welding additives, using an industrial camera to capture a molten pool image, preprocessing the molten pool image, and obtaining a molten pool data set; using the molten pool data set to train an anomaly detection model for unsupervised learning, the anomaly detection model includes a forward knowledge distillation and a reverse knowledge distillation process; using the trained anomaly detection model for reasoning to obtain an anomaly detection result. The method for detecting anomalies in fusion welding additives based on unsupervised learning of the present invention adopts an unsupervised learning algorithm, and only requires normal data for training. The network model adopts a multi-student-teacher distillation model, and the anomaly detection accuracy can achieve good results for various morphological and quality anomalies that occur during the welding process, such as cracks, offsets, holes, pores, slag inclusions, and other defects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a fusion welding additive anomaly detection method based on unsupervised learning, belonging to the technical field of industrial anomaly detection. Background Art

[0002] The concept of intelligent welding manufacturing has been proposed with the vigorous development of artificial intelligence and big data. It is a new production operation paradigm based on multi-sensor and artificial intelligence technology. It can not only perceive the changes in welding behavior in real time through multi-sensors, but also use prior knowledge (such as molten pool morphology characteristics, weld morphology, etc.) to diagnose welding conditions, control welding processes and make reliable decisions. In the welding process, there are often interferences such as strong arc, spatter, ultra-high temperature and dust, in addition to improper welding parameter settings and improper weld size. These interferences may cause unpredictable abnormal defects on the surface of the molten welding material, such as cracks, offsets, hole defects, pores, slag inclusions and other defects. These defects will cause problems in geometric dimensions and tolerances and poor performance, which brings severe tests and challenges to welding process status monitoring. Therefore, we urgently need to develop a welding process monitoring technology with high reliability, which can detect abnormalities in time and take corresponding measures when abnormalities are found.

[0003] Therefore, anomaly detection has become a feasible and promising quality assurance technology. The mainstream methods of traditional machine learning mainly include methods based on statistical models, methods based on image decomposition, and methods based on sparse coding reconstruction. Although unsupervised image anomaly detection methods based on traditional machine learning have achieved certain results, these methods often require the design of manual features and complex expert prior knowledge, and also face problems such as poor robustness, slow detection speed, and low accuracy.

[0004] In recent years, the application of deep learning in the field of anomaly detection has provided a more reliable and efficient solution for industrial production. Methods based on deep learning often have high accuracy and can adapt to anomaly detection tasks of different types and scales. However, in welding manufacturing, it is often difficult to obtain a large amount of defective data, and because the causes of anomalies are diverse and unpredictable, it is difficult to use one or several anomalies that have occurred to represent all anomalies. Therefore, if an anomaly detection algorithm based on supervised learning is used to detect various anomalies in welding, it is often difficult to achieve satisfactory results. Therefore, there is an urgent need for a new solution, namely an unsupervised algorithm, which does not rely on labeled data and only needs to be trained on normal images. It can automatically learn features and patterns from these unlabeled normal data and discover potential anomalies or abnormal areas without knowing the abnormal labels in advance.

[0005] Knowledge distillation is one of the most popular structures in unsupervised anomaly detection. The scheme consists of a descriptive teacher network and a randomly initialized student network. However, most existing teacher-student network methods focus on anomaly detection in natural images, but perform poorly in the welding process. In this context, how to make the unsupervised anomaly detection algorithm match the fusion welding additive process and deeply explore the morphological quality characteristics of the molten pool to achieve accurate detection has become an important research direction.

[0006] Therefore, a fusion welding additive anomaly detection method based on unsupervised learning is needed to solve the above problems. Summary of the invention

[0007] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a method for detecting anomalies in fusion welding additives based on unsupervised learning.

[0008] A method for detecting abnormalities in fusion welding additives based on unsupervised learning comprises the following steps:

[0009] 1. In the process of fusion welding additive, the molten pool image is captured by an industrial camera to obtain a molten pool data set;

[0010] 2. Using the melt pool data set obtained in step 1 to train an unsupervised learning anomaly detection model, the anomaly detection model includes a forward knowledge distillation process and a reverse knowledge distillation process;

[0011] 3. Use the anomaly detection model trained in step 2 to perform inference and obtain the anomaly detection results.

[0012] Furthermore, the industrial camera is replaced by an active lighting camera, which is arranged on a robotic arm of the welding equipment. The active lighting camera includes a camera, a narrow-band pass filter and a monochromatic laser. The narrow-band pass filter is arranged in front of the lens of the camera, and the monochromatic laser illuminates the imaging object.

[0013] The narrow bandpass filter blocks almost all process light from entering the camera sensor, except for the light emitted in the narrow transmission band of the filter. The laser light emitted by the monochromatic laser illuminates the imaging object, and the laser illumination only needs to be greater than the process light intensity in the narrow transmission band of the filter. The active illumination camera is fixed on the mechanical arm of the welding robot, and the active illumination camera is used to capture the molten pool image during welding.

[0014] Furthermore, the anomaly detection model in step 2 includes a teacher network, a first student network and a second student network.

[0015] The forward knowledge distillation process is the knowledge transfer of the teacher network to the first student network, the teacher network is a network pre-trained on ImageNet for image classification tasks, and the teacher network and the first student network have the same architecture;

[0016] The reverse knowledge distillation process is the knowledge transfer from the teacher network to the second student network, and the teacher network and the second student network are encoder and decoder structures.

[0017] Furthermore, the forward knowledge distillation process constructs a teacher-student network, implicitly models the normal image feature distribution, and obtains the teacher feature and student characteristics .

[0018] Furthermore, the reverse knowledge distillation process constructs an encoding-decoding network, where the teacher network is the encoder and the second student network is the decoder, implicitly models the normal image feature distribution, and obtains the teacher feature and student characteristics .

[0019] Furthermore, the reverse knowledge distillation process also includes a multi-feature fusion module, which first downsamples the shallow features of the teacher network using one or more 3×3 convolutional layers with a step size of 2, and then processes the downsampling results using a 1×1 convolutional layer with a step size of 1 and a ResBlock module.

[0020] Information that helps students decode is retained.

[0021] Furthermore, the loss function of the anomaly detection model in step 2 is expressed as follows:

[0022] ,

[0023] In the formula, and are two hyperparameters used to balance and The impact of is the loss function of the forward knowledge distillation process, is the loss function of the reverse knowledge distillation process;

[0024] ,

[0025] In the formula, L represents the number of network feature layers used, where , is the two-dimensional anomaly map of the lth layer, is the feature map height of the lth layer, is the feature map width of the lth layer, represents the teacher characteristics in the forward knowledge distillation process, represents the student characteristics in the forward knowledge distillation process, represents the teacher features in the reverse knowledge distillation process, Represents the student features in the reverse knowledge distillation process.

[0026] Furthermore, the reasoning process of step 3 utilizes the multi-scale anomaly map fusion module, including the following steps:

[0027] 51. Input the test image into the anomaly detection model, generate 3 feature maps with different resolutions in the forward knowledge distillation process and the reverse knowledge distillation process, and generate a total of 6 feature maps ;

[0028] 52. Add feature maps with the same resolution;

[0029] 53. Use the bilinear interpolation method to adjust the feature maps of different resolutions to the same resolution and multiply them to obtain the final anomaly score map:

[0030]

[0031] in, represents the test image, Represents element-wise multiplication.

[0032] Invention principle: The knowledge distillation method based on unsupervised learning uses a general scheme to implicitly model the feature distribution of normal data. The scheme includes a descriptive teacher network and a randomly initialized student network. During the training process, the student network is trained on normal data in order to imitate the output of the teacher network. During the inference process, assuming that the generalization ability of the student network outside the normal data is very poor, the anomaly score of each input pixel can be obtained through the regression error of the student network.

[0033] Beneficial effects: The fusion welding additive anomaly detection method based on unsupervised learning of the present invention adopts an unsupervised learning algorithm and only requires normal data for training. The network model adopts a multi-student-teacher distillation model. For various morphological and quality anomalies that occur during the welding process, such as cracks, offsets, holes, pores, slag inclusions and other defects, the anomaly detection accuracy can achieve good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the anomaly detection method of fusion welding additive based on unsupervised learning;

[0035] Figure 2 It is a structural diagram of an active lighting camera;

[0036] Figure 3 This is a schematic diagram of the structure of the anomaly detection model for unsupervised learning;

[0037] Figure 4 This is a schematic diagram of the structure of the multi-feature fusion module in the reverse knowledge distillation process;

[0038] Figure 5 The reasoning process of the anomaly detection model, including the multi-scale anomaly map fusion module;

[0039] Figure 6 It is the measurement process of cosine similarity;

[0040] Figure 7 These are various types of molten pool images collected during the fusion welding additive process. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings to more clearly and completely illustrate the technical solutions of the present invention.

[0042] See also Figure 1 and Figure 2 As shown, the fusion welding additive anomaly detection method based on unsupervised learning of the present invention comprises the following steps:

[0043] 1. In the process of fusion welding additive, the molten pool image is captured by an industrial camera to obtain a molten pool data set;

[0044] 2. Use the melt pool data set obtained in step 1 to train an unsupervised learning anomaly detection model. The anomaly detection model includes a forward knowledge distillation process and a reverse knowledge distillation process.

[0045] 3. Use the anomaly detection model trained in step 2 to perform inference and obtain the anomaly detection results.

[0046] See also Figure 2 As shown, the hardware components used in the present invention are an industrial camera and a welding robot. The industrial camera is fixed on the mechanical arm of the welding robot, and the industrial camera is used to capture the molten pool image during welding. Figure 7 These are various types of molten pool images captured by industrial cameras.

[0047] Among them, the industrial camera of the present invention can also use an active lighting camera, which is composed of a camera, a narrow band pass filter and a monochromatic laser. The narrow band pass filter blocks almost all process light from entering the camera sensor, except for the light emitted in the narrow transmission band of the filter. The laser emitted by the monochromatic laser illuminates the imaging object, and the laser illumination only needs to be greater than the process light intensity within the narrow transmission band of the filter. The active lighting camera is fixed on the mechanical arm of the welding robot, and the active lighting camera is used to capture the molten pool image during welding.

[0048] The various types of melt pool images collected are processed and classified. Since an unsupervised learning algorithm is used, the training set consists only of normal melt pool images, and the test set consists of normal melt pool images and abnormal melt pool images.

[0049] Model design and training:

[0050] The design of the multi-student-teacher framework model is particularly critical. The core of the model lies in whether the student network can effectively learn the output of the teacher network for normal data features, thereby improving the ability to detect abnormal data.

[0051] Specific design:

[0052] The framework diagram of the multi-student-teacher network is as follows Figure 3 shown.

[0053] First, the forward knowledge distillation process. The forward knowledge distillation stage is the knowledge transfer of the teacher network to the first student network. The teacher network is a network pre-trained on ImageNet for image classification tasks. To reduce information loss, the student network uses the same architecture as the teacher network. At the same time, during the distillation process, we only update the parameters of the student and freeze all the parameters of the teacher to ensure that the TS model achieves the best convergence effect. The feature tensor of the lth layer output of the teacher network and the first student network is and .

[0054] Second, the reverse knowledge distillation process. The reverse knowledge distillation stage is the knowledge transfer of the teacher network to the second student network. The teacher and the student are the encoder and decoder structures. The feature tensor of the lth layer output of the teacher network and the second student network is and .

[0055] In the reverse knowledge distillation process, the present invention proposes a multi-feature fusion module, see Figure 4 . It aggregates both low-level and high-level features of the teacher model, building a rich embedding for the reconstruction of normal data features. A trainable first-class embedding module is introduced to project the high-dimensional representation of the fused teacher network into a low-dimensional space. In order to align the representations in the feature cascade, the shallow features are downsampled by one or more 3×3 convolutional layers (conv-3) with a stride of 2, and then a 1×1 convolutional layer (conv-1) with a stride of 1 is used to obtain rich and compact features, and the information that is beneficial to the student network for decoding is retained through a first-class embedding module (Res Block).

[0056] Third, cosine similarity loss. In the process of knowledge distillation, the present invention uses cosine similarity loss as distillation loss. Cosine similarity can measure the possibility of abnormality, because the cosine similarity of the output features of the teacher model and the student model is lower in the abnormal area and higher in the normal area. Figure 6 This explains why cosine similarity can be used to search for anomalies. The angle of the feature vector in the abnormal area is larger than that in the normal area, and a larger angle means a lower cosine similarity. Therefore, cosine similarity is useful in searching for abnormal areas.

[0057] The loss function formula of forward knowledge distillation is expressed as follows:

[0058] ,

[0059] Considering multi-scale knowledge distillation, we Accumulate to get the average scalar loss function:

[0060] ,

[0061] The loss function formula of reverse knowledge distillation is expressed as follows:

[0062] ,

[0063] Wherein, L represents the number of network feature layers used in the experiment, and L=3 in the present invention.

[0064] The total loss function during the training phase is:

[0065] ,

[0066] in, and are two hyperparameters used to balance the influence of the two parts.

[0067] See also Figure 5 As shown in Figure 2, the model reasoning process uses a multi-scale anomaly map fusion module and includes the following steps:

[0068] 51. Pass the test image into the model and generate 3 feature maps with different resolutions in each of the two stages, for a total of 6 feature maps ;

[0069] 52. Add feature maps with the same resolution;

[0070] 53. Use bilinear interpolation to adjust the feature maps of different resolutions to the same resolution and multiply them to obtain the final anomaly score map. The formula is as follows:

[0071] ,

[0072] in, represents the test image, Represents element-wise multiplication.

[0073] Training process:

[0074] The first step is model training. The network training environment configuration in the example of the present invention is Windows 11, the CPU is Intel core-i5 12400F, the graphics card is Nvidia GeForce RTX-4060, and the memory is DDR4@2666MHz 32GB.

[0075] In the second step, the present invention verifies the effectiveness and superiority of the designed fusion welding additive anomaly detection model through data set testing and comparative experiments.

[0076] In the third step, the present invention verifies the advantages of the fusion welding additive anomaly detection model in the present invention compared with other anomaly detection models through comparative experiments.

[0077] It can be seen from Table 1 and Table 2 that the fusion welding additive anomaly detection model designed in the present invention has superior and accurate detection capabilities for anomalies generated during the welding process.

[0078] Table 1 Performance of the proposed method compared with other models on the hole defect dataset

[0079]

[0080] Table 2 Performance of the present invention compared with other models on the overpenetration dataset

[0081]

[0082] Table 3 Performance of the present invention on public datasets

[0083]

[0084] It can be seen from Table 3 that the model designed by the present invention performs well in public datasets and has a certain degree of universality.

[0085] The knowledge distillation method based on unsupervised learning implicitly models the feature distribution of normal data using a general scheme. The scheme consists of a descriptive teacher network and a randomly initialized student network. During training, the student network is trained on normal data in order to mimic the output of the teacher network. During inference, assuming that the generalization ability of the student network is poor beyond normal data, the anomaly score of each input pixel can be obtained through the regression error of the student network.

[0086] The fusion welding additive anomaly detection method based on unsupervised learning of the present invention adopts an unsupervised learning algorithm and only requires normal data for training. The network model adopts a multi-student-teacher distillation model. The anomaly detection accuracy can achieve good results for various morphological and quality anomalies that occur during the welding process, such as cracks, offsets, holes, pores, slag inclusions and other defects.

Claims

1. A method for detecting abnormalities in fusion welding additives based on unsupervised learning, characterized in that: The following steps are involved:

1. In the process of fusion welding additive, the molten pool image is captured by an industrial camera to obtain a molten pool data set; 2. Using the melt pool data set obtained in step 1 to train an unsupervised learning anomaly detection model, the anomaly detection model includes a forward knowledge distillation process and a reverse knowledge distillation process; The anomaly detection model in step 2 includes a teacher network, a first student network, and a second student network. The forward knowledge distillation process is the knowledge transfer of the teacher network to the first student network, the teacher network is a network pre-trained on ImageNet for image classification tasks, and the teacher network and the first student network have the same architecture; The reverse knowledge distillation process is the knowledge transfer from the teacher network to the second student network, and the teacher network and the second student network are encoder and decoder structures; The forward knowledge distillation process constructs a teacher-student network and implicitly models the normal image feature distribution to obtain the teacher feature and student characteristics ; The reverse knowledge distillation process constructs an encoding-decoding network, where the teacher network is the encoder and the second student network is the decoder, implicitly models the normal image feature distribution, and obtains the teacher feature and student characteristics ; The reverse knowledge distillation process also includes a multi-feature fusion module, which first downsamples the shallow features of the teacher network using one or more 3×3 convolutional layers with a step size of 2, and then processes the downsampled results using a 1×1 convolutional layer with a step size of 1 and a ResBlock module; 3. Use the anomaly detection model trained in step 2 to perform inference and obtain anomaly detection results; The reasoning process of step 3 uses the multi-scale anomaly map fusion module and includes the following steps:

51. Input the test image into the anomaly detection model, generate 3 feature maps with different resolutions in the forward knowledge distillation process and the reverse knowledge distillation process, and generate a total of 6 feature maps ; 52. Add feature maps with the same resolution; 53. Use the bilinear interpolation method to adjust the feature maps of different resolutions to the same resolution and multiply them to obtain the final anomaly score map: , in, represents the test image, Represents element-wise multiplication.

2. The method for detecting abnormalities in fusion welding additives based on unsupervised learning according to claim 1, characterized in that: The loss function of the anomaly detection model in step 2 is expressed as follows: , In the formula, and are two hyperparameters used to balance and The impact of is the loss function of the forward knowledge distillation process, is the loss function of the reverse knowledge distillation process; , In the formula, L represents the number of network feature layers used, where , is the two-dimensional anomaly map of the lth layer, is the feature map height of the lth layer, is the feature map width of the lth layer, represents the teacher characteristics in the forward knowledge distillation process, represents the student characteristics in the forward knowledge distillation process, represents the teacher features in the reverse knowledge distillation process, Represents the student features in the reverse knowledge distillation process.