Additive manufacturing process defect detection method based on multi-sensor data fusion

Abstract

The invention belongs to the technical field of additive manufacturing, and particularly relates to an additive manufacturing process defect detection method based on multi-sensor data fusion. According to the method, multi-modal information such as a temperature field signal, a molten pool signal, a spectral signal and a sound signal in an electric arc additive manufacturing process is combined; special elements are monitored on line in the printing process, a neural network of a Transform architecture is adopted for data processing, a multi-head attention mechanism in the Transform network can effectively capture data correlation in a global range, the advantage of signal complementation between different sensors can be better played, and the printing accuracy is improved. Effective detection of the surface appearance and internal defects of the formed weld bead in the electric arc additive manufacturing process is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of additive manufacturing, and in particular relates to an additive manufacturing process defect detection method based on multi-sensor data fusion. Background Art

[0002] In recent years, additive manufacturing technology has developed rapidly around the world, becoming a research hotspot in countries around the world and an intelligent manufacturing technology that occupies the commanding heights of the new round of scientific and technological innovation competition. It is regarded as one of the disruptive technologies that will give birth to the fourth industrial revolution and determine the future economy. Laser-arc composite additive manufacturing technology, based on arc additive technology, further improves the deposition efficiency and is suitable for the manufacture of large complex parts such as large aerospace and ships. However, the complex metallurgical nature of metal additive manufacturing causes the forming quality of the workpiece to be affected by a variety of process parameters, and the stability and quality reliability of its process quality are still the main reasons that limit its widespread application in high-tech fields. In addition, uneven material composition, inappropriate real-time process parameters, and fluctuations in external working conditions are prone to defects such as pores, inclusions, cracks, and deformation; at the same time, the layer-by-layer accumulation characteristics of metal additive manufacturing make the current layer affect the formation of the next layer, and even cause the final printed parts to fail. Therefore, it is very important to use effective information of all stages and types in the metal additive manufacturing process to construct a high-precision and efficient online monitoring model.

[0003] The invention patent with publication number CN117849054A proposes a symmetry detection system and method for arc additive manufacturing defects. This method uses a camera to collect images of the deposited layer, combined with a deep learning method and camera calibration, to achieve the accuracy of defect recognition and to determine the number and symmetry of defects on both sides of the deposited layer. Its disadvantage is that only an optical camera is used to collect signals, but the strong arc light in the arc additive manufacturing process will cause serious interference to the image quality.

[0004] The invention patent with the publication number CN116559196A proposes a data fusion additive manufacturing detection system and method, which uses magneto-optical imagers, infrared cameras, high-speed cameras and acoustic sensors to collect data, extract surface morphology information, temperature field information, acoustic signals, and magnetic signal features, uses Adam optimizer to optimize, and uses convolutional neural networks and recurrent neural networks to build deep learning networks for data processing to achieve defect recognition and classification. Its shortcomings are that it directly uses long short-term memory networks to process acoustic signals, ignoring the frequency domain information of the sound; on the other hand, long short-term memory networks and recurrent neural networks need to process input features sequentially, which cannot fully utilize the characteristics of GPU parallel computing and is not conducive to the training of large data sets. Summary of the invention

[0005] In view of the technical problems of insufficient signal collection and data processing in existing additive manufacturing defect detection methods, the present invention proposes an additive manufacturing process defect detection method which utilizes an infrared thermal imager, a molten pool camera, a spectrometer and a microphone as signal acquisition equipment to effectively detect the surface morphology and internal defects of the formed weld during the printing process.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] The defect detection method of additive manufacturing process based on multi-sensor data fusion includes the following steps:

[0008] (1) Use a molten pool camera to capture images of the molten pool area, use an infrared thermal imager to capture infrared images directly below the welding torch, use a spectrometer to extract elemental spectra of hydrogen and argon elements to obtain characteristic spectra, and use a microphone to collect sound signals;

[0009] (2) The molten pool area image and the infrared image are processed to obtain the same number of frames, and the sound signal and the spectral signal are divided according to the number of frames of the molten pool area image and the infrared image to obtain each frame signal. Then, the data is labeled and divided into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0010] (3) Extract the Mel-frequency cepstral coefficient of the sound signal to obtain the corresponding time-Mel-frequency cepstral coefficient image; crop the redundant areas of the molten pool area image and the infrared image, and only retain the molten pool weld area of ​​the infrared thermal imager and the molten pool area of ​​the molten pool camera to obtain the ROI image; then fill the outer areas of the time-Mel-frequency spectral coefficient image, the infrared image, and the molten pool area image with blanks to make the three images of the same size;

[0011] (4) Build a convolutional neural network + Transformer neural network;

[0012] (5) The ROI image and the time-Mel spectrum coefficient image are used as the input values ​​of the convolutional neural network, and the convolutional neural network is used to perform preliminary feature extraction; the extracted two-dimensional features are flattened to become one-dimensional vector features, and then the characteristic spectrum lines are added to the end of the features to form a new feature vector, and then an additional set of class tokens are added as subsequent classification input features. After the entire variable is positionally encoded, it is used as the input value of the Transformer network. After the Transformer model, a multi-layer perceptron and a softmax layer are connected as the classification output of the entire model;

[0013] (6) Use the divided training set and validation set to test the effectiveness and accuracy of the model. After its effectiveness and accuracy are guaranteed, use the test set to confirm whether the model is overfitting.

[0014] Preferably, the sound information processing process in step (4) is to further divide each frame of the sound signal into frames, and then perform Fourier transform on it. The transformed spectrum data passes through a Mel filter group to obtain Mel energy. Finally, the energy logarithm is discrete cosine transformed to obtain the MFCC coefficient. The MFCC coefficients of different subframes are mapped into a grayscale image to obtain the MFCC image of the frame signal.

[0015] Preferably, during further frame processing, a Hamming window is used to perform windowing processing on the sub-frame sound signal after the frame division.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are:

[0017] 1. The present invention combines multimodal information such as temperature field signals, molten pool signals, spectral signals and sound signals in the arc additive manufacturing process to achieve effective detection of the surface morphology and internal defects of the formed weld in the arc additive manufacturing process.

[0018] 2. Use a spectrometer to extract the elemental spectra of hydrogen and argon to obtain characteristic spectra, and perform online monitoring of special elements during the printing process. This can detect internal defects caused by moisture on the welding wire or equipment lubricating grease invading the weld, making up for the problem that conventional methods are difficult to detect such defects, and effectively expanding the detection dimension.

[0019] 3. The Transformer neural network is used for data processing. The multi-head attention mechanism in the Transformer network can effectively capture the data correlation on a global scale and better leverage the signal complementarity between different sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the structure of the detection system based on the defect detection method of the additive manufacturing process of the present invention;

[0021] Figure 2 A network structure diagram of the defect detection method for additive manufacturing process of the present invention;

[0022] In the above figures: 1. substrate; 2. printing component; 3. infrared thermal imager; 4. welding gun; 5. molten pool camera; 6. microphone spectrum probe; 7. spectrum probe; 8. data acquisition card; 9. spectrometer; 10. computer. DETAILED DESCRIPTION

[0023] In order to better understand the present invention, a detailed description is given below in conjunction with the accompanying drawings and embodiments.

[0024] Example

[0025] A defect detection method for additive manufacturing process based on multi-sensor data fusion, which is based on the detection system such as Figure 1 As shown, it includes an infrared thermal imager 3, a welding gun 4, a molten pool camera 5, a microphone 6, a spectrometer probe 7, a data acquisition card 8, a spectrometer 9 and a computer 10. The infrared thermal imager 3 and the molten pool camera 5 are directly connected to the computer 10, the microphone 6 is connected to the computer 10 through the data acquisition card 8, and the spectrometer 9 probe is connected to the computer 10.

[0026] like Figure 2 As shown, the defect detection method comprises the following steps:

[0027] 1. When printing, install the substrate 1 on the printing tooling platform, start printing the printed component 2, use the molten pool camera to capture the image of the molten pool area, use the infrared thermal imager to capture the infrared image directly below the welding gun, use the spectrometer to extract the elemental spectrum of hydrogen and argon elements to obtain the characteristic spectrum, and use the microphone to collect the sound signal.

[0028] Residual water vapor during the printing process can cause potential defects, so consider monitoring the hydrogen element contained in the water. Since the hydrogen atom emits an electromagnetic wave with a wavelength of 656.27nm when it transitions from the second excited state to the first excited state, it is easy to obtain and according to the NIST database, HI 656.27nm has the largest relative intensity, so this characteristic spectral line is selected. At the same time, in order to avoid the false alarm of a single hydrogen element spectral line, the Ar I 675.28nm spectral line is added, because its wavelength is close to the HI 656.27nm spectral line and it also has a high relative intensity.

[0029] 2. Perform preliminary preprocessing on the information collected by the infrared thermal imager 3, the molten pool camera 5, the microphone 6 and the spectrometer 9. Specifically, the molten pool area image and the infrared image collected by the infrared thermal imager and the molten pool camera are inserted or extracted to obtain the same number of frames. The sound signal and the spectral signal are divided according to the number of frames of the molten pool area image and the infrared image to obtain each frame signal, and then the data is marked and divided into a training set, a validation set and a test set in a ratio of 7:2:1.

[0030] 3. Perform time-frequency two-dimensional processing on the sound signal: For each frame of the sound signal, further perform frame processing. In order to prevent spectrum leakage, the sub-frame sound signal after the frame is divided is windowed, and the present embodiment uses a Hamming window. Thereafter, Fourier transform is performed on it, and the transformed spectrum data passes through the Mel filter group to obtain the Mel energy. Finally, the energy logarithm is discrete cosine transformed, that is, the MFCC coefficient. The MFCC coefficients of different sub-frames are mapped to a grayscale image, that is, the MFCC image of the frame signal is obtained.

[0031] The infrared image and the molten pool image are cropped to remove the redundant areas, and only the molten pool weld area of ​​the infrared image and the molten pool area of ​​the molten pool image are retained as the ROI part. Then the outer areas of the sound time-frequency image, infrared image and molten pool image are filled with blanks to ensure the consistency of the image sizes of the three. Considering that the molten pool image, infrared image and time-Mel cepstrum image of the acoustic signal are all grayscale images, their pixel sizes are unified to 96*96 through interpolation filling or downsampling.

[0032] 4. Build the framework of convolutional neural network + Transformer neural network. And set the corresponding hyperparameters. In the convolutional neural network part, the first block uses a single convolution layer with a convolution kernel of 3, a stride of 3, and a channel number of 32. In the second block, the first convolution layer adopts a residual structure of 3 convolutions. The size of the convolution kernel of the first layer is 7, the padding is 3, and the convolution is followed by a layer normalization. The size of the convolution kernel of the second convolution layer is 1, the number of channels is 128, and it is followed by a GELU activation function. The size of the convolution kernel of the third layer is 1, the number of channels is 32, and the output of this layer is added to the input of this block. The third block uses a convolution layer with a convolution kernel of 1 and a channel of 3. The output of the convolution part is 32*32*3.

[0033] In the Transformer structure, the 32*32*3 image obtained by convolution is divided into 8*8*3 size and two spectral features are flattened. That is, the input token dimension is 194, the batch is 16, the number of attention heads is selected as 2, and the encoder and decoder are selected as 2 layers.

[0034] The multilayer perceptron uses 194 as the input dimension, the number of neurons in the first layer is 32, the number of neurons in the second layer is 8, and the number of neurons in the third layer is 4. The learning rate is 0.001, and the adam optimizer is used.

[0035] 5. The time-frequency image of the sound, the infrared thermal imager, and the ROI image of the molten pool camera are used as the input of the convolutional neural network, and the convolutional neural network is used to perform preliminary feature extraction. The extracted two-dimensional features are flattened and transformed into one-dimensional vector features; then the characteristic spectral lines of the spectrum screening are added to the end of the feature to form a new feature vector. An additional set of class tokens are added as subsequent classification input features.

[0036] After position encoding, the entire variable is used as the input of the Transformer network. After the Transformer model, it is connected to a multi-layer perceptron and a softmax layer as the classification output of the entire model. The use of multiple input features can more comprehensively reflect the online printing information of the additive manufacturing process. The introduction of convolutional neural networks can perform preliminary extraction of image features, reduce the amount of data, and reduce the computational load. The transformer network can better control global information and has a better understanding of the abstract time-frequency images converted from acoustic signals.

[0037] 6. Use the divided training set and validation set to test the effectiveness and accuracy of the model. After its effectiveness and accuracy are guaranteed, use the test set to confirm whether the model is overfitting. The Transformer architecture often has a large number of parameters, which is prone to overfitting when there is insufficient training data, resulting in low versatility of the model. By setting a validation set, the overfitting ability of the model can be effectively monitored on the validation set, and the hyperparameters can be adjusted in time to prevent overfitting and select the optimal hyperparameters.

[0038] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A defect detection method for additive manufacturing process based on multi-sensor data fusion, characterized in that: The following steps are involved: (1) Use a molten pool camera to capture images of the molten pool area, use an infrared thermal imager to capture infrared images directly below the welding torch, use a spectrometer to extract elemental spectra of hydrogen and argon elements to obtain characteristic spectra, and use a microphone to collect sound signals; (2) The molten pool area image and the infrared image are processed to obtain the same number of frames, and the sound signal and the spectral signal are divided according to the number of frames of the molten pool area image and the infrared image to obtain each frame signal. Then, the data is labeled and divided into a training set, a validation set, and a test set in a ratio of 7:2:1; (3) Extract the Mel-frequency cepstral coefficient of the sound signal to obtain the corresponding time-Mel-frequency cepstral coefficient image; crop the redundant areas of the molten pool area image and the infrared image, and only retain the molten pool weld area of ​​the infrared thermal imager and the molten pool area of ​​the molten pool camera to obtain the ROI image; then fill the outer areas of the time-Mel-frequency spectral coefficient image, the infrared image, and the molten pool area image with blanks to make the three images of the same size; (4) Build a convolutional neural network + Transformer neural network; (5) The ROI image and the time-Mel cepstral coefficient image are used as the input values ​​of the convolutional neural network, and the convolutional neural network is used to perform preliminary feature extraction; the extracted two-dimensional features are flattened to become one-dimensional vector features, and then the characteristic spectrum lines are added to the end of the features to form a new feature vector, and then an additional set of class tokens are added as subsequent classification input features. After the entire variable is positionally encoded, it is used as the input value of the Transformer network. After the Transformer model, a multi-layer perceptron and a softmax layer are connected as the classification output of the entire model; (6) Use the divided training set and validation set to test the effectiveness and accuracy of the model. After its effectiveness and accuracy are guaranteed, use the test set to confirm whether the model is overfitting.

2. The additive manufacturing process defect detection method based on multi-sensor data fusion according to claim 1, characterized in that: The sound information processing process in step (4) is to further divide each frame of the sound signal into frames, then perform Fourier transform, pass the transformed spectrum data through the Mel filter group to obtain Mel energy, and finally perform discrete cosine transform on the energy logarithm, that is, MFCC coefficients. The MFCC coefficients of different subframes are mapped into grayscale images to obtain the MFCC image of the frame signal.

3. The additive manufacturing process defect detection method based on multi-sensor data fusion according to claim 2, characterized in that: When further framing is performed, the sub-frame sound signal after framing is subjected to windowing processing using a Hamming window.

Images