Metal additive manufacturing defect detection method and system based on generative adversarial network
By generating adversarial networks to convert acoustic signals into images and perform feature matching and fusion, the accuracy and efficiency issues of defect detection in arc additive manufacturing are solved, and high-precision defect recognition is achieved, which is suitable for a variety of additive manufacturing methods.
Patent Information
- Application Number
- CN202510930223.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-07
- Publication Date
- 2025-10-10
AI Technical Summary
Existing arc additive manufacturing defect detection methods mainly rely on molten pool image monitoring, which has poor environmental adaptability, limited viewing angle and lack of comprehensiveness, resulting in the omission of local defects. In addition, the acoustic signal acquisition and processing process is cumbersome, making it difficult to achieve high-precision defect identification.
A generative adversarial network-based method is used to convert one-dimensional acoustic signals into two-dimensional images through the Markov transfer field, and a conditional generative adversarial network is used to generate a pseudo-Markov image, which is then matched and fused with the melt pool image to identify the defect type.
It realizes high-precision defect identification without real-time acquisition of acoustic signals, improves the accuracy and stability of defect identification, is applicable to various additive manufacturing methods, and has good promotion value.
Smart Images
Figure CN120766033A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to additive manufacturing defect detection technology, and in particular to a metal additive manufacturing defect detection method and system based on a generative adversarial network. BACKGROUND
[0002] Directed Energy Deposition with Arc (DED-Arc) is a direct energy deposition process based on arc welding. Compared with other additive manufacturing technologies, DED-Arc has unique advantages such as high deposition rate, low manufacturing cost, and high material utilization rate. Although DED-Arc technology has made great progress in recent years, since its forming process is essentially based on arc welding technology, it is subject to many interference factors during the deposition process, such as environmental changes, process parameter fluctuations, and metallurgical chemical properties, resulting in many uncertainties. Therefore, it is difficult to determine the current state of the material deposition process, which indirectly leads to inaccurate control, resulting in low precision and poor quality of the formed parts. This is one of the biggest challenges limiting the development and widespread industrial application of DED-Arc and other metal additive manufacturing technologies. Therefore, real-time detection and accurate identification of defects during the manufacturing process are crucial.
[0003] Existing DED-Arc defect detection methods mainly rely on molten pool image monitoring. However, molten pool images have poor environmental adaptability, are limited by viewing angle, and lack comprehensiveness, which can easily lead to the omission of local defects. Using molten pool images alone cannot effectively obtain all the features of the current state. In contrast, acoustic signals have comprehensiveness, real-time nature, and non-contact nature, and can reflect changes in process parameters and the formation mechanism of defects, providing an accurate, real-time, and reliable means for defect monitoring. At the same time, recent research has shown that fusing information from different sources during metal additive manufacturing can obtain more features of the current state, thereby achieving better recognition results. Therefore, obtaining and fusing multiple sources of information is currently a more mainstream approach.
[0004] Currently, it is cumbersome to collect acoustic signals and convert them into two-dimensional images. Various noises are generated during the collection process, and noise filtering is required. Noise that cannot be filtered out also needs to be denoised and other various treatments. After the current experiment processes the acoustic signals, new experiments need to be processed, resulting in a relatively cumbersome process for collecting acoustic signals each time. SUMMARY
[0005] To solve the problems of the prior art, the present application aims to provide a metal additive manufacturing defect detection method and system based on a generative adversarial network, which can achieve high detection accuracy without collecting acoustic signals each time.
[0006] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:
[0007] A metal additive manufacturing defect detection method based on a generative adversarial network comprises the following steps:
[0008] S1: By setting different process parameters, the acoustic signals and melt pool images of various defect types in additive manufacturing printing are extracted and pre-processed;
[0009] S2: The pre-processed acoustic signals of different defect types are converted into Markov images through the Markov transfer field;
[0010] S3: Different defect types are used as conditions and input into a trained Markov image generator together with random noise to generate several pseudo-Markov images of different defect types, wherein the Markov image generator is a conditional generator in a conditional generative adversarial network, which is used to generate pseudo-Markov images based on the defect type and random noise, and the discriminator of the conditional generative adversarial network is used to distinguish the authenticity of the pseudo-Markov images from the real Markov images;
[0011] S4: Inputting the molten pool image to be inspected into a trained image feature matching network, wherein the image feature matching network is used to match the molten pool image with several pseudo-Markov images of different defect types generated by the Markov image generator, and select the pseudo-Markov image with the highest similarity for output;
[0012] S5: fusing the pseudo-Markov image output by the image feature matching network with the melt pool image to be detected using a fusion network to obtain a fused image, wherein the fusion network is used to fuse the melt pool image and the pseudo-Markov image into one image;
[0013] S6: Input the fused image into a trained defect recognition network to identify the defect type. The defect recognition network is used to identify the defect type of the input fused image.
[0014] Furthermore, step S1 specifically includes:
[0015] S1-1: By setting different process parameters, the acoustic signals and melt pool images of different defects in additive manufacturing printing are collected;
[0016] S1-2: Perform wavelet decomposition on the acoustic signal to divide it into different frequency bands, remove the corresponding DC component frequency, and then reconstruct the acoustic signal;
[0017] S1-3: Use wavelet denoising algorithm to filter the acoustic signal and filter the background noise during printing;
[0018] S1-4: The acoustic signal is initially and tail cut off, only leaving the acoustic signal of the printing process.
[0019] Further, the step S2 specifically comprises:
[0020] S2-1: Set the sliding window and the sliding step;
[0021] S2-2: According to the preset sliding window and the sliding step, the preprocessed acoustic signal is slid, thereby extracting several acoustic signals;
[0022] S2-3: For each acoustic signal, the acoustic signal is converted into a Markov picture by a Markov transition field, thereby obtaining several Markov pictures.
[0023] Further, the condition generator is specifically a deep convolution generation network (DCGAN) architecture based on transpose convolution, and the discriminator is a deep convolution network.
[0024] Further, the picture feature matching network is specifically a twin network.
[0025] Further, the training process of the picture feature matching network comprises:
[0026] Each collected molten pool picture and a pseudo Markov picture belonging to the same defect type as the molten pool picture are taken as a set of positive sample data, and each collected molten pool picture and any pseudo Markov picture belonging to a different defect type from the molten pool picture are taken as a set of negative sample data; based on the several positive sample data and negative sample data, the picture feature matching network is trained.
[0027] Further, the loss function used when training the picture feature matching network is:
[0028]
[0029] d = ‖X a -X b ‖2
[0030] In the formula, L() represents the loss function, X a ,X b respectively represent the molten pool picture and the pseudo Markov picture in a set of samples, Y represents the matching result, 1 for matching and 0 for mismatching, m is a set safety distance, and d represents the Euclidean distance of X a ,X b .
[0031] Further, the fusion network takes the pseudo Markov picture as the R channel, takes the molten pool picture as the G channel, sets the B channel as a channel value with an intensity lower than a preset threshold, and finally outputs an RGB two-dimensional image.
[0032] Further, the defect identification network adopts Efficientnet-B1 as a backbone network.
[0033] A metal additive manufacturing defect detection system based on a generative adversarial network, the system comprising:
[0034] An information acquisition module is configured to extract acoustic signals and molten pool pictures of various defect types in additive manufacturing printing by setting different process parameters and pre-process the acoustic signals and molten pool pictures.
[0035] An acoustic signal conversion module is configured to convert the pre-processed acoustic signals of different defect types into Markov pictures through a Markov transition field.
[0036] A Markov picture generator is configured to input different defect types as conditions and random noise to generate pseudo Markov pictures of different defect types, wherein the Markov picture generator is a condition generator in a conditional generative adversarial network, the condition generator is configured to generate pseudo Markov pictures according to defect types and random noise, and a discriminator of the conditional generative adversarial network is configured to distinguish the pseudo Markov pictures from real Markov pictures.
[0037] A picture feature matching network is configured to input the molten pool picture to be detected into the trained picture feature matching network, wherein the picture feature matching network is configured to match the molten pool picture with the pseudo Markov pictures of different defect types generated by the Markov picture generator and output the pseudo Markov picture with the highest similarity.
[0038] A fusion network is configured to fuse the pseudo Markov picture output by the picture feature matching network with the molten pool picture to be detected to obtain a fusion picture, wherein the fusion network is configured to fuse the molten pool picture and the pseudo Markov picture into one picture.
[0039] A defect identification network is configured to identify the defect type of the fusion picture.
[0040] Compared with the prior art, the present application has the following advantages:
[0041] 1. The present application does not need to collect acoustic signals, but only needs to match according to the molten pool picture to obtain the corresponding pseudo Markov picture of the molten pool picture, and then identify the defect through the fusion picture of the molten pool picture and the pseudo Markov picture, thereby achieving the effect of enriching the current state features of the picture, and the effect is improved with the continuous expansion of the effective data set, thereby realizing higher defect identification accuracy.
[0042] 2、The application converts one-dimensional sound signals into two-dimensional images through Markov transition field (MTF) technology, thereby effectively mapping the dynamic change characteristics of the sound signals into static images, so that the defect recognition network can directly process complex time sequence data, and the method can not only capture the implicit patterns and dependencies in the time sequence, but also retain the global characteristics of the time sequence through the Markov matrix, including the trend and pattern of state transition, thereby realizing more accurate defect classification.
[0043] 3、The application constructs a conditional GAN based on a WGAN-GP architecture, introduces category information in the generator through conditional batch normalization (CBN) and in the discriminator through a projection discriminator, so that the model can generate images of a specific category, and uses a Wasserstein loss and a gradient penalty to improve training stability and improve the quality of generated images.
[0044] 4、The data set of the application can identify common discontinuities and porosity defects in the printing process, and also has the potential to predict other defects, and only needs to collect sound signals of other defects and convert them into pictures, and fill them into the data set, so that it can generate the ability to generate the defect, and provide more accurate defect classification for identifying the molten pool state in the electric arc additive manufacturing process. This method is also applicable to process defect detection of various additive manufacturing methods such as laser cladding, laser powder bed melting, and electron beam additive manufacturing, and has good popularization value.
[0045] 5、The application only uses the R and G channels of the RGB picture, and the B channel is set to low intensity, which does not affect defect recognition. Therefore, other feature sources can be collected and fused on this basis to further improve the defect recognition accuracy. The generator can also be used to generate other difficult-to-collect information, and the classification accuracy can be improved by fusing the B channel according to the above. BRIEF DESCRIPTION OF DRAWINGS
[0046] Figure 1 is a flowchart of a metal additive manufacturing defect detection method based on a generative adversarial network provided by an embodiment of the application;
[0047] Figure 2 is a structural schematic diagram of a sound signal acquisition device provided by an embodiment of the application;
[0048] Figure 3 is a method schematic diagram of one-dimensional time domain sound signal preprocessing provided by an embodiment of the application;
[0049] Figure 4 is a method schematic diagram of converting one-dimensional time domain sound signals into two-dimensional space domain Markov graphs provided by an embodiment of the application;
[0050] Figure 5 is a training process of a conditional generative adversarial network provided by an embodiment of the application;
[0051] Figure 6 This is a partial flow chart of a metal additive manufacturing defect detection method provided by an embodiment of the present invention;
[0052] Figure 7 Schematic diagram of the training process of the defect recognition network provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.
[0054] Example 1
[0055] The embodiment of the present invention provides a metal additive manufacturing defect detection method based on a generative adversarial network, such as Figure 1 As shown, the method includes the following steps:
[0056] S1: By setting different process parameters, the acoustic signals and melt pool images of various defect types in additive manufacturing printing are extracted and preprocessed.
[0057] This step specifically includes:
[0058] S1-1: By setting different process parameters, the acoustic signals and melt pool images of different defects in additive manufacturing printing are collected.
[0059] like Figure 2 As shown, the acoustic signal acquisition device includes a central controller (1), a sound acquisition card (2), an acoustic signal detector (3), an electric welding gun head (4), a wire feeding and gas feeding port (5), a six-axis robotic arm (6), an argon shielding gas cylinder (7), a welding machine (8) and a control cabinet (9), a Q235 substrate (10), and a workbench (11); the device is installed and connected as shown in the figure; in the arc additive manufacturing process, the dynamic fluctuation of the molten pool, welding spatter and other phenomena will generate specific acoustic signals. When collecting, first, a high-precision acoustic signal detector is used to collect the acoustic signal in the arc additive manufacturing process. The collected acoustic signal is sound pressure data, which is collected using the DAQ module of Labview, wherein the sampling frequency is 40000Hz, and the sound pressure data is saved as a TDMS file. The detector is installed in a side-by-side manner to avoid interference that may affect the sensor, such as high temperature, high humidity and metal spatter. In this embodiment, during the acquisition process, by adjusting the process parameters of arc additive manufacturing, such as current, voltage, shielding gas flow rate, and scanning speed, acoustic signals with three typical defect characteristics are designed: discontinuity defects, porosity defects, and normal state signals.
[0060] The molten pool images are collected using a camera.
[0061] S1-2: Wavelet decomposition is performed on the acoustic signal, the acoustic signal is divided into different frequency bands, and the corresponding direct current component frequency is removed before the acoustic signal is reconstructed by inverse wavelet transform.
[0062] This can eliminate the direct current noise caused by the start and closing of the arc, and ensure that the residual signal better reflects the acoustic characteristics in the actual manufacturing process.
[0063] S1-3: The wavelet denoising algorithm is used to filter the acoustic signal, and the background noise in the printing process is filtered, as shown in Figure 3
[0064] Specifically, the Labview self-contained wavelet denoising module is used to filter the interference noise in the acoustic signal, and the specific parameter configuration is db04 wavelet, soft threshold and four-level decomposition, which filters out the background noise caused by the environmental noise (such as wind, equipment vibration, etc.) in the arc additive manufacturing process. This method can effectively retain the useful components of the signal while suppressing the interference of irrelevant noise.
[0065] S1-4: The acoustic signal is truncated at the beginning and end, leaving only the acoustic signal of the printing process.
[0066] Among them, the noise of arc starting and closing is usually large in amplitude and concentrated, usually located at the beginning and end of the signal. By observing the starting and ending time period of the one-dimensional time domain acoustic signal, the unnecessary open and close arc noise components are truncated, and the effective signal in the welding process is retained. After these preprocessing steps, the final signal will have a higher signal-to-noise ratio, providing clear signal data for subsequent feature extraction.
[0067] S2: The acoustic signals of different defect types after preprocessing are converted into Markov pictures through Markov transition field.
[0068] This step specifically includes:
[0069] S2-1: Set the sliding window and sliding step.
[0070] Specifically, in PyCharm, import the necessary libraries, import TdmsFile, import numpy, MTF library and matplotlib.pyplot library, wherein TdmsFile is used to open the above acoustic signal file saved in Tdms format, and the data of a specific channel is read and converted into a NumPy array. The data is sliced, and a sliding window with a data length of m=20000 is selected, and 20000 data points are extracted each time; the sliding step n=4000.
[0071] S2-2: The preprocessed acoustic signal is slid according to the preset sliding window and sliding step, thereby extracting several acoustic signals.
[0072] According to the preset sliding window and sliding step, the data in each sliding window is extracted as a sound signal segment. Finally, a sufficient number of processed data segments are obtained, and these data segments are independent time series segments.
[0073] S2-3: For each sound signal segment, a Markov picture is generated by converting the sound signal segment into a Markov picture through a Markov transition field, thereby obtaining a plurality of Markov pictures.
[0074] The one-dimensional sound signal is converted into a two-dimensional spatial domain image through the introduced MTF library, as shown in Figure 4 20000 data points are quantized and divided into 8 different quantile boxes. The quantization process uses quantile strategy to quantize the data points, and the frequency of transferring a data point in a quantile box to other quantile boxes (including the quantile box where the data point originally locates) is constructed into a transition frequency matrix W to form a Markov transition field M. The formula of the transition frequency matrix and the Markov transition field is as follows:
[0075]
[0076]
[0077] In the formula, w ij represents the frequency of transferring the quantile box i to the quantile box j, Q represents the quantization level (state number), q i , q j respectively represent the i-th and j-th quantization barrels (states), x1, x N respectively represent the original values of the first to N-th sampling points on the one-dimensional time series signal.
[0078] Matplotlib is used to set the color mapping of the image using rainbow color tones, set the origin of the image at the lower left corner, hide the coordinate axis, so that the image display is clearer, and finally map the Markov transition field into a spatial domain image display with a size of 128x128.
[0079] S3: Different defect types are input into the trained Markov picture generator as conditions together with random noise to generate a plurality of pseudo Markov pictures of different defect types.
[0080] In the formula, w Figure 5As shown, the Markov picture generator is a conditional generator in a conditional generative adversarial network (Conditional GAN). The conditional generative adversarial network is a WGAN-GP (Wasserstein GAN with Gradient Penalty) based architecture. The conditional generator is a transposed convolution based deep convolutional generative network (DCGAN) architecture, and the discriminator is a deep convolutional network (Projection Discriminator). The conditional generator is used to generate pseudo Markov pictures according to defect types and random noise, and the discriminator of the conditional generative adversarial network is used to distinguish the pseudo Markov pictures and the real Markov pictures
[0081] The defect type labels are set to discontinuous, normal, and pore respectively.
[0082] Generator input: noise vector z (shape [batch_size, z_dim]) and class label y (shape [batch_size]), batch_size is the input batch size, and z_dim is the dimension of the noise vector. Generator architecture: DCGAN (Deep Convolutional Generative Network) style architecture based on transposed convolution, and conditional batch normalization (CBN) is added. Running process: label embedding, convert integer class label y to a dense vector y_emb of embedding_dim dimensions, a fully connected layer maps noise z to a high-dimensional vector, and then reshapes it into a small spatial feature map. Here, only z is used, and y_emb is not directly combined. Conditional information is injected through the subsequent CBN layer. Apply conditional batch normalization to the initial feature map to inject class information y_emb. It is composed of multiple ConvTranspose2d layers, which gradually enlarge the spatial size of the feature map (4x4->8x8->16x16->32x32->64x64->128x128), while reducing the number of channels. Each upsampling layer is followed by a ConditionalBatchNorm2d layer (using y_emb) and a LeakyReLU activation function. CBN continuously injects class information here to guide the generation process. The last transposed convolution layer directly outputs 3 channels (corresponding to RGB) with a size of 128x128. Use the Tanh activation function to scale the output value to the range [-1, 1], which is usually matched with the normalization method of the input data. Initialize the weights of the convolution and linear layers using a normal distribution with a mean of 0 and a standard deviation of 0.02.
[0083] The generator formula: G(z, y) = Tanh(ConvTranspose_final(LeakyReLU(CBN_4(ConvTranspose_4(LeakyReLU(CBN_3(ConvTranspose_3(LeakyReLU(CBN_2(ConvTranspose_2(LeakyReLU(CBN_1(ConvTranspose_1(LeakyReLU(CBN_0(Reshape(Linear_initial(z)), E_G(y)))), E_G(y)))), E_G(y)))), E_G(y)))), E_G(y))))) ). Where E_G(y): converts the class label y to an embedding vector. Linear_initial(z): projects the noise z through a fully connected layer. Reshape(...): reshapes the above projection result to the initial 4x4 feature map. CBN_0(H, y_emb): applies the first conditional batch normalization to the feature map H (i.e. the result of Reshape(...)), conditioned on E_G(y). LeakyReLU(...): applies the LeakyReLU activation function. ConvTranspose_1(H_prev): the first transposed convolutional layer, input is the output of the previous layer H_prev. CBN_1(H, y_emb): applies conditional batch normalization to the output of ConvTranspose_1, H. And so on, until ConvTranspose_4 and CBN_4. ConvTranspose_final(H_prev): the last transposed convolutional layer, converts the feature map to a 3-channel image. Tanh(...): the final Tanh activation function, scales the output pixel values to [-1, 1].
[0084] Discriminator input: image x (shape [batch_size, 3, 128, 128]) and class label y (shape [batch_size]). Discriminator architecture: a deep convolutional network based on convolution (Convolution) combined with the idea of spectral normalization (Spectral Normalization) and projection discriminator (Projection Discriminator). Running process: label embedding (label_embedding), the same as the generator, converts y to y_emb, down-sampling block, composed of multiple Conv2d layers, gradually reduces the spatial size of the feature map (128x128->64x64->32x32->16x16->8x8->4x4), while increasing the number of channels, spectral normalization (spectral_norm) is applied to all convolution layers and projection linear layers, spectral normalization stabilizes the training of GAN (especially WGAN) by limiting the Lipschitz constant of the network layer, which helps to meet the requirements of WGAN theory or alleviate mode collapse, a LeakyReLU activation function is used after each convolution layer. Batch normalization (BN) or layer normalization (LN) is not used here because BN may introduce dependencies between samples within a batch, which conflicts with the goal of GP. After a series of convolutions, a feature map h with shape [batch_size, ndf x 16, 4, 4] is obtained, where ndf represents the basic number of discriminator feature maps. Unconditional score (conv_uncond): a final convolution layer (kernel=4, stride=1, padding=0) converts the 4x4 feature map h into a single scalar value uncond_score (shape [batch_size, 1]). This score represents the basic score of the discriminator's judgment of the "authenticity" of the input image, and does not directly depend on the label. Projection discriminator: the final feature map h is compressed into a vector h_pooled (shape [batch_size, ndf x 16]) through adaptive average pooling. The class embedding y_emb is projected to the same dimension as h_pooled through a spectrally normalized linear layer. The inner product between h_pooled and the projected label embedding is calculated. This inner product projection_score (shape [batch_size, 1]) measures the matching degree of image features and given labels.
[0085] The discriminator formula: D(x, y) = SN(Conv_uncond(Backbone(x))) + <AvgPool(Backbone(x)), SN(Linear_proj(E_D(y)))>. Where x: input image (including Markov real image x_real or generated pseudo Markov image x_fake). Y: category label (integer). E_D(y): label embedding layer in the discriminator, which maps the integer label y to a vector y_emb_D of embedding_dim dimensions. SN(Layer(·)): apply spectral normalization (Spectral Normalization) to the weights of Layer. Spectral normalization stabilizes the training of GAN (especially WGAN) by limiting the Lipschitz constant of the network layer. Conv_i(·): the i-th convolutional layer. LeakyReLU(·): Leaky RectifiedLinear Unit activation function. Backbone(x): represents the backbone convolutional network of the discriminator, which is used to extract features from the input image x. h_feat: the final feature map output by Backbone(x) (for example, the output after conv5 in the code, with a shape similar to [batch, ndf x 16, 4, 4]). Conv_uncond(·): a convolutional layer used to calculate the unconditional score from h_feat. It usually maps h_feat to a scalar (or a [batch, 1, 1, 1] tensor, which is then flattened). AvgPool(·): adaptive average pooling layer (Adaptive Average Pooling). It pools the spatial dimensions of each channel of the feature map h_feat into a value, resulting in a vector. Linear_proj(·): a fully connected layer used to project the label embedding y_emb_D into a space with the same dimensions as AvgPool(h_feat). <A, B>: represents the inner product between vectors A and B. If A and B are vectors of the same length, then <A, B> = Σ(A i ×B i ).
[0086] Through the joint training of the conditional generator and the discriminator, the conditional generator can generate corresponding pseudo Markov pictures according to the defect category label. Therefore, the labels of different defect types are input into the trained Markov picture generator as conditions together with random noise to obtain a number of pseudo Markov pictures of different defect types for subsequent matching.
[0087] S4: input the molten pool picture to be detected into the trained picture feature matching network.
[0088] The picture feature matching network is used to match the molten pool picture and the pseudo Markov picture of several different defect types generated by the Markov picture generator, and output the pseudo Markov picture with the highest similarity. The picture feature matching network is specifically a twin network. The training process of the picture feature matching network includes:
[0089] Each molten pool picture collected and the pseudo Markov picture belonging to the same defect type as the molten pool picture are taken as a set of positive sample data, and each molten pool picture collected and any pseudo Markov picture belonging to a different defect type from the molten pool picture are taken as a set of negative sample data; based on the several positive sample data and negative sample data, the picture feature matching network is trained; and the loss function used is:
[0090]
[0091] d = ‖X a -X b ‖2
[0092] In the formula, L() represents the loss function, X a ,X b respectively represent the molten pool picture and the pseudo Markov picture in a set of sample data, Y represents the matching result, 1 for matching and 0 for not matching, m is a set safety distance, and d represents the Euclidean distance of X a ,X b . When the distance of (X a ,X b ) is less than m, the loss function will become 0, which makes X a ,X b similar but not the same, and can ensure the generalization ability of the algorithm.
[0093] S5: The pseudo Markov picture output by the picture feature matching network is fused with the molten pool picture to be detected by using the fusion network to obtain a fusion picture. As shown in Figure 6 .
[0094] The fusion network is used to fuse the molten pool picture and the pseudo Markov picture into one picture. The picture fusion network mainly has three functions: first, the load_generator function: load the pre-trained generator weight of the specified path (model_path). It will get the model parameters (z_dim, num_classes, ngf, embedding_dim) from the config to instantiate the Generator class. Load weight: try to load the model_path. It can handle two cases: one is that the file is directly a model state dictionary, and the other is that the file is a complete checkpoint dictionary containing the generator_state_dict key. Error handling: if there is a size mismatch error (for example, the loaded model structure does not completely match the Generator defined in the current code) when loading, it will try to load only the weight layer with matching name and size, and print the loading information. Set the model to evaluation mode (generator.eval()). Second, the get_zrct_images function: randomly select num_images real images from the specific class subdirectory (class_name) under the specified zrct_root directory. Use similar image transformations (Resize, ToTensor, Normalize) as in training to load and preprocess these images. Return the loaded image batch tensor and the corresponding original file path list. These images will be used as the G channel in the fusion process. Third, the generate_fused_images function: set: create the output directory, set the device. Load the model: call load_generator to load the pre-trained model. Generate by class: iterate over each class in config['sub_dirs']: 1. Create the label tensor for this class (labels). 2. Generate random noise z. 3. Generate R channel images using the loaded generator (r_channel_images = generator(z, labels)). 4. Call get_zrct_images to load real images from the ZRCT directory for the corresponding class as the G channel (g_channel_images). 5. Call the fuse_images function to create fused RGB images (fused_images) using the generated R channel, loaded G channel, and b_channel_intensity in the configuration. 6. Save the images: save the generated R channel images, G channel images from ZRCT, and the final fusion images respectively. Use the save_image function, note that ×0.5+0.5 is to inverse normalize the [-1, 1] range image to the range in order to save as a standard image file. 7. Print the saving information to indicate the source of each channel.
[0095] S6: input the fusion picture into the trained defect recognition network to identify the defect type.
[0096] The defect recognition network is used to identify the defect type of the input fusion picture.
[0097] As shown in Figure 7 During training, the data set is divided into a training set, a validation set, and a test set in a ratio of 7:1:2. Image enhancement is performed during training, and the enhancement methods include random cropping, horizontal flipping, normalization, and random erasing. Then the input is input into the defect recognition network for training. The defect recognition network uses Efficientnet-B1 as the backbone network for feature extraction and learning of the input training set. The AdamW optimizer is used. The CosineAnnealingLrUpdater is used. The training set is input into the defect recognition network for training, and a preliminary image classification model is obtained after the defect recognition network is trained. The validation set is input into the defect recognition network for validation, and the hyperparameters are adjusted according to the recognition and classification effect of the sound signal features in the image. After adjusting the final parameters, the test is performed using the real MTF picture data set as the standard. All subsequent different types of data sets are trained and tested using the hyperparameters.
[0098] The performance of the model is verified by the validation set, and the hyperparameters are adjusted according to the performance to make the model achieve ideal performance and obtain the final image classification model. The performance parameters include Precision, Recall, and F1-Score, and the formulas are as follows:
[0099]
[0100] In the formula, precision represents the accuracy rate, recall represents the recall rate, True Positive (TP) is the number of samples predicted as i and actually as i, and False Negative (FN) is the number of samples actually as i but predicted as other categories.
[0101] The defect recognition results and the confusion matrix verified by the real MTF picture data set are shown in Tables 1 and 2. The number of continuous, normal, and air hole defects is 493, 524, and 541, respectively. According to the high precision of defect recognition, it is shown that the data set collection has provided sufficient information.
[0102] Table 1 Defect recognition results:
[0103] Class Precision Recall F1-Score Discontinuous 96.04 98.98 97.49 Normal 99.02 97.12 98.06 Air hole 99.07 98.15 98.6
[0104] Table 2 Confusion matrix:
[0105] Discontinuous Normal Air hole Discontinuous 97 1 0 Normal 2 101 1 Air hole 2 0 106
[0106] The defect recognition results and confusion matrix using the pseudo Markov picture dataset for verification are shown in Table 3 and Table 4, and the number of discontinuity, normal and porosity is 600, 599 and 599 respectively. The accuracy of defect recognition reaches a very high accuracy, indicating that the generated pseudo Markov picture contains sufficient feature information.
[0107] Table 3 Defect recognition results:
[0108] Class Precision Recall F1-Score Discontinuous 99.17 100.0 99.59 Normal 100.0 99.16 99.58 Air hole 100.0 100.0 100.0
[0109] Table 4 Confusion matrix:
[0110] Discontinuous Normal Air hole Discontinuous 120 0 0 Normal 1 118 0 Air hole 0 0 119
[0111] The results and confusion matrix using the real molten pool picture dataset for verification are shown in Table 5 and Table 6, and the number of discontinuity, normal and porosity is 600, 599 and 599 respectively.
[0112] The accuracy of defect recognition also reaches a very high accuracy, indicating that the real molten pool picture contains sufficient feature information.
[0113] Table 5 Defect recognition results:
[0114] Class Precision Recall F1-Score Discontinuous 100.0 100.0 100.0 Normal 100.0 99.16 99.58 Air hole 99.17 100.0 99.58
[0115] Table 6 Confusion matrix:
[0116] Discontinuous Normal Air hole Discontinuous 120 0 0 Normal 0 118 1 Air hole 0 0 119
[0117] It can be observed that under the same defect recognition model and the same hyperparameters, the above recognition accuracy is very high, but there are still misclassified errors, and there are certain features that cause the model to be unable to recognize. As shown in Table 7 and Table 8, the results and confusion matrix using the fusion picture dataset for verification both reach 100%, indicating that this method makes the fusion picture contain sufficient information, so that the model can well extract and recognize the defect features in the image and accurately classify. The following data uses the same model and dataset as above.
[0118] Table 7 Defect recognition results:
[0119] Class Precision Recall F1-Score Discontinuous 100.0 100.0 100.0 Normal 100.0 100.0 100.0 Air hole 100.0 100.0 100.0
[0120] Table 8 Confusion matrix:
[0121] Discontinuous Normal Air hole Discontinuous 120 0 0 Normal 0 119 0 Air hole 0 0 119
[0122] Example Two
[0123] The embodiment of the present application provides a metal additive manufacturing defect detection system based on a generative adversarial network, and the system comprises:
[0124] An information acquisition module is used for acquiring acoustic signals and molten pool pictures of various defect types in additive manufacturing printing and performing preprocessing;
[0125] An acoustic signal conversion module is used for converting the acoustic signals of various defect types after preprocessing into Markov pictures through Markov transition fields;
[0126] A Markov picture generator is used for inputting various defect types as conditions and random noises to generate pseudo Markov pictures of various defect types, wherein the Markov picture generator is a conditional generator in a conditional generative adversarial network, the conditional generator is used for generating pseudo Markov pictures according to defect types and random noises, and a discriminator of the conditional generative adversarial network is used for discriminating the pseudo Markov pictures and real Markov pictures;
[0127] A picture feature matching network is used for inputting the molten pool picture to be detected into the trained picture feature matching network, wherein the picture feature matching network is used for matching the molten pool picture and the pseudo Markov pictures of various defect types generated by the Markov picture generator, and outputting the pseudo Markov picture with the highest similarity;
[0128] A fusion network is used for fusing the pseudo Markov picture output by the picture feature matching network and the molten pool picture to be detected to obtain a fusion picture, and the fusion network is used for fusing the molten pool picture and the pseudo Markov picture into one picture;
[0129] A defect recognition network is used for recognizing the defect type of the fusion picture.
[0130] The system provided by the embodiment of the present application can be used to execute the method provided by the embodiment one of the present application, and has the corresponding functions and beneficial effects of the execution method.
[0131] It is worth noting that in the embodiments of the above system, each unit and module included is only divided according to the function logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only used for convenient mutual distinction, and are not used to limit the protection scope of the present application.
[0132] The above-described embodiments are merely illustrative for the present application, wherein the modules illustrated as separate components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, i.e., can be located in one place or can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purposes of the present embodiments. It is clear to those skilled in the art that the various embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized only by hardware, as long as the functions or effects can be realized.
[0133] It should be understood that the above embodiments and descriptions in the specification are only the principles, main features and advantages of the present application, and various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the protection scope of the present application.
Claims
1. A metal additive manufacturing defect detection method based on generative adversarial networks, characterized in that: The method comprises the following steps: S1: By setting different process parameters, the acoustic signals and melt pool images of various defect types in additive manufacturing printing are extracted and pre-processed; S2: The pre-processed acoustic signals of different defect types are converted into Markov images through the Markov transfer field; S3: Different defect types are used as conditions and input into a trained Markov image generator together with random noise to generate several pseudo-Markov images of different defect types, wherein the Markov image generator is a conditional generator in a conditional generative adversarial network, which is used to generate pseudo-Markov images based on the defect type and random noise, and the discriminator of the conditional generative adversarial network is used to distinguish the authenticity of the pseudo-Markov images from the real Markov images; S4: Inputting the molten pool image to be inspected into a trained image feature matching network, wherein the image feature matching network is used to match the molten pool image with several pseudo-Markov images of different defect types generated by the Markov image generator, and select the pseudo-Markov image with the highest similarity for output; S5: fusing the pseudo-Markov image output by the image feature matching network with the melt pool image to be detected using a fusion network to obtain a fused image, wherein the fusion network is used to fuse the melt pool image and the pseudo-Markov image into one image; S6: Input the fused image into a trained defect recognition network to identify the defect type. The defect recognition network is used to identify the defect type of the input fused image.
2. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 1 is characterized in that: Step S1 specifically includes: S1-1: By setting different process parameters, the acoustic signals and melt pool images of different defects in additive manufacturing printing are collected; S1-2: Perform wavelet decomposition on the acoustic signal to divide it into different frequency bands, remove the corresponding DC component frequency, and then reconstruct the acoustic signal; S1-3: Use wavelet denoising algorithm to filter the acoustic signal and filter the background noise during printing; S1-4: Cut off the initial and tail parts of the acoustic signal, leaving only the acoustic signal of the printing process.
3. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 1, characterized in that: Step S2 specifically includes: S2-1: Set the sliding window and sliding step size; S2-2: Slide the pre-processed acoustic signal according to a preset sliding window and sliding step size to extract several segments of acoustic signal; S2-3: For each acoustic signal, convert it into a Markov picture through the Markov transition field, thereby obtaining several Markov pictures.
4. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 1, characterized in that: The condition generator is specifically a deep convolutional generation network DCGAN architecture based on transposed convolution, and the discriminator is a deep convolutional network.
5. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 1, characterized in that: The image feature matching network is specifically a twin network.
6. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 1, characterized in that: The training process of the image feature matching network includes: Each collected melt pool image and a pseudo-Markov image of the same defect type as the melt pool image are taken as a set of positive sample data, and each collected melt pool image and any pseudo-Markov image of a different defect type as the melt pool image are taken as a set of negative sample data; based on a number of positive sample data and negative sample data, the image feature matching network is trained.
7. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 6, characterized in that: The loss function used in the training of the image feature matching network is: d=‖X a -X b ‖2 Where L() represents the loss function, X a ,X b They represent the melt pool image and pseudo-Markov image in a set of samples respectively, Y represents the matching result, matching is 1, and mismatch is 0, m is the set safety distance, and d represents X a ,X b The Euclidean distance.
8. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 1, characterized in that: The fusion network uses the pseudo-Markov map as the R channel, the melt pool image as the G channel, and the B channel as the channel value with intensity lower than a preset threshold, and finally outputs an RGB two-dimensional image.
9. The metal additive manufacturing defect detection method based on generative adversarial network according to claim 1, characterized in that: The defect recognition network uses Efficientnet-B1 as the backbone network.
10. A metal additive manufacturing defect detection system based on generative adversarial networks, characterized in that: The system comprises: The information acquisition module is used to extract acoustic signals and melt pool images of various defect types in additive manufacturing printing by setting different process parameters and perform pre-processing; Acoustic signal conversion module, used to convert the pre-processed acoustic signals of different defect types into Markov images through Markov transfer fields; A Markov image generator, configured to input different defect types as conditions along with random noise to generate a plurality of pseudo-Markov images of different defect types. The Markov image generator is a conditional generator in a conditional generative adversarial network, configured to generate pseudo-Markov images based on the defect type and random noise, and a discriminator in the conditional generative adversarial network is configured to distinguish the authenticity of the pseudo-Markov images from that of real Markov images. An image feature matching network is used to input the molten pool image to be detected into a trained image feature matching network, wherein the image feature matching network is used to match the molten pool image with several pseudo-Markov images of different defect types generated by the Markov image generator, and select the pseudo-Markov image with the highest similarity for output; A fusion network is used to fuse the pseudo-Markov image output by the image feature matching network with the melt pool image to be detected to obtain a fused image. The fusion network is used to fuse the melt pool image and the pseudo-Markov image into one image. Defect recognition network, used to identify the defect type of the fused image.