Multi-class defect detection method and system for molten powder laying image of laser powder bed

Abstract

The invention provides a multi-class defect monitoring method and system for fusion powder laying images of a laser powder bed, and the method comprises the steps: firstly superposing the input images of the laser powder bed before and after fusion powder laying to form a two-channel image stack, then constructing four branch input stacks for the original image stack, enabling the A branch input stack to be a detail feature, and enabling the B branch input stack to be a detail feature; the B branch input stack is a local feature, the C branch input stack is a global feature, and the D branch input stack is a pixel coordinate; then features extracted by the four branches are spliced into fusion features along channel dimensions, the fusion features enter a classification layer and a softmax layer, corresponding preset category probability output is obtained, and finally multi-category defect detection of the laser powder bed molten powder laying image is achieved. According to the method, multi-class defect pixel-level detection of the molten powder laying image of the laser powder bed is realized, network parameters are greatly reduced due to a parallel network structure and a relatively shallow network layer number, and the trained network can realize multi-class defect pixel-level detection of the molten powder laying image of the laser powder bed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of powder image defect detection of laser powder bed fusion process, and particularly relates to a laser powder bed fusion powder image multi-class defect detection method and system. BACKGROUND

[0002] As the most advanced metal additive manufacturing technology, laser powder bed fusion (LPBF) has been widely used in the manufacturing of key metal parts in the fields of aerospace, medical health and so on. However, since the LPBF forming process involves the coupling of multiple factors, it is a major challenge to ensure the quality stability and consistency of the LPBF formed parts. The powder laying quality is an important macroscopic characterization in the process monitoring of LPBF, and the powder laying defects will directly affect the forming quality of the current layer and affect the subsequent forming process, thereby causing quality hidden dangers for the formed parts. Timely and accurate detection and evaluation of the defects in the powder laying process can help to find abnormal conditions in the powder laying process, and is of great significance for the optimization of the forming process and the improvement of the part quality.

[0003] In recent years, high-precision and high-automation powder laying defect monitoring technologies are mostly concentrated in the field of computer vision, such as image segmentation technology. Traditional machine learning relies on manual feature design, which is difficult to capture complex texture features and has poor generalization; although deep learning models achieve semantic segmentation, they are insufficient in dealing with class imbalance problems (such as long-tail distribution data), and have large model parameter quantity and high computational resource consumption. Existing methods such as Fischer using Xception model (Monitoring of the powderbed quality in metal additive manufacturing using deep transfer learning), Xu Xiaojing building a SLM powder laying detection system based on machine vision (Research on automatic detection and classification of laser selective melting powder laying defects based on machine vision), Liu Lixin using deep learning Halcon model (Research on intelligent powder laying technology of selective laser melting based on machine vision), Zhao et al. using FasterR-CNN model, SSD model and YOLOv5 model (Layer-wise multi-defect detection for laser powder bed fusion using deep learning algorithm with visual explanation) have limited pixel-by-pixel detection capability for multi-class defects of additive manufacturing powder image, and the network trained to segment the powder image often needs a large amount of labeled data. In the industrial field, image data labeling requires a lot of manpower and the full participation of professional personnel, and it takes a long time and is difficult to form a data set with rich labeled data, which also becomes an obstacle to the development of deep neural networks in the industrial field. How to improve the pixel-by-pixel detection accuracy of multi-class defects of additive manufacturing powder image, form an automatic real-time powder image defect detection process, and regularize a large amount of powder image data into a high-quality labeled data set is a problem to be solved. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a laser powder bed fusion powder laying image multi-class defect detection method and system, which introduces a multi-scale convolutional network to extract powder image features of different scales in parallel, and fuses features of different scales, realizes pixel-level detection of multi-class defects of laser powder bed fusion powder laying image, and the parallel network structure and shallow network layer number also greatly reduce the network parameters. The trained network can realize pixel-level detection of multi-class defects of laser powder bed fusion powder laying image.

[0005] To achieve the above purposes, the present application adopts the following technical solutions:

[0006] A laser powder bed fusion powder laying image multi-class defect monitoring method, comprising the following steps:

[0007] Step 1: Based on the laser powder bed fusion powder laying image dataset, use each layer of powder laying before and after two visible light images as network input, and stack them into two-channel image stack input; Wherein each layer of powder laying before and after two images corresponds to a pixel-level annotation file, and a plurality of preset class labels are provided;

[0008] Step 2: Process the two-channel powder laying before and after image stack in step 1 for multi-scale and position information, and construct four branch input stacks, A branch input stack is a detail feature, which is cut from the original image, the size is N*N pixels, the size is adjusted according to the actual situation to adapt to different camera resolutions, B branch input stack is a local feature, C branch input stack is a global feature, and D branch input stack is a pixel coordinate;

[0009] Step 3: Convolution is performed on the A branch input stack in step 2 to extract the detail feature, and the output maintains the original size, and the output size is N*N*H1 feature map, H1 is the output feature channel number of A branch network;

[0010] Step 4: The B branch input stack in step 2 enters the U-Net network, extracts the local feature, and outputs the same size as the A branch input stack through up-sampling, and the output size is N*N*H2 feature map, H2 is the output feature channel number of B branch network;

[0011] Step 5: The C branch input stack in step 2 is convolved and pooled to extract the global feature, which is compressed into a feature vector with a length of 1*H3 through a fully connected layer, and then expanded to the size of the A branch input stack through a bilinear interpolation algorithm, and the output size is N*N*H3 feature map, H3 is the output feature channel number of C branch network;

[0012] Step 6: The feature maps output by the above A branch input stack, B branch input stack and C branch input stack are spliced along the channel dimension, and then the two-channel coordinates of the D branch input stack are stacked to obtain a fusion feature map with a size of N*N*(H1+H2+H3+2);

[0013] Step 7: The fusion feature of step 6 is input into the classification layer, mapped to the preset multiple categories, and then input into the softmax layer (normalization exponential layer) to obtain the category probability of each pixel, and finally realize the pixel-level defect monitoring of the powder laying image.

[0014] The four branch input stacks in step 2 are obtained by the following process:

[0015] Step 2.1: The original image is divided into slices with fixed pixel size, the slice size is N*N, and all the slices are stacked together to form the A branch input stack;

[0016] Step 2.2: The A branch input stack is sampled using the bilinear interpolation algorithm, scaled to b*b pixel size, and the B branch input stack is formed;

[0017] Step 2.3: The original image stack is sampled using the bilinear interpolation algorithm to obtain the C branch input stack with c*c pixel size;

[0018] Step 2.4: The coordinates of the pixels in each slice of the A branch input stack in the original image are calculated, and the center is normalized to form the D branch input stack with N*N*2.

[0019] The detail feature extraction network in step 3 is constructed as follows: the input A stack with N*N is mirror filled with 4 pixels, enters a convolution layer, and the activation function uses ReLU, and the output size is N*N*H1 feature map.

[0020] The local feature extraction network in step 4 is constructed as follows: first, the B branch input stack with b*b is mirror filled with 2 pixels and input into the U-Net network, which contains four parts of encoder, decoder, skip connection and full connection layer; The encoder part contains 5 times of convolution and 5 times of pooling operation, which reduces the feature to 4*4 size; Use the full connection layer to compress it into a feature vector with length 1*256 dimensions, then the feature vector enters the decoder, uses the nearest neighbor upsampling method to gradually recover to b*b pixel size, after each upsampling, first use 1*1 convolution to reduce the number of channels, then jump connection with the corresponding scale of the encoding features, and then a 3*3 convolution is used to fuse the features; The decoder outputs a feature map of b*b*H2, and finally uses the bilinear interpolation algorithm to upsample to the size of the B branch input stack N*N, and outputs a feature map of N*N*H2 size.

[0021] The global feature extraction network in step 5 is constructed as follows: the C branch input stack is first mirror filled with 2 pixels, then enters three convolution layers, the activation function uses ReLU, and after each convolution layer, a pooling layer is entered, and after extracting global features, a full connection layer is used to compress the feature into a feature vector with length 1*H3, a Dropout layer is connected after the full connection layer to prevent overfitting, p=0.5, and finally the bilinear interpolation algorithm is used to upsample the 1*H3 feature vector to expand it to a feature map of N*N*H3.

[0022] The same size feature maps (H1, H2, H3 channels respectively) output by the A branch input stack, the B branch input stack, the C branch input stack and 2 channels of the D branch input stack in step 6 are spliced along the channel dimension to fuse into a comprehensive feature vector map with a size of N*N*(H1+H2+H3+2).

[0023] Step 7 makes the fused features in step 6 enter the classification layer, first enters a bottleneck layer, then enters a BatchNorm layer, uses ReLU as the activation function, then uses 1*1 convolution to map the (H1+H2+H3+2)-dimensional features to a preset multi-class output vector, directly calculates the class probability of each pixel through a softmax layer after the classification layer, and the calculation formula of the softmax layer is as follows:

[0024]

[0025] Wherein n classes is the total number of preset classes, p j is the preset jth non-normalized class distribution, q j is the preset jth normalized probability, p k is the kth non-normalized class distribution, e is the base of natural logarithm, and k [1, n classes ].

[0026] A laser powder bed fusion powder laying image multi-class defect monitoring system can realize the laser powder bed fusion powder laying image multi-class defect monitoring method.

[0027] Compared with the prior art, the beneficial effects of the present application are:

[0028] The present application can realize high-precision detection of multi-class defects of the powder laying image by using three branches to extract powder laying image features of different scales in parallel, and only uses a relatively shallow network layer, greatly reduces the parameter quantity and the calculation cost, improves the speed of pixel-level defect detection of the image, and realizes the detection of multi-class defects of the powder laying image.

[0029] The present application overcomes the problems of low precision of laser powder bed fusion powder laying image multi-class defect detection, long time consumption and great difficulty in forming a data set with rich labeled data in an industrial environment, and provides a feasible method for realizing real-time powder laying image multi-class pixel-level defect detection of laser powder bed fusion technology and large-scale automatic labeling of powder laying image data to form a high-quality data set. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The flowchart of the embodiment method of the present application.

[0031] Figure 2The image before and after powder laying of a certain 2-layer laser powder bed fusion method of an embodiment of the application.

[0032] Figure 3 The corresponding multi-class defect detection result of the input 2-layer laser powder bed powder laying image output of the method of an embodiment of the application. DETAILED DESCRIPTION

[0033] In order to illustrate the technical solutions of the application, 20 layers of laser powder bed fusion images with pixel-level labeling before and after powder laying are taken as training data, the proposed network is trained, and pixel-level multi-class defect detection of the laser powder bed fusion image is taken as an example for illustration.

[0034] REFERENCE Figure 1 A multi-class defect monitoring method for laser powder bed fusion powder laying images, comprising the following steps:

[0035] Step 1: using 20 layers of laser powder bed fusion images with pixel-level labeling before and after powder laying, using the visible light images before and after powder laying of each layer as network input, and stacking them into two-channel image stacks; wherein the powder laying image of each layer corresponds to a pixel-level labeling file, and the preset labels include normal powder laying, normal fusion, doctor blade scratch, incomplete powder laying, swelling, fragment, warping, and smoke;

[0036] Step 2: multi-scale and position information processing is performed on the two-channel powder laying before and after image stacks in step 1, and four branch input stacks are constructed, the A branch input stack is a detail feature, which is cut from the original image, the size is N*N pixels, the size is adjusted according to the actual situation to adapt to different camera resolutions, the B branch input stack is a local feature, the C branch input stack is a global feature, and the D branch input stack is a pixel coordinate;

[0037] The four branch input stacks are obtained through the following process:

[0038] Step 2.1: the original image is divided into slices according to a fixed size of 256 pixels, the slice size is 256*256, and after division, all the slices are stacked together to form the A branch input stack;

[0039] Step 2.2: the A branch input stack is down-sampled by using a bilinear interpolation algorithm, and is scaled to 128*128 size to form the B branch input stack;

[0040] Step 2.3: the original image stack is down-sampled by using a bilinear interpolation algorithm to obtain a C branch input stack with a size of 32*32;

[0041] Step 2.4: Calculate the coordinates of the pixels in each slice of the A branch input stack in the original image, and perform center normalization to obtain a 2-channel D branch input stack; the D branch input stack is encoded by the pixel coordinates in each slice in the x coordinate channel and the y coordinate channel, and after center normalization, the coordinate position encoding is obtained, the coordinates of the D branch input stack are aligned with each slice of the B branch input stack, forming a D branch input stack with a size of 256*256 and 2 channels;

[0042] Step 3: Convolve the A branch input stack in step 2 to extract detailed features, output the original size, and output 128 channels; the detailed feature extraction network is constructed as follows: the input is a 256*256 A branch input stack, which is filled with 4 pixels of mirror image, enters a convolution layer, the activation function uses ReLU, the convolution kernel size is 11*11, and the output size is 256*256*128;

[0043] Step 4: The B branch input stack in step 2 enters the U-Net network, extracts local features, and outputs the same size as the A branch input stack through upsampling, and outputs 32 channels; the local feature extraction network is constructed as follows: first, the 128*128 B branch input stack is filled with 2 pixels of mirror image and then enters the U-Net network; the U-Net network includes an encoder, a decoder, a skip connection, and a fully connected layer; the encoder part includes 5 times of convolution and 5 times of 2*2 pooling, which reduces the feature to 4*4 size, and uses a fully connected layer to compress it into a feature vector with a length of 256; the decoder starts with the feature vector, uses nearest neighbor upsampling to gradually restore the size, and after each upsampling, a 1*1 convolution is used to reduce the number of channels, then the corresponding scale of the encoded features is connected through a skip connection, and then a 3*3 convolution is used to fuse the features; the decoder outputs a 128*128, 32-channel feature map, which is upsampled to the size of the A branch input stack through bilinear interpolation, and finally outputs a feature map with a size of 256*256*32;

[0044] Step 5: Convolve and pool the C branch input stack in step 2 to extract global features, compress them into a feature vector with a length of 16 through a fully connected layer, then expand them back to the size of the A branch input stack through bilinear interpolation, and output 16 channels; the global feature extraction network is constructed as follows: the C branch input stack is first filled with 2 pixels of mirror image with a size of 36*36, then enters three layers of convolution+ReLU and three layers of 2*2 pooling layer to extract global features, then enters two layers of fully connected layer to compress the features into a length of 16 dimensions, both layers of fully connected layer are connected with a Dropout layer (p=0.5) to prevent overfitting, and finally the 16-dimensional features are expanded to the size of the A branch input stack through bilinear interpolation upsampling, and finally a feature map with a size of 256*256*16 is output;

[0045] Step 6: The feature maps output by the A branch input stack, the B branch input stack, and the C branch input stack are spliced along the channel dimension, and then the two-channel coordinates of the D branch input stack are superimposed to obtain a fused feature with a size of 256*256*178; in step 6, the feature maps with the same size output by the A branch input stack, the B branch input stack, and the C branch input stack (128, 32, and 16 channels, respectively) and the D branch input stack (2 channels) are spliced along the channel dimension to form a comprehensive feature vector map with a size of 225*225*178 and 178 dimensions per pixel;

[0046] Step 7: The fused feature in step 6 is input into a classification layer to be mapped into 8 label categories to be classified, first enters a bottleneck layer, then enters a BatchNorm layer, uses ReLU as an activation function, then uses a 1*1 convolution to map the 178-dimensional feature into a preset 8-category output vector, and then enters a softmax layer to obtain the class probability of each pixel, and finally realizes pixel-level multi-class defect detection of the powder laying image; wherein the calculation formula of the softmax layer is as follows:

[0047]

[0048] Wherein n classes is the total number of preset categories, p j is the preset jth non-normalized category distribution, q j is the preset jth normalized probability, p k is the kth non-normalized category distribution, e is the base of the natural logarithm, and k [1, n classes ].

[0049] In the training process, the Adam optimizer is used, the learning rate is 0.0001, the first and second exponential decay rates are set to 0.9 and 0.999, and the corresponding epsilon is set to 0.0001. The loss function uses cross-entropy loss;

[0050] The final training result can reach a pixel accuracy of 0.9342 within 200 epochs, wherein the pixel accuracy of each category is as follows: normal powder laying 0.9386, normal fusion 0.9823, scraper scratch 0.3324, incomplete powder laying 0.9172, swelling 0.8523, fragment 0.8542, warping 0.7901, and smoke dust 0.8894. In the corresponding labeled visual image, each label corresponds to a color: normal powder laying: gray, normal fusion: black, scraper scratch: green, incomplete powder laying: blue, swelling: yellow, fragment: magenta, warping: cyan, and smoke dust: orange.

[0051] Reference Figure 2 , Figure 2 is the image of the laser powder bed fusion of the embodiment of the application before and after 2 layers of powder laying,Figure 2 The left two images in the middle are taken before powder laying, and the right two images are taken after powder laying. Referring to Figure 3 , Figure 3 The corresponding multi-class defect detection results of the input 2-layer laser powder bed fusion powder laying image output of the embodiment of the present application can be seen. The system can accurately identify the printing state of each pixel position before and after powder laying of each printing layer. Different colors of pixels correspond to different preset categories.

[0052] A laser powder bed fusion powder laying image multi-class defect monitoring system can implement the above laser powder bed fusion powder laying image multi-class defect monitoring method.

Claims

1. A multi-category defect monitoring method for laser powder bed fusion powder laying images, characterized in that: The following steps are involved: Step 1: Based on the laser powder bed fusion powder laying image dataset, two visible light images before and after each layer of powder laying are used as network input and superimposed into a two-channel image stack input. Each layer of powder laying has a corresponding pixel-level annotation file with preset multi-category labels. Step 2: Perform multi-scale and position information processing on the two-channel image stacks before and after powder coating in step 1 to construct four branch input stacks. The A branch input stack is the detail feature, which is segmented from the original image and has a size of N*N pixels. The size is adjusted according to the actual situation to adapt to different camera resolutions. The B branch input stack is the local feature, the C branch input stack is the global feature, and the D branch input stack is the pixel coordinate. Step 3: Convolve the A branch input stack in step 2 to extract detailed features, and keep the output size as N*N*H1 feature map, where H1 is the number of output feature channels of the A branch network; Step 4: The B branch input stack in step 2 enters the U-Net network, and after local feature extraction, the output is upsampled to match the size of the A branch input stack. The output size is N*N*H2 feature map, where H2 is the number of feature channels of the B branch network output. Step 5: Perform convolution pooling on the C-branch input stack in step 2 to extract global features. The feature vector is compressed into a length of 1*H3 through a fully connected layer, and then expanded to the size of the A-branch input stack through a bilinear interpolation algorithm. The output size is N*N*H3 feature map, where H3 is the number of feature channels of the C-branch network output. Step 6: Concatenate the feature maps output by the A-branch input stack, the B-branch input stack, and the C-branch input stack along the channel dimension, and then superimpose the two channel coordinates of the D-branch input stack to obtain a fused feature map of size N*N*(H1+H2+H3+2); Step 7: Input the fused features of step 6 into the classification layer, map them into multiple preset categories, and then enter the softmax layer (normalized exponential layer) to obtain the category probability of each pixel, ultimately achieving pixel-level defect monitoring of the powder spreading image.

2. The method according to claim 1, characterized in that The four branch input stacks constructed in step 2 are obtained by the following process: Step 2.1: Divide the original image into slices of fixed pixel size, with a slice size of N*N. After division, all slices are stacked together to form the input stack of branch A; Step 2.2: Use bilinear interpolation to sample the input stack of branch A and scale it to b*b pixel size to form the input stack of branch B. Step 2.3: Use bilinear interpolation algorithm to sample the original image stack to obtain a C-branch input stack of c*c pixel size; Step 2.4: Calculate the coordinates of the pixels in each slice of the A branch input stack in the original image and perform center normalization to form an N*N*2 D branch input stack.

3. The method according to claim 1, characterized in that The detail feature extraction network in step 3 is constructed as follows: the input A stack of N*N is mirror-padding by 4 pixels, enters a convolutional layer, uses ReLU as the activation function, and outputs a feature map of size N*N*H1.

4. The method according to claim 1, wherein The local feature extraction network in step 4 is constructed as follows: First, the b*b B branch input stack is mirrored and padded with 2 pixels and then input into the U-Net network. The U-Net network consists of four parts: encoder, decoder, skip connection and fully connected layer. The encoder part contains 5 convolutions and 5 pooling operations to reduce the feature size to 4*4. The fully connected layer is used to compress it into a feature vector with a length of 1*256. The feature vector then enters the decoder and is gradually restored to the b*b pixel size using the nearest neighbor upsampling method. After each upsampling, a 1*1 convolution is first used to reduce the number of channels, and then it is skipped with the encoded feature of the corresponding scale, and then a 3*3 convolution is used to fuse the features. The decoder outputs a feature map of b*b*H2, and finally uses a bilinear interpolation algorithm to upsample to the B branch input stack size N*N, and outputs a feature map of size N*N*H2.

5. The method according to claim 1, wherein The global feature extraction network in step 5 is constructed as follows: the C branch input stack is first mirrored and padded with 2 pixels, and then enters the three convolutional layers. The activation function uses ReLU. After each convolutional layer, it enters the pooling layer. After extracting the global features, it enters the fully connected layer to compress the features into a feature vector of length 1*H3. A Dropout layer is followed by the fully connected layer to prevent overfitting, p=0.

5. Finally, the bilinear interpolation algorithm is used for upsampling to expand this 1*H3 feature vector to an N*N*H3 feature map.

6. The method according to claim 1, characterized in that In step 6, the same-sized feature maps output by the A-branch input stack, the B-branch input stack, the C-branch input stack, and the 2 channels of the D-branch input stack are spliced ​​along the channel dimension and fused into a comprehensive feature vector map of size N*N*(H1+H2+H3+2).

7. The method according to claim 1, characterized in that Step 7 allows the fused features in step 6 to enter the classification layer, first entering a bottleneck layer, then entering the BatchNorm layer. The activation function uses ReLU, and then a 1*1 convolution is used to map the (H1+H2+H3+2)-dimensional features into a preset multi-category output vector. After the classification layer, the softmax layer is directly used to calculate the category probability of each pixel. The calculation formula of the softmax layer is as follows: where n classes is the total number of preset categories, p j is the preset unnormalized category distribution of the jth category, q j is the preset normalized probability of class j, p k is the unnormalized category distribution of the kth category, e is the base of the natural logarithm, k∈[1,n classes ].

8. A multi-category defect monitoring system for laser powder bed fusion powder spreading images, characterized by: The method for monitoring multi-category defects in laser powder bed fusion powder laying images can be implemented as described in any one of claims 1 to 7.

Images