An additive manufacturing online monitoring method and system based on edge computing

Abstract

The present invention belongs to the technical field of additive manufacturing, and discloses an online monitoring method and system for additive manufacturing based on edge computing. The method of the present invention includes establishing a data set for additive manufacturing, constructing an online monitoring model for additive manufacturing, using the data set to train the online monitoring model to obtain a trained online monitoring model, deploying the trained online monitoring model on an edge computing device, and using the edge computing device to obtain online monitoring information of additive manufacturing. The system includes an edge computing device and a high-speed camera; the high-speed camera is used to shoot the additive manufacturing process, and transmit the captured video sequence to the edge computing device; the edge computing device is deployed with a trained online monitoring model, and the edge computing device is used to obtain online monitoring information of additive manufacturing. The present invention can realize stable, low-latency, high-resolution, and high-precision online monitoring of additive manufacturing process characteristics, and can also reduce the requirements for high-speed camera equipment.

Description

Technical Field

[0001] The present invention belongs to the technical field of additive manufacturing, and more specifically, relates to an additive manufacturing online monitoring method and system based on edge computing. Background Art

[0002] Additive manufacturing is a near-net-shape process that directly prints raw materials such as metals into products. It has extremely high application value in the fields of aerospace, automobiles and ships, micro-nano manufacturing, biomedical engineering, etc. However, due to its special layer-by-layer printing mechanism, the manufacturing process tends to have large defects, and problems such as poor density, anisotropy, local deformation and stress concentration may occur. Therefore, reasonable and efficient detection methods are needed for defect detection and quality inspection.

[0003] In recent years, AI-based image processing algorithms have developed rapidly, and a variety of image processing algorithms have been applied to online monitoring of additive manufacturing, providing new methods for feature monitoring and quality control of the additive manufacturing process. Additive manufacturing process monitoring requires high performance from high-speed cameras. However, current high-speed cameras have problems such as high prices and low resolution for high-speed shooting, making it difficult to support high-precision image segmentation. In addition, AI-based image processing algorithms require large amounts of computation. The industry usually decouples data acquisition and data processing, and uses remote servers to execute algorithm models, which leads to problems such as high communication delays and unstable data transmission. Summary of the invention

[0004] The present invention provides an additive manufacturing online monitoring method and system based on edge computing to solve the problems in the prior art that additive manufacturing online monitoring is difficult to achieve high-precision image segmentation, has high communication delay, and has unstable data transmission.

[0005] The present invention provides an online monitoring method for additive manufacturing based on edge computing, comprising the following steps:

[0006] Building a dataset for additive manufacturing;

[0007] Constructing an online monitoring model for additive manufacturing, the online monitoring model comprising a video super-resolution model and a feature segmentation model; the video super-resolution model takes a video sequence as input, the feature segmentation model takes an output of the video super-resolution model as input, and the feature segmentation model outputs a feature segmentation map;

[0008] The video super-resolution model includes a spatiotemporal encoder, a residual feature extraction module, a query matrix generation module, a key matrix generation module and a value matrix generation module; the pixel resolution of any frame image in the input video sequence is (h0 / r, w0 / r, 1), h0 and w0 are the number of pixels of the image in the height and width directions respectively, and r is the multiple of super-resolution reconstruction; the video sequence is encoded by the spatiotemporal encoder and input into the residual feature extraction module to obtain a feature map matrix, and the feature map matrix is ​​respectively input into the query matrix generation module, the key matrix generation module and the value matrix generation module to obtain a query matrix, a key matrix and a value matrix respectively; a total feature map is obtained based on the query matrix, the key matrix and the value matrix; the total feature map is pixel-rearranged to obtain a super-resolution reconstruction result with a pixel resolution of (h0, w0, 1);

[0009] Using the data set to train the online monitoring model to obtain a trained online monitoring model;

[0010] The trained online monitoring model is deployed on an edge computing device, and the edge computing device is used to obtain online monitoring information of additive manufacturing.

[0011] Preferably, the residual feature extraction module includes n cascaded residual blocks; each of the residual blocks includes a first 3D convolutional layer with a convolution kernel size of 1x1, a first batch of normalization layers, a second 3D convolutional layer with a convolution kernel size of 3x3, and a second batch of normalization layers; the number of convolution kernels of the first 3D convolutional layer is 32*index, index is index, 1≤index≤n; the number of convolution kernels of the second 3D convolutional layer is 32.

[0012] Preferably, the query matrix generation module, the key matrix generation module and the value matrix generation module have the same structure, and all include k cascaded blocks; the structures of the first k-1 blocks all include a third 3D convolution layer with a convolution kernel size of 1x1 and a third batch normalization layer, and the number of convolution kernels of the third 3D convolution layer is 32n, where n is the number of residual blocks included in the residual feature extraction module; the structure of the last block includes a fourth 3D convolution layer with a convolution kernel size of 3x3 and a fourth batch normalization layer, and the number of convolution kernels of the fourth 3D convolution layer is 1.

[0013] Preferably, the query matrix generation module, the key matrix generation module and the value matrix generation module have different weights after training, and matrices with the same shape but different values ​​can be obtained by using the query matrix generation module, the key matrix generation module and the value matrix generation module.

[0014] Preferably, the input video sequence is denoted as {X t-N ,…,X t,..,X t+N}; The query matrix is ​​recorded as Q t ; The key matrix is ​​denoted as {K i}, tN≤i≤t+N, i≠t; the value matrix is ​​recorded as {V i}, tN≤i≤t+N; the value matrix of the current frame is recorded as V t ; The total feature map is calculated as follows:

[0015]

[0016] Where FM t is the total feature map.

[0017] Preferably, the feature segmentation model includes an encoder and a decoder; the encoder uses the Xception network as a backbone to extract features, and the encoder outputs the feature extraction results output by the video super-resolution model and the feature extraction results after spatial pyramid pooling to the decoder respectively; the decoder uses 1×1 convolution and 4x upsampling to fuse the two feature extraction results and output a high-resolution feature segmentation map.

[0018] Preferably, the data set includes a video super-resolution data set and a feature segmentation data set;

[0019] The additive manufacturing process is photographed by a high-speed camera to obtain an original image sequence in RGB color space; a reference brightness matrix sequence is obtained by RGB-YCbCr color space conversion; and a resolution of the reference brightness matrix sequence is reduced to 1 / r of the original resolution by using a nearest neighbor interpolation method to obtain a low-resolution brightness matrix sequence;

[0020] The video super-resolution model is trained using the video super-resolution dataset, with the low-resolution brightness matrix sequence as input and the reference brightness matrix sequence as output;

[0021] The feature segmentation model is trained using the feature segmentation data set, with the reference brightness matrix as input and a feature segmentation map annotated with different feature contours in the additive manufacturing process as output.

[0022] Preferably, when training the online monitoring model, the mean square error is used as the loss function, and the Adam optimizer is selected to tune the network weights.

[0023] Preferably, the trained online monitoring model is quantized and pruned by the edge computing hardware SDK to obtain a model that is compatible with the hardware of the edge computing device; a model session is created in the edge computing device and placed in the back end as a service, the relative positions of the high-speed camera and the metal powder bed are fixed, the video sequence captured by the high-speed camera is connected to the back end interface, and the edge computing device is used to obtain online monitoring information of additive manufacturing.

[0024] On the other hand, the present invention provides an additive manufacturing online monitoring system based on edge computing, comprising: an edge computing device and a high-speed camera; the high-speed camera is used to shoot the additive manufacturing process and transmit the captured video sequence to the edge computing device; a trained online monitoring model is deployed on the edge computing device, and the edge computing device is used to obtain online monitoring information of additive manufacturing;

[0025] The additive manufacturing online monitoring system based on edge computing is used to execute the steps in the above-mentioned additive manufacturing online monitoring method based on edge computing.

[0026] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0027] The online monitoring model of additive manufacturing constructed by the present invention includes a video super-resolution model and a feature segmentation model, wherein the video super-resolution model includes a spatiotemporal encoder, a residual feature extraction module, a query matrix generation module (Gen_Q), a key matrix generation module (Gen_K) and a value matrix generation module (Gen_V); the pixel resolution of any frame image in the input video sequence is (h0 / r, w0 / r, 1), h0 and w0 are the number of pixels of the image in the height and width directions respectively, and r is the multiple of super-resolution reconstruction; the video sequence is encoded by the spatiotemporal encoder and input into the residual feature extraction module to obtain a feature map matrix, and the feature map matrix is ​​respectively input into the query matrix generation module, the key matrix generation module, and the value matrix generation module to obtain the query matrix, the key matrix, and the value matrix respectively; the total feature map is obtained based on the query matrix, the key matrix and the value matrix; the total feature map is pixel-rearranged to obtain a super-resolution reconstruction result with a pixel resolution of (h0, w0, 1). The present invention uses the data set of additive manufacturing to train the online monitoring model to obtain a trained online monitoring model; the trained online monitoring model is deployed on the edge computing device, and the edge computing device is used to obtain the online monitoring information of additive manufacturing. Compared with the method adopted by the general visual Transformer to convert the features into one dimension, and then calculate the feature vectors such as Q, K, V, so that the network has local attention, the present invention does not convert the feature matrix into one dimension, but retains its original two-dimensional shape to generate Q, K, V feature matrices so that the network has spatial global attention, which not only saves the amount of calculation, but also retains the spatiotemporal information, and improves the efficiency while improving the calculation speed. The video super-resolution model constructed by the visual Transformer proposed by this method has better image quality than the mainstream super-resolution network reconstruction at the same inference speed, and takes the lead in the peak signal-to-noise ratio PSNR image quality index, surpassing the classic interpolation algorithm and most super-resolution neural network models. Since the result of super-resolution reconstruction contains more explicit image features, the accuracy of feature extraction results is significantly improved compared with direct extraction. At the same time, since the present invention deploys the trained online monitoring model on the edge computing device, the inference speed is fast and the inference time is much shorter than the CPU inference time, which is helpful for timely feedback of the manufacturing process. By applying the video super-resolution model, only the original 1 / r 2 The data can be sent by the receiving end (for example, the original image resolution is 300 pixels × 300 pixels, but only 75 pixels × 75 pixels of data need to be sent to reconstruct a high-quality image with almost the same quality as the original image), and high-quality and high-reduction data can be obtained by super-resolution reconstruction at the receiving end, which greatly reduces the bandwidth and rate requirements of communication, and has lower latency and higher stability under the same communication conditions. In summary, the present invention can achieve stable low-latency, high-resolution, and high-precision online monitoring of additive manufacturing process characteristics, and can also reduce the requirements for high-speed camera equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of an additive manufacturing online monitoring system based on edge computing provided by an embodiment of the present invention;

[0029] Figure 2 Flowchart for building a video super-resolution dataset;

[0030] Figure 3 Schematic diagram for establishing a feature segmentation dataset;

[0031] Figure 4 This is the structural diagram of the video super-resolution model;

[0032] Figure 5 Schematic diagram of feature segmentation model;

[0033] Figure 6 This is a super-resolution effect image obtained based on the video sequence;

[0034] Figure 7 A comparison diagram of super-resolution results obtained by the present invention and other methods;

[0035] Figure 8 A comparison diagram of feature segmentation results obtained by the present invention and other methods;

[0036] Fig. 9 This is a performance comparison chart of the present invention and other models in terms of peak signal-to-noise ratio and inference time. DETAILED DESCRIPTION

[0037] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] Embodiment 1:

[0039] Embodiment 1 provides an additive manufacturing online monitoring method based on edge computing, comprising the following steps:

[0040] Step 1: Create a dataset for additive manufacturing.

[0041] Step 2: Construct an online monitoring model for additive manufacturing, wherein the online monitoring model includes a video super-resolution model and a feature segmentation model; the video super-resolution model takes a video sequence as input, the feature segmentation model takes the output of the video super-resolution model as input, and the feature segmentation model outputs a feature segmentation map.

[0042] Among them, the video super-resolution model includes a spatiotemporal encoder, a residual feature extraction module, a query matrix generation module, a key matrix generation module and a value matrix generation module.

[0043] The pixel resolution of any frame image in the input video sequence is (h0 / r, w0 / r, 1), h0 and w0 are the number of pixels of the image in the height and width directions respectively, and r is the multiple of super-resolution reconstruction; the video sequence is encoded by the spatiotemporal encoder and input into the residual feature extraction module to obtain a feature map matrix, and the feature map matrix is ​​respectively input into the query matrix generation module, the key matrix generation module, and the value matrix generation module to obtain a query matrix, a key matrix, and a value matrix respectively; a total feature map is obtained based on the query matrix, the key matrix, and the value matrix; the total feature map is subjected to pixel rearrangement to obtain a super-resolution reconstruction result with a pixel resolution of (h0, w0, 1).

[0044] Specifically, the residual feature extraction module includes n cascaded residual blocks; each of the residual blocks includes a first 3D convolution layer with a convolution kernel size of 1x1, a first batch of normalization layers, a second 3D convolution layer with a convolution kernel size of 3x3, and a second batch of normalization layers; the number of convolution kernels of the first 3D convolution layer is 32*index, index is index, 1≤index≤n; the number of convolution kernels of the second 3D convolution layer is 32.

[0045] The query matrix generation module, the key matrix generation module and the value matrix generation module have the same structure, and all include k cascaded blocks; the structures of the first k-1 blocks all include a third 3D convolution layer with a convolution kernel size of 1x1 and a third batch normalization layer, and the number of convolution kernels of the third 3D convolution layer is 32n, where n is the number of residual blocks included in the residual feature extraction module; the structure of the last block includes a fourth 3D convolution layer with a convolution kernel size of 3x3 and a fourth batch normalization layer, and the number of convolution kernels of the fourth 3D convolution layer is 1.

[0046] The query matrix generation module, the key matrix generation module and the value matrix generation module have different weights after training. By using the query matrix generation module, the key matrix generation module and the value matrix generation module, matrices with the same shape but different values ​​can be obtained.

[0047] Specifically, the input video sequence is denoted as {X t-N ,…,X t ,..,X t+N}; The query matrix is ​​recorded as Q t ; The key matrix is ​​denoted as {K i}, tN≤i≤t+N, i≠t; the value matrix is ​​recorded as {V i}, tN≤i≤t+N; the value matrix of the current frame is recorded as V t ; The total feature map is calculated as follows:

[0048]

[0049] Where FM t is the total feature map.

[0050] Specifically, the feature segmentation model includes an encoder and a decoder; the encoder uses the Xception network as a backbone to extract features, and the encoder outputs the feature extraction results output by the video super-resolution model and the feature extraction results after spatial pyramid pooling to the decoder respectively; the decoder uses 1×1 convolution and 4x upsampling to fuse the two feature extraction results and output a high-resolution feature segmentation map.

[0051] Step 3: Use the data set to train the online monitoring model to obtain a trained online monitoring model.

[0052] Step 4: Deploy the trained online monitoring model on an edge computing device, and use the edge computing device to obtain online monitoring information of additive manufacturing.

[0053] Wherein, the data set includes a video super-resolution data set and a feature segmentation data set. The additive manufacturing process is photographed with a high-speed camera to obtain an original image sequence in the RGB color space; a reference brightness matrix sequence is obtained by converting the RGB-YCbCr color space; the nearest neighbor interpolation method is used to reduce the resolution of the reference brightness matrix sequence to the original 1 / r to obtain a low-resolution brightness matrix sequence. The video super-resolution model is trained using the video super-resolution data set, with the low-resolution brightness matrix sequence as input and the reference brightness matrix sequence as output. The feature segmentation model is trained using the feature segmentation data set, with the reference brightness matrix as input and a feature segmentation map annotated with different feature contours in the additive manufacturing process as output.

[0054] When training the online monitoring model, the mean square error can be used as the loss function, and the Adam optimizer can be used to tune the network weights.

[0055] In addition, the trained online monitoring model can be quantized and pruned by the edge computing hardware SDK to obtain a model that is compatible with the hardware of the edge computing device; a model session is created in the edge computing device and placed in the back end as a service, the relative positions of the high-speed camera and the metal powder bed are fixed, and the video sequence captured by the high-speed camera is connected to the back end interface, and the edge computing device is used to obtain online monitoring information of additive manufacturing.

[0056] Embodiment 2:

[0057] Example 2 provides an additive manufacturing online monitoring system based on edge computing, including: an edge computing device and a high-speed camera; the high-speed camera is used to film the additive manufacturing process and transmit the captured video sequence to the edge computing device; a trained online monitoring model is deployed on the edge computing device, and the edge computing device is used to obtain online monitoring information of additive manufacturing.

[0058] The additive manufacturing online monitoring system based on edge computing provided in Example 2 is used to execute the steps in the additive manufacturing online monitoring method based on edge computing as described in Example 1. The functions of each component in Example 2 correspond to the steps in the method provided in Example 1, and therefore can be understood in conjunction with the description of Example 1 and will not be repeated here.

[0059] The steps of the present invention are further described below.

[0060] Step 1: Establish an additive manufacturing dataset, that is, construct an additive manufacturing process video dataset.

[0061] (1) Video Super-Resolution Dataset.

[0062] The additive manufacturing process is video-shot. Assume that the manufacturing process video has a total of M frames. Starting from the beginning, the frame interval is 1 and 2N+1 frames are continuously captured to obtain the original image sequence {Im t-N ,…,Im t ,..,Im t+N}(N+1≤t≤MN),Im t represents the tth original image in the original image sequence, where the size of each element matrix (i.e., pixel resolution) is (h0, w0, 3), where h0 and w0 are the number of pixels in the height and width directions of the original image, respectively. Secondly, the reference brightness matrix sequence {GT t-N ,…,GT t ,..,GT t+N}(N+1≤t≤MN), GT t is the tth reference brightness matrix in the reference brightness matrix sequence, where the size of each element matrix is ​​(h0, w0, 1). The nearest neighbor interpolation method is used to reduce the resolution of the reference brightness matrix sequence to 1 / r of the original to obtain a low-resolution brightness matrix sequence {X t-N ,…,X t ,..,X t+N}(N+1≤t≤MN), X t is the t-th low-resolution brightness matrix in the low-resolution brightness matrix sequence, wherein the size of each element matrix is ​​(h0 / r, w0 / r, 1).

[0063] The video super-resolution model is trained using the video super-resolution dataset, with a low-resolution brightness matrix sequence {X t-N ,…,X t ,..,X t+N} as input, taking the tth reference brightness matrix GT in the reference brightness matrix sequence t is the output.

[0064] In step 1, the middle frame of the continuous 2N+1 frame video sequence is super-reconstructed, that is, the N+1th frame in the video sequence.

[0065] The matrix transformation method in step 1 is to convert the RGB color space into the YCbCr color space, and the conversion formula is:

[0066]

[0067] Among them, R, G, B represent the component matrices of the image to be converted in the three color channels of R, G, and B respectively, and Y, Cb, and Cr represent the component matrices of the converted image in terms of brightness, blue concentration offset, and red concentration offset.

[0068] (2) Feature segmentation dataset.

[0069] The additive manufacturing video shot by the aforementioned high-speed camera is saved frame by frame, and different feature contours in the manufacturing process, such as the molten pool, are manually annotated.

[0070] The feature segmentation model is trained using the feature segmentation dataset, with the reference brightness matrix GT t is the input and the corresponding manual annotation result is the output.

[0071] Step 2: Build an online monitoring model for additive manufacturing.

[0072] The online monitoring model for additive manufacturing includes a video super-resolution model based on an attention mechanism and a feature segmentation model based on a DeepLab network. The video super-resolution model takes the video sequence captured by a high-speed camera as input, and the feature segmentation model takes the output of the video super-resolution model as input. The entire model ultimately outputs a high-resolution feature segmentation map.

[0073] (1) Video super-resolution model ViTSR based on attention mechanism.

[0074] Construct a spatiotemporal encoder to encode the input sequence {X t-N ,…,X t ,..,X t+N} concatenates the encodings in time and space in its last dimension. For any frame X in the input sequence iThe temporal encoder (TE) of (-N≤i≤N) represents the relative position of the frame in the sequence. Its matrix size is (h0 / r,w0 / r,1), and each position is filled with The spatial encoder (SE) of any frame in the input sequence represents the position of any pixel in the frame, which is represented by the horizontal encoder SE. x and longitudinal SE y Composition, SE x and SE y The size of is (h0,w0,1). Let (m,n) be any frame X i The position coordinates of a pixel in , 1≤m≤w0, 1≤n≤h0, then the corresponding spatial encoding values ​​are as follows:

[0075] SE x (m,n)=sin(mπ / w0-π / 2)

[0076] SE y (m,n)=sin(nπ / h0-π / 2)

[0077] Therefore, the input sequence {X t-N ,…,X t ,..,X t+N}The matrix shape of any element after splicing spatiotemporal coding is (h0,w0,4).

[0078] Define a residual feature extraction module. This module consists of n cascaded residual blocks. Each residual block contains a 3D convolution layer with a convolution kernel size of 1x1, a batch normalization layer, a 3D convolution layer with a convolution kernel size of 3x3, and a batch normalization layer. The two 3D convolution layers in each residual block have convolution kernels of 32*index and 32 according to their index index (1≤index≤n). The input of each residual block is the concatenation of the feature maps output by all previous residual blocks.

[0079] Define a query matrix generation module Gen_Q, which consists of k cascaded blocks. The structure of the first k-1 blocks contains 32n 3D convolution layers with a convolution kernel size of 1x1 and a batch normalization layer. The structure of the last block is a 3D convolution layer with a convolution kernel size of 3x3 and a batch normalization layer.

[0080] Define a key matrix generation module Gen_K, which consists of k cascaded blocks. The structure of the first k-1 blocks contains 32n 3D convolution layers with a convolution kernel size of 1x1 and a batch normalization layer. The structure of the last block is 2N 3D convolution layers with a convolution kernel size of 3x3 and a batch normalization layer.

[0081] Define a value matrix generation module Gen_V, which consists of k cascaded blocks. The structure of the first k-1 blocks contains 32n 3D convolution layers with a convolution kernel size of 1x1 and a batch normalization layer. The structure of the last block is 2N+1 3D convolution layers with a convolution kernel size of 3x3 and a batch normalization layer.

[0082] Based on the above description, it can be seen that the structures of the above three matrix generation modules are exactly the same, but the weights are different after training. In the subsequent steps, these three modules can be used to obtain matrices with the same shape but different values.

[0083] Construct a super-resolution model, and the network input is the input part of the data set in step 1 {X t-N ,…,X t ,..,X t+N}, the size is (h0 / r, w0 / r, 1); after being encoded by the spatiotemporal encoder TSE, it is then input into the residual feature extraction module to obtain the feature map matrix FM, and FM is input into the Gen_Q, Gen_K, and Gen_V modules (corresponding to Figure 4 The contents of the dashed box in the “Visual Transformer” in the t The query matrix Q t , key matrix {K i}(tN≤i≤t+N, i≠t), value matrix {V i}(tN≤i≤t+N). Using the query matrix Q t With the key matrix {K i}After obtaining the attention matrix and the value matrix {V i} and then multiplied with the value matrix V of the current frame t Add together to get the total feature map FM t , the specific calculation is as follows:

[0084]

[0085] The total feature map FM t Pixels are rearranged to obtain a super-resolution reconstruction result with a pixel resolution of (h0, w0, 1).

[0086] (2) Feature segmentation model FCN.

[0087] The feature segmentation model includes an encoder and a decoder. The encoder uses the open source Xception network as the backbone to extract features, and outputs the output feature extraction results and the feature extraction results after spatial pyramid pooling to the decoder respectively. The decoder uses 1×1 convolution and 4x upsampling to fuse the two results to obtain the final high-resolution segmentation result.

[0088] Step 3: Use the data set to train the online monitoring model, that is, train the neural network model.

[0089] Specifically, the data set produced in step 1 is used to train the neural network model constructed in step 2, the mean square error is used as the loss function, and the Adam optimizer is selected to tune the network weights, and finally a trained online monitoring model is obtained.

[0090] Step 4: Deploy the model to the edge computing device.

[0091] The trained online monitoring model is deployed on an edge computing device, and the edge computing device is used to obtain online monitoring information of additive manufacturing.

[0092] Specifically, the online monitoring model trained in step 3 can be quantized and pruned using the edge computing hardware SDK to obtain a model that is compatible with the edge computing hardware. A model session is created on the edge computing device and placed on the back end as a service. The relative positions of the high-speed camera and the metal powder bed are fixed, and the video stream captured by the high-speed camera is connected to the back-end interface to obtain the online monitoring results of the additive manufacturing process.

[0093] The present invention is described below with reference to the parameters.

[0094] Assume N=1, M=1000, h0=300, w0=300, n=6, k=2, r=4, the convolution step size is 1, and the activation function is the relu function.

[0095] The process of the present invention is mainly divided into four steps: data collection, model construction, model training, and model deployment. The integrated system is as follows: Figure 1 As shown. First, a high-speed camera is used to shoot the additive manufacturing process to build a data set, and then the network model is built and trained through the TensorFlow framework. Finally, the trained model is quantized, pruned and deployed on the edge hardware detection system. When the online monitoring method is applied, a high-speed camera is used to shoot the video of the additive manufacturing process. The edge computing device obtains a continuous 3-frame image sequence to perform super-resolution reconstruction on the current frame. Then, a feature extraction model is used to extract features such as plasma arc and molten pool from the current frame to achieve the purpose of online monitoring of the additive manufacturing process features.

[0096] Step 1: Create a dataset for additive manufacturing.

[0097] (1) Video Super-Resolution Dataset.

[0098] See also Figure 2 , the additive manufacturing process is video-shot. The manufacturing process video has 1000 frames in total. Three frames are captured continuously from the beginning with a frame interval of 1, thereby obtaining the original image sequence {Im t}(1≤t≤1000), the reference brightness matrix sequence of the additive manufacturing image is obtained by RGB-YCbCr color space conversion {GT t}(1≤t≤1000), the formula is as follows:

[0099]

[0100] Among them, R, G, B represent the component matrices of the image to be converted on the three color channels of R, G, and B, respectively, and Y, Cb, and Cr represent the component matrices of the converted image on brightness, blue concentration offset, and red concentration offset. Here, only the data of the Y channel is taken, that is, only the brightness information of the image is retained to compress the amount of data and reduce the amount of calculation of the subsequent neural network. Therefore, any matrix GT in the brightness matrix sequence t The dimensions are (300,300,1).

[0101] The brightness matrix sequence {GT t The resolution of} is reduced to 1 / 4 of the original to obtain a low-resolution brightness matrix sequence {X t}(1≤t≤1000).

[0102] The video super-resolution model is trained using the video super-resolution dataset, with a low-resolution brightness matrix sequence {X t} as input, with the reference brightness matrix sequence {GT t} as output.

[0103] The original image sequence of the additive manufacturing process described in step 1 {Im t}, the size of each element matrix is ​​(300, 300, 3), and the reference brightness matrix sequence {GT t The size of each element matrix in} is (300, 300, 1); the low-resolution brightness matrix sequence {X t}The size of each element matrix is ​​(75,75,1).

[0104] (2) Feature segmentation dataset.

[0105] See also Figure 3 , the feature segmentation model is trained using the feature segmentation dataset, with video frames as input and corresponding manual annotation results as output.

[0106] The additive manufacturing video shot by the aforementioned high-speed camera was saved frame by frame, and the Labelme feature annotation tool was used to manually annotate different feature contours in the manufacturing process, including the molten pool and plasma arc.

[0107] Step 2: Build an online monitoring model for additive manufacturing.

[0108] (1) Video super-resolution model ViTSR based on attention mechanism.

[0109] The structure of the video super-resolution network (Visual Transformer Super Resolution, ViTSR) can be found in Figure 4 , construct a spatiotemporal encoder for the low-resolution brightness matrix sequence {X t-N ,…,X t ,..,X t+N} concatenates the encodings in time and space in its last dimension. For any frame X in the input sequence i The temporal encoder (TE) of (-1≤i≤1) indicates the relative position of the frame in the sequence. Its matrix size is (300, 300, 1), and each position is filled with The spatial encoder (SE) of any frame in the input sequence represents the position of any pixel in the frame, which is represented by the horizontal encoder SE. x and longitudinal SE y Composition, SE x and SE y The size of is (300, 300, 1). Let (m,n) be any frame X i The position coordinates of a pixel in , 1≤m≤w0, 1≤n≤h0, then the corresponding spatial encoding values ​​are as follows:

[0110] SE x (m,n)=sin(mπ / 300-π / 2)

[0111] SE y (m,n)=sin(nπ / 300-π / 2)

[0112] Define a residual feature extraction module, which consists of 6 cascaded residual blocks. Each residual block contains a 3D convolution layer with a convolution kernel size of 1x1, a batch normalization layer, a 3D convolution layer with a convolution kernel size of 3x3, and a batch normalization layer. The two 3D convolution layers in each residual block have convolution kernels of 32*index and 32 according to their index index (1≤index≤5). The input of each residual block is the concatenation of the feature maps output by all previous residual blocks.

[0113] Define a query matrix generation module Gen_Q, which consists of two cascaded blocks. The structure of the first block contains 160 3D convolution layers with a convolution kernel size of 1x1 and a batch normalization layer. The structure of the last block is 1 3D convolution layer with a convolution kernel size of 3x3 and a batch normalization layer.

[0114] Define a key matrix generation module Gen_K, which consists of 2 cascaded blocks. The structure of the first block contains 160 3D convolution layers with a convolution kernel size of 1x1 and a batch normalization layer. The structure of the last block is 2 3D convolution layers with a convolution kernel size of 3x3 and a batch normalization layer, where r is the multiple of super-resolution reconstruction.

[0115] Define a value matrix generation module Gen_V, which consists of k cascaded blocks. The structure of the first block contains 160 3D convolution layers with a convolution kernel size of 1x1 and a batch normalization layer. The structure of the last block consists of 3 3D convolution layers with a convolution kernel size of 3x3 and a batch normalization layer, where r is the multiple of super-resolution reconstruction.

[0116] That is Figure 4 Construct a multi-branch neural network, the network input is the input part of the data set in step 1, that is, the low-resolution brightness matrix sequence {X t-1 ,X t ,X t+1}, the dimensions are (75, 75, 1). t After the residual feature extraction module, it is input into the Gen_Q, Gen_K, and Gen_V modules to obtain X t The query matrix Q t , key matrix {K i-1 , K i+1}、Value matrix {V i-1 ,V i ,V i+1}. Using the query matrix Q t With the key matrix {K i}After obtaining the attention matrix and the value matrix {V i} and then multiplied with the value matrix V of the current frame t Add together to get the total feature map FM t , the specific calculation is as follows:

[0117]

[0118] The total feature map FM t Rearrange pixels to obtain high-frequency feature maps for super-resolution reconstruction X t Perform r times bicubic interpolation to obtain the low-frequency feature map B t ,Will With B t Add together to get the super-resolution reconstruction brightness matrix Y with a size of (300, 300, 1) t result.

[0119] (2) Feature segmentation model FCN.

[0120] join Figure 5 The feature segmentation model includes an encoder and a decoder. The encoder uses the open source Xception network as the backbone to extract features, and outputs the output feature extraction results and the feature extraction results after spatial pyramid pooling to the decoder respectively. The decoder uses 1×1 convolution and 4x upsampling to fuse the two results to obtain the final high-resolution segmentation result.

[0121] Specifically, Figure 5 As shown in (a), the feature segmentation model receives the super-resolution reconstruction brightness matrix Y t As input, the Xception network is used to extract image features. The image features are calculated in two ways, one is directly output to the decoder, and the other is output to the decoder after pooling the image features through the spatial pyramid and image pooling. The decoder uses 1×1 convolution to decode the image features, and uses 1×1 convolution and 4 times upsampling to decode the pooled features. After merging the two feature maps, 1×1 convolution and 4 times upsampling are used to obtain the final segmentation result. Figure 5 (b) in Figure 2 shows the detailed structure of the Xception backbone network.

[0122] Step 3: Use the dataset to train the online monitoring model.

[0123] This step uses the data set created in step 1 and the neural network built in step 2. The mean square error is used as the loss function, and the Adam optimizer is selected to tune the network weights. The training is iterated 1000 times, and the model with the smallest error is retained.

[0124] Figure 6 In order to utilize the trained online monitoring model, the super-resolution effect map based on the video sequence is obtained. Specifically, Figure 6 The results of 4x super-resolution reconstruction of continuous additive manufacturing process images by the video super-resolution network proposed in the present invention are demonstrated (shooting rate: 30,000 frames / second, T=1 / 30 millisecond). The super-resolution reconstructed images are clear in details, continuous in image, and free of artifacts.

[0125] In order to better verify the effect of the present invention, the present invention is compared with several existing conventional methods below.

[0126] Existing super-resolution methods include Bicubic, SRCNN, FSRCNN, ESPCN, DUF, TGA, see Figure 7 The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the super-resolution reconstruction results of this method (ViTSR) are better than those of other super-resolution methods and interpolation methods.

[0127] The feature segmentation results obtained using the commonly used feature extraction methods are as follows: Figure 8As shown in (a) to (f) of Figure 8 (a) in corresponds to the triangle method, such as Figure 8 (b) in corresponds to the maximum entropy method, such as Figure 8 (c) in corresponds to the watershed method, such as Figure 8 (d) in corresponds to the Otsu method, such as Figure 8 (e) in corresponds to the basic global threshold method, such as Figure 8 (f) in FIG. 1 corresponds to the full convolutional network method. Under the condition of 75 pixels * 75 pixels as input, the segmentation result obtained by using the present invention is shown in FIG. Figure 8 As shown in (g) in FIG. 1 , it can be seen from the comparison that the online monitoring model proposed in the present invention has higher resolution and higher accuracy than other methods for segmenting the plasma arc and the molten pool under the same conditions.

[0128] Step 4: Deploy the model to the edge computing device.

[0129] The relative positions of the high-speed camera and the screening mechanism are fixed, and the video super-resolution model ViTSR trained in step 3 is quantized and pruned to obtain the lightweight inference model ViTSR'. Combined with the feature segmentation model FCN, it is deployed on the hardware computing platform to determine in real time whether there is any abnormality in the printing process.

[0130] Fig. 9 The video super-resolution method of this method and other methods show the average value of inference time and Peak Signal to Noise Ratio (PSNR) on the test set. It can be seen that this method has the characteristics of short inference time and high image quality index (PSNR).

[0131] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. An online monitoring method for additive manufacturing based on edge computing, characterized in that: The following steps are involved: Building a dataset for additive manufacturing; Constructing an online monitoring model for additive manufacturing, the online monitoring model comprising a video super-resolution model and a feature segmentation model; the video super-resolution model takes a video sequence as input, the feature segmentation model takes an output of the video super-resolution model as input, and the feature segmentation model outputs a feature segmentation map; The video super-resolution model includes a spatiotemporal encoder, a residual feature extraction module, a query matrix generation module, a key matrix generation module and a value matrix generation module; the pixel resolution of any frame image in the input video sequence is (h0 / r, w0 / r, 1), h0 and w0 are the number of pixels of the image in the height and width directions respectively, and r is the multiple of super-resolution reconstruction; the video sequence is encoded by the spatiotemporal encoder and input into the residual feature extraction module to obtain a feature map matrix, and the feature map matrix is ​​respectively input into the query matrix generation module, the key matrix generation module and the value matrix generation module to obtain a query matrix, a key matrix and a value matrix respectively; a total feature map is obtained based on the query matrix, the key matrix and the value matrix; the total feature map is pixel-rearranged to obtain a super-resolution reconstruction result with a pixel resolution of (h0, w0, 1); Using the data set to train the online monitoring model to obtain a trained online monitoring model; The trained online monitoring model is deployed on an edge computing device, and the edge computing device is used to obtain online monitoring information of additive manufacturing.

2. The method for online monitoring of additive manufacturing based on edge computing according to claim 1, characterized in that: The residual feature extraction module includes n cascaded residual blocks; each of the residual blocks includes a first 3D convolutional layer with a convolution kernel size of 1x1, a first batch of normalization layers, a second 3D convolutional layer with a convolution kernel size of 3x3, and a second batch of normalization layers; the number of convolution kernels of the first 3D convolutional layer is 32*index, index is index, 1≤index≤n; the number of convolution kernels of the second 3D convolutional layer is 32.

3. The method for online monitoring of additive manufacturing based on edge computing according to claim 1, characterized in that: The query matrix generation module, the key matrix generation module and the value matrix generation module have the same structure, and all include k cascaded blocks; the structures of the first k-1 blocks all include a third 3D convolution layer with a convolution kernel size of 1x1 and a third batch normalization layer, and the number of convolution kernels of the third 3D convolution layer is 32n, where n is the number of residual blocks included in the residual feature extraction module; the structure of the last block includes a fourth 3D convolution layer with a convolution kernel size of 3x3 and a fourth batch normalization layer, and the number of convolution kernels of the fourth 3D convolution layer is 1.

4. The method for online monitoring of additive manufacturing based on edge computing according to claim 3 is characterized in that: The query matrix generation module, the key matrix generation module and the value matrix generation module have different weights after training. By using the query matrix generation module, the key matrix generation module and the value matrix generation module, matrices with the same shape but different values ​​can be obtained.

5. The method for online monitoring of additive manufacturing based on edge computing according to claim 1, characterized in that: The input video sequence is denoted as {X t-N ,…,X t ,..,X t+N }; The query matrix is ​​recorded as Q t ; The key matrix is ​​denoted as {K i }, tN≤i≤t+N, i≠t; the value matrix is ​​recorded as {V i }, tN≤i≤t+N; the value matrix of the current frame is recorded as V t ; The total feature map is calculated as follows: Where FM t is the total feature map.

6. The method for online monitoring of additive manufacturing based on edge computing according to claim 1, characterized in that: The feature segmentation model includes an encoder and a decoder; the encoder uses the Xception network as the backbone to extract features, and the encoder outputs the feature extraction results output by the video super-resolution model and the feature extraction results after spatial pyramid pooling to the decoder respectively; the decoder uses 1×1 convolution and 4 times upsampling to fuse the two feature extraction results and output a high-resolution feature segmentation map.

7. The method for online monitoring of additive manufacturing based on edge computing according to claim 1, characterized in that: The data set includes a video super-resolution data set and a feature segmentation data set; The additive manufacturing process is photographed using a high-speed camera to obtain an original image sequence in the RGB color space; a reference brightness matrix sequence is obtained through RGB-YCbCr color space conversion; Using the nearest neighbor interpolation method to reduce the resolution of the reference brightness matrix sequence to 1 / r of the original resolution, to obtain a low-resolution brightness matrix sequence; The video super-resolution model is trained using the video super-resolution dataset, with the low-resolution brightness matrix sequence as input and the reference brightness matrix sequence as output; The feature segmentation model is trained using the feature segmentation data set, with the reference brightness matrix as input and a feature segmentation map annotated with different feature contours in the additive manufacturing process as output.

8. The method for online monitoring of additive manufacturing based on edge computing according to claim 1, characterized in that: When training the online monitoring model, the mean square error is used as the loss function, and the Adam optimizer is selected to tune the network weights.

9. The method for online monitoring of additive manufacturing based on edge computing according to claim 1, characterized in that: The trained online monitoring model is quantized and pruned using the edge computing hardware SDK to obtain a model that is compatible with the hardware of the edge computing device; a model session is created in the edge computing device and placed in the backend as a service, the relative positions of the high-speed camera and the metal powder bed are fixed, the video sequence captured by the high-speed camera is connected to the backend interface, and the edge computing device is used to obtain online monitoring information of additive manufacturing.

10. An additive manufacturing online monitoring system based on edge computing, characterized in that: include: edge computing devices and high-speed cameras; The high-speed camera is used to shoot the additive manufacturing process and transmit the captured video sequence to the edge computing device; A trained online monitoring model is deployed on the edge computing device, and the edge computing device is used to obtain online monitoring information of additive manufacturing; The additive manufacturing online monitoring system based on edge computing is used to execute the steps in the additive manufacturing online monitoring method based on edge computing as described in any one of claims 1 to 9.

Images