Method and device for acquiring molten pool features based on improved U-Net

Abstract

The present application belongs to the field of machine vision technology, and specifically discloses a method and device for acquiring melt pool features based on an improved U-Net, wherein the method includes: collecting an image of the melt pool, segmenting the image of the melt pool, and acquiring a partial image of the melt pool; inputting the partial image of the melt pool into an improved U-Net model to acquire melt pool features; the improved U-Net model includes an encoder, a jump connection, a decoder, and an output layer, and the encoding area of ​​the encoder includes a bneck module of a MobileNet V3 module. The present application realizes online detection of melt pool features through a lightweight model with high detection accuracy and high speed.

Description

Technical Field

[0001] The present application belongs to the field of machine vision technology, and more specifically, to a method and device for acquiring molten pool features based on an improved U-Net. Background Art

[0002] Laser additive manufacturing is considered to be a technology that can promote the next industrial revolution. This technology is widely used in industrial manufacturing, aerospace, medicine, shipbuilding and other fields. However, due to its inherent characteristics such as rapid heating and cooling, defects such as porosity, cracks and warping deformation are prone to occur during its forming process. The molten pool is the most basic unit in additive manufacturing. The characteristics of the molten pool reflect the stability of the manufacturing process and are also the root cause of defects. Therefore, the extraction and analysis of the molten pool geometry, so as to feedback and control the key parameters in the additive manufacturing process, is currently an effective means to improve the quality stability and process repeatability of additively manufactured parts.

[0003] In the related technology, there are two main methods for detecting melt pool features: traditional image segmentation algorithms and semantic segmentation algorithms based on deep learning. However, traditional segmentation algorithms mainly use the differential geometry characteristics of images, which are seriously affected by grayscale changes and fail to achieve true semantic segmentation and complete accurate identification and positioning of the melt pool. The semantic segmentation algorithm has high detection accuracy but has problems such as too many parameters and long processing time, which makes it impossible to observe a large number of details in the manufacturing process. In addition, due to the high computational requirements and the limited computing performance and storage resources of embedded devices, these devices are not suitable for large-scale deployment. Therefore, how to provide a lightweight model with high detection accuracy and fast speed to realize online detection of melt pool features has become an urgent problem to be solved. Summary of the invention

[0004] In view of the defects of the related art, the purpose of this application is to provide a method and device for acquiring molten pool features based on an improved U-Net, aiming to solve the problems of low feature recognition accuracy and slow speed during molten pool detection in the related art.

[0005] In a first aspect, the present application provides a method for acquiring melt pool features based on an improved U-Net, the method comprising:

[0006] Collecting an image of the molten pool, segmenting the image of the molten pool, and obtaining a partial image of the molten pool;

[0007] Input the partial image of the melt pool into the improved U-Net model to obtain the melt pool features;

[0008] Among them, the improved U-Net model includes an encoder, a skip connection, a decoder and an output layer, and the encoding area of ​​the encoder includes the bneck module of the MobileNet V3 module.

[0009] In some embodiments, the encoder is a multi-layer vertical progressive structure, each layer includes at least one bneck module; the decoder includes multiple layers, and the multiple layers of the decoder are symmetrical with the vertical progressive structure of the encoder; the output feature maps of each layer of the encoder are transmitted to the multiple layers in the decoder through jump connections, and the jump connections include convolutional attention modules CBAM.

[0010] In some embodiments, the image of the melt pool portion is input into the improved U-Net model to obtain melt pool features, including:

[0011] Inputting the melt pool partial image into the encoder so that at least one bneck module of each layer performs feature extraction on the melt pool partial image;

[0012] The output feature maps of each layer of the encoder are transmitted to the decoder through skip connections to obtain the feature maps output by the decoder;

[0013] Through the 1×1 convolution of the output layer, the feature map output by the decoder is converted into a 2D channel feature map. Each pixel is classified one by one through the Softmax classifier, and the white pixels are obtained as the melt pool features.

[0014] In some embodiments, the step of performing feature extraction by the bneck module includes:

[0015] Use 1×1 convolution to increase the dimension of the melt pool part image input to the bneck module;

[0016] The 3×3 depthwise separable convolution is used to extract features from the melt pool image after the dimension increase operation, and a feature map is obtained in which each channel has undergone independent convolution operation.

[0017] Perform global average pooling on the feature map through the SE module, adjust the weight of each channel of the feature map, and obtain the feature map after channel weight adjustment;

[0018] Use a 1×1 convolution kernel to reduce the dimension of the feature map after channel weight adjustment to obtain the initial output feature map of the bneck module;

[0019] Based on the melt pool partial image and the output feature map of the input bneck module, a residual connection operation is performed to obtain the output feature map of the bneck module.

[0020] In some embodiments, the improved U-Net model is trained based on the following steps:

[0021] Obtain partial image samples of the melt pool and construct an initial improved U-Net model. The VGG convolutional neural network in the backbone network of the initial improved U-Net model is replaced by the MobileNet V3 module.

[0022] The initial improved U-Net model is trained by using partial image samples of the melt pool, loss functions and evaluation indicators to obtain an improved U-Net model;

[0023] Among them, the loss function is expressed as:

[0024]

[0025] Among them, y∈{0,1} represents the labeled value, x∈{0,1} represents the model output, 0 and 1 represent the melt pool feature and background feature respectively; M represents the penalty map of the melt pool part image sample, L represents the loss function, L' represents the intermediate form of the loss function, and m represents the total number of samples;

[0026] The evaluation index is expressed as:

[0027]

[0028] Among them, p ji represents the number of pixels j predicted as pixels i, p ij Indicates the number of pixels i predicted as pixels j, p ii It indicates the number of pixels predicted as i pixels, i and j correspond to the melt pool feature and background feature respectively.

[0029] In some embodiments, obtaining a sample of a molten pool portion image includes:

[0030] Collecting original image data of the molten pool, and preprocessing the original image data to obtain preprocessed image data;

[0031] The Labelme segmentation and annotation tool is used to annotate the melt pool part and / or the background part of the melt pool part image samples obtained by preprocessing to obtain labeled melt pool part image samples, wherein the labeled melt pool part image samples are divided into training samples, test samples and verification samples.

[0032] In some embodiments, preprocessing the original image data to obtain preprocessed image data includes:

[0033] The original image data is subjected to filtering processing, image graying processing, image graying stretching processing and molten pool partial image cropping in sequence, so that the resolution size of the obtained molten pool partial image samples is uniform.

[0034] In a second aspect, the present application provides a molten pool feature acquisition device based on an improved U-Net, the device comprising:

[0035] An image acquisition module, used for acquiring an image of the molten pool, segmenting the image of the molten pool, and obtaining a partial image of the molten pool;

[0036] A feature acquisition module is used to input the partial image of the melt pool into the improved U-Net model to obtain the melt pool features;

[0037] Among them, the improved U-Net model includes an encoder, a skip connection, a decoder and an output layer, and the encoding area of ​​the encoder includes the bneck module of the MobileNet V3 module.

[0038] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0040] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0041] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the related art:

[0042] (1) The model was constructed using the improved U-Net neural network, and the model with the best performance was selected through multiple iterations of training and MIoU. Finally, the model with the highest accuracy was selected for practical application, thus ensuring that the model can achieve high-precision melt pool image segmentation.

[0043] (2) The improved U-Net neural network introduces the SE module and the convolutional attention mechanism in the step connection part, so that the network pays attention to spatial and channel information, suppresses invalid features, and speeds up the network reasoning. The introduction of deep separable convolution and inverted residual modules can reduce the total number of model parameters, achieve network compression and lightweight, and enable this solution to achieve online real-time detection of the melt pool size.

[0044] (3) The improved U-Net model has fewer parameters, which is conducive to deployment in embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 It is a flow chart of a method for acquiring molten pool features based on an improved U-Net provided in an embodiment of the present application;

[0047] Figure 2 It is a schematic diagram of the structure of the improved U-Net model provided in the embodiment of the present application;

[0048] Figure 3 It is a schematic diagram of the structure of the CBAM attention mechanism provided in the embodiment of the present application;

[0049] Figure 4 It is a structural diagram of the bneck module provided in an embodiment of the present application;

[0050] Figure 5 It is a structural schematic diagram of a molten pool feature acquisition device based on an improved U-Net provided in an embodiment of the present application;

[0051] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Figure 1 is a flow chart of a method for acquiring molten pool features based on an improved U-Net provided in an embodiment of the present application, such as Figure 1 As shown, the method at least includes the following steps:

[0054] S101, collecting an image of a molten pool, segmenting the image of the molten pool, and obtaining a partial image of the molten pool.

[0055] Optionally, the molten pool may be a molten pool in laser additive manufacturing, or a molten pool in the manufacturing of other materials. The embodiments of the present application take the molten pool in laser additive manufacturing as an example, but this is not intended to limit the molten pool.

[0056] Specifically, in order to obtain the melt pool characteristics and detect the melt pool conditions in real time, a high-speed camera can be used to collect the melt pool image flow information in laser additive manufacturing, and through preprocessing operations such as image segmentation, the minimum bounding box of the melt pool in the image can be cut out as an improved U-Net model.

[0057] Optionally, after obtaining the original image of the molten pool, the original image of the molten pool can be preprocessed first. The preprocessing may include image filtering, image grayscale, image grayscale stretching and molten pool area cropping. In the embodiment of the present application, the collected molten pool image is preprocessed to eliminate noise, reduce the impact of noise on the image and highlight the molten pool object.

[0058] Optionally, after obtaining the melt pool image flow information, sampling can be performed based on a certain period, and then preprocessing operations such as image segmentation can be performed to cut out the minimum bounding box of the melt pool in the image. This can be used as an improved U-Net model to effectively reduce the amount of calculation.

[0059] Optionally, when performing image cutting, the image size can be adjusted (for example, by performing preprocessing operations such as image stretching) before cutting to ensure that the resolution size of the minimum bounding box of the molten pool in the cut image (the molten pool part image) is unified to a size suitable for the input of the improved U-Net model, for example, it can be unified to a resolution size of 224×224.

[0060] For example, the specific steps of image preprocessing may include:

[0061] Step a: Use a 3×3 square window to perform median filtering on the molten pool image. This method removes noise while ensuring good edge characteristics of the molten pool image.

[0062] Step b: grayscale the molten pool image using the average value method.

[0063] Step c: using exponential transformation to highlight the molten pool part in the molten pool image.

[0064] Step d: Select the object with the largest bright surface area and expand its minimum bounding box to the RoI area with a resolution of 224×224, and then cut the minimum bounding box.

[0065] S102, input the partial image of the molten pool into the improved U-Net model to obtain the molten pool features.

[0066] Among them, the improved U-Net model includes an encoder, a skip connection, a decoder and an output layer, and the encoding area of ​​the encoder includes the bneck module of the MobileNet V3 module.

[0067] Specifically, there will still be some background pixels in the minimum bounding box of the molten pool in the cut image (partial image of the molten pool), so the molten pool features can be obtained by inputting the partial image of the molten pool into the improved U-Net model; wherein, the improved U-Net model is a pre-trained model, and the trained improved U-Net model is deployed on the embedded development board to segment the molten pool image in laser additive manufacturing collected in real time, so as to realize real-time detection of molten pool features.

[0068] The improved U-Net model in the embodiment of the present application adopts the MobileNet V3 module to replace the VGG convolutional neural network adopted in the backbone network of the U-Net model in the related art; the SE module in the bneck module in the MobileNet V3 module can enhance the feature representation, and improve the expressiveness of the model by dynamically adjusting the weights of each channel in the feature map. The MobileNetV3 structure is usually more streamlined and optimized for edge computing. It uses fewer computing resources while still maintaining high performance, so that the computational efficiency of the model is also guaranteed; and the MobileNetV3 structure reduces the number of convolutional layers, which can improve the computational efficiency of the model.

[0069] In some embodiments, the encoder is a multi-layer vertical progressive structure, each layer includes at least one bneck module; the decoder includes multiple layers, and the multiple layers of the decoder are symmetrical with the vertical progressive structure of the encoder; the output feature maps of each layer of the encoder are transmitted to the multiple layers in the decoder through jump connections, and the jump connections include convolutional attention modules CBAM.

[0070] Figure 2 is a schematic diagram of the structure of the improved U-Net model provided in the embodiment of the present application, such as Figure 2 As shown in the figure, the improved U-Net model includes an encoder, a skip connection, a decoder and an output layer. The encoder is a multi-level vertical progressive structure, and each level includes at least one bneck module; the bneck module includes operations such as depthwise separable convolution, SE module, inverted residual module, lightweight attention model, h swish activation function, etc.; among them, the SE module can enhance the feature representation and improve the expressiveness of the model by dynamically adjusting the weights of each channel in the feature map. The depthwise separable convolution significantly reduces the number of parameters and the amount of calculation; the inverted residual module reduces information loss by introducing short-circuit connections, making the network easier to train and more computationally efficient.

[0071] like Figure 2As shown in the figure, the decoder of the improved U-Net model also includes multiple layers, and its structure is symmetrical to that of the encoder; for example, the encoder includes 4 layers, and the decoder also includes 4 layers. The output feature map of each stage of the encoder is transmitted to the decoder through a skip connection, realizing the fusion of shallow information and deep information, and providing more semantic information for the decoding process;

[0072] Figure 3 : is a schematic diagram of the structure of the CBAM attention mechanism provided in the embodiment of the present application, such as Figure 3 As shown in the figure, the skip connection introduces the Convolutional Block Attention Module (CBAM), which enables the improved U-Net model to focus on the feature information and channel information of the target pixel, suppress invalid features, and improve the training efficiency of the improved U-Net model.

[0073] In some embodiments, the image of the melt pool portion is input into the improved U-Net model to obtain melt pool features, including:

[0074] Inputting the melt pool partial image into the encoder so that at least one bneck module of each layer performs feature extraction on the melt pool partial image;

[0075] The output feature maps of each layer of the encoder are transmitted to the decoder through skip connections to obtain the feature maps output by the decoder;

[0076] Through the 1×1 convolution of the output layer, the feature map output by the decoder is converted into a 2D channel feature map. Each pixel is classified one by one through the Softmax classifier, and the white pixels are obtained as the melt pool features.

[0077] Specifically, the improved U-Net model includes an encoder, a skip connection, a decoder and an output layer, and the encoding area of ​​the encoder includes the bneck module of the MobileNet V3 module; when applying the trained improved U-Net model to obtain the melt pool feature, the melt pool part image can be input into the encoder, at least one bneck module of each layer performs feature extraction on the melt pool part image, and the output feature map of each layer of the encoder is transmitted to the decoder through the skip connection to obtain the feature map output by the decoder, and the feature map output by the decoder is converted into a 2D channel feature map through the 1×1 convolution of the output layer, and each pixel is classified one by one through the Softmax classifier, so that the white pixels are the melt pool features and the black pixels are the background features.

[0078] In some embodiments, Figure 4 is a schematic diagram of the structure of the bneck module provided in the embodiment of the present application, such as Figure 4 As shown, the steps of feature extraction by the bneck module include:

[0079] Use 1×1 convolution to increase the dimension of the melt pool part image input to the bneck module;

[0080] The 3×3 depthwise separable convolution is used to extract features from the melt pool image after the dimension increase operation, and a feature map is obtained in which each channel has undergone independent convolution operation.

[0081] Perform global average pooling on the feature map through the SE module, adjust the weight of each channel of the feature map, and obtain the feature map after channel weight adjustment;

[0082] Use a 1×1 convolution kernel to reduce the dimension of the feature map after channel weight adjustment to obtain the initial output feature map of the bneck module;

[0083] Based on the melt pool partial image and the output feature map of the input bneck module, a residual connection operation is performed to obtain the output feature map of the bneck module.

[0084] Specifically, the feature map of the melt pool input to the bneck module is subjected to feature extraction by the bneck structure, which includes the following steps, wherein the size (height and width) and the number of channels (depth) of the feature map are known:

[0085] Step a: The first 1×1 convolution (dimensionality increase).

[0086] A 1×1 convolution kernel is used to perform a convolution operation on the melt pool part image (Feature Map) of the input bneck module. The number of output channels is usually several times more than the number of input channels (for example, it may be 6 times in MobileNetV3), and a feature map with unchanged spatial size but increased number of channels is obtained.

[0087] Step b: Depthwise Convolution.

[0088] Use the same number of convolution kernels as the number of channels of the melt pool image after the dimension increase operation (each convolution kernel acts on only one channel) to perform a convolution operation to obtain a feature map with the same spatial size and number of channels, but each channel of the feature map has undergone an independent convolution operation;

[0089] Step c, SE module.

[0090] After the depthwise separable convolution, the feature maps of each channel that have undergone independent convolution operations are added to the SE module. The SE module first performs global average pooling on the feature maps of each channel that have undergone independent convolution operations, then learns the importance weights of each channel through two fully connected layers, and finally applies these weights back to the original feature maps for feature map scaling to obtain feature maps with channel weight adjustment.

[0091] Step d, second 1×1 convolution (dimensionality reduction).

[0092] The feature map after the SE module is convolved again with a 1×1 convolution kernel. The number of output channels is determined according to the design (usually less than the number of output channels of the first 1×1 convolution), and a feature map with unchanged spatial size but reduced number of channels is obtained. This feature map is the initial output feature map of the bneck structure.

[0093] Step e: residual connection.

[0094] When the number of input channels is equal to the number of output channels, the molten pool part image input to the BNeck module is added to the initial output feature map to obtain a feature map that combines the original input information and the feature information extracted from the BNeck structure as the output feature map of the BNeck module.

[0095] Optionally, the activation function of the bneck module adopts ReLU6 and / or Hard-Swish.

[0096] Among them, the ReLU6 activation function can be expressed as:

[0097] ReLU(6)=min(max(0,x))

[0098] Among them, the Hard-Swish activation function can be expressed as:

[0099]

[0100] Among them, x represents the input of the activation function.

[0101] In some embodiments, the improved U-Net model is trained based on the following steps:

[0102] Obtain partial image samples of the melt pool and construct an initial improved U-Net model. The VGG convolutional neural network in the backbone network of the initial improved U-Net model is replaced by the MobileNet V3 module.

[0103] The initial improved U-Net model is trained by using partial image samples of the melt pool, loss functions and evaluation indicators to obtain an improved U-Net model;

[0104] Among them, the loss function is expressed as:

[0105]

[0106] Where y∈{0,1} represents the labeled value, x∈{0,1} represents the model output, 0 and 1 represent the melt pool feature and background feature respectively; M represents the penalty map of the melt pool image sample, L represents the loss function, L' represents the intermediate form of the loss function, and m represents the total number of samples;

[0107] The evaluation index is expressed as:

[0108]

[0109] Among them, p ji represents the number of pixels j predicted as pixels i, p ij Indicates the number of pixels i predicted as pixels j, p ii It indicates the number of pixels predicted as i pixels, i and j correspond to the melt pool feature and background feature respectively.

[0110] Specifically, the training set, test set, and validation set in the partial image samples of the molten pool can be used to train, test, and validate the initial improved U-Net model, respectively. First, the training set is input into the initial improved U-Net model for iterative training; and the final model is optimized in combination with the loss function and evaluation index for real-time detection of the molten pool size in laser additive manufacturing.

[0111] Optionally, when defining the loss function, a penalty map (M) can be first created for each melt pool part image sample to specify a penalty factor for each pixel, wherein a high penalty factor is assigned to a highlight interference part (such as spatter), and the loss function is improved by using a specific penalty factor.

[0112] The loss function can be expressed as follows:

[0113]

[0114] Among them, y∈{0,1} represents the labeled value, and x∈{0,1} represents the model output, which has two categories: melt pool features (category 1) and background features (category 2).

[0115] Optionally, in order to minimize the loss function, an adaptive momentum estimation method (Adam) may be used as an optimization algorithm.

[0116] Optionally, the evaluation index can be MIoU, which is expressed as follows:

[0117]

[0118] Among them, p ji represents the number of pixels j predicted as pixels i, pij Indicates the number of pixels i predicted as pixels j, p ji It indicates the number of pixels predicted as i pixels, i and j correspond to the melt pool feature and background feature respectively.

[0119] In some embodiments, obtaining a sample of a molten pool portion image includes:

[0120] Collecting original image data of the molten pool, and preprocessing the original image data to obtain preprocessed image data;

[0121] The Labelme segmentation and annotation tool is used to annotate the melt pool part and / or the background part of the melt pool part image samples obtained by preprocessing to obtain labeled melt pool part image samples, wherein the labeled melt pool part image samples are divided into training samples, test samples and verification samples.

[0122] In some embodiments, preprocessing the original image data to obtain preprocessed image data includes:

[0123] The original image data is subjected to filtering processing, image graying processing, image graying stretching processing and molten pool partial image cropping in sequence, so that the resolution size of the obtained molten pool partial image samples is uniform.

[0124] Specifically, when making the melt pool partial image samples, the original image data of the melt pool can be first collected, and the original image data can be filtered, grayed, stretched, and cropped in sequence to make the resolution of the obtained melt pool partial image samples uniform, such as a RoI area with a resolution of 224×224, and then the minimum bounding box can be cut to obtain the melt pool partial image; the Labelme segmentation and annotation tool is used to annotate the melt pool part and / or background part at the pixel level on the preprocessed melt pool partial image samples to obtain labeled melt pool partial image samples, and the samples are divided into training set, test set, and validation set in a ratio of 6:2:2.

[0125] In one embodiment, the training method of the improved U-Net model provided in the embodiment of the present application includes the following steps:

[0126] Step 1: Use a high-speed camera to capture a dynamic molten pool image of laser additive manufacturing;

[0127] Step 2: pre-process the collected molten pool image to eliminate noise and highlight the molten pool object.

[0128] Furthermore, in step 2, the specific steps of image preprocessing include:

[0129] Step a: Use a 3×3 square window to perform median filtering on the molten pool image. This method removes noise while ensuring good edge characteristics of the molten pool image.

[0130] Step b: grayscale the image using the average value method.

[0131] Step c: using exponential transformation to highlight the molten pool part in the molten pool image.

[0132] Step d: Select the object with the largest bright surface area and expand its minimum bounding box to a 224×224 RoI area.

[0133] Step 3: Expert annotation is performed on the preprocessed melt pool images to construct a data set. The specific method of expert annotation is: use the Labelme tool to annotate the melt pool images at the pixel level, including melt pool features and background features, which are category 0 and category 1 respectively. The acquired melt pool image samples are divided into training set, test set and validation set in a ratio of 6:2:2.

[0134] Step 4: Build an improved U-Net model, which includes: encoder, skip connection, decoder and output layer.

[0135] Furthermore, the encoder presents a vertical progressive structure, which is used to gradually and deeply extract the features of the image samples of the melt pool, and the encoding area introduces the bneck module in MobileNet V3. Generally, the deeper the position, the more bneck modules are required, and the more advanced the abstract features extracted are. The bneck module helps to reduce the model calculation burden and the number of parameters while maintaining important semantic information. Furthermore, the bneck module includes the following stages:

[0136] Step a: Use 1×1 ordinary convolution to increase the dimension.

[0137] Step b: Use 3×3 depth-wise separable convolution to extract features in high-dimensional space.

[0138] Step c: introduce the SE module to dynamically adjust the weight of each channel. The activation functions used are as follows:

[0139] ReLU(6)=min(max(0,x))

[0140]

[0141] Step d: Continue with the 1×1 convolution down the channel, connecting with a residual link if and only if the input and output have the same number of channels.

[0142] Decoder: Restores image information layer by layer. Its structure is symmetrical with the encoder and has 4 stages. Each stage contains two 3×3 convolutions and an upsampling layer implemented by 2×2 deconvolution. After each stage, the scale of the output feature map is doubled and the channel dimension is halved.

[0143] Skip connection: The output feature map in each stage of the encoder is transmitted to the decoder through a skip connection. The connection here introduces a convolutional attention module (CBAM) to achieve the fusion of shallow information and deep information, providing more semantic information for the decoding process.

[0144] The output layer uses 1×1 convolution to convert the feature map into a 2D channel feature map, and classifies each pixel one by one through the Softmax classifier to obtain a segmentation map with white pixels as melt pool features and black pixels as background features.

[0145] Step 5: Input the preprocessed data set into the improved U-Net neural network model constructed above, perform iterative training, and optimize the model by combining the loss function and evaluation index.

[0146] Furthermore, the loss function is customized: first, a penalty map (M) is created for each sample of the melt pool image to specify the penalty factor for each pixel, where the highlight interference part (such as spatter) will be assigned a high penalty factor. The specific formula is as follows:

[0147]

[0148] y∈{0,1} represents the labeled value, and x∈{0,1} represents the model output, which has two categories: melt pool (category 1) and background (category 2).

[0149] Furthermore, the AdamW algorithm is used to continuously modify the network parameters to reduce the loss function. This can effectively enhance the stability of model training and accelerate the convergence speed, while overcoming the local convergence problem that may be encountered during the training process, thereby further improving the accuracy and speed of melt pool feature extraction.

[0150] Furthermore, the evaluation index uses MIoU, and the formula is as follows:

[0151]

[0152] Among them, p ji represents the number of pixels j predicted as pixels i, p ij Indicates the number of pixels i predicted as pixels j, p ii It indicates the number of pixels predicted as i pixels, i and j correspond to the melt pool feature and background feature respectively.

[0153] In an embodiment of the present application, a trained network model is deployed on an embedded development board to segment the molten pool image collected in real time during laser additive manufacturing, thereby achieving real-time detection of molten pool features.

[0154] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0155] Figure 5 is a schematic diagram of the structure of a molten pool feature acquisition device based on an improved U-Net provided in an embodiment of the present application, such as Figure 5 As shown, the device at least includes:

[0156] An image acquisition module 501 is used to acquire an image of the molten pool, segment the image of the molten pool, and obtain a partial image of the molten pool;

[0157] A feature acquisition module 502 is used to input the image of the molten pool portion into the improved U-Net model to obtain the molten pool features;

[0158] Among them, the improved U-Net model includes an encoder, a skip connection, a decoder and an output layer, and the encoding area of ​​the encoder includes the bneck module of the MobileNet V3 module.

[0159] In some embodiments, the encoder is a multi-layer vertical progressive structure, each layer includes at least one bneck module; the decoder includes multiple layers, and the multiple layers of the decoder are symmetrical with the vertical progressive structure of the encoder; the output feature maps of each layer of the encoder are transmitted to the multiple layers in the decoder through jump connections, and the jump connections include convolutional attention modules CBAM.

[0160] In some embodiments, the feature acquisition module 502 is specifically used to:

[0161] Inputting the melt pool partial image into the encoder so that at least one bneck module of each layer performs feature extraction on the melt pool partial image;

[0162] The output feature maps of each layer of the encoder are transmitted to the decoder through skip connections to obtain the feature maps output by the decoder;

[0163] Through the 1×1 convolution of the output layer, the feature map output by the decoder is converted into a 2D channel feature map. Each pixel is classified one by one through the Softmax classifier, and the white pixels are obtained as the melt pool features.

[0164] In some embodiments, the step of performing feature extraction by the bneck module includes:

[0165] Use 1×1 convolution to increase the dimension of the melt pool part image input to the bneck module;

[0166] The 3×3 depthwise separable convolution is used to extract features from the melt pool image after the dimension increase operation, and a feature map is obtained in which each channel has undergone independent convolution operation.

[0167] Perform global average pooling on the feature map through the SE module, adjust the weight of each channel of the feature map, and obtain the feature map after channel weight adjustment;

[0168] Use a 1×1 convolution kernel to reduce the dimension of the feature map after channel weight adjustment to obtain the initial output feature map of the bneck module;

[0169] Based on the melt pool partial image and the output feature map of the input bneck module, a residual connection operation is performed to obtain the output feature map of the bneck module.

[0170] In some embodiments, the improved U-Net model is trained based on the following steps:

[0171] Obtain partial image samples of the melt pool and construct an initial improved U-Net model. The VGG convolutional neural network in the backbone network of the initial improved U-Net model is replaced by the MobileNet V3 module.

[0172] The initial improved U-Net model is trained by using partial image samples of the melt pool, loss functions and evaluation indicators to obtain an improved U-Net model;

[0173] Among them, the loss function is expressed as:

[0174]

[0175] Among them, y∈{0,1} represents the labeled value, x∈{0,1} represents the model output, 0 and 1 represent the melt pool feature and background feature respectively; M represents the penalty map of the melt pool part image sample, L represents the loss function, L' represents the intermediate form of the loss function, and m represents the total number of samples;

[0176] The evaluation index is expressed as:

[0177]

[0178] Among them, p ji represents the number of pixels j predicted as pixels i, p ij Indicates the number of pixels i predicted as pixels j, p ii It indicates the number of pixels predicted as i pixels, i and j correspond to the melt pool feature and background feature respectively.

[0179] In some embodiments, obtaining a sample of a molten pool portion image includes:

[0180] Collecting original image data of the molten pool, and preprocessing the original image data to obtain preprocessed image data;

[0181] The Labelme segmentation and annotation tool is used to annotate the melt pool part and / or the background part of the melt pool part image samples obtained by preprocessing to obtain labeled melt pool part image samples, wherein the labeled melt pool part image samples are divided into training samples, test samples and verification samples.

[0182] In some embodiments, preprocessing the original image data to obtain preprocessed image data includes:

[0183] The original image data is subjected to filtering processing, image graying processing, image graying stretching processing and molten pool partial image cropping in sequence, so that the resolution size of the obtained molten pool partial image samples is uniform.

[0184] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0185] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0186] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device. The device may include: at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the above embodiment.

[0187] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device may include: a processor (processor) 601, a communication interface (Communications Interface) 602, a memory (memory) 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call the software instructions in the memory 603 to execute the method described in the above embodiment.

[0188] In addition, the logic instructions in the above-mentioned memory 603 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application.

[0189] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0190] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0191] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0192] The method steps in the embodiments of the present application can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0193] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0194] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0195] It is easy for those skilled in the art to understand that the above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for acquiring molten pool features based on improved U-Net, characterized in that: The method comprises: Collecting an image of the molten pool, segmenting the image of the molten pool, and obtaining a partial image of the molten pool; Inputting the partial image of the molten pool into the improved U-Net model to obtain the molten pool features; The improved U-Net model includes an encoder, a skip connection, a decoder and an output layer, and the encoding area of ​​the encoder includes a bneck module of the MobileNet V3 module.

2. The method for acquiring molten pool features based on improved U-Net according to claim 1, characterized in that: The encoder is a multi-layered longitudinal progressive structure, each layer includes at least one backneck module; the decoder includes multiple layers, and the multiple layers of the decoder are symmetrical with the longitudinal progressive structure of the encoder; The output feature maps of each level of the encoder are transmitted to multiple levels in the decoder via skip connections, and the skip connections include convolutional attention modules (CBAM).

3. The method for acquiring molten pool features based on improved U-Net according to claim 2, characterized in that: The step of inputting the molten pool portion image into an improved U-Net model to obtain molten pool features includes: Inputting the melt pool partial image into an encoder so that at least one bneck module of each layer performs feature extraction on the melt pool partial image; Transmitting the output feature maps of each layer of the encoder to the decoder through the skip connection to obtain the feature maps output by the decoder; Through the 1×1 convolution of the output layer, the feature map output by the decoder is converted into a 2D channel feature map. Each pixel is classified one by one through the Softmax classifier, and the white pixels are obtained as the melt pool features.

4. The method for acquiring molten pool features based on improved U-Net according to claim 3, characterized in that: The steps of feature extraction by the bneck module include: Use 1×1 convolution to perform dimension-up operation on the melt pool part image input to the bneck module; The 3×3 depthwise separable convolution is used to extract features from the melt pool image after the dimension increase operation, and a feature map is obtained in which each channel has undergone independent convolution operation. Performing global average pooling on the feature map through the SE module, adjusting the weight of each channel of the feature map, and obtaining a feature map after channel weight adjustment; Use a 1×1 convolution kernel to reduce the dimension of the feature map after channel weight adjustment to obtain the initial output feature map of the bneck module; Based on the molten pool partial image input to the bneck module and the output feature map, a residual connection operation is performed to obtain the output feature map of the bneck module.

5. The method for acquiring molten pool features based on improved U-Net according to any one of claims 1 to 4, characterized in that: The improved U-Net model is trained based on the following steps: Obtain partial image samples of the melt pool and construct an initial improved U-Net model, in which the VGG convolutional neural network in the backbone network is replaced by a MobileNet V3 module; The initial improved U-Net model is trained by using the partial image samples of the molten pool, the loss function and the evaluation index to obtain the improved U-Net model; Wherein, the loss function is expressed as: Among them, y∈{0,1} represents the labeled value, x∈{0,1} represents the model output, 0 and 1 represent the melt pool feature and background feature respectively; M represents the penalty map of the melt pool part image sample, L represents the loss function, L' represents the intermediate form of the loss function, and m represents the total number of samples; The evaluation index is expressed as: Among them, p ji represents the number of pixels j predicted as pixels i, p ij Indicates the number of pixels i predicted as pixels j, p ii It indicates the number of pixels predicted as i pixels, i and j correspond to the melt pool feature and background feature respectively.

6. The method for acquiring molten pool features based on improved U-Net according to claim 5, characterized in that: The step of obtaining a partial image sample of the molten pool includes: Collecting original image data of the molten pool, and preprocessing the original image data to obtain preprocessed image data; The Labelme segmentation and annotation tool is used to annotate the melt pool part and / or the background part of the melt pool part image samples obtained by preprocessing to obtain labeled melt pool part image samples, wherein the labeled melt pool part image samples are divided into training samples, test samples and verification samples.

7. The method for acquiring molten pool features based on improved U-Net according to claim 6, characterized in that: The preprocessing of the original image data to obtain preprocessed image data includes: The original image data is subjected to filtering processing, image grayscale processing, image grayscale stretching processing and molten pool partial image cropping in sequence, so that the resolution size of the obtained molten pool partial image samples is uniform.

8. A molten pool feature acquisition device based on improved U-Net, characterized in that: The device comprises: An image acquisition module, used for acquiring an image of the molten pool, segmenting the image of the molten pool, and obtaining a partial image of the molten pool; A feature acquisition module, used for inputting the partial image of the molten pool into the improved U-Net model to obtain the molten pool features; The improved U-Net model includes an encoder, a skip connection, a decoder and an output layer, and the encoding area of ​​the encoder includes a bneck module of the MobileNet V3 module.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images