Molten pool image segmentation method and device

Abstract

The invention discloses a molten pool image segmentation method and device, and belongs to the technical field of image processing, and the method comprises the steps: constructing a molten pool image segmentation model which comprises a deep convolution encoder and a multi-scale convolution decoder; lightweight processing is carried out on the to-be-segmented welding image according to a deep convolution encoder, and pyramid features are obtained; and performing multi-scale feature extraction on the pyramid features according to a multi-scale convolution decoder to obtain a target molten pool image. The welding image to be segmented is subjected to lightweight processing through the deep convolution encoder, so that the data processing amount can be effectively reduced, and the real-time performance of data acquisition is improved; the pyramid features are subjected to multi-scale feature extraction through the multi-scale convolution decoder, and the features of the image can be obtained from multiple angles, so that the reliability of the target molten pool image is effectively ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for segmenting a molten pool image. Background Art

[0002] Welding deviation refers to the phenomenon that the actual welding trajectory deviates from the expected trajectory due to unpredictable factors during the normal welding process. The existence of welding deviation will affect the quality and performance of the welded joint, and thus affect the overall welding quality.

[0003] A molten pool is generated during the welding process, and the shape of the molten pool determines the shape of the weld. When the molten pool is narrow and deep, the weld formation coefficient is small. This shape is conducive to penetration, a small heat-affected zone, and improved welding thermal efficiency. However, a too small weld formation coefficient may cause defects such as pores and cracks. Therefore, the morphology of the molten pool has an important influence on the welding quality. In order to monitor the welding quality, it is necessary to obtain clear and reliable molten pool images. However, the current technology for obtaining molten pool images is either time-consuming and cannot meet the needs of real-time monitoring, or the image pixels are low and the reliability is not high.

[0004] Therefore, in the process of obtaining the molten pool image, the existing method has the problem of being unable to obtain reliable images in real time. Summary of the invention

[0005] In view of this, it is necessary to provide a molten pool image segmentation method and device to solve the problem that reliable images cannot be obtained in real time during the process of obtaining molten pool images in the prior art.

[0006] In order to solve the above problems, the present invention provides a molten pool image segmentation method, comprising: Construct a melt pool image segmentation model, which includes a deep convolutional encoder and a multi-scale convolutional decoder; The welding image to be segmented is lightly processed according to the deep convolution encoder to obtain pyramid features; Multi-scale feature extraction is performed on the pyramid features according to the multi-scale convolution decoder to obtain the target melt pool image.

[0007] In a possible implementation, the deep convolution encoder includes a target backbone network and a dilated convolution module; the welding image to be segmented is lightweight processed according to the deep convolution encoder to obtain a pyramid feature, including: Perform lightweight segmentation on the welding image to be segmented according to the target backbone network to obtain lightweight features; According to the dilated convolution module, a dilated convolution operation is performed on the lightweight features to obtain pyramid features.

[0008] In one possible implementation, the target backbone network uses the first twelve layers of MobileNetV3-Small; Among them, the eleventh and twelfth layers both use dilated convolution; The atrous convolution module consists of three atrous convolution layers with different dilation rates.

[0009] In a possible implementation, the multi-scale convolution decoder includes a large core grouping attention gate, a multi-scale convolution attention module, an up-convolution block, and a segmentation head; and the multi-scale feature extraction is performed on the pyramid feature according to the multi-scale convolution decoder to obtain a target melt pool image, including: The first transition feature is obtained by fusing pyramid features through large core group attention gate optimization and skip connection to capture spatial context information; Refine the mapping of the first transition feature through a multi-scale convolutional attention module to obtain a second transition feature; The segmentation head segments and outputs a target melt pool image according to the second transition feature; Part of the second transition features are upsampled through the upconvolution block, and the first transition features are optimized according to the upsampled image.

[0010] In a possible implementation, the multi-scale convolutional attention module includes a channel attention block, a spatial attention block, and an efficient multi-scale convolutional block; refining the mapping of the first transition feature through the multi-scale convolutional attention module to obtain the second transition feature includes: Assign a value to the channel of the first transition feature according to the channel attention block to obtain a channel dimension value; The spatial attention block extracts local features from the first transition feature according to the channel dimension value to obtain local key features; The local key features are refined through the efficient multi-scale convolution block to obtain the second transition feature.

[0011] In a possible implementation, assigning a value to the channel of the first transition feature according to the channel attention block to obtain a channel dimension value includes: Adaptive maximum pooling and adaptive average pooling are performed on the spatial dimension of the first transition feature to extract channel maximum pooling features and channel average pooling features; Perform the first point convolution, activation and second point convolution processing on the channel maximum pooling feature and the channel average pooling feature respectively to obtain the channel maximum pooling feature map and the channel average pooling feature map; Perform secondary activation on the channel maximum pooling feature map and the average pooling feature map to obtain the channel attention weight; The channel attention weight is merged into the first transition feature through the Hadamard product to obtain the channel dimension value.

[0012] In a possible implementation, the spatial attention block extracts local features from the first transition feature according to the channel dimension value to obtain local key features, including: The maximum and average values ​​of the first transition features are pooled along the channel dimension; The maximum value and the average value are convolved through a large kernel convolution layer to obtain local context features; The attention weight of the local context feature is calculated according to the activation function, and the first transition feature is adjusted according to the attention weight to obtain the local key feature.

[0013] In a possible implementation, the local key features are refined by an efficient multi-scale convolution block to map the feature to obtain the second transition feature, including: The number of channels of the local key features is expanded through the first point-by-point convolution, and the local key features are subjected to batch normalization, activation function processing and multi-scale deep convolution in sequence to obtain deep convolution features; The second point-by-point convolution and batch normalization operations are performed on the deep convolution features in sequence to obtain the second transition features.

[0014] In a possible implementation, after performing multi-scale feature extraction on the pyramid features according to the multi-scale convolution decoder to obtain the target melt pool image, the following further includes: Obtaining the target melt pool image by melt pool method cutting angle; Welding deviation is determined based on the weld pool method.

[0015] In order to solve the above problems, the present invention also provides a molten pool image segmentation device, comprising: A melt pool image segmentation model construction module is used to construct a melt pool image segmentation model, which includes a deep convolution encoder and a multi-scale convolution decoder; A pyramid feature acquisition module is used to perform lightweight processing on the welding image to be segmented according to a deep convolution encoder to obtain pyramid features; The target melt pool image acquisition module is used to perform multi-scale feature extraction on the pyramid features according to the multi-scale convolution decoder to obtain the target melt pool image.

[0016] The beneficial effects of adopting the above-mentioned embodiments are as follows: the present invention provides a molten pool image segmentation method, which performs lightweight processing on the welding image to be segmented through a deep convolution encoder, and obtains pyramid features in a targeted manner, which can effectively reduce the amount of data processing and improve the real-time performance of data acquisition; multi-scale feature extraction is performed on the pyramid features through a multi-scale convolution decoder to obtain a target molten pool image, and the features of the image can be obtained from multiple angles, thereby effectively ensuring the reliability of the target molten pool image; in summary, by constructing a molten pool image segmentation model including a deep convolution encoder and a multi-scale convolution decoder, real-time acquisition of a target molten pool image with rich features is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of an embodiment of a melt pool image segmentation method provided by the present invention; Figure 2 A schematic diagram of the structure of an embodiment of a deep convolutional encoder provided by the present invention; Figure 3 A schematic diagram of the structure of an inverted residual network structure according to an embodiment of the present invention; Figure 4 A schematic diagram of the structure of an embodiment of a depthwise separable convolution process provided by the present invention; Figure 5 A schematic diagram of the structure of an embodiment of a dilated convolution module provided by the present invention; Figure 6 A schematic diagram of a process for obtaining a target molten pool image according to an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an embodiment of a multi-scale convolutional decoder provided by the present invention; Figure 8 A schematic diagram of a flow chart of an embodiment of obtaining a second transition characteristic provided by the present invention; Fig. 9 A schematic diagram of a structure of an embodiment of a multi-scale convolutional attention module provided by the present invention; Fig.10 A schematic diagram of the structure of an embodiment of a channel attention block provided by the present invention; Fig.11 A schematic diagram of the structure of an embodiment of a spatial attention block provided by the present invention; Fig.12 A schematic diagram of the structure of an embodiment of an efficient multi-scale convolution block provided by the present invention; Fig.13 A schematic diagram of the structure of an embodiment of an efficient up-convolution block provided by the present invention; Fig.14 A schematic diagram of the result of an embodiment of calibrating a molten pool image provided by the present invention; Fig.15A structural block diagram of an embodiment of a molten pool image segmentation device provided by the present invention; Fig.16 This is a structural block diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of the present invention and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0019] Molten pool refers to the liquid metal part with a certain geometric shape formed on the weldment under the action of welding heat source. During the fusion welding process, the parent metal part on the weldment melts into a pool due to the heat of the welding arc, forming a molten pool. The morphological characteristics of the molten pool usually include its shape, size, depth and the flow of liquid metal. These characteristics are affected by many factors such as welding current, arc voltage, welding speed, type and diameter of welding rod or wire, material and thickness of weldment, etc.

[0020] Welding deviation refers to the deviation between the actual weld shape, size, position, etc. and the design requirements during the welding process. This deviation may be caused by many factors, including the selection of welding process parameters, the performance of welding equipment, the skill level of the operator, and the welding environment.

[0021] The DeepLabV3+ model is a deep learning model that performs well in the field of semantic segmentation. It uses an encoder-decoder structure and introduces key technologies such as atrous convolution and atrous spatial pyramid pooling (ASPP). These technologies enable the DeepLabV3+ model to capture rich contextual information and clear object boundaries while maintaining efficient calculations, thereby achieving high-precision semantic segmentation.

[0022] MobileNetV3 is a lightweight convolutional neural network structure proposed by Google in 2019, designed to improve speed and accuracy on mobile devices.

[0023] MobileNetV3-Small is a lightweight convolutional neural network structure variant in the MobileNetV3 series proposed by Google in 2019. It is specially optimized for speed and efficiency, making it an ideal choice for resource-limited applications.

[0024] During the fusion welding process, the shape of the molten pool can be intuitively presented by the molten pool image. By analyzing the molten pool image, the welding deviation can be effectively determined. However, the current technology for obtaining the molten pool image is either time-consuming and cannot meet the needs of real-time monitoring, or the image pixels are low and the reliability is not high.

[0025] Therefore, in the process of obtaining the molten pool image, the existing method has the problem of being unable to obtain reliable images in real time.

[0026] In order to solve the above problems, the present invention provides a molten pool image segmentation method and device, which are described in detail below.

[0027] Figure 1 A schematic diagram of a flow chart of an embodiment of a melt pool image segmentation method provided by the present invention, as shown in FIG. Figure 1 As shown, the melt pool image segmentation method includes: S101: construct a melt pool image segmentation model, which includes a deep convolution encoder and a multi-scale convolution decoder; S102: performing lightweight processing on the welding image to be segmented according to the deep convolution encoder to obtain a pyramid feature; S103: Perform multi-scale feature extraction on the pyramid features according to the multi-scale convolution decoder to obtain a target melt pool image.

[0028] In this embodiment, the welding image to be segmented is lightweight processed by a deep convolution encoder, and the pyramid features are obtained in a targeted manner, which can effectively reduce the amount of data processing and improve the real-time performance of data acquisition. The pyramid features are extracted at multiscale by a multiscale convolution decoder to obtain the target molten pool image, and the image features can be obtained from multiple angles, thereby effectively ensuring the reliability of the target molten pool image. In summary, by constructing a molten pool image segmentation model including a deep convolution encoder and a multiscale convolution decoder, real-time acquisition of the target molten pool image with rich features is achieved.

[0029] It should be noted that the melt pool image segmentation model (Deeplab-EMCAD segmentation model) mainly consists of two parts: deep convolutional encoder (Deeplab) and multi-scale convolutional decoder (EMCAD).

[0030] As a preferred embodiment, in S102, the deep convolution encoder includes a target backbone network and a hole convolution module. In order to perform lightweight processing on the welding image to be segmented according to the deep convolution encoder to obtain pyramid features, first, the welding image to be segmented is lightweight segmented according to the target backbone network to obtain lightweight features; then, the hole convolution module is used to perform a hole convolution operation on the lightweight features to obtain pyramid features.

[0031] It should be noted that the target backbone network uses the first twelve layers of MobileNetV3-Small; Among them, the eleventh and twelfth layers both use dilated convolution; The atrous convolution module consists of three atrous convolution layers with different dilation rates.

[0032] In a specific embodiment, if Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of an embodiment of the deep convolutional encoder provided by the present invention. backbone represents the backbone network, MobileNetV3 represents the lightweight deep learning model, 3×3ConvRate6 represents the convolutional layer when Rate=6, 3×3Conv Rate12 represents the convolutional layer when Rate=12, and 3×3ConvRate24 represents the convolutional layer when Rate=24.

[0033] Regarding the backbone network, in order to improve the lightweight characteristics of the network and speed up the training speed, MobileNetV3 is selected as the backbone network of the Deeplab-EMCAD encoder in this embodiment. MobileNetV3, with its efficient inverted residual structure as the core, can significantly reduce the computational complexity while ensuring accuracy. Compared with other heavier backbone networks, after integrating the MobileNetV3 backbone in the encoder, its lightweight advantage can be fully utilized to achieve real-time segmentation of the molten pool image in the welding environment. Figure 3 As shown, Figure 3 A schematic diagram of the structure of an inverted residual network structure according to an embodiment of the present invention.

[0034] Among them, Conv2d 1×1 means using a 1×1 convolution kernel in the convolution operation; BatchNorm (BatchNormalization), Relu6 refers to the Relu activation function that limits the maximum output value to 6; DepthwiseCon2d 3×3 means using a 3×3 convolution kernel for depthwise convolution.

[0035] The inverted residual structure of MobileNetV3 performs dimensionality increase and feature extraction through 1×1 and 3×3 depthwise separable convolutions, and finally integrates the feature information through dimensionality reduction convolutions, so that the network can obtain sufficient feature information while reducing parameters.

[0036] Among them, the core of MobileNetV3 - depthwise separable convolution is a convolution operation that effectively reduces the number of model parameters and computational load. It decomposes the standard convolution into two independent steps. The first step is depthwise convolution, which convolves each input channel separately. The second step is pointwise convolution, which uses a 1×1 convolution kernel to fuse channel information of the features extracted by depthwise convolution. The whole process is as follows: Figure 4 As shown, Figure 4 A schematic diagram of the structure of an embodiment of the depthwise separable convolution process provided by the present invention. Depthwise Convolution represents depthwise convolution; Input represents input; Filters*3 represents that when the input feature map has 3 channels, the depthwise convolution will use 3 convolution kernels (one for each channel). "Filters*3" does not refer to the total number of convolution kernels, but to the case where each input channel corresponds to a convolution kernel; Map*3 represents that when the input feature map has M channels, the depthwise convolution will use M convolution kernels, and each convolution kernel only acts on the corresponding input channel. Therefore, the output of the depthwise convolution will be M feature maps, and each feature map corresponds to the output of an input channel.

[0037] Pointwise Convolution means point-by-point convolution; Filters*4 refers to the number of convolution kernels of point-by-point convolution, which means that the point-by-point convolution layer has 3 convolution kernels, each of which acts on the output channels of all depth-wise convolutions and outputs a new channel; Map*4 means that 4 1×1 convolution kernels are used in the point-by-point convolution, thereby outputting 4 new feature maps.

[0038] Assume that the input feature map size is DF×DF , the number of channels is M , the convolution kernel size is DK×DK , the number of output channels is N , then the standard convolution calculation amount is M×N×DF×DF×DK×DK The depth-wise separable convolution divides the standard convolution into two steps. The first step is the depth-wise convolution. M The convolution is performed on each channel, and the computational cost is M×DF×DF×DK×DK ; The second step is point-by-point convolution, using N 1 × 1 ×M The convolution kernel is used for convolution, and the calculation amount is M×N×DF×DF , so the total amount of calculation is M×DF ×DF×DK×DK + M×N×F×DF From formula (1), we can conclude that the computational complexity of depthwise separable convolution is (1 / N + 1 / (D K ×D K)) times, greatly reducing the amount of computation, thereby reducing the model size and training time.

[0039] (1) In order to further reduce the number of model parameters of the network, this application only uses the first 12 layers of MobileNetV3-Small, and modifies the original network parameters for 4 downsamplings. At the same time, the 11th and 12th layers use dilated convolutions to enhance feature extraction capabilities. Specifically, the network structure is as follows: Input represents the input size; Operator represents the network layer; Exp size represents the first layer of 1×1 convolution in the bottleneck, and the dimension is increased to a certain extent (the first bottleneck does not have a 1×1 convolution dimension increase operation); #out represents the number of channels output by the bottleneck; SE represents whether the SE module is used; NL represents which activation function is used; HS represents HardSwish, RE represents ReLU; s represents the step size (s=2, the length and width become half of the original), and the optimized MobileNetV3-Small network structure is shown in Table 1: Table 1 Optimized MobileNetV3-Small network structure

[0040] Furthermore, the Atrous Spatial Pyramid Poling (ASPP) module captures multi-scale information through atrous convolutions with different dilation rates, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an embodiment of the atrous convolution module provided by the present invention. Rate represents the expansion rate. The ASPP module has 4 parallel atrous convolution layers. The size of each convolution kernel is 3×3, but their expansion rates are different, namely 6, 12, 18, and 24. These different expansion rates (Rate) represent different receptive fields. When Rate=6, the receptive field is relatively small and pays more attention to local details; when Rate=12, the receptive field is slightly expanded and can capture more context information; when Rate=18, the receptive field is further increased and can focus on context in a larger range; when Rate=24, it has the largest receptive field and can capture global information.

[0041] However, when collecting images in real time and encoding them during the welding process, the four parallel hole convolution layers will increase the amount of calculation, causing a certain time delay when the model segments the molten pool image from the welding image, resulting in the failure to obtain the molten pool method angle in time, which will be detrimental to the adjustment of welding deviations during the welding process. For this reason, in this embodiment, only the convolution layers at Rate=6, Rate=12, and Rate=24 are retained, which not only reduces the amount of calculation, but also ensures the integrity of the acquired molten pool feature information from local to overall.

[0042] It should be noted that the number of retained convolutional layers and the expansion rate can also be adjusted according to actual conditions, such as retaining the convolutional layers when Rate=6, Rate=18, and Rate=24, or only retaining the convolutional layers when Rate=12 and Rate=24, without any restriction here.

[0043] In this embodiment, the inverted residual structure based on MobileNetV3 is used for dimensionality increase and feature extraction through 1×1 and 3×3 depth-separable convolutions, and finally the feature information is integrated through dimensionality reduction convolutions, so that the network can still obtain sufficient feature information while reducing parameters. After integrating the MobileNetV3 backbone in the encoder, its lightweight advantage can be fully utilized to achieve real-time segmentation of the molten pool image in the welding environment; and, since the ASPP module can capture multi-scale information, it effectively guarantees the ability of the deep convolution encoder to capture multi-scale information, ensuring that while reducing the model size, it still has excellent segmentation accuracy and adaptability to complex welding scenarios.

[0044] As a preferred embodiment, in S103, the multi-scale convolution decoder includes a large core grouping attention gate, a multi-scale convolution attention module, an upper convolution block and a segmentation head; in order to perform multi-scale feature extraction on the pyramid feature according to the multi-scale convolution decoder to obtain a target melt pool image, such as Figure 6 As shown, Figure 6 A schematic diagram of a process for obtaining a target molten pool image according to an embodiment of the present invention includes: S601: The first transition feature is obtained by fusing pyramid features through large core group attention gate optimization and skip connection to capture spatial context information; S602: Refine the mapping of the first transition feature through a multi-scale convolutional attention module to obtain a second transition feature; S603: The segmentation head segments and outputs a target molten pool image according to the second transition feature; Part of the second transition features are upsampled through the upconvolution block, and the first transition features are optimized according to the upsampled image.

[0045] In a specific embodiment, after extracting three pyramid features from the deep convolutional encoder, as shown in FIG. Figure 7 As shown, Figure 7 A schematic diagram of the structure of an embodiment of a multi-scale convolutional decoder provided by the present invention, wherein the multi-scale convolutional decoder comprises a large kernel group attention gate (LGAG), a multi-scale convolutional attention module (MSCAM), an upper convolution block (EUCB) and a segmentation head (SH), and outputs three prediction graphs P1, P2, and P3.

[0046] Specifically, the three pyramid features are refined by LGAG and then by MSCAM. After each MSCAM, the SH segmentation head is used to generate the segmentation map of this stage. Among them, P2 and P3 will also obtain more local information through the EUCB module respectively, and combine their own global information to generate the final segmentation output.

[0047] In this embodiment, the large core group attention gate optimizes the feature map fused with the skip connection through the gated attention mechanism, the up-convolution block is responsible for upsampling the features, the segmentation head (SH) generates the segmentation output, and the multi-scale convolution attention module is used to robustly enhance the feature map.

[0048] As a preferred embodiment, in S601, in order to improve the segmentation accuracy, a new large-kernel groups attention gate (LGAG) is used. By combining the feature map with the learned attention coefficient, the activation effect of relevant features is gradually enhanced, while irrelevant features are suppressed. This method adjusts the information flow of each stage of the network through the gating signal derived from the high-level features, thereby improving the accuracy of image segmentation. Unlike the Attention UNet that uses 1×1 convolution to process the gate signal g (features from the skip connection) and the input feature map x (upsampled features), the qatt function here uses separate 3×3 group convolution GCs respectively. g and GC x to process g and x. Then batch normalization is used to normalize these convolutional features and merge them by element-wise addition. The generated feature map is then activated by the ReLU layer. After that, the feature map is further converted to a single-channel form using a 1×1 convolution and a BN layer. Finally, the attention coefficient is obtained by the Sigmoid activation function. This coefficient is dot-multiplied with the input feature x to obtain the attention gate feature LGAG (g, x). LGAG can be expressed as Equation (2) and Equation (3): (2) (3) Since 3×3 large kernel group convolution is used in qatt (.), LGAG can capture more spatial context information at a lower computational cost, thus better meeting the real-time requirements of welding pool image segmentation.

[0049] As a preferred embodiment, in S602, the multi-scale convolution attention module includes a channel attention block, a spatial attention block and an efficient multi-scale convolution block; in order to refine the mapping of the first transition feature through the multi-scale convolution attention module to obtain the second transition feature, such as Figure 8 As shown, Figure 8 A schematic diagram of a process for obtaining a second transition characteristic according to an embodiment of the present invention includes: S801: assigning a value to the channel of the first transition feature according to the channel attention block to obtain a channel dimension value; S802: The spatial attention block extracts local features from the first transition feature according to the channel dimension value to obtain local key features; S803: Refine the feature mapping of the local key features through an efficient multi-scale convolution block to obtain a second transition feature.

[0050] In a specific embodiment, if Fig. 9 As shown, Fig. 9 A structural schematic diagram of an embodiment of a multi-scale convolutional attention module provided by the present invention is provided. The multi-scale convolutional attention module (MSCAM) is composed of a channel attention block (CAB(.)), a spatial attention block (SAB(.)) and an efficient multi-scale convolutional block (MSCB(.)).

[0051] Among them, the channel attention block CAB is used to emphasize the features of related channels, the spatial attention block SAB focuses on local context information, and MSCB enhances the multi-scale representation of the feature map. MSCAM (.) is shown in formula (4): (4) Here, x is the input tensor.

[0052] As a preferred embodiment, in S801, in order to assign values ​​to the channels of the first transition feature according to the channel attention block and obtain the channel dimension value, first, adaptive maximum pooling and adaptive average pooling are performed on the spatial dimensions of the first transition feature, respectively, to extract the channel maximum pooling features and the channel average pooling features; then, the channel maximum pooling features and the channel average pooling features are respectively subjected to the first point convolution, activation, and second point convolution processing to obtain the channel maximum pooling feature map and the channel average pooling feature map; next, the channel maximum pooling feature map and the average pooling feature map are secondary activated to obtain the channel attention weight; finally, the channel attention weight is merged into the first transition feature through the Hadamard product to obtain the channel dimension value.

[0053] It should be noted that the channel maximum pooling feature is sequentially subjected to the first point convolution, activation and second point convolution processing to obtain the channel maximum pooling feature map; the channel average pooling feature is sequentially subjected to the first point convolution, activation and second point convolution processing to obtain the channel average pooling feature map, and the operation steps are exactly the same.

[0054] like Fig.10 As shown, Fig.10 This is a structural schematic diagram of an embodiment of a channel attention block provided by the present invention. AMP represents adaptive maximum pooling, AAP represents adaptive average pooling, Conv1×1 C / r represents, ReLU represents ReLU activation, Conv1×1 represents a convolution operation using a convolution kernel of size 1×1, + represents feature map addition, Sigmoid represents Sigmoid activation, and o represents Hadamard product.

[0055] The channel attention block assigns different importance to each channel, highlighting relevant features and suppressing irrelevant information. CAB first performs adaptive maximum pooling (Pm(.)) and adaptive average pooling (Pa(.)) on the spatial dimensions (i.e., height and width) of the input feature map to extract the key features of each channel. Then, the number of channels is reduced by point convolution (C1(.)) (reduction rate r = 1 / 16) and ReLU (R) activation is used. Next, the number of channels is restored using point convolution (C2(.)). Finally, after adding the two restored feature maps, the channel attention weights are obtained by Sigmoid (σ) activation, and finally these weights are merged into the input x using the Hadamard product. CAB(.) is defined by equation (5): (5) Among them, Pm (.) corresponds to AMP in the figure, and Pa (.) corresponds to AAP in the figure.

[0056] As a preferred embodiment, in S802, in order to extract local features of the first transition feature according to the channel dimension value and obtain local key features, the spatial attention block first collects the maximum value and average value of the first transition feature along the channel dimension; then, convolution operation is performed on the maximum value and average value through a large kernel convolution layer to obtain local context features; finally, the attention weight of the local context feature is calculated according to the activation function, and the first transition feature is adjusted according to the attention weight to obtain the local key features.

[0057] like Fig.11 As shown, Fig.11A schematic diagram of the structure of an embodiment of the spatial attention block provided by the present invention, wherein Channel Max represents the maximum value of the channel; Channel Average represents the average value of the channel; C represents a pooling operation; Conv7×7 Pad:3 represents a convolution operation with a convolution kernel size of 7×7 and a padding size of 3; Sigmoid represents Sigmoid function activation.

[0058] The Spatial Attention Block (SAB) in MSCAM simulates the attention process of the human brain and focuses on specific parts of the input image. SAB first pools the maximum value (Chmax(.)) and the average value (Chavg(.)) along the channel dimension to focus on local features. Then, a large kernel convolution layer is used to enhance the local contextual relationship between features. Subsequently, Sigmoid activation (σ) is applied to calculate the attention weights. Finally, these weights are provided to the input x for more targeted processing of information. SAB(.) can be defined using Equation (6): (6) As a preferred embodiment, in S803, in order to refine the feature mapping of the local key features through an efficient multi-scale convolution block to obtain the second transition feature, first, the number of channels of the local key features is expanded through a first point-by-point convolution, and the local key features are sequentially subjected to batch normalization, activation function processing and multi-scale deep convolution to obtain deep convolution features; then, the deep convolution features are sequentially subjected to a second point-by-point convolution and batch normalization operations to obtain the second transition feature.

[0059] Specifically, Fig.12 As shown, Fig.12 A schematic diagram of the structure of an embodiment of an efficient multi-scale convolution block provided by the present invention, wherein Conv1×1, C*2 represents a convolution operation with a convolution kernel size of 1×1, and the number of channels is doubled; BN (Batch Normalization); BN, ReLU6 represents batch normalization, and activation is performed by a Relu activation function with a maximum output value limited to 6; MSDC represents multi-scale depthwise convolution; Conv1×1 represents a convolution operation with a convolution kernel size of 1×1; PWC represents point-by-point convolution; PWCq×q; PWCs×s; ReLU6 represents activation by a Relu activation function with a maximum output value limited to 6; Channel Shuffle represents channel shuffling.

[0060] The efficient multi-scale convolution block (MSCB) in MSCAM is a multi-scale convolution block used to enhance the features generated on the expansion path. The design of MSCB follows the reverse residual block (IRB) structure of MobileNetV2, but performs depthwise convolution at different scales and uses channel shuffle operations to enhance the information integration between channels. Specifically, in MSCB, the number of channels is first expanded (expansion factor = 2) using pointwise convolution PWC1 (.), and then processed by batch normalization (BN (.)) and ReLU6 activation function (R6 (.)). Next, multi-scale depthwise convolution (MSDC (.)) is applied to capture multi-scale contextual information, while enhancing the relationship between channels through channel shuffle. Finally, the number of channels is converted back to the original number using pointwise convolution PWC2 (.) and BN (.) operations, thereby encoding the inter-channel dependency. The expression of MSCB is shown in Equation (7): (7) The parallel MSDC (.) with different kernel (ks) sizes can be expressed by equation (8): (8) Among them, DWCB ks (x) is expressed by formula (9): (9) Here, DWC ks (.) is a depthwise convolution with kernel size ks. BN(.) and R6(.) are batch normalization and ReLU6 activation, respectively. In addition, MSDC(.) uses the recursively updated input x, where the input x is residually connected to the previous DWCB ks (.) for better regularization, as shown in formula (10): (10) like Fig.13 As shown, Fig.13 A schematic diagram of the structure of an embodiment of an efficient upconvolution block provided by the present invention, wherein Up2x represents an upsampling operation; DWC3×3 represents a 3×3 depth convolution; BN represents batch normalization; ReLU represents a ReLU activation function; and Conv1×1 represents a 1×1 convolution operation.

[0061] The Efficient up_convolution block (EUCB) upsamples the feature map of the current stage to match the size and resolution of the feature map of the next skip connection. EUCB first uses an upsampling operation Up (.) with a scale factor of 2 to enhance the feature map. Then, the upsampled feature map is enhanced by 3×3 depthwise convolution DWC (.), batch normalization BN (.), and ReLU (.) activation. Finally, a 1×1 convolution C is used. 1×1 (.) to reduce the channel to match the next stage. The expression of EUCB (.) is shown in formula (11): (11) Since EUCB uses less computationally intensive depthwise convolutions instead of standard 3×3 convolutions, it exhibits high computational efficiency in the melt pool image upsampling task.

[0062] The three segmentation heads of the EMCAD decoder produce three prediction maps P1, P2, and P3 at their respective stages. In order to comprehensively optimize each prediction map and their combination, a loss aggregation method called MUTATION is used, which calculates the loss of all possible combinations of prediction maps obtained by the three segmentation heads. Since there are three prediction maps, there are a total of 2 3 -1 = 7 unique prediction combinations (excluding the empty set), and then add up these losses. The goal is to minimize this cumulative combined loss during training, so that the predictions of all segmentation heads and all their combinations are as accurate as possible. Then the total loss L total It can be expressed by formula (12): (12) Among them, L P1 , L P2 and L P3 is the loss of each individual prediction graph, L P1+P2 , L P1+P3 and L P2+P3 is the loss of the combination of two prediction graphs, L P1+P2+P3 is the loss of the combination of the three prediction graphs, and α, β, γ, δ, ε, ζ, and η are the weights assigned to each loss term to balance their contribution to the total loss.

[0063] Through the above method, the deep convolution encoder is used to perform lightweight processing on the welding image to be segmented, and the pyramid features are obtained in a targeted manner, which can effectively reduce the amount of data processing and improve the real-time performance of data acquisition. The multi-scale convolution decoder is used to extract multi-scale features of the pyramid features to obtain the target molten pool image, and the features of the image can be obtained from multiple angles, thereby effectively ensuring the reliability of the target molten pool image. That is, the molten pool image segmentation model is used to realize the real-time acquisition of the target molten pool image with rich features.

[0064] After obtaining the target molten pool image, in order to directly link the molten pool image with the welding quality, it is also necessary to obtain the molten pool method cutting angle of the target molten pool image; and perform welding deviation judgment based on the molten pool method cutting angle.

[0065] Specifically, when observing the shape of the molten pool during normal welding, it can be found that in one welding cycle, that is, when the welding gun swings from the left limit to the right limit and then from the right limit to the left limit, the molten pool is shaped like an upper semicircle, such as Fig.14 As shown, Fig.14 The schematic diagram of the result of the first embodiment of the calibrated molten pool image provided by the present invention shows that a perpendicular bisector L1 passing through the semicircular molten pool and a horizontal chord L2 passing through the center of the semicircular molten pool can be found. With L1 and L2 as coordinate axes and their intersection as the coordinate center point O, the horizontal axis L2 will generate two intersection points (x1, y1) and (x2, y2) with the arc boundary of the semicircular molten pool. With these two intersection points as tangent points, two tangents intersecting with the longitudinal axis L1 can be drawn, and their intersection points are (x3, y3) and (x4, y4). At this time, the tangent and L1 will generate two angles. Therefore, the angle between the tangent and the perpendicular bisector L1 is defined as the molten pool method tangent angle, which is recorded as θ1 and θ2 respectively, and can be expressed by equations (13) and (14): (13) (14) Then, under ideal welding conditions, the molten pool in a single welding cycle is symmetrically semicircular, and the normal cutting angles θ1 and θ2 have relatively consistent angle values. That is, under ideal conditions, θ1≈θ2, which means that the molten pool boundary shape is symmetrical, the welding trajectory is stable, and the welding has not deviated. However, when welding deviation occurs during the welding process, the movement trajectory of the welding gun deviates from the ideal path, resulting in an asymmetric change in the molten pool shape, which is manifested as a significant angle change of θ1 and θ2. This angle change can quantify the degree of deviation during welding, obtain the welding deviation amount, and provide basic conditions for subsequent deviation adjustment.

[0066] In order to verify whether the deviation measurement principle meets the actual needs, the welding gun position is adjusted from 1.5mm to the left to the normal position to 1.5mm to the right, with a step size of 0.1mm, to collect the molten pool image and determine the deviation corresponding to each image. Then, after extracting the molten pool image through the Deeplab-EMCAD model, the deviation measurement principle described above is used to find the normal tangent using the Hough linear transformation, and the intersection is calculated. Then, the angle is calculated using the inverse tangent function and marked. Subsequently, a total of 31 sets of data corresponding to the normal cutting angle of different deviations are selected from the collected data, drawn into a scatter plot, and its linear regression equation is calculated. The linear regression of the deviation and the molten pool normal cutting angle scatter plot is obtained: d = -0.01θ1+0.05θ2-0.83 Where, d is the welding deviation, d<0 is left deviation, d>0 is right deviation, θ1 and θ2 are the cutting angles of the molten pool method, This formula indicates that during the welding process, the molten pool cutting angle and the welding deviation present a certain linear relationship.

[0067] In the above manner, not only the target molten pool image with rich features can be obtained in real time through the molten pool image segmentation model, but also the real-time monitoring of welding quality can be achieved by combining the deviation determination principle described above.

[0068] In a specific real-time example, in order to verify the effect of the molten pool image segmentation model in real-time segmentation of the molten pool image during the welding process, it was verified on an all-position pipeline welding experimental platform.

[0069] Four networks, including classic segmentation networks SegNet, PSPNet, UNet, Yolov8, and DeepLabV3+, were used as experimental objects for comparison. After 100 rounds of training, it was found that the average intersection-over-union ratio, average pixel accuracy, and inference time of the molten pool image segmentation model of the present application were optimal, and the number of parameters was also the smallest. This shows that the molten pool image segmentation model can accurately segment the molten pool target while ensuring that the real-time algorithm can meet the needs of the welding industry site.

[0070] In order to solve the above problems, the present invention also provides a molten pool image segmentation device, such as Fig.15 As shown, Fig.15 This is a structural block diagram of an embodiment of a molten pool image segmentation device provided by the present invention. The molten pool image segmentation device 1500 includes: A melt pool image segmentation model construction module 1501 is used to construct a melt pool image segmentation model, and the melt pool image segmentation model includes a deep convolution encoder and a multi-scale convolution decoder; A pyramid feature acquisition module 1502 is used to perform lightweight processing on the welding image to be segmented according to a deep convolution encoder to obtain a pyramid feature; The target melt pool image acquisition module 1503 is used to perform multi-scale feature extraction on the pyramid features according to the multi-scale convolution decoder to obtain the target melt pool image.

[0071] like Fig.16 As shown, the present invention also provides an electronic device 1600. The electronic device 1600 includes a processor 1601, a memory 1602 and a display 1603. Fig.16 Only some components of the electronic device 1600 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0072] In some embodiments, the processor 1601 may be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 1602, such as the melt pool image segmentation method of the present invention.

[0073] In some embodiments, processor 1601 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, processor 1601 may be local or remote. In some embodiments, processor 1601 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0074] In some embodiments, the memory 1602 may be an internal storage unit of the electronic device 1600, such as a hard disk or memory of the electronic device 1600. In other embodiments, the memory 1602 may also be an external storage device of the electronic device 1600, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 1600.

[0075] Furthermore, the memory 1602 may include both an internal storage unit of the electronic device 1600 and an external storage device. The memory 1602 is used to store application software installed in the electronic device 1600 and various data.

[0076] In some embodiments, the display 1603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 1603 is used to display information of the electronic device 1600 and to display a visual user interface. The components 1601-1603 of the electronic device 1600 communicate with each other via a system bus.

[0077] In one embodiment, when the processor 1601 executes the melt pool image segmentation program in the memory 1602, the following steps may be implemented: Construct a melt pool image segmentation model, which includes a deep convolutional encoder and a multi-scale convolutional decoder; The welding image to be segmented is lightly processed according to the deep convolution encoder to obtain pyramid features; Multi-scale feature extraction is performed on the pyramid features according to the multi-scale convolution decoder to obtain the target melt pool image.

[0078] It should be understood that: when the processor 1601 executes the molten pool image segmentation program in the memory 1602, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0079] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 1600 mentioned, and the electronic device 1600 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1600 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0080] Correspondingly, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions of the molten pool image segmentation method provided in the above-mentioned method embodiments.

[0081] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0082] The melt pool image segmentation method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A melt pool image segmentation method, characterized in that: include: Constructing a melt pool image segmentation model, wherein the melt pool image segmentation model includes a deep convolution encoder and a multi-scale convolution decoder; Lightweight processing is performed on the welding image to be segmented according to the deep convolution encoder to obtain a pyramid feature; The pyramid features are subjected to multi-scale feature extraction according to the multi-scale convolution decoder to obtain a target molten pool image.

2. The molten pool image segmentation method according to claim 1, characterized in that: The deep convolution encoder includes a target backbone network and a hole convolution module; the light-weight processing of the welding image to be segmented is performed according to the deep convolution encoder to obtain a pyramid feature, including: Performing lightweight segmentation on the welding image to be segmented according to the target backbone network to obtain lightweight features; A dilated convolution operation is performed on the lightweight feature according to the dilated convolution module to obtain a pyramid feature.

3. The molten pool image segmentation method according to claim 2, characterized in that: The target backbone network uses the first twelve layers of MobileNetV3-Small; Among them, the eleventh and twelfth layers both use dilated convolution; The dilated convolution module includes three dilated convolution layers with different expansion rates.

4. The molten pool image segmentation method according to claim 1, characterized in that: The multi-scale convolution decoder includes a large core grouping attention gate, a multi-scale convolution attention module, an upper convolution block and a segmentation head; the multi-scale feature extraction of the pyramid feature according to the multi-scale convolution decoder to obtain a target molten pool image includes: The pyramid features are fused through the large core group attention gate optimization and skip connection to capture spatial context information and obtain a first transition feature; Refining the mapping of the first transition feature through the multi-scale convolutional attention module to obtain a second transition feature; The segmentation head segments and outputs the target molten pool image according to the second transition feature; Part of the second transition feature is upsampled by the upconvolution block, and the first transition feature is optimized according to the upsampled image.

5. The molten pool image segmentation method according to claim 4, characterized in that: The multi-scale convolution attention module includes a channel attention block, a spatial attention block and an efficient multi-scale convolution block; the mapping of the first transition feature is refined by the multi-scale convolution attention module to obtain the second transition feature, including: Assigning a value to the channel of the first transition feature according to the channel attention block to obtain a channel dimension value; The spatial attention block extracts local features from the first transition features according to the channel dimension value to obtain local key features; The local key features are refined by the efficient multi-scale convolution block to map the features and obtain the second transition features.

6. The molten pool image segmentation method according to claim 5, characterized in that: The assigning a value to the channel of the first transition feature according to the channel attention block to obtain a channel dimension value includes: Performing adaptive maximum pooling and adaptive average pooling on the spatial dimensions of the first transition feature respectively, and extracting channel maximum pooling features and channel average pooling features; Performing first point convolution, activation, and second point convolution processing on the channel maximum pooling feature and the channel average pooling feature, respectively, to obtain a channel maximum pooling feature map and a channel average pooling feature map; Performing secondary activation on the channel maximum pooling feature map and the average pooling feature map to obtain a channel attention weight; The channel attention weight is merged into the first transition feature through the Hadamard product to obtain a channel dimension value.

7. The molten pool image segmentation method according to claim 5, characterized in that: The spatial attention block extracts local features from the first transition feature according to the channel dimension value to obtain local key features, including: Aggregating the maximum and average values ​​of the first transition characteristics along the channel dimension; Performing a convolution operation on the maximum value and the average value through a large kernel convolution layer to obtain a local context feature; The attention weight of the local context feature is calculated according to the activation function, and the first transition feature is adjusted according to the attention weight to obtain a local key feature.

8. The molten pool image segmentation method according to claim 5, characterized in that: The step of refining the feature mapping of the local key features by the efficient multi-scale convolution block to obtain the second transition feature includes: Expanding the number of channels of the local key features through a first point-by-point convolution, and sequentially performing batch normalization processing, activation function processing, and multi-scale deep convolution on the local key features to obtain deep convolution features; The second point-by-point convolution and batch normalization operations are sequentially performed on the deep convolution features to obtain second transition features.

9. The molten pool image segmentation method according to claim 1, characterized in that: After performing multi-scale feature extraction on the pyramid feature according to the multi-scale convolution decoder to obtain a target molten pool image, the method further includes: Acquire a molten pool method cut angle of the target molten pool image; The welding deviation is determined according to the molten pool method.

10. A molten pool image segmentation device, characterized in that: include: A molten pool image segmentation model construction module, used to construct a molten pool image segmentation model, wherein the molten pool image segmentation model includes a deep convolution encoder and a multi-scale convolution decoder; A pyramid feature acquisition module, used for performing lightweight processing on the welding image to be segmented according to the deep convolution encoder to obtain pyramid features; The target melt pool image acquisition module is used to perform multi-scale feature extraction on the pyramid features according to the multi-scale convolution decoder to obtain a target melt pool image.

Images