Vision-based weld joint identification method, system and equipment and medium

Abstract

The invention discloses a vision-based weld joint identification method, system and device and a medium, and the method comprises the following steps: carrying out the feature extraction of an RGB image and a depth image, and obtaining a multi-stage RGB coding feature and a multi-stage depth coding feature; performing feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level fusion features; and decoding the multi-level fusion features, and enhancing local features to obtain a welding seam recognition result. According to the welding seam identification method, a cross-window remote dependency relationship can be established by using the Swin Transform, two different modal features of the RGB image and the depth image are extracted, complementary information of different modals is fully fused through multi-modal feature fusion and feature enhancement, and local details are enhanced and supplemented, so that the welding seam identification performance is improved, and the welding seam identification efficiency is improved. And a welding seam identification result is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a vision-based weld recognition method, system, equipment and medium. Background Art

[0002] Traditional target recognition methods only use RGB images as input images. When dealing with complex scenes, they are often affected by factors such as illumination changes, occlusion, and background interference, and cannot achieve satisfactory results. In recent years, with the increasing popularity of depth cameras such as Kinect and RealSense, RGB-D target recognition has quickly become a hot research direction. Since RGB images only contain texture and color information, while depth images can provide geometric information such as the spatial structure, shape, and boundaries of the target, RGB-D target recognition helps improve the accuracy and robustness of recognition in complex scenes. However, it cannot be ignored that there are significant differences in information between RGB images and depth images. How to effectively integrate the complementary information of these two different forms has become a key issue in RGB-D target recognition. Existing fusion methods do not fully consider the differences and complementarities between the two modalities, and cannot guarantee the effective fusion of the two modal information.

[0003] Transformer is a deep neural network model based on the self-attention mechanism. It was originally applied in the field of natural language processing (NLP), especially in machine translation, text summarization, question-answering systems, etc., and has achieved remarkable results. In recent years, some scholars have proposed introducing the Transformer architecture into visual tasks and proposed Vision Transformer (ViT). The core idea of ​​ViT is to regard the image as a sequence of pixels and use the Transformer's self-attention mechanism to learn the feature representation of the image. Compared with traditional convolutional neural networks, ViT does not rely on local convolution operations, but adopts global context information, which makes it show powerful capabilities in processing image tasks. Therefore, it has gradually become the core feature extraction network in the fields of target recognition, image classification, image segmentation, etc. However, the block operation of ViT will destroy local details, resulting in block effects, which affects the performance of target recognition.

[0004] In the weld image, since the weld is in the shape of a thin strip and occupies a relatively small proportion in the entire image, it is easily affected by lighting, background, etc., making the weld target inconspicuous. At the same time, due to different acquisition angles and acquisition distances, the clarity of the weld target varies greatly. In view of the above characteristics of the weld image, the existing vision-based target recognition method is used to identify the weld in the image, but the recognition accuracy is low. Summary of the invention

[0005] The main problem to be solved by the present invention is to provide a vision-based weld recognition method, system, equipment and medium to solve the problem of low recognition accuracy when using existing vision-based target recognition methods to recognize welds in images.

[0006] To solve the above technical problems, a technical solution adopted by the present invention is to provide a vision-based weld recognition method, which includes the following steps: performing feature extraction on RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; performing feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and multi-level depth coding features to obtain multi-level fusion features; decoding the multi-level fusion features and enhancing local features to obtain weld recognition results.

[0007] In some embodiments, before extracting features from the RGB image and the depth image respectively to obtain multi-level RGB coding features and multi-level depth coding features, the step also includes: preprocessing the RGB image and the depth image, converting the depth image from a single channel to a three-channel, and reshaping the RGB image and the depth image to the same size.

[0008] In some embodiments, the feature extraction of the RGB image and the depth image is performed to obtain multi-level RGB coding features and multi-level depth coding features, including the steps of: using a Swin Transformer network to extract four layers of features from the RGB image and the depth image, respectively, to obtain the i-th layer of RGB coding features and the i-th layer deep encoding feature i=1,2,3,4; when i=1, the i-th layer RGB encoding feature The size of the RGB image is 1 / 2 of the size of the RGB image, and the i-th layer deep encoding feature The size of is 1 / 2 of the size of the depth image; when i=2,3, the i-th layer RGB encoding feature The size is the RGB encoding feature of the (i-1)th layer The i-th layer has a depth encoding feature of 1 / 2 the size of The size is the (i-1)th layer depth encoding feature ; when i=4, the i-th layer RGB encoding feature The size of the (i-1) layer RGB encoding feature The size of the i-th layer is the same as that of The size and depth encoding features of the (i-1)th layer The same size.

[0009] In some embodiments, the multi-level RGB encoding features and the multi-level depth encoding features are subjected to feature enhancement and cross-modal feature fusion to obtain multi-level fusion features, including the steps of: using a spatial attention weight matrix to perform feature enhancement on the multi-level RGB encoding features and the multi-level depth encoding features, respectively, to obtain multi-level RGB enhanced features and multi-level depth enhanced features; using a coordinate attention mechanism to perform cross-modal feature fusion on the multi-level RGB enhanced features and the multi-level depth enhanced features to obtain the multi-level fusion features.

[0010] In some embodiments, the multi-level fusion features are decoded and local features are enhanced to obtain weld identification results, including the steps of: decoding the multi-level fusion features to generate significant decoding features; using the RGB image to enhance local features in the significant decoding features to obtain the weld identification results.

[0011] The present invention also provides a vision-based weld recognition system, the system includes a weld recognition network, and the weld recognition network includes: a feature extraction module, used to extract features from RGB images and depth images respectively, to obtain multi-level RGB coding features and multi-level depth coding features; a feature fusion module, used to perform feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and multi-level depth coding features, to obtain multi-level fusion features; a decoding module, used to decode the multi-level fusion features and enhance local features to obtain weld recognition results.

[0012] In some embodiments, the feature fusion module includes a spatial attention enhancement module and a coordinate attention fusion module. The spatial attention enhancement module is used to use the spatial attention weight matrix to perform feature enhancement on the multi-level RGB encoding features and the multi-level depth encoding features respectively to obtain multi-level RGB enhanced features and multi-level depth enhanced features; the coordinate attention fusion module is used to use the coordinate attention mechanism to perform cross-modal feature fusion on the multi-level RGB enhanced features and the multi-level depth enhanced features to obtain the multi-level fusion features.

[0013] In some embodiments, the decoding module includes a residual convolution module and a local feature enhancement module. The residual convolution module is used to decode the multi-level fusion features to generate significant decoding features; the local feature enhancement module is used to use the RGB image to enhance the local features in the significant decoding features to obtain the weld recognition result.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0016] The beneficial effects of the present invention are as follows: the present invention discloses a vision-based weld recognition method, system, device and medium, the method comprising the following steps: extracting features from RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; performing feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level fusion features; decoding the multi-level fusion features, and enhancing local features to obtain weld recognition results. The weld recognition method of the present invention can use Swin Transformer to establish a cross-window remote dependency relationship, extract features of two different modes, RGB images and depth images, and fully fuse the complementary information of different modes through multi-modal feature fusion and feature enhancement, enhance and supplement local details, thereby improving the performance of weld recognition and making the weld recognition results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of an embodiment of a vision-based weld identification method of the present invention;

[0018] Figure 2 It is a principle diagram of an embodiment of a vision-based weld identification method of the present invention;

[0019] Figure 3 It is a specific flow chart of step S1 in an embodiment of a vision-based weld recognition method of the present invention;

[0020] Figure 4 It is a data flow diagram in Swin Transformer Block in an embodiment of a vision-based weld recognition method of the present invention;

[0021] Figure 5 It is a specific flow chart of step S2 in an embodiment of a vision-based weld recognition method of the present invention;

[0022] Figure 6 It is a data flow diagram in CIM in an embodiment of a vision-based weld identification method of the present invention;

[0023] Figure 7 It is a specific flow chart of step S3 in an embodiment of a vision-based weld recognition method of the present invention;

[0024] Figure 8 It is a data flow diagram in RCM in an embodiment of a vision-based weld recognition method of the present invention;

[0025] Fig. 9 It is a data flow diagram in FEM in an embodiment of a vision-based weld identification method of the present invention;

[0026] Fig.10 It is a structural block diagram of an embodiment of a vision-based weld recognition system of the present invention;

[0027] Fig.11 It is a specific structural diagram of a feature fusion module in an embodiment of a vision-based weld recognition system of the present invention;

[0028] Fig.12 It is a specific structural diagram of a decoding module in an embodiment of a vision-based weld recognition system of the present invention;

[0029] Fig.13 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention;

[0030] Fig.14 It is a schematic block diagram of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0031] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0032] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0033] like Figure 1 As shown, a vision-based weld identification method of the present invention is shown, comprising the following steps:

[0034] Step S1: extract features from the RGB image and the depth image respectively to obtain multi-level RGB coding features and multi-level depth coding features.

[0035] Combination Figure 2It should be noted that before step S1, the RGB image 1 and the depth image 2 need to be preprocessed, the depth image 2 is converted from a single channel to a three-channel by grayscale mapping, and the size (height and width) of the RGB image 1 and the depth image 2 are reshaped to 384×384. Through preprocessing, the channels of the RGB image 1 and the depth image 2 are both 3, and the sizes of the RGB image 1 and the depth image 2 are the same, so that the two can be connected in batches.

[0036] like Figure 2 As shown, in step S1, two Swin Transformer networks are used as feature extraction modules 101 to extract four layers of features from RGB image 1 and depth image 2 respectively, and obtain the i-th layer RGB encoding feature and the i-th layer deep encoding feature i=1,2,3,4; when i=1, the i-th layer RGB encoding feature The size of is half of the size of the RGB image, and the i-th layer deep encoding feature The size of is 1 / 2 of the depth image size; when i=2,3, the i-th layer RGB encoding feature The size is the RGB encoding feature of the (i-1)th layer The i-th layer has a depth encoding feature of 1 / 2 the size of The size is the (i-1)th layer depth encoding feature ; when i=4, the i-th layer RGB encoding feature The size of the (i-1) layer RGB encoding feature The size of is the same as that of the deep encoding feature of the i-th layer The size and depth encoding features of the (i-1)th layer The same size.

[0037] Among them, the SwinTransformer network is a deep learning model, and the "Swin" in its name refers to "Shifted window", i.e. "shifted window". The characteristic of this model is that it adopts a multi-level structure and uses the method of shifted windows, with emphasis on the characteristics of its Hierarchical (multi-level) and Shifted Windows (shifted windows). The Swin Transformer network has strong flexibility in multi-scale modeling, and can reduce the quadratic computational complexity to linear computational complexity. Compared with traditional visual Transformer, the SwinTransformer network can improve the representation ability of features while maintaining computational efficiency by shifting windows, so the present invention uses SwinTransformer as a feature extraction network.

[0038] like Figure 2 As shown, in this embodiment, the SwinTransformer network adopts the Swin-B version, which has a smaller model and can balance performance and resource consumption to meet the weld recognition requirements in practical applications. The SwinTransformer network includes three components or modules: Patch Embedding, Swin Transformer Block, and Patch Merging, and these modules are combined and used in a certain order.

[0039] Specifically, combined Figure 3 As shown, step S1 includes the following steps:

[0040] Step S11: Segment the RGB image and the depth image into a plurality of image blocks of the same size and without overlapping.

[0041] Among them, using Patch Embedding, the data processing process of Patch Embedding is to cut the input image into multiple small areas (patches), and then convert these areas into vector representations through linear projection for subsequent processing and analysis. This process not only involves dividing the image into small areas of fixed size, but also converting these small areas into high-dimensional vectors through linear layers. These vectors are then input into the Transformer model for tasks such as image recognition.

[0042] In this embodiment, the RGB image 1 and the depth image 2 are respectively divided into 96 image blocks (small areas) of the same size and non-overlapping, and the size of each image block is 4×4.

[0043] Step S12: Encode using the image blocks corresponding to the RGB image and the depth image respectively to obtain multi-level RGB encoding features and multi-level depth encoding features.

[0044] Combination Figure 2 , the multi-level RGB encoding features include four layers of RGB encoding features, and the multi-level depth encoding features include four layers of depth encoding features. When extracting the first three layers of RGB / depth encoding features, the feature extraction process of each layer of RGB / depth encoding features is: first use the corresponding Swin Transformer Block for feature extraction, and then use the corresponding Patch Merging to reduce the dimension of the extracted features, so as to obtain the required first three layers of RGB / depth encoding features; when extracting the 4th layer of RGB / depth encoding features, only use the corresponding Swin Transformer Block for feature extraction, and you can directly get the 4th layer of RGB / depth encoding features.

[0045] Among them, the number of image blocks input and output of each Swin Transformer Block remains unchanged, and all image blocks obtained after segmentation of RGB image 1 / depth image 2 constitute block features.

[0046] like Figure 4 As shown in the figure, the Swin Transformer Block is divided into two parts. In the first part, the block features are first normalized by the corresponding normalization layer LN (Layer Norm), and then passed through the window self-attention operation layer W-MSA (Window-Multihead Self Attention). W-MSA is a windowed multi-head self-attention module, which can divide the normalized block features into windows one by one. Each window is composed of multiple image blocks. Attention operation is performed in each window to obtain the first attention feature, which can effectively reduce the computational complexity. Then, the first attention feature is connected to the block feature by the first jump connection to obtain the first connection feature; secondly, the first connection feature is normalized and fully connected by the corresponding normalization layer LN and multilayer perceptron MLP (Multilayer Perceptron) in turn to obtain the first intermediate feature, and then the first intermediate feature is connected to the first connection feature by the second jump connection to obtain the second connection feature.

[0047] Furthermore, in the second part, the second connection feature is first normalized by the corresponding normalization layer LN, and then passed through the sliding window self-attention layer SW-MSA (Shifted Window-Multihead Self Attention). SW-MSA is a multi-head self-attention module based on a shifted window, which is used to perform information exchange between windows on the normalized second connection feature to obtain the second attention feature, and then the second attention feature is connected to the second connection feature by a third jump connection to obtain the third connection feature; secondly, the third connection feature is normalized and fully connected by the corresponding normalization layer LN and the multi-layer perceptron MLP in turn to obtain the second intermediate feature, and then the second intermediate feature is connected to the third connection feature by a fourth jump connection to obtain the fourth connection feature.

[0048] Combination Figure 2 , the fourth connection feature is input into the corresponding Patch Merging, so that the fourth connection feature is reduced in dimension, and the number of channels becomes twice that of the fourth connection feature, thereby obtaining the corresponding first-layer RGB / depth encoding feature.

[0049] Correspondingly, the output of each layer of Patch Merging is directly connected to the next layer of Swin Transformer Block as the input of the next layer of Swin Transformer Block. When i=2,3, the process of extracting the RGB / depth encoding features of the i-th layer is similar to the process of extracting the RGB / depth encoding features of the first layer; when i=4, the process of extracting the RGB / depth encoding features of the fourth layer is similar to the process of extracting the fourth connection features, which will not be repeated here.

[0050] In this embodiment, when extracting the first three layers of RGB / depth coding features, after the feature extraction stage of each Swin TransformerBlock is completed, Patch Merging is used to halve the height and width of the extracted features and double the number of channels; and when extracting the 4th layer of RGB / depth coding features, only Swin Transformer Block is used for feature extraction. Therefore, it can be obtained that the size of the first layer of RGB coding features and the first layer of depth coding features are both 192×192, and the number of channels is 6; the size of the second layer of RGB coding features and the second layer of depth coding features are both 96×96, and the number of channels is 12; the size of the third layer of RGB coding features and the third layer of depth coding features are both 48×48, and the number of channels is 24; the size of the fourth layer of RGB coding features and the fourth layer of depth coding features are both 48×48, and the number of channels is 24.

[0051] In this process, multi-scale encoding features are obtained from the RGB image 1 and the depth image 2, the feature size is gradually compressed, and the number of channels is gradually increased, so as to achieve the gradual enhancement of semantic information from low-level to high-level.

[0052] Step S2: Perform feature enhancement and cross-modal feature fusion on the multi-level RGB encoding features and the multi-level depth encoding features to obtain multi-level fusion features.

[0053] It should be noted that RGB image 1 contains a lot of texture information, while depth image 2 focuses on spatial position information. On the one hand, the same target position in multimodal images should be the same, so the features of different modalities need to be aligned first. On the other hand, RGB image 1 and depth image 2 each emphasize different important information, so they can use each other's complementary information to enrich their own features. Based on the above two aspects, combined with Figure 2 and Figure 6The present invention designs a feature fusion module 102, which includes a four-layer cross-modal feature fusion module (Cross-ModalIntersectional Fusion Module, CIM). CIM adopts spatial attention mechanism (Spatial Attention, SA) and coordinate attention mechanism (Coordinate Attention, CA), which are used for feature enhancement and cross-modal feature fusion of different modal features respectively.

[0054] Specifically, if Figure 5 As shown, step S2 includes the following sub-steps:

[0055] Step S21: Use the spatial attention weight matrix to perform feature enhancement on the multi-level RGB encoding features and the multi-level depth encoding features respectively to obtain multi-level RGB enhanced features and multi-level depth enhanced features.

[0056] It should be noted that the multi-level RGB coding features include four layers of RGB coding features, and the multi-level depth coding features include four layers of depth coding features. Therefore, the corresponding multi-level RGB enhancement features and multi-level depth enhancement features include four layers of RGB enhancement features and four layers of depth enhancement features, respectively. The following describes step S21 by introducing in detail how to use the i-th layer CIM to obtain the i-th layer RGB enhancement features and the i-th layer depth enhancement features. Among them, the i-th layer CIM here is for Figure 2 The i-th CIM corresponding from top to bottom in the four-layer CIM.

[0057] First, the i-th layer RGB encoding feature and the i-th layer deep encoding feature They are respectively input into the corresponding spatial attention layer SA to obtain the spatial attention weight matrices corresponding to the two; at the same time, the i-th layer RGB encoding feature and the i-th layer deep encoding feature After multiplication, the corresponding spatial attention layer SA is input to obtain the spatial attention weight matrix corresponding to the two. The specific operation can be expressed by the following formula:

[0058]

[0059] Among them, SA represents the spatial attention operation, Represents the RGB encoding feature of the i-th layer The corresponding spatial attention weight matrix, Represents the deep encoding feature of the i-th layer The corresponding spatial attention weight matrix, SA i Represents the RGB encoding feature of the i-th layer and the i-th layer deep encoding feature The corresponding spatial attention weight matrix.

[0060] The specific process of spatial attention operation is as follows: First, the i-th layer RGB encoding feature of an input Or the i-th layer deep encoding feature Or other features are respectively subjected to global maximum pooling and global average pooling in the channel dimension to obtain two feature maps with a channel number of 1, namely the maximum pooling feature map and the average pooling feature map; then, the maximum pooling feature map and the average pooling feature map are spliced ​​according to the channel to obtain a spliced ​​feature map with a channel number of 2; finally, the spliced ​​feature map is convolved and then activated using the Sigmoid activation function to obtain the RGB encoding feature of the i-th layer Or the i-th layer deep encoding feature Or the spatial attention weight matrix corresponding to other features. In this process, the spatial attention operation focuses on the location of effective information on the feature map, performs pooling in the channel dimension, and compresses the channel size to facilitate the subsequent learning of spatial features.

[0061] Furthermore, the i-th layer RGB encoding feature Multiply by SA i and The sum of the i-th layer deep encoding features Multiply by SA i and The sum of the i-th layer RGB enhanced features and the i-th layer depth enhanced features is obtained to utilize each other's important information. The specific operation can be expressed by the following formula:

[0062]

[0063] Among them, F i r represents the RGB enhanced features of the i-th layer, F i d represents the i-th layer depth enhancement feature.

[0064] Step S22: Using the coordinate attention mechanism, perform cross-modal feature fusion on the multi-level RGB enhanced features and the multi-level depth enhanced features to obtain multi-level fusion features.

[0065] In this embodiment, the multi-level fusion feature includes four layers of fusion features. For the i-th layer of fusion features, the i-th layer of RGB enhancement features F i r and the i-th layer depth enhancement feature F i d Through the fusion of the coordinate attention mechanism, the i-th layer fusion features can be obtained.

[0066] Combination Figure 6 , first the i-th layer RGB enhanced feature F i r and the i-th layer depth enhancement feature F i d Splicing to get the i-th layer RGB-D feature F i , Secondly, the i-th layer RGB-D features are input into the horizontal pooling layer HP and the vertical pooling layer WP for pooling, and then the pooling results are input into the splicing convolution layer ConvBS for splicing and convolution operations, and then the features output by the splicing convolution layer ConvBS are input into the horizontal coordinate attention layer HCA and the vertical coordinate attention layer WCA respectively. After the coordinate attention operation, the horizontal coordinate attention feature map F is obtained. h And the vertical coordinate attention feature map F w , and combine the two with the i-th layer RGB-D feature F i Fusion, and convolution through the Conv layer, to obtain the i-th layer fusion feature F i f The specific fusion process can be expressed by the following formula:

[0067] F i =Cat(F i r ,F i d ),

[0068] F h / w =CA h / w (ConvBS(P h (F i ),P w (F i ))),

[0069] F i f =Conv(F i ×F h ×F w ),

[0070] Among them, Cat represents the splicing operation along the channel dimension (corresponding to Figure 6 In );F i represents the RGB-D feature of the i-th layer; P h and P w Represents one-dimensional horizontal average pooling and one-dimensional vertical average pooling, respectively, which are used to make the i-th layer RGB-D feature F i The height h and width w of become 1; ConvBS(x,y) means that x and y are concatenated first, and then the concatenated result is convolved; CA h and CA wrepresents the coordinate attention operation along the height h and width w directions respectively; F h and F w They represent the horizontal coordinate attention feature map and the vertical coordinate attention feature map generated along the height h and width w directions respectively; Conv represents the convolution operation; F i f Represents the fusion features of the i-th layer.

[0071] In this process, each layer of CIM first enhances the corresponding RGB encoding features and depth encoding features through the cross-spatial attention mechanism, and then fuses the enhanced features of the same scale of the two modalities through the fusion part, which can effectively realize the information complementarity between different modalities.

[0072] Step S3: Decode the multi-level fusion features and enhance the local features to obtain the weld recognition results.

[0073] Specifically, if Figure 7 As shown, step S3 includes the following sub-steps:

[0074] Step S31: Decode the multi-level fusion features to generate significant decoding features.

[0075] Combination Figure 2 Since some edge detail information is often lost during the feature extraction process, and the upsampling process during the decoding process will also introduce certain noise information, the present invention designs a decoding module 103, which includes three layers of residual convolutional modules (RCM). Each layer of RCM can be regarded as a hierarchical progressive residual decoder, which gradually integrates low-level features to supplement detail information during the decoding process, and reduces the influence of noise through residual convolution blocks.

[0076] In this embodiment, the 4th layer fusion feature F obtained by the 4th layer CIM is first i 4 The third-layer fusion feature F obtained by the third-layer CIM i 3 Input into the third layer RCM, the third layer RCM decodes to obtain the first significant feature p3; then the first significant feature p3 obtained by the third layer RCM and the second layer fusion feature F obtained by the second layer CIM are combined. i 2 Input into the second layer RCM, the second layer RCM is decoded to obtain the second significant feature p2; then the second significant feature p2 obtained by the second layer RCM and the first layer fusion feature F obtained by the first layer CIM are combined. i 1 Input into the first layer RCM, the first layer RCM decoding generates significant decoding features p1. The specific process can be expressed by the following formula:

[0077]

[0078] Here, RCM represents the decoding operation within the corresponding RCM.

[0079] It should be noted that Figure 2 The three layers of RCM from top to bottom correspond to the first layer RCM, the second layer RCM, and the third layer RCM.

[0080] like Figure 2 and Figure 8 As shown, the specific operation in the j-th layer RCM can be expressed by the following formula:

[0081]

[0082] Among them, when j = 3, When j = 2, 1, UP represents an upsampling operation; PW1 represents a point convolution operation with a convolution kernel size of 1×1 and 4 channels; PW2 represents a point convolution operation with a convolution kernel size of 1×1 and 1 channel; represents the GELU activation function; LN represents the normalization operation; DW represents the depth convolution operation with a convolution kernel size of 7×7 and a channel number of 1.

[0083] In this embodiment, RCM is used to gradually fuse low-level features and improve features using residual convolution, thereby obtaining more accurate significant decoding features for weld information.

[0084] Step S32: using the RGB image, enhancing the local features in the significant decoding features to obtain the weld recognition result.

[0085] Due to the block operation of Patch Embedding in step S1, the obtained significant decoding feature p1 may have problems of block effect and detail destruction. Figure 2 and Fig. 9 Inspired by the advantages of convolutional neural networks in processing local details, the decoding module 103 also includes a local feature enhancement module (Feature Enhancement Module, FEM), which can be used to supplement detail information and further improve the performance of the network, making the weld recognition result obtained by the network more accurate.

[0086] like Fig. 9 As shown in the figure, FEM introduces the first two layers VGG Block1 and VGGBlock2 in the convolutional neural network VGG16 model to extract two shallow features with rich texture details in RGB image 1, which are represented as the first shallow feature V 112 and the second shallow feature V224 .

[0087] First, the significant decoding feature p1 is input into the first convolution activation layer BC1. The first convolution activation layer BC1 includes a BaseConv2d layer. The BaseConv2d layer contains a convolution layer and a ReLu activation function.

[0088] After the significant decoding feature p1 is sequentially subjected to the corresponding convolution and activation operations through the first convolution activation layer BC1, the feature output by the first convolution activation layer BC1 is upsampled through the upsampling layer UP to obtain the first shallow feature V 112 The first decoded feature T with the same resolution 112 The specific process can be expressed by the following formula:

[0089] T 112 =UP(Baseconv(p1)),

[0090] Among them, BaseConv means performing corresponding convolution and activation operations in sequence.

[0091] Furthermore, considering that simply using concatenated fusion features cannot effectively capture the detail information embedded in certain channels, the channel attention mechanism (CA) is used here to find important channels with detail information for adaptive fusion while retaining the salient target body. 112 With the first shallow feature V 112 After splicing, the corresponding channel attention layer CA is input to perform channel attention operation to obtain the first local enhanced feature.

[0092] Then, after the first local enhancement feature is input into the corresponding second convolution activation layer BC2 for corresponding convolution and activation operations, the feature output by the corresponding second convolution activation layer BC2 is upsampled through the upsampling layer UP to obtain the second shallow feature V 224 The second decoded feature T with the same resolution 224 Then the second decoding feature T 224 With the second shallow feature V 224 After splicing, the corresponding channel attention layer CA is input to perform channel attention operation to obtain the second local enhanced feature.

[0093] Finally, the second local enhancement feature is input into the corresponding second convolution activation layer BC2 for corresponding convolution and activation operations to obtain the final weld recognition result S out Among them, the second convolution activation layer BC2 includes two BaseConv2d layers, and the specific process can be expressed by the following formula:

[0094] T 224 =UP(Baseconv(Baseconv(CA(Cat(T 112 ,V 112 ))))),

[0095] S out =Baseconv(Baseconv(CA(Cat(T 224 ,V 224 )))),

[0096] Among them, CA represents the channel attention operation.

[0097] In this embodiment, a FEM is added at the end of the network to achieve local detail enhancement and supplementation, thereby outputting a weld recognition result with a high weld recognition accuracy.

[0098] Based on the same inventive concept, the present invention also provides a vision-based weld recognition system, combined with Figure 2 and Fig.10 As shown, the system includes a weld identification network (Weld Identification Network, WINet), WINet includes:

[0099] The feature extraction module 101 is used to extract features from the RGB image 1 and the depth image 2 respectively to obtain multi-level RGB coding features and multi-level depth coding features.

[0100] The feature fusion module 102 is used to perform feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level fusion features.

[0101] The decoding module 103 is used to decode the multi-level fusion features and enhance the local features to obtain the weld recognition result.

[0102] like Figure 2 As shown, after the RGB image 1 and the depth image 2 are input into WINet, the RGB image 1 and the depth image 2 can be processed by the various component modules in the network, and finally the required weld recognition result can be output through the decoding module 103.

[0103] Furthermore, the system also includes a preprocessing module, which is used to preprocess the RGB image 1 and the depth image 2, reshape the RGB image 1 and the depth image 2, make the RGB image 1 and the depth image 2 have the same size, and convert the depth image 2 from a single channel to a three-channel.

[0104] Furthermore, if Fig.11As shown, the feature fusion module 102 includes a spatial attention enhancement module 104 and a coordinate attention fusion module 105. The spatial attention enhancement module 104 is used to utilize the spatial attention weight matrix to perform feature enhancement on the multi-level RGB encoding features and the multi-level depth encoding features respectively to obtain multi-level RGB enhanced features and multi-level depth enhanced features; the coordinate attention fusion module 105 is used to utilize the coordinate attention mechanism to perform cross-modal feature fusion on the multi-level RGB enhanced features and the multi-level depth enhanced features to obtain multi-level fusion features.

[0105] Furthermore, if Fig.12 As shown, the decoding module 103 includes a residual convolution module 106 and a local feature enhancement module 107. The residual convolution module 106 is used to decode the multi-level fusion features to generate significant decoding features; the local feature enhancement module 107 is used to use the RGB image to enhance the local features in the significant decoding features to obtain the weld recognition result.

[0106] In the present application, other technical features of the above-mentioned vision-based weld recognition system are the same as the features disclosed in the above-mentioned method embodiment and will not be repeated here.

[0107] Based on the same inventive concept, the present application also provides an electronic device, which includes a processor, a memory and a communication circuit, and the processor is connected to the memory and the communication circuit respectively; wherein the communication circuit is used for communication connection, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above method.

[0108] See also Fig.13 The electronic device described in the embodiment of the present application may specifically include a processor 210 and a memory 220. The memory 220 is coupled to the processor 210.

[0109] The processor 210 is used to control the operation of the electronic device. The processor 210 may also be referred to as a CPU (Central Processing Unit). The processor 210 may be an integrated circuit chip having the ability to process signals. The processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 210 may also be any conventional processor, etc.

[0110] The memory 220 is used to store computer programs, and may be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 220 may include one or more computer-readable storage media, which may be non-transitory. The memory 220 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage terminals, flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 220 is used to store at least one program code.

[0111] The processor 210 is used to execute the computer program stored in the memory 220 to implement the methods described in the various method embodiments of the present application.

[0112] In some embodiments, the electronic device may further include: a peripheral terminal interface 230 and at least one peripheral terminal. The processor 210, the memory 220 and the peripheral terminal interface 230 may be connected via a bus or a signal line. Each peripheral terminal may be connected to the peripheral terminal interface 230 via a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes: at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260 and a power supply 270.

[0113] The peripheral terminal interface 230 may be used to connect at least one peripheral terminal related to I / O (Input / output) to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral terminal interface 230 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 210, the memory 220, and the peripheral terminal interface 230 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0114] The radio frequency circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 240 communicates with the communication network and other IoT devices through electromagnetic signals, and the radio frequency circuit 240 is the communication circuit of the electronic device. The radio frequency circuit 240 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 240 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identity module card, and the like. The radio frequency circuit 240 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 240 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0115] The display screen 250 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 250 is a touch display screen, the display screen 250 also has the ability to collect touch signals on the surface or above the surface of the display screen 250. The touch signal can be input to the processor 210 as a control signal for processing. At this time, the display screen 250 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 250 can be one, set on the front panel of the electronic device; in other embodiments, the display screen 250 can be at least two, respectively set on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 250 can be a flexible display screen, set on a curved surface or a folding surface of the electronic device. Even, the display screen 250 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0116] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 210 for processing, or input them into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0117] The power supply 270 is used to power various components in the electronic device. The power supply 270 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0118] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0122] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above method.

[0123] See also Fig.14 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0124] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0125] It can be seen that the present invention discloses a vision-based weld recognition method, system, device and medium, and the method includes the following steps: extracting features from RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; performing feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and multi-level depth coding features to obtain multi-level fusion features; decoding the multi-level fusion features, and enhancing local features to obtain weld recognition results. The weld recognition method of the present invention can use Swin Transformer to establish a cross-window remote dependency relationship, extract features of two different modes of RGB images and depth images, and fully fuse the complementary information of different modes through multi-modal feature fusion and feature enhancement, enhance and supplement local details, thereby improving the performance of weld recognition and making the weld recognition results more accurate.

[0126] The above are only embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent structural transformations made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are included in the protection scope of the present invention.

Claims

1. A vision-based weld recognition method, characterized in that: The method comprises the following steps: Perform feature extraction on RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; Performing feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level fusion features; The multi-level fusion features are decoded, and local features are enhanced to obtain weld recognition results.

2. The weld identification method according to claim 1, characterized in that: Before extracting features from the RGB image and the depth image to obtain multi-level RGB coding features and multi-level depth coding features, the method further includes the following steps: The RGB image and the depth image are preprocessed, the depth image is converted from a single channel to a three-channel image, and the RGB image and the depth image are reshaped to the same size.

3. The weld identification method according to claim 2, characterized in that: The method of extracting features from the RGB image and the depth image respectively to obtain multi-level RGB coding features and multi-level depth coding features comprises the following steps: Using the Swin Transformer network, four layers of features are extracted from the RGB image and the depth image respectively to obtain the i-th layer of RGB encoding features. and the i-th layer deep encoding feature Among them, when i=1, the i-th layer RGB encoding feature The size of the RGB image is 1 / 2 of the size of the RGB image, and the i-th layer deep encoding feature The size of is 1 / 2 of the size of the depth image; When i=2,3, the i-th layer RGB encoding feature The size is the RGB encoding feature of the (i-1)th layer The i-th layer has a depth encoding feature of 1 / 2 the size of The size is the (i-1)th layer depth encoding feature 1 / 2 the size; When i=4, the i-th layer RGB encoding feature The size of the (i-1) layer RGB encoding feature The size of the i-th layer is the same as that of The size and depth encoding features of the (i-1)th layer The same size.

4. The weld identification method according to claim 1, characterized in that: The step of performing feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level fusion features includes the following steps: Using the spatial attention weight matrix, the multi-level RGB encoding features and the multi-level depth encoding features are respectively enhanced to obtain multi-level RGB enhanced features and multi-level depth enhanced features; The coordinate attention mechanism is used to perform cross-modal feature fusion on the multi-level RGB enhanced features and the multi-level depth enhanced features to obtain the multi-level fusion features.

5. The weld identification method according to claim 1, characterized in that: The decoding of the multi-level fusion features and the enhancement of local features to obtain weld recognition results include the following steps: Decoding the multi-level fusion features to generate significant decoding features; The RGB image is used to enhance local features in the significant decoding features to obtain the weld recognition result.

6. A vision-based weld recognition system, characterized in that: The system includes a weld identification network, and the weld identification network includes: A feature extraction module is used to extract features from RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; A feature fusion module, used to perform feature enhancement and cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level fusion features; The decoding module is used to decode the multi-level fusion features and enhance local features to obtain weld recognition results.

7. The weld identification system according to claim 6, characterized in that: The feature fusion module includes a spatial attention enhancement module and a coordinate attention fusion module. The spatial attention enhancement module is used to use the spatial attention weight matrix to perform feature enhancement on the multi-level RGB coding features and the multi-level depth coding features, respectively, to obtain multi-level RGB enhanced features and multi-level depth enhanced features; The coordinate attention fusion module is used to utilize the coordinate attention mechanism to perform cross-modal feature fusion on the multi-level RGB enhanced features and the multi-level depth enhanced features to obtain the multi-level fusion features.

8. The weld identification system according to claim 6, characterized in that: The decoding module includes a residual convolution module and a local feature enhancement module. The residual convolution module is used to decode the multi-level fusion features to generate significant decoding features; The local feature enhancement module is used to utilize the RGB image to enhance the local features in the significant decoding features to obtain the weld recognition result.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images