Welding defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a welding defect detection method and device, electronic equipment and a storage medium. The method comprises the steps that multiple frames of to-be-detected molten pool images in a to-be-detected molten pool video are obtained; inputting the to-be-detected molten pool images into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to the to-be-detected molten pool images; and inputting the molten pool image features corresponding to all the to-be-detected molten pool images into a welding defect detection module of a welding defect detection model to obtain welding defects corresponding to the to-be-detected molten pool videos. According to the method, the welding defects can be detected through the multiple frames of molten pool images in the molten pool video, and the welding defect detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of welding technology, and in particular to a welding defect detection method, device, electronic equipment and storage medium. Background Art

[0002] Welding defects refer to defects formed in the weld joint during the welding process. At present, the influence of various process parameters on the porosity of additive welds in actual production is studied by using X-ray flaw detection and pixel statistics, and the molten pool porosity defects generated under process parameters with higher porosity are selected as experimental data.

[0003] The existing method uses a single-frame molten pool image to detect welding defects. However, in real welding scenarios, the molten pool area is in a molten state. Welding and the occurrence of welding defects are a dynamic process. Many welding defects need to be judged through the change process of the molten pool. Therefore, it is inaccurate to use a single-frame image to detect welding defects. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a welding defect detection method, device, electronic device and storage medium, which can detect welding defects through multiple frames of molten pool images in a molten pool video, thereby improving the accuracy of detecting welding defects.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting welding defects, the method for detecting welding defects comprising:

[0006] Acquire multiple frames of images of the molten pool to be detected in the molten pool video to be detected;

[0007] Input each molten pool image to be detected into the feature extraction module of the welding defect detection model respectively to obtain the molten pool image features corresponding to each molten pool image to be detected;

[0008] The molten pool image features corresponding to all the molten pool images to be detected are input into the welding defect detection module of the welding defect detection model to obtain the welding defects corresponding to the molten pool videos to be detected.

[0009] In a possible implementation, each molten pool image to be detected is input into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected, including:

[0010] The molten pool images to be detected are respectively input into the lightweight network units in the feature extraction module to obtain a plurality of initial image features corresponding to the molten pool images to be detected;

[0011] For all initial image features corresponding to the molten pool image to be detected, feature splicing is performed in the order of image size from small to large, so as to obtain the first intermediate image features corresponding to the molten pool image to be detected;

[0012] All initial image features corresponding to the molten pool image to be detected are added element by element in the order of image size from small to large, so as to obtain a second intermediate image feature corresponding to the molten pool image to be detected;

[0013] The first intermediate image features and the second intermediate image features corresponding to the molten pool image to be detected are fused to obtain the molten pool image features corresponding to the molten pool image to be detected.

[0014] In a possible implementation manner, the initial image features include a first initial image feature, a second initial image feature, a third initial image feature, and a fourth initial image feature;

[0015] The image size of the first initial image feature, the image size of the second initial image feature, the image size of the third initial image feature, and the image size of the fourth initial image feature gradually increase.

[0016] In a possible implementation manner, all initial image features corresponding to the molten pool image to be detected are spliced ​​in order of image size from small to large to obtain a first intermediate image feature corresponding to the molten pool image to be detected, including:

[0017] Determine the first initial image feature corresponding to the molten pool image to be detected as the current convolution image feature;

[0018] The initial image features corresponding to the molten pool image to be detected are traversed in order from small to large image sizes;

[0019] The image features obtained by upsampling the current convolution image features and the initial image features of the next traversal are concatenated to obtain concatenated image features;

[0020] Performing depth-separable convolution on the spliced ​​image features to obtain the latest convolution image features; wherein the feature scale of the latest convolution image features is the same as the feature scale of the next traversed initial image features;

[0021] If the currently traversed initial image feature is the third initial image feature, the latest convolution image feature is determined as the first intermediate image feature; otherwise, the traversal continues.

[0022] In a possible implementation, the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected are fused to obtain the molten pool image feature corresponding to the molten pool image to be detected, including:

[0023] Adding the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element, obtaining the first molten pool image feature;

[0024] Multiplying the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element, obtaining a second molten pool image feature;

[0025] The first molten pool image feature and the second molten pool image feature corresponding to the molten pool image to be detected are feature spliced ​​to obtain a final molten pool image feature corresponding to the molten pool image to be detected.

[0026] In a possible implementation, the welding defect detection model is trained by the following steps:

[0027] Acquire multiple frames of molten pool sample images in the molten pool sample video, and actual welding defects corresponding to the molten pool sample video;

[0028] Input each molten pool sample image into the feature extraction module in the welding defect detection model respectively, and obtain the fifth initial sample image feature, the first intermediate sample image feature, and the molten pool sample image feature corresponding to each molten pool image to be detected;

[0029] Obtaining target training features according to the fifth initial sample image features and the first intermediate sample image features;

[0030] Inputting the molten pool sample image features corresponding to all molten pool sample images into the welding defect detection module of the welding defect detection model to obtain the detected welding defects corresponding to the molten pool sample videos;

[0031] The welding defect detection model is trained based on the target training features corresponding to all molten pool sample images, the actual welding defects corresponding to the molten pool sample videos, and the detected welding defects.

[0032] In a second aspect, an embodiment of the present application further provides a welding defect detection device, the welding defect detection device comprising:

[0033] An acquisition module is used to acquire multiple frames of images of the molten pool to be detected in the molten pool video to be detected;

[0034] An input module, used to input each molten pool image to be detected into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected;

[0035] The input module is also used to input the molten pool image features corresponding to all the molten pool images to be detected into the welding defect detection module of the welding defect detection model to obtain the welding defects corresponding to the molten pool videos to be detected.

[0036] In a possible implementation, the input module is specifically used to input the molten pool image to be detected into the lightweight network unit in the feature extraction module respectively, to obtain multiple initial image features corresponding to the molten pool image to be detected; for all the initial image features corresponding to the molten pool image to be detected, feature splicing is performed in the order of image size from small to large, to obtain the first intermediate image feature corresponding to the molten pool image to be detected; for all the initial image features corresponding to the molten pool image to be detected, element-by-element addition is performed in the order of image size from small to large, to obtain the second intermediate image feature corresponding to the molten pool image to be detected; the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected are merged to obtain the molten pool image feature corresponding to the molten pool image to be detected.

[0037] In a possible implementation, the initial image features include a first initial image feature, a second initial image feature, a third initial image feature, and a fourth initial image feature; the image size of the first initial image feature, the image size of the second initial image feature, the image size of the third initial image feature, and the image size of the fourth initial image feature gradually increase.

[0038] In a possible implementation, the input module is further used for:

[0039] Determine the first initial image feature corresponding to the molten pool image to be detected as the current convolution image feature;

[0040] The initial image features corresponding to the molten pool image to be detected are traversed in order from small to large image sizes;

[0041] The image features obtained by upsampling the current convolution image features and the initial image features of the next traversal are concatenated to obtain concatenated image features;

[0042] Performing depth-separable convolution on the spliced ​​image features to obtain the latest convolution image features; wherein the feature scale of the latest convolution image features is the same as the feature scale of the next traversed initial image features;

[0043] If the currently traversed initial image feature is the third initial image feature, the latest convolution image feature is determined as the first intermediate image feature; otherwise, the traversal continues.

[0044] In a possible implementation, the input module is further used for:

[0045] Adding the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element, obtaining the first molten pool image feature;

[0046] Multiplying the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element, obtaining a second molten pool image feature;

[0047] The first molten pool image feature and the second molten pool image feature corresponding to the molten pool image to be detected are feature spliced ​​to obtain a final molten pool image feature corresponding to the molten pool image to be detected.

[0048] In a possible embodiment, the device also includes a training module; the training module is specifically used to obtain multiple frames of molten pool sample images in the molten pool sample video, and the actual welding defects corresponding to the molten pool sample video; each molten pool sample image is input into the feature extraction module in the welding defect detection model respectively, and the fifth initial sample image feature, the first intermediate sample image feature, and the molten pool sample image feature corresponding to each molten pool image to be detected are obtained; according to the fifth initial sample image feature and the first intermediate sample image feature, the target training feature is obtained; the molten pool sample image features corresponding to all molten pool sample images are input into the welding defect detection module of the welding defect detection model, and the detected welding defects corresponding to the molten pool sample video are obtained; the welding defect detection model is trained according to the target training features corresponding to all molten pool sample images, the actual welding defects corresponding to the molten pool sample video, and the detected welding defects.

[0049] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of any welding defect detection method as described in the first aspect.

[0050] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the welding defect detection method as described in any one of the first aspects are executed.

[0051] The embodiment of the present application provides a method, device, electronic device and storage medium for detecting welding defects, the method comprising: obtaining multiple frames of molten pool images to be detected in a molten pool video to be detected; inputting each molten pool image to be detected into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected; inputting molten pool image features corresponding to all molten pool images to be detected into a welding defect detection module of a welding defect detection model to obtain welding defects corresponding to the molten pool video to be detected. The present application can detect welding defects through multiple frames of molten pool images in a molten pool video, thereby improving the accuracy of detecting welding defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flow chart of a welding defect detection method provided in an embodiment of the present application is shown;

[0054] Figure 2 A schematic diagram showing molten pool images of different welding defects provided in an embodiment of the present application is shown;

[0055] Figure 3 A flow chart showing another welding defect detection method provided in an embodiment of the present application;

[0056] Figure 4 A structural schematic diagram of a welding defect detection model provided in an embodiment of the present application is shown;

[0057] Figure 5 A schematic structural diagram of a welding defect detection device provided in an embodiment of the present application is shown;

[0058] Figure 6 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0059] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0060] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0061] In order to enable those skilled in the art to use the contents of this application, the following implementation is provided in conjunction with a specific application scenario, the "autonomous driving field". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the "autonomous driving field", it should be understood that this is only an exemplary embodiment.

[0062] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0063] A welding defect detection method provided in an embodiment of the present application is described in detail below.

[0064] Reference Figure 1 As shown, it is a schematic diagram of a flow chart of a welding defect detection method provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:

[0065] S101, obtaining multiple frames of images of a molten pool to be detected in a video of the molten pool to be detected.

[0066] In the implementation manner of the present application, the molten pool refers to a liquid metal part of a certain shape formed on the weldment during welding. A video of the molten pool formed during welding is obtained to obtain a video of the molten pool to be detected; the duration of the video of the molten pool to be detected is a preset duration, generally set to 1 second, and contains 20 to 30 frames of images per second. A preset number of images (generally 10 frames, by limiting the number can reduce the detection workload and improve the detection efficiency) in the video of the molten pool to be detected are determined as the initial molten pool images to be detected;

[0067] Furthermore, the python toolkit labelme is used to annotate the molten pool area, molten hole area and weld fusion line after welding in the initial molten pool image to be detected; the annotated initial molten pool image to be detected is converted into an image mask form to obtain the final molten pool image to be detected; in the final molten pool image to be detected, the annotated area in the initial molten pool image to be detected is displayed using a first specific color (such as white), and the non-annotated area in the initial molten pool image to be detected is displayed using a second specific color (such as black), so that more attention can be paid to the annotated area when determining welding defects.

[0068] The image mask refers to the use of a specific image or function to cover or mask the target area to control the image processing area or the image processing process. The image mask is mainly used to extract the region of interest (ROI).

[0069] Here, welding defects include porosity defects, lack of fusion defects and swing abnormalities. During the occurrence of porosity defects, the characteristic changes in the molten pool area are obviously different from other welding defects; during the occurrence of lack of fusion defects, the shape changes of the molten holes in the molten pool area are obviously different from other defects; during the occurrence of swing abnormalities, the shape changes of the weld fusion line after welding in the molten pool area are obviously different from other defects; therefore, it is necessary to mark the molten pool area, the molten hole area and the weld fusion line after welding in the initial molten pool image to be detected and display them in specific colors to improve the accuracy of detecting welding defects. Figure 2 Shown is a schematic diagram of molten pool images of different welding defects provided in an embodiment of the present application.

[0070] S102, inputting each molten pool image to be detected into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected.

[0071] In the embodiment of the present application, the welding defect detection model includes a feature extraction module and a welding defect detection module; the feature extraction module includes a lightweight network unit, a feature processing unit and an image segmentation unit;

[0072] Specifically, the molten pool image to be detected is input into the feature extraction module of the welding defect detection model through the following steps to obtain the molten pool image features corresponding to the molten pool image to be detected:

[0073] Step 1: Input the molten pool image to be detected into the lightweight network unit in the feature extraction module to obtain multiple initial image features corresponding to the molten pool image to be detected.

[0074] In the embodiment of the present application, before the molten pool image to be detected is input into the lightweight network unit in the feature extraction module, the size of the molten pool image to be detected is unified to 416*416*, and converted into an image with three RGB channels and a tensor shape of [3,416,416], and then input into the lightweight network unit. Among them, the lightweight network unit uses the lightweight network model MoblieNetV2 as the backbone to perform preliminary feature extraction.

[0075] Here, the outputs of different scales of the intermediate layers of the lightweight network unit are extracted, namely the initial first initial image features [320, 13, 13], the second initial image features [96, 26, 26], the initial third initial image features [32, 52, 52], and the fourth initial image features [24, 104, 104]; the initial first initial image features and the initial third initial image features are respectively input into the CBAM (Convolutional Block Attention Module) unit to obtain the final first initial image features and the final third initial image features.

[0076] Among them, the image size of the first initial image feature, the image size of the second initial image feature, the image size of the third initial image feature, and the image size of the fourth initial image feature gradually increase. In addition, the feature size of the fourth initial image feature is one-fourth of the image size of the molten pool image to be detected; the image size of the third initial image feature is one-half of the feature size of the fourth initial image feature; the image size of the second initial image feature is one-half of the feature size of the third initial image feature; and the image size of the first initial image feature is one-half of the feature size of the second initial image feature. The first value in the feature scale is the feature dimension, and the second value and the third value are the height and width of the feature dimension, respectively. For example, the first initial image feature is [320,13,13], where 320 is the feature dimension, and 13 and 13 are the height and width of the feature dimension, respectively.

[0077] Here, CBAM can increase the model's attention to small targets and improve the detection accuracy of welding defects.

[0078] Step 2: For all initial image features corresponding to the molten pool image to be detected, feature splicing is performed in the order of image size from small to large to obtain the first intermediate image features corresponding to the molten pool image to be detected.

[0079] In the implementation mode of the present application, all the initial image features corresponding to the molten pool image to be detected are input into the image segmentation unit, and all the initial image features corresponding to the molten pool image to be detected are feature spliced ​​in the order of image size from small to large, so as to obtain the first intermediate image feature corresponding to the molten pool image to be detected. Specifically, the first initial image feature corresponding to the molten pool image to be detected is determined as the current convolution image feature; the initial image features corresponding to the molten pool image to be detected are traversed in sequence in the order of image size from small to large; the image feature obtained after upsampling the current convolution image feature and the next traversed initial image feature are feature spliced ​​to obtain the spliced ​​image feature; the spliced ​​image feature is subjected to a depth-separable convolution to obtain the latest convolution image feature. Among them, the feature scale of the latest convolution image feature is the same as the feature scale of the next traversed initial image feature; if the current traversed initial image feature is the third initial image feature, the latest convolution image feature is determined as the first intermediate image feature; otherwise, the traversal is continued.

[0080] Specifically, the first initial image feature corresponding to the molten pool image to be detected is determined as the current convolution image feature; first, the first initial image feature is traversed; the image feature obtained after upsampling the first initial image feature (the feature size of the image feature is the same as the feature size of the second initial image feature) and the second initial image feature are feature-concatenated to obtain a concatenated image feature (the same feature size as the second initial image feature); the concatenated image feature is depth-separable convolution performed to obtain the latest convolution image feature (the same feature scale as the second initial image feature); since the currently traversed first initial image feature is not the third initial image feature, the second initial image feature is traversed;

[0081] The image feature obtained by upsampling the current convolution image feature (the feature size of the image feature is the same as the feature size of the third initial image feature) and the third initial image feature are feature-concatenated to obtain a concatenated image feature (the feature size is the same as the third initial image feature); the concatenated image feature is depth-separable convolution performed to obtain the latest convolution image feature (the feature scale is the same as the third initial image feature); since the second initial image feature currently traversed is not the third initial image feature, the third initial image feature is started to be traversed;

[0082] The image feature obtained by upsampling the current convolution image feature (the feature size of the image feature is the same as the feature size of the fourth initial image feature) and the fourth initial image feature are spliced ​​to obtain a spliced ​​image feature (the same feature size as the fourth initial image feature); the spliced ​​image feature is subjected to depthwise separable convolution to obtain the latest convolution image feature (the same feature scale as the fourth initial image feature); since the third initial image feature currently traversed is the third initial image feature, the latest convolution image feature is determined as the first intermediate image feature; the final feature scale of the first intermediate image feature is [24,104,104].

[0083] Here, the image segmentation unit is composed of the Encoder part of the U-Net network, and a skip connection is used in the depthwise separable convolution Depconv, and the convolution kernel of the convolution 3*3 is set to 7*7 to increase the receptive field and improve the perception of small targets. The convolution layer is replaced by an improved depthwise separable convolution, that is, a skip connection is used in the depthwise separable convolution in this application, and a 7*7 convolution kernel is adopted, which can increase the receptive field and improve the perception of small targets.

[0084] Step 3: For all initial image features corresponding to the molten pool image to be detected, add them element by element in the order of image size from small to large to obtain the second intermediate image features corresponding to the molten pool image to be detected.

[0085] In the embodiment of the present application, all the initial image features corresponding to the molten pool image to be detected are input into the feature processing unit, and the second intermediate image features corresponding to the molten pool image to be detected are obtained by element-by-element addition in the order of image size from small to large. Specifically, it includes: the image features obtained by bilinear interpolation upsampling the first initial image features corresponding to the molten pool image to be detected are added element-by-element to the second initial image features corresponding to the molten pool image to be detected, so as to obtain the first spliced ​​image features (with the same feature scale as the second initial image features); the image features obtained by bilinear interpolation upsampling the first spliced ​​image features are added element-by-element to the third initial image features corresponding to the molten pool image to be detected, so as to obtain the second spliced ​​image features (with the same feature scale as the third initial image features); the image features obtained by bilinear interpolation upsampling the second spliced ​​image features are added element-by-element to the fourth initial image features corresponding to the molten pool image to be detected, so as to obtain the second intermediate image features (with the same feature scale as the fourth initial image features, which is [24,104,104]).

[0086] Here, the feature processing unit uses bilinear interpolation upsampling and element-by-element addition operations to form an overall parameter-free branch, which is of great significance for reducing the amount of calculation and improving the real-time performance of the model.

[0087] Step 4: Fuse the first intermediate image features and the second intermediate image features corresponding to the molten pool image to be detected to obtain the molten pool image features corresponding to the molten pool image to be detected.

[0088] In the implementation mode of the present application, the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected are added element by element to obtain the first molten pool image feature; the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected are multiplied element by element to obtain the second molten pool image feature; the first molten pool image feature and the second molten pool image feature corresponding to the molten pool image to be detected are convolved to obtain the final molten pool image feature corresponding to the molten pool image to be detected. The feature scale of the final molten pool image feature is [48,104,104].

[0089] S103, inputting the molten pool image features corresponding to all the molten pool images to be detected into the welding defect detection module of the welding defect detection model to obtain the welding defects corresponding to the molten pool videos to be detected.

[0090] In an implementation manner of the present application, the molten pool image features corresponding to all molten pool images to be detected are input into a long short-term memory (LSTM) network to obtain an initial welding defect feature vector; the initial welding defect feature vector is input into a fully connected layer in a welding defect detection module to obtain a target welding defect feature vector; a softmax function is used to map the values ​​of each dimension of the target welding defect feature vector to between 0 and 1 to obtain the detection probability of each welding defect; wherein each dimension corresponds to a welding defect; the greater the detection probability corresponding to the dimension, the greater the probability that the corresponding welding defect exists in the molten pool video to be detected; the welding defect with the highest detection probability is determined as the welding defect corresponding to the molten pool video to be detected.

[0091] The embodiment of the present application provides a method for detecting welding defects, the method comprising: obtaining multiple frames of molten pool images to be detected in a molten pool video to be detected; inputting each molten pool image to be detected into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected; inputting molten pool image features corresponding to all molten pool images to be detected into a welding defect detection module of a welding defect detection model to obtain welding defects corresponding to the molten pool video to be detected. The present application can detect welding defects through multiple frames of molten pool images in a molten pool video, thereby improving the accuracy of detecting welding defects.

[0092] Reference Figure 3 As shown, it is a flow chart of another welding defect detection method provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:

[0093] S301, obtaining multiple frames of molten pool sample images in a molten pool sample video, and actual welding defects corresponding to the molten pool sample video.

[0094] In the implementation mode of the present application, a molten pool camera is used to collect molten pool sample videos. Specifically, a Fronius welding machine is used, single pulse mode welding is adopted, and welding video data of each stage of base, filling and capping is collected. Solid welding wire is used, model GML-w56, base plate thickness is 16mm, and left and right rupture angles are 30°.

[0095] A total of 285 molten pool sample videos were collected in this experiment. Each sample is 1s long, including 20 to 30 frames of images per second. 10 frames of images in the video per second are taken as molten pool sample images. The sample videos include 90 molten pool sample videos of normal welding status, 90 molten pool sample videos of porosity defects, 90 molten pool sample videos of unfused defects, and 15 molten pool sample videos with abnormal swing (all as test sets).

[0096] Specifically, the python toolkit labelme is used to annotate the molten pool area, molten hole area and post-weld weld fusion line in the molten pool sample image; the annotated molten pool sample image is converted into an image mask form to obtain the final molten pool sample image; in the final molten pool sample image, the annotated area in the molten pool sample image is displayed using a first specific color (such as white), and the non-annotated area in the molten pool sample image is displayed using a second specific color (such as black).

[0097] S302, input each molten pool sample image into the feature extraction module in the welding defect detection model respectively, and obtain the fifth initial sample image feature, the first intermediate sample image feature, and the molten pool sample image feature corresponding to each molten pool image to be detected.

[0098] Specifically, the molten pool sample image is input into the feature extraction module in the welding defect detection model through the following steps to obtain the fifth initial sample image feature, the first intermediate sample image feature, and the molten pool sample image feature corresponding to the molten pool image to be detected:

[0099] Step 1: Input the melt pool sample image into the lightweight network unit in the feature extraction module to obtain the first initial sample image feature, the second initial sample image feature, the third initial sample image feature, the fourth initial sample image feature, and the fifth initial sample image feature corresponding to the melt pool sample image.

[0100] In an embodiment of the present application, before the melt pool sample image is input into the lightweight network unit in the feature extraction module, the size of the melt pool sample image is unified to 416*416*, and converted into an RGB three-channel image with a tensor shape of [3,416,416] before being input into the lightweight network unit.

[0101] Here, the outputs of different scales of the intermediate layers of the lightweight network unit are extracted, namely the initial first initial sample image features [320, 13, 13], the second initial sample image features [96, 26, 26], the initial third initial sample image features [32, 52, 52], the fourth initial sample image features [24, 104, 104], and the initial fifth initial sample image features [16, 208, 208]; the initial first initial sample image features, the initial third initial sample image features, and the initial fifth initial sample image features are respectively input into the CBAM (Convolutional Block Attention Module) unit to obtain the final first initial sample image features, the final third sample initial image features, and the final fifth sample initial image features.

[0102] Among them, the image size of the first initial sample image feature, the image size of the second initial sample image feature, the image size of the third initial sample image feature, the image size of the fourth initial sample image feature, and the image size of the fifth initial sample image feature gradually increase. In addition, the feature size of the fifth initial sample image feature is half of the image size of the melt pool sample image; the feature size of the fourth initial sample image feature is half of the feature size of the fifth initial sample image feature; the image size of the third initial sample image feature is half of the feature size of the fourth initial sample image feature; the image size of the second initial sample image feature is half of the feature size of the third initial sample image feature; and the image size of the first initial sample image feature is half of the feature size of the second initial sample image feature.

[0103] Step 2: For all initial sample image features corresponding to the melt pool sample image, feature splicing is performed in order from small to large image size to obtain the first intermediate sample image features corresponding to the melt pool sample image.

[0104] In an implementation manner of the present application, the implementation process of performing feature stitching on all initial sample image features corresponding to the melt pool sample image in the order of image size from small to large refers to the implementation process of "performing feature stitching on all initial image features corresponding to the melt pool image to be detected in the order of image size from small to large, to obtain the first intermediate image features corresponding to the melt pool image to be detected" in step S102.

[0105] Step 3: For all initial sample image features corresponding to the melt pool sample image, add them element by element in the order of image size from small to large to obtain the second intermediate sample image features corresponding to the melt pool sample image.

[0106] In an implementation manner of the present application, for all initial sample image features corresponding to the melt pool sample image, element-by-element addition is performed in order of image size from small to large, with reference to the implementation process of "for all initial image features corresponding to the melt pool image to be detected, element-by-element addition is performed in order of image size from small to large to obtain the second intermediate image features corresponding to the melt pool image to be detected" in step S102.

[0107] Step 4: Fuse the first intermediate sample image features and the second intermediate sample image features corresponding to the molten pool sample image to obtain the molten pool sample image features corresponding to the molten pool sample image.

[0108] In the implementation manner of the present application, the implementation process of fusing the first intermediate sample image features and the second intermediate sample image features corresponding to the melt pool sample image refers to the implementation process of "fusing the first intermediate image features and the second intermediate image features corresponding to the melt pool image to be detected to obtain the melt pool image features corresponding to the melt pool image to be detected" in step S102.

[0109] S303: Obtain target training features according to the fifth initial sample image features and the first intermediate sample image features.

[0110] In the implementation of the present application, the decoder part consists of two convolutional layers and a depth-separable convolution, compressing the dimension to n_seg_class (the number of segmentation categories, 4 in the present invention), the scale size is the input image size, and the output is out_seg. The calculation process is as follows:

[0111] Feature_1=Conv_1(E).

[0112] Feature_2 = up(U, (H, W))

[0113] Out_seg=Depconv(conv_2(cat(Feature_1, Feature_2)))

[0114] Among them, E is the fifth initial sample image feature; (H, W) is the size of the fifth initial sample image feature, U is the first intermediate sample image feature, Conv_1 is the first convolution layer, conv_2 is the second convolution layer, up is upsampling, Depconv is depth-separable convolution, and Out_seg is the target training feature.

[0115] S304. Input the molten pool sample image features corresponding to all molten pool sample images into the welding defect detection module of the welding defect detection model to obtain the welding defects corresponding to the molten pool sample videos.

[0116] In an embodiment of the present application, the molten pool sample image features corresponding to all molten pool sample images are input into a long short-term memory (LSTM) network to obtain an initial welding defect sample feature vector; the initial welding defect sample feature vector is input into a fully connected layer in a welding defect detection module to obtain a target welding defect sample feature vector; a softmax function is used to map the values ​​of each dimension of the target welding defect sample feature vector to between 0 and 1 to obtain the detection probability of each welding defect; wherein each dimension corresponds to a welding defect; the greater the detection probability corresponding to the dimension, the greater the probability that the corresponding welding defect exists in the molten pool sample video; the welding defect with the highest detection probability is determined as the welding defect corresponding to the molten pool sample video.

[0117] S305, training a welding defect detection model according to target training features corresponding to all molten pool sample images, actual welding defects corresponding to molten pool sample videos, and detected welding defects.

[0118] like Figure 4 As shown, it is a structural schematic diagram of the welding defect detection model provided in an embodiment of the present application; the model extracts image features through MobileNetV2, uses CBAM (Convolutional Block Attention Module) to extract more detailed features on feature maps of different scales, and designs upsampling layers and convolution layers to further process features, and uses the Encoder part of the improved U-Net network to segment the molten pool area, and uses LSTM to model the temporal relationship between multiple frame images to realize the detection of welding defects.

[0119] An embodiment of the present application provides another method for detecting welding defects, which can train a welding defect detection model and improve the detection accuracy of welding defects.

[0120] Based on the same inventive concept, the embodiment of the present application also provides a welding defect detection device corresponding to the welding defect detection method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned welding defect detection method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0121] Reference Figure 5 FIG. 1 is a schematic diagram of a welding defect detection device provided in an embodiment of the present application, and the welding defect detection device includes:

[0122] An acquisition module 501 is used to acquire multiple frames of images of a molten pool to be detected in a video of the molten pool to be detected;

[0123] An input module 502 is used to input each molten pool image to be detected into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected;

[0124] The input module 502 is also used to input the molten pool image features corresponding to all the molten pool images to be detected into the welding defect detection module of the welding defect detection model to obtain the welding defects corresponding to the molten pool videos to be detected.

[0125] In a possible implementation, the input module 502 is specifically used to input the molten pool image to be detected into the lightweight network unit in the feature extraction module, respectively, to obtain multiple initial image features corresponding to the molten pool image to be detected; for all the initial image features corresponding to the molten pool image to be detected, feature splicing is performed in the order of image size from small to large, to obtain the first intermediate image feature corresponding to the molten pool image to be detected; for all the initial image features corresponding to the molten pool image to be detected, element-by-element addition is performed in the order of image size from small to large, to obtain the second intermediate image feature corresponding to the molten pool image to be detected; the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected are merged to obtain the molten pool image feature corresponding to the molten pool image to be detected.

[0126] In a possible implementation, the initial image features include a first initial image feature, a second initial image feature, a third initial image feature, and a fourth initial image feature; the image size of the first initial image feature, the image size of the second initial image feature, the image size of the third initial image feature, and the image size of the fourth initial image feature gradually increase.

[0127] In a possible implementation, the input module 502 is further configured to:

[0128] Determine the first initial image feature corresponding to the molten pool image to be detected as the current convolution image feature;

[0129] The initial image features corresponding to the molten pool image to be detected are traversed in order from small to large image sizes;

[0130] The image features obtained by upsampling the current convolution image features and the initial image features of the next traversal are concatenated to obtain concatenated image features;

[0131] Performing depth-separable convolution on the spliced ​​image features to obtain the latest convolution image features; wherein the feature scale of the latest convolution image features is the same as the feature scale of the next traversed initial image features;

[0132] If the currently traversed initial image feature is the third initial image feature, the latest convolution image feature is determined as the first intermediate image feature; otherwise, the traversal continues.

[0133] In a possible implementation, the input module 502 is further configured to:

[0134] Adding the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element, obtaining the first molten pool image feature;

[0135] Multiplying the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element, obtaining a second molten pool image feature;

[0136] The first molten pool image feature and the second molten pool image feature corresponding to the molten pool image to be detected are feature spliced ​​to obtain a final molten pool image feature corresponding to the molten pool image to be detected.

[0137] In a possible embodiment, the device also includes a training module 503; the training module 503 is specifically used to obtain multiple frames of molten pool sample images in the molten pool sample video, and the actual welding defects corresponding to the molten pool sample video; each molten pool sample image is input into the feature extraction module in the welding defect detection model respectively, and the fifth initial sample image feature, the first intermediate sample image feature, and the molten pool sample image feature corresponding to each molten pool image to be detected are obtained; according to the fifth initial sample image feature and the first intermediate sample image feature, the target training feature is obtained; the molten pool sample image features corresponding to all molten pool sample images are input into the welding defect detection module of the welding defect detection model, and the detected welding defects corresponding to the molten pool sample video are obtained; the welding defect detection model is trained according to the target training features corresponding to all molten pool sample images, the actual welding defects corresponding to the molten pool sample video, and the detected welding defects.

[0138] The embodiment of the present application provides a welding defect detection device, which includes: an acquisition module 501, which is used to acquire multiple frames of molten pool images to be detected in the molten pool video to be detected; an input module 502, which is used to input each molten pool image to be detected into the feature extraction module of the welding defect detection model, respectively, to obtain the molten pool image features corresponding to each molten pool image to be detected; the input module 502 is also used to input the molten pool image features corresponding to all molten pool images to be detected into the welding defect detection module of the welding defect detection model, to obtain the welding defects corresponding to the molten pool video to be detected. The present application can detect welding defects through multiple frames of molten pool images in the molten pool video, thereby improving the accuracy of detecting welding defects.

[0139] like Figure 6As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the above-mentioned welding defect detection method.

[0140] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memory and processor, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, the above-mentioned welding defect detection method can be executed.

[0141] Corresponding to the above-mentioned welding defect detection method, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned welding defect detection method are executed.

[0142] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0143] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0145] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the information processing method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0146] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting welding defects, characterized in that: The method comprises: Acquire multiple frames of images of the molten pool to be detected in the molten pool video to be detected; Input each molten pool image to be detected into the feature extraction module of the welding defect detection model respectively to obtain the molten pool image features corresponding to each molten pool image to be detected; The molten pool image features corresponding to all the molten pool images to be detected are input into the welding defect detection module of the welding defect detection model to obtain the welding defects corresponding to the molten pool videos to be detected.

2. The method for detecting welding defects according to claim 1, characterized in that: The steps of inputting each molten pool image to be detected into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected include: Inputting the to-be-detected molten pool images into the lightweight network units in the feature extraction module respectively to obtain a plurality of initial image features corresponding to the to-be-detected molten pool images; For all initial image features corresponding to the molten pool image to be detected, feature stitching is performed in ascending order of image size to obtain a first intermediate image feature corresponding to the molten pool image to be detected; For all initial image features corresponding to the molten pool image to be detected, adding them element by element in ascending order of image size to obtain a second intermediate image feature corresponding to the molten pool image to be detected; The first intermediate image features and the second intermediate image features corresponding to the molten pool image to be detected are fused to obtain the molten pool image features corresponding to the molten pool image to be detected.

3. The method for detecting welding defects according to claim 2, characterized in that: The initial image features include a first initial image feature, a second initial image feature, a third initial image feature, and a fourth initial image feature; The image size of the first initial image feature, the image size of the second initial image feature, the image size of the third initial image feature, and the image size of the fourth initial image feature gradually increase.

4. The method for detecting welding defects according to claim 3, characterized in that: The step of performing feature stitching on all initial image features corresponding to the molten pool image to be detected in order of image size from small to large to obtain a first intermediate image feature corresponding to the molten pool image to be detected includes: Determine the first initial image feature corresponding to the molten pool image to be detected as the current convolution image feature; The initial image features corresponding to the molten pool image to be detected are traversed in order from small to large image sizes; The image features obtained by upsampling the current convolution image features and the initial image features of the next traversal are concatenated to obtain concatenated image features; Performing depth-separable convolution on the spliced ​​image features to obtain the latest convolution image features; wherein the feature scale of the latest convolution image features is the same as the feature scale of the initial image features of the next traversal; If the currently traversed initial image feature is the third initial image feature, the latest convolution image feature is determined as the first intermediate image feature; otherwise, the traversal continues.

5. The method for detecting welding defects according to claim 2, characterized in that: The step of fusing the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected to obtain the molten pool image feature corresponding to the molten pool image to be detected includes: Adding the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element, obtaining a first molten pool image feature; Multiplying the first intermediate image feature and the second intermediate image feature corresponding to the molten pool image to be detected element by element to obtain a second molten pool image feature; The first molten pool image feature and the second molten pool image feature corresponding to the molten pool image to be detected are feature spliced ​​to obtain a final molten pool image feature corresponding to the molten pool image to be detected.

6. The method for detecting welding defects according to any one of claims 2 to 5, characterized in that: The welding defect detection model is trained by the following steps: Acquire multiple frames of molten pool sample images in a molten pool sample video, and actual welding defects corresponding to the molten pool sample video; Inputting each molten pool sample image into the feature extraction module in the welding defect detection model respectively, and obtaining the fifth initial sample image feature, the first intermediate sample image feature, and the molten pool sample image feature corresponding to each molten pool image to be detected; Obtaining a target training feature according to the fifth initial sample image feature and the first intermediate sample image feature; Inputting the molten pool sample image features corresponding to all molten pool sample images into the welding defect detection module of the welding defect detection model to obtain the detected welding defects corresponding to the molten pool sample videos; The welding defect detection model is trained according to the target training features corresponding to all molten pool sample images, the actual welding defects corresponding to the molten pool sample videos, and the detected welding defects.

7. A welding defect detection device, characterized in that: The device comprises: An acquisition module is used to acquire multiple frames of images of the molten pool to be detected in the molten pool video to be detected; An input module, used to input each molten pool image to be detected into a feature extraction module of a welding defect detection model to obtain molten pool image features corresponding to each molten pool image to be detected; The input module is also used to input the molten pool image features corresponding to all the molten pool images to be detected into the welding defect detection module of the welding defect detection model to obtain the welding defects corresponding to the molten pool videos to be detected.

8. The welding defect detection device according to claim 7, characterized in that: The input module is specifically used for: Inputting the to-be-detected molten pool images into the lightweight network units in the feature extraction module respectively to obtain a plurality of initial image features corresponding to the to-be-detected molten pool images; For all initial image features corresponding to the molten pool image to be detected, feature stitching is performed in ascending order of image size to obtain a first intermediate image feature corresponding to the molten pool image to be detected; For all initial image features corresponding to the molten pool image to be detected, adding them element by element in ascending order of image size to obtain a second intermediate image feature corresponding to the molten pool image to be detected; The first intermediate image features and the second intermediate image features corresponding to the molten pool image to be detected are fused to obtain the molten pool image features corresponding to the molten pool image to be detected.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the welding defect detection method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the welding defect detection method according to any one of claims 1 to 6 are executed.

Images