An image processing method for automatic welding of transmission tower components

Abstract

The present invention discloses an image processing method for automatic welding of transmission tower components, which belongs to the technical field of tower production. The processing method comprises: obtaining an initial image captured during the automatic welding process, performing channel expansion on the initial image to obtain a first-order feature matrix; using multiple feature generation modules to process the first-order feature matrix to obtain a second-order feature matrix; inputting the second-order feature matrix into an image reconstruction module, and outputting an improved image. The present invention creatively designs a semi-dense inner cross feature extraction mechanism in the first subunit, which can better identify and remove other noise interference while retaining key image information such as welds. The second subunit effectively models the importance of each image area through its internal view transformation calibration mechanism, integrates the first result with the second result, and enables the feature generation module to have a more accurate and robust learning ability for various information.

Description

Technical Field

[0001] The invention belongs to the technical field of iron tower production, and in particular relates to an image processing method for automatic welding of transmission iron tower components. Background Art

[0002] During the production process of transmission towers, a lot of welding work is required, for example, Figure 1 The figure shows one section of a steel pipe tower. During fabrication, flanges are welded to both ends of the steel pipe, and multiple triangular ribs are welded between the pipe and the flange. Traditional manual welding methods suffer from low efficiency, high workload, and uncontrollable welding quality. Therefore, existing technologies are moving towards automation and intelligentization.

[0003] In automated welding systems, optical cameras capture on-site images, then use built-in algorithms to identify and locate welds. These systems also monitor and evaluate welding quality in real time, automatically identifying welding defects. However, in actual welding operations and quality assessments, various factors often interfere, causing misleading information in the captured images and leading to unstable automated identification and positioning accuracy. Examples include poor lighting conditions at the work site, smoke generated during welding that affects vision, and misleading dirt and scratches on component surfaces. Summary of the Invention

[0004] In view of this, the present invention provides an image processing method for automatic welding operations of transmission tower components, so as to improve the recognition, positioning and evaluation accuracy during the automated welding operation.

[0005] The technical solution is as follows: The present invention provides an image processing method for automatic welding of transmission tower components, comprising the following steps:

[0006] Acquire an initial image captured during the automatic welding process, perform channel expansion on the initial image, and obtain a first-order feature matrix;

[0007] After processing the first-order feature matrix using a plurality of feature generation modules arranged in series, a second-order feature matrix is obtained;

[0008] Inputting the second-order characteristic matrix into an image reconstruction module, and outputting the image reconstruction module to obtain an improved image;

[0009] The feature generation module is provided with a first subunit and a second subunit. After performing a feature extraction operation based on the first subunit, a first result is obtained. After performing a feature extraction operation based on the second subunit, a second result is obtained. The first result and the second result are integrated to obtain an inner-order feature matrix output by the feature generation module.

[0010] Furthermore, the first subunit in the first feature generation module takes the first-order feature matrix as input, and the first subunit in the subsequent feature generation module takes the inner-order feature matrix output by its upstream feature generation module as input.

[0011] Furthermore, the internal calculation process of the first subunit includes:

[0012] After processing the feature matrix input to the first sub-unit based on the first extractor and the first selector respectively, a first feature map and a second feature map are obtained;

[0013] Subtract the first characteristic map from the second characteristic map (subtract the first characteristic map from the difference of the second characteristic map) to obtain a third characteristic map;

[0014] Performing a Hadamard product on the first feature map and the second feature map to obtain a fourth feature map;

[0015] Inputting the first feature map, the third feature map, and the fourth feature map into a second extractor, and after calculation, the second extractor outputting a fifth feature map;

[0016] Inputting the second feature map, the third feature map, and the fourth feature map into a second selector, and after calculation, the second selector outputs a sixth feature map;

[0017] The fifth feature map and the sixth feature map are integrated to obtain the first result.

[0018] Furthermore, the first extractor and the second extractor each include a convolution operation layer and a first excitation layer arranged in sequence;

[0019] Or / and, the first selector and the second selector both include a deformable convolution layer and a second excitation layer arranged in sequence.

[0020] Furthermore, the second subunit takes the third feature map and the fourth feature map as input.

[0021] Furthermore, the internal calculation process of the second subunit includes:

[0022] Performing global average pooling processing on the third feature map in the channel dimension to obtain a first data set;

[0023] performing expansion processing on the first data set to obtain a second data set;

[0024] The second data set is sequentially subjected to the first convolution and the first activation processes to obtain a third data set;

[0025] Performing global maximum pooling processing on the fourth feature map in the spatial dimension to obtain a fourth data set;

[0026] The fourth data set is integrated with the third data set to obtain the second result.

[0027] Further, integrating the first result and the second result to obtain an inner-order feature matrix output by the feature generation module includes: broadcasting the second result along the spatial dimension to obtain a third result, and then multiplying the third result by the first result element-by-element to obtain the inner-order feature matrix output by the feature generation module;

[0028] or / and, integrating the fifth feature map with the sixth feature map to obtain the first result, including: concatenating the fifth feature map with the sixth feature map to obtain a seventh feature map, and then sequentially subjecting the seventh feature map to a second convolution and a second activation process to obtain the first result;

[0029] Or / and, integrating the fourth data set with the third data set to obtain the second result, including: adding the corresponding elements of the fourth data set and the third data set to obtain a fifth data set, and the fifth data set is sequentially subjected to a third convolution and a third activation processing to obtain the second result.

[0030] Furthermore, before inputting the second-order characteristic matrix into the image reconstruction module, the second-order characteristic matrix is first processed using the adaptation feature map, and then the second-order characteristic matrix processed by the adaptation feature map is input into the image reconstruction module.

[0031] Furthermore, the adaptation module takes the first data set generated in each second subunit as input, and after calculation, the adaptation module generates the adaptation feature map; the adaptation feature map completes the processing of the second-order feature matrix by multiplying the second-order feature matrix element by element.

[0032] Furthermore, the calculation process within the adaptation module includes:

[0033] splicing all first data sets together to obtain a first adaptation matrix;

[0034] The first adaptation matrix is sequentially subjected to a fourth convolution and a fourth activation process to obtain a second adaptation matrix;

[0035] The second adaptation matrix is sequentially processed by the PixelShuffle layer and the fifth activation process to obtain a third adaptation matrix;

[0036] The third adaptation matrix is broadcasted along the channel dimension to obtain the adaptation feature map.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) Actual tests show that the conventional encoder-decoder network architecture cannot cope well with the image optimization task in the automatic welding operation scenario. Some existing technologies filter too many details of the image after processing, eliminating interference factors such as smoke and dirt, but also weaken the visual performance of the weld, further deteriorating the recognition and positioning effect. Some existing technologies have limited effect on eliminating interference factors such as smoke and dirt in the image, resulting in the processed image also having the problem of poor recognition and positioning effect, which is difficult to meet the needs of actual production. The present invention creatively designs a semi-dense inner cross feature extraction mechanism in the first subunit. Through the cooperation of two extractors and two selectors, the first subunit has a feature extraction effect that is more compatible with the image optimization target in the automatic welding operation scenario. While retaining key image information such as the weld, it can better identify and remove other noise interference.

[0039] (2) In order to improve the feature generation module's ability to identify partitions of large-scale image information when extracting features, the second subunit of the present invention takes the third feature map and the fourth feature map (containing a large amount of cross-regional information) as input, and effectively models the importance of each image region through its internal view transformation calibration mechanism, integrating the first result with the second result, so that the feature generation module has a more accurate and robust learning ability for various information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a structural diagram of one section of a steel pipe tower.

[0041] Figure 2 FIG. 4 is a schematic diagram of an image processing flow according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in further detail below with reference to the accompanying drawings.

[0043] This embodiment provides an image processing method for automatic welding of transmission tower components. Figure 2 As shown, the following steps are included:

[0044] S100: Acquire an initial image captured during the automatic welding process, perform channel expansion on the initial image, and obtain a first-order feature matrix. Specifically, in this embodiment, a convolution operation is performed on the initial image using a starting convolution layer to obtain a first-order feature matrix. The number of channels in the first-order feature matrix can be set according to actual conditions. In this embodiment, the number of channels in the first-order feature matrix is 48. The convolution kernel size and step size of the starting convolution layer can be set according to actual conditions. In this embodiment, the convolution kernel size of the starting convolution layer is 3*3, and the step size is 1.

[0045] S200: Then, four feature generation modules arranged in series are used to process the first-order feature matrix to obtain a second-order feature matrix. The first feature generation module uses the first-order feature matrix as input, and each subsequent feature generation module uses the inner-order feature matrix output by the previous feature generation module as input, until the inner-order feature matrix output by the last feature generation module becomes the second-order feature matrix.

[0046] S300: Input the second-order feature matrix into the image reconstruction module. After calculation, the image reconstruction module outputs an improved image. The image reconstruction module can be implemented using existing conventional technologies. Specifically, in this embodiment, the image reconstruction module includes a fifth convolutional layer, a sixth activation layer, and a sixth convolutional layer connected in sequence. The convolution kernel size of the fifth and sixth convolutional layers is 3*3, and the stride size is 1. The sixth activation layer uses the ReLU function. Finally, the sixth convolutional layer outputs an improved image with 3 channels.

[0047] Furthermore, the feature generation module includes a first subunit and a second subunit. After performing a feature extraction operation based on the first subunit, a first result is obtained. After performing a feature extraction operation based on the second subunit, a second result is obtained. The first result and the second result are integrated to obtain an inner-order feature matrix output by the feature generation module. The size of the inner-order feature matrix output by each feature generation module is equal to the size of the first-order feature matrix.

[0048] Within the first subunit, the first extractor and the second extractor both include a convolution operation layer and a first excitation layer arranged in sequence. In this embodiment, the convolution operation layer has a convolution kernel size of 5*5, a step size of 1, and the first excitation layer is a ReLU function. The first selector and the second selector both include a deformable convolution layer and a second excitation layer arranged in sequence. The convolution kernel size of the deformable convolution layer is 3*3, the step size is 1, and the second excitation layer is a PReLU function. The sizes of the first feature map, the second feature map, the third feature map, the fourth feature map, the fifth feature map, and the sixth feature map generated within all first subunits are equal to the size of the first-order feature matrix. The convolution kernel size of the second convolution is 3*3, the step size is 1, the second activation uses the ReLU function, and the size of the first result is also equal to the size of the first-order feature matrix.

[0049] The second sub-unit takes the third feature map and the fourth feature map generated by the first sub-unit in the same feature generation module as input. The internal calculation process of the second sub-unit includes:

[0050] Performing global average pooling processing on the third feature map in the channel dimension, compressing the number of channels of the third feature map to 1, and obtaining a first data set;

[0051] Expand the first data set (the first data set can be expanded by processing it with the flatten layer) to obtain a second data set, which is a one-dimensional vector;

[0052] The second data set is sequentially processed by the first convolution and the first activation to obtain the third data set; wherein the convolution kernel size of the first convolution is 1*1, and the first activation is the sigmoid function;

[0053] Performing global maximum pooling processing on the fourth feature map in the spatial dimension to obtain a fourth data set, where the fourth data set is a one-dimensional vector and has the same length as the third data set;

[0054] The fourth data set is integrated with the third data set to obtain a second result.

[0055] The above-mentioned integration of the fourth data set and the third data set to obtain the second result includes: adding the corresponding elements of the fourth data set and the third data set to obtain a fifth data set, and the fifth data set is sequentially processed by the third convolution (convolution kernel size is 1*1) and the third activation (sigmoid function) to obtain the second result.

[0056] The way to integrate the first result and the second result is the existing conventional technology. The height and width dimensions of the second result are both 1, and its channel dimension is equal to the channel dimension of the first result. The second result is broadcast along the spatial dimension to obtain a third result, so that the height, width and channel dimensions of the third result are all equal to the height, width and channel dimensions of the first result. Since it is a broadcast operation, in the third result, all eigenvalues within the same channel are obtained by copying the eigenvalues of the corresponding channel position of the second result. The third result is then multiplied element by element with the corresponding element in the first result to obtain the inner-order feature matrix output by the feature generation module.

[0057] In some other embodiments, the first data set generated by each second subunit is also input into the adaptation module, which generates an adapted feature map after calculation. The size of the adapted feature map is equal to the size of the second-order feature matrix. The adapted feature map is multiplied element-by-element by the second-order feature matrix to complete the processing of the second-order feature matrix. The second-order feature matrix processed by the adapted feature map is then input into the image reconstruction module.

[0058] Specifically, the calculation process within the adaptation module may include:

[0059] splicing all first data sets together to obtain a first adaptation matrix;

[0060] The first adaptation matrix is sequentially processed by the fourth convolution and the fourth activation to obtain a second adaptation matrix; as an example but not limiting, the convolution kernel size of the fourth convolution is 3*3, the stride is 2, the fourth activation uses the ReLU function, the height and width of the second adaptation matrix are both half of the second-order feature matrix, and the number of channels is 4;

[0061] The second adaptation matrix is processed by the PixelShuffle layer and the fifth activation (logistics function) to obtain the third adaptation matrix. The height and width of the third adaptation matrix are equal to the height and width of the second-order feature matrix, and the number of channels of the third adaptation matrix is 1;

[0062] The third adaptation matrix is broadcast along the channel dimension so that the height, width and channel size of the matrix are equal to those of the second feature matrix, thereby obtaining an adapted feature map.

[0063] The adaptation module first uses the fourth convolution to reduce the dimensionality of the first adaptation matrix, and then expands the feature information in the spatial direction through the PixelShuffle layer. The adaptation module cooperates with the second subunit to realize cross-view and cross-level joint learning of feature information, so that the method provided by the present invention can better handle the situation where multiple complex interference information is mixed and embedded in the image at the same time, and the applicable scenarios are more diversified.

[0064] In specific implementation, the starting convolution layer, all feature generation modules and image generation modules together constitute a complete convolutional neural network. The convolutional neural network is first trained using a training set, and then the trained starting convolution layer, feature generation module and image generation module are used to perform the above-mentioned image processing.

[0065] Finally, it should be noted that the above description is only a preferred embodiment of the present invention. Under the guidance of the present invention, ordinary technicians in this field can make various similar expressions without violating the purpose and claims of the present invention. Such changes fall within the scope of protection of the present invention.

Claims

1. An image processing method for automatic welding of transmission tower components, characterized in that: The following steps are involved: Acquire an initial image captured during the automatic welding process, perform channel expansion on the initial image, and obtain a first-order feature matrix; After processing the first-order feature matrix using a plurality of feature generation modules arranged in series, a second-order feature matrix is obtained; Inputting the second-order characteristic matrix into an image reconstruction module, and outputting the image reconstruction module to obtain an improved image; The feature generation module is provided with a first subunit and a second subunit. After performing a feature extraction operation based on the first subunit, a first result is obtained. After performing a feature extraction operation based on the second subunit, a second result is obtained. The first result and the second result are integrated to obtain an inner-order feature matrix output by the feature generation module. The internal calculation process of the first subunit includes: After processing the feature matrix input to the first sub-unit based on the first extractor and the first selector respectively, a first feature map and a second feature map are obtained; Subtracting the first characteristic map from the second characteristic map to obtain a third characteristic map; Performing a Hadamard product on the first feature map and the second feature map to obtain a fourth feature map; Inputting the first feature map, the third feature map, and the fourth feature map into a second extractor, and after calculation, the second extractor outputting a fifth feature map; Inputting the second feature map, the third feature map, and the fourth feature map into a second selector, and after calculation, the second selector outputs a sixth feature map; Integrating the fifth feature map and the sixth feature map to obtain the first result; The second subunit takes the third feature map and the fourth feature map as input, and the internal calculation process of the second subunit includes: Performing global average pooling processing on the third feature map in the channel dimension to obtain a first data set; performing expansion processing on the first data set to obtain a second data set; The second data set is sequentially subjected to the first convolution and the first activation processes to obtain a third data set; Performing global maximum pooling processing on the fourth feature map in the spatial dimension to obtain a fourth data set; Integrating the fourth data set with the third data set to obtain the second result; Integrating the first result with the second result to obtain an inner-order feature matrix output by the feature generation module includes: broadcasting the second result along the spatial dimension to obtain a third result, and then multiplying the third result by the first result element-by-element to obtain the inner-order feature matrix output by the feature generation module.

2. The image processing method for automatic welding of transmission tower components according to claim 1, characterized in that: The first subunit in the first feature generation module takes the first-order feature matrix as input, and the first subunit in the subsequent feature generation modules takes the inner-order feature matrix output by its upstream feature generation module as input.

3. The image processing method for automatic welding of transmission tower components according to claim 2, characterized in that: The first extractor and the second extractor each include a convolution operation layer and a first excitation layer arranged in sequence; Or / and, the first selector and the second selector both include a deformable convolution layer and a second excitation layer arranged in sequence.

4. The image processing method for automatic welding of transmission tower components according to claim 3, characterized in that: Integrating the fifth feature map with the sixth feature map to obtain the first result, including: concatenating the fifth feature map with the sixth feature map to obtain a seventh feature map, and then sequentially subjecting the seventh feature map to a second convolution and a second activation process to obtain the first result; Or / and, integrating the fourth data set with the third data set to obtain the second result, including: adding the corresponding elements of the fourth data set and the third data set to obtain a fifth data set, and the fifth data set is sequentially subjected to a third convolution and a third activation processing to obtain the second result.

5. The image processing method for automatic welding of transmission tower components according to claim 4, characterized in that: Before the second-order characteristic matrix is input into the image reconstruction module, the second-order characteristic matrix is first processed using the adaptation feature map, and then the second-order characteristic matrix processed by the adaptation feature map is input into the image reconstruction module.

6. The image processing method for automatic welding of transmission tower components according to claim 5, characterized in that: The adaptation module takes the first data set generated in each second subunit as input, and generates the adaptation feature map after calculation; the adaptation feature map completes the processing of the second-order feature matrix by multiplying the second-order feature matrix element by element.

7. The image processing method for automatic welding of transmission tower components according to claim 6, characterized in that: The internal calculation process of the adaptation module includes: splicing all first data sets together to obtain a first adaptation matrix; The first adaptation matrix is sequentially subjected to a fourth convolution and a fourth activation process to obtain a second adaptation matrix; The second adaptation matrix is sequentially processed through the PixelShuffle layer and the fifth activation process to obtain a third adaptation matrix; the third adaptation matrix is broadcasted along the channel dimension to obtain the adaptation feature map.

Images