Image processing method for automatic welding operation of power transmission tower parts

Abstract

The invention discloses an image processing method for automatic welding operation of power transmission tower parts, and belongs to the technical field of tower production. The processing method comprises the steps that an initial image shot in the automatic welding process is obtained, channel expansion is conducted on the initial image, and a first-order feature matrix is obtained; processing the first-order feature matrix by using a plurality of feature generation modules to obtain a second-order feature matrix; and inputting the second-order feature matrix into an image reconstruction module, and outputting to obtain an improved image. According to the method, a semi-dense internal cross feature extraction mechanism is creatively designed in the first subunit, and other noise interference can be better identified and removed while key image information such as a welding seam is reserved. And the second subunit performs effective modeling on the importance of each image region through an internal view transformation calibration mechanism, and integrates the first result and the second result, so that the feature generation module has more accurate and more stable learning ability for various information.

Description

Technical Field

[0001] The invention belongs to the technical field of iron tower production, and in particular relates to an image processing method for automatic welding of transmission iron tower components. Background Art

[0002] During the production process of transmission towers, a lot of welding work is required, for example, Figure 1 The figure shows one section of a steel pipe tower. During processing, flanges need to be welded at both ends of the steel pipe, and multiple triangular ribs need to be welded between the steel pipe and the flange. The traditional method of manual welding has the disadvantages of low efficiency, high work intensity, and uncontrollable welding quality. Therefore, existing technologies are developing towards automation and intelligence.

[0003] In the automatic welding system, the optical camera is used to collect the on-site images, and then the weld is identified and located through the system's built-in algorithm. In addition, the welding quality can be monitored and evaluated in real time, and welding defects can be automatically identified. However, in the actual welding operation and quality evaluation process, there are often many factors that interfere, resulting in misleading information in the collected images, leading to unstable automatic identification and positioning accuracy. For example, the lighting conditions at the work site are poor, the smoke generated during the welding process affects the line of sight, and the dirt and scratches on the surface of the parts are misleading. Summary of the invention

[0004] In view of this, the present invention provides an image processing method for automatic welding of transmission tower components, so as to improve the recognition, positioning and evaluation accuracy during the automatic welding operation.

[0005] The technical solution is as follows: The present invention provides an image processing method for automatic welding of transmission tower components, comprising the following steps:

[0006] Acquire an initial image captured during the automatic welding process, perform channel expansion on the initial image, and obtain a first-order feature matrix;

[0007] After processing the first-order feature matrix using a plurality of feature generation modules arranged in series, a second-order feature matrix is ​​obtained;

[0008] Inputting the second-order characteristic matrix into an image reconstruction module, the image reconstruction module outputs an improved image;

[0009] The feature generation module is provided with a first subunit and a second subunit. After performing a feature extraction operation based on the first subunit, a first result is obtained. After performing a feature extraction operation based on the second subunit, a second result is obtained. The first result and the second result are integrated to obtain an inner-order feature matrix output by the feature generation module.

[0010] Furthermore, the first subunit in the first feature generation module takes the first-order feature matrix as input, and the first subunit in the subsequent feature generation module takes the inner-order feature matrix output by its upstream feature generation module as input.

[0011] Furthermore, the internal calculation process of the first subunit includes:

[0012] After processing the feature matrix input to the first sub-unit based on the first extractor and the first selector respectively, a first feature map and a second feature map are obtained;

[0013] Subtract the first characteristic map from the second characteristic map (subtract the first characteristic map from the difference of the second characteristic map) to obtain a third characteristic map;

[0014] Performing a Hadamard product of the first feature map and the second feature map to obtain a fourth feature map;

[0015] Inputting the first feature map, the third feature map and the fourth feature map into a second extractor, and after calculation, the second extractor outputs a fifth feature map;

[0016] The second feature map, the third feature map and the fourth feature map are input into a second selector, and after calculation, the second selector outputs a sixth feature map;

[0017] The fifth feature map is integrated with the sixth feature map to obtain the first result.

[0018] Furthermore, the first extractor and the second extractor each include a convolution operation layer and a first excitation layer arranged in sequence;

[0019] Or / and, the first selector and the second selector both include a deformable convolution layer and a second excitation layer arranged in sequence.

[0020] Furthermore, the second subunit takes the third feature map and the fourth feature map as input.

[0021] Furthermore, the internal calculation process of the second subunit includes:

[0022] Performing global average pooling processing on the third feature map in the channel dimension to obtain a first data set;

[0023] Performing expansion processing on the first data set to obtain a second data set;

[0024] The second data set is sequentially subjected to the first convolution and the first activation processes to obtain a third data set;

[0025] Performing global maximum pooling processing on the fourth feature map in the spatial dimension to obtain a fourth data set;

[0026] The fourth data set is integrated with the third data set to obtain the second result.

[0027] Further, integrating the first result with the second result to obtain an inner-order feature matrix output by the feature generation module includes: broadcasting the second result along the spatial dimension to obtain a third result, and then multiplying the third result by the first result element by element to obtain the inner-order feature matrix output by the feature generation module;

[0028] or / and, integrating the fifth feature map with the sixth feature map to obtain the first result, including: concatenating the fifth feature map with the sixth feature map to obtain a seventh feature map, and then sequentially subjecting the seventh feature map to a second convolution and a second activation process to obtain the first result;

[0029] Or / and, integrating the fourth data set with the third data set to obtain the second result, including: adding corresponding elements of the fourth data set and the third data set to obtain a fifth data set, and the fifth data set is successively subjected to a third convolution and a third activation process to obtain the second result.

[0030] Furthermore, before the second-order feature matrix is ​​input into the image reconstruction module, the second-order feature matrix is ​​first processed using an adaptation feature map, and then the second-order feature matrix processed by the adaptation feature map is input into the image reconstruction module.

[0031] Furthermore, the adaptation module takes the first data set generated in each second subunit as input, and after calculation, the adaptation module generates the adaptation feature map; the adaptation feature map completes the processing of the second-order feature matrix by multiplying the second-order feature matrix element by element.

[0032] Furthermore, the calculation process inside the adaptation module includes:

[0033] All the first data sets are concatenated to obtain a first adaptation matrix;

[0034] The first adaptation matrix is ​​sequentially subjected to a fourth convolution and a fourth activation process to obtain a second adaptation matrix;

[0035] The second adaptation matrix is ​​processed by the PixelShuffle layer and the fifth activation in sequence to obtain a third adaptation matrix;

[0036] The third adaptation matrix is ​​broadcasted along the channel dimension to obtain the adaptation feature map.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) Actual tests show that the conventional encoder-decoder network architecture cannot cope well with the image optimization task in the automatic welding operation scenario. Some existing technologies filter too many details of the processed images, eliminating interference factors such as smoke and dirt, but also weaken the visual performance of the weld, further deteriorating the recognition and positioning effects. Some existing technologies have limited effects on eliminating interference factors such as smoke and dirt in the image, resulting in the processed images also having poor recognition and positioning effects, which is difficult to meet the needs of actual production. The present invention creatively designs a semi-dense inner cross feature extraction mechanism in the first subunit. Through the cooperation of two extractors and two selectors, the first subunit has a feature extraction effect that is more matched with the image optimization target in the automatic welding operation scenario. While retaining key image information such as welds, it can better identify and remove other noise interference.

[0039] (2) In order to improve the feature generation module's ability to identify partitions of a wide range of image information when extracting features, the second subunit of the present invention takes the third feature map (containing a large amount of cross-regional information) and the fourth feature map as input, and effectively models the importance of each image region through its internal view transformation calibration mechanism, and integrates the first result with the second result, so that the feature generation module has a more accurate and robust learning ability for various information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a structural diagram of one section of a steel tube tower.

[0041] Figure 2 FIG. 4 is a schematic diagram of an image processing flow according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0043] This embodiment provides an image processing method for automatic welding of transmission tower components. Figure 2 As shown, the following steps are included:

[0044] S100, obtaining an initial image captured during the automatic welding process, performing channel expansion on the initial image, and obtaining a first-order feature matrix. Specifically, in this embodiment, a convolution operation is performed on the initial image using a starting convolution layer to obtain a first-order feature matrix. The number of channels of the first-order feature matrix can be set according to actual conditions. In this embodiment, the number of channels of the first-order feature matrix is ​​48, and the convolution kernel size and step size of the starting convolution layer can be set according to actual conditions. In this embodiment, the convolution kernel size of the starting convolution layer is 3*3, and the step size is 1.

[0045] S200, then use four feature generation modules arranged in series to process the first-order feature matrix to obtain a second-order feature matrix. The first feature generation module uses the first-order feature matrix as input, and each subsequent feature generation module uses the inner-order feature matrix output by the previous feature generation module as input, until the inner-order feature matrix output by the last feature generation module is the second-order feature matrix.

[0046] S300, input the second-order feature matrix into the image reconstruction module, and after calculation, the image reconstruction module outputs an improved image. The image reconstruction module can be implemented using existing conventional technologies. Specifically, in this embodiment, the image reconstruction module includes a fifth convolutional layer, a sixth activation layer, and a sixth convolutional layer connected in sequence. The convolution kernel size of the fifth convolutional layer and the sixth convolutional layer is 3*3, and the step size is 1. The sixth activation layer uses the ReLU function, and finally the sixth convolutional layer outputs an improved image with 3 channels.

[0047] Furthermore, a first subunit and a second subunit are provided in the feature generation module. After performing a feature extraction operation based on the first subunit, a first result is obtained. After performing a feature extraction operation based on the second subunit, a second result is obtained. The first result and the second result are integrated to obtain an inner-order feature matrix output by the feature generation module. The size of the inner-order feature matrix output by each feature generation module is equal to the size of the first-order feature matrix.

[0048] Inside the first subunit, the first extractor and the second extractor both include sequentially arranged convolution operation layers and first excitation layers. In this embodiment, the convolution operation layer has a convolution kernel size of 5*5, a step size of 1, and the first excitation layer is a ReLU function; the first selector and the second selector both include sequentially arranged deformable convolution layers (deformableconvolution) and second excitation layers, the deformable convolution layer has a convolution kernel size of 3*3, a step size of 1, and the second excitation layer is a PReLU function. The sizes of the first feature map, the second feature map, the third feature map, the fourth feature map, the fifth feature map, and the sixth feature map generated inside all the first subunits are equal to the size of the first-order feature matrix, the convolution kernel size of the second convolution is 3*3, the step size is 1, the second activation uses the ReLU function, and the size of the first result is also equal to the size of the first-order feature matrix.

[0049] The second subunit takes the third feature map and the fourth feature map generated by the first subunit in the same feature generation module as input, and the internal calculation process of the second subunit includes:

[0050] Performing global average pooling processing on the third feature map in the channel dimension, compressing the number of channels of the third feature map to 1, and obtaining a first data set;

[0051] Expand the first data set (the first data set can be expanded by processing it with the flatten layer) to obtain a second data set, which is a one-dimensional vector;

[0052] The second data set is processed by the first convolution and the first activation in sequence to obtain a third data set; wherein the convolution kernel size of the first convolution is 1*1, and the first activation is a sigmoid function;

[0053] Performing global maximum pooling processing on the fourth feature map in the spatial dimension to obtain a fourth data set, where the fourth data set is a one-dimensional vector, and the length of the fourth data set is equal to the length of the third data set;

[0054] The fourth data set is integrated with the third data set to obtain a second result.

[0055] The above-mentioned integration of the fourth data set and the third data set to obtain the second result includes: adding the corresponding elements of the fourth data set and the third data set to obtain a fifth data set, and the fifth data set is successively processed by the third convolution (the convolution kernel size is 1*1) and the third activation (sigmoid function) to obtain the second result.

[0056] The way of integrating the first result and the second result is the existing conventional technology. The height and width dimensions of the second result are both 1, and its channel dimension is equal to the channel dimension of the first result. The second result is broadcasted along the spatial dimension to obtain the third result, so that the height, width and channel dimensions of the third result are equal to the height, width and channel dimensions of the first result. Since a broadcast operation is performed, in the third result, all eigenvalues ​​within the same channel are obtained by copying the eigenvalues ​​of the corresponding channel position of the second result. Then the third result is multiplied element by element with the corresponding elements in the first result to obtain the inner-order feature matrix output by the feature generation module.

[0057] In some other embodiments, the first data set generated in each second subunit is also input into the adaptation module, and after calculation, the adaptation module generates an adaptation feature map. The size of the adaptation feature map is equal to the size of the second-order feature matrix. The adaptation feature map is multiplied element by element with the second-order feature matrix to complete the processing of the second-order feature matrix. The second-order feature matrix processed by the adaptation feature map is then input into the image reconstruction module.

[0058] Specifically, the calculation process inside the adaptation module may include:

[0059] All the first data sets are concatenated to obtain a first adaptation matrix;

[0060] The first adaptation matrix is ​​sequentially processed by the fourth convolution and the fourth activation to obtain the second adaptation matrix; as an example but not limiting, the convolution kernel size of the fourth convolution is 3*3, the step size is 2, the fourth activation uses the ReLU function, the height and width of the second adaptation matrix are both half of the second-order feature matrix, and the number of channels is 4;

[0061] The second adaptation matrix is ​​processed by the PixelShuffle layer and the fifth activation (logistics function) in turn to obtain the third adaptation matrix. The height and width of the third adaptation matrix are equal to the height and width of the second-order feature matrix, and the number of channels of the third adaptation matrix is ​​1;

[0062] The third adaptation matrix is ​​broadcasted along the channel dimension so that the height, width and channel size of the matrix are equal to those of the second feature matrix, thereby obtaining an adapted feature map.

[0063] The adaptation module first uses the fourth convolution to reduce the dimension of the first adaptation matrix, and then expands the feature information in the spatial direction through the PixelShuffle layer. The adaptation module cooperates with the second subunit to realize cross-view and cross-level joint learning of feature information, so that the method provided by the present invention can better handle the situation where multiple complex interference information is mixed and embedded in the image at the same time, and the applicable scenarios are more diversified.

[0064] In specific implementation, the starting convolution layer, all feature generation modules and image generation modules together constitute a complete convolutional neural network. The convolutional neural network is first trained using a training set, and then the trained starting convolution layer, feature generation module and image generation module are used to perform the above-mentioned image processing.

[0065] Finally, it should be noted that the above description is only a preferred embodiment of the present invention. Under the guidance of the present invention, ordinary technicians in this field can make various similar expressions without violating the purpose and claims of the present invention, and such changes all fall within the scope of protection of the present invention.

Claims

1. An image processing method for automatic welding of transmission tower components, characterized in that: The following steps are involved: Acquire an initial image captured during the automatic welding process, perform channel expansion on the initial image, and obtain a first-order feature matrix; After processing the first-order feature matrix using a plurality of feature generation modules arranged in series, a second-order feature matrix is ​​obtained; Inputting the second-order characteristic matrix into an image reconstruction module, the image reconstruction module outputs an improved image; The feature generation module is provided with a first subunit and a second subunit. After performing a feature extraction operation based on the first subunit, a first result is obtained. After performing a feature extraction operation based on the second subunit, a second result is obtained. The first result and the second result are integrated to obtain an inner-order feature matrix output by the feature generation module.

2. The image processing method for automatic welding of transmission tower components according to claim 1, characterized in that: The first subunit in the first feature generation module takes the first-order feature matrix as input, and the first subunit in the subsequent feature generation modules takes the inner-order feature matrix output by its upstream feature generation module as input.

3. The image processing method for automatic welding of transmission tower components according to claim 2, characterized in that: The internal calculation process of the first subunit includes: After processing the feature matrix input to the first sub-unit based on the first extractor and the first selector respectively, a first feature map and a second feature map are obtained; Subtract the first characteristic map from the second characteristic map to obtain a third characteristic map; Performing a Hadamard product of the first feature map and the second feature map to obtain a fourth feature map; Inputting the first feature map, the third feature map and the fourth feature map into a second extractor, and after calculation, the second extractor outputs a fifth feature map; The second feature map, the third feature map and the fourth feature map are input into a second selector, and after calculation, the second selector outputs a sixth feature map; The fifth feature map is integrated with the sixth feature map to obtain the first result.

4. The image processing method for automatic welding of transmission tower components according to claim 3 is characterized in that: The first extractor and the second extractor each include a convolution operation layer and a first excitation layer arranged in sequence; Or / and, the first selector and the second selector both include a deformable convolution layer and a second excitation layer arranged in sequence.

5. The image processing method for automatic welding of transmission tower components according to claim 4, characterized in that: The second subunit takes the third feature map and the fourth feature map as input.

6. The image processing method for automatic welding of transmission tower components according to claim 5, characterized in that: The internal calculation process of the second subunit includes: Performing global average pooling processing on the third feature map in the channel dimension to obtain a first data set; Performing expansion processing on the first data set to obtain a second data set; The second data set is sequentially subjected to the first convolution and the first activation processes to obtain a third data set; Performing global maximum pooling processing on the fourth feature map in the spatial dimension to obtain a fourth data set; The fourth data set is integrated with the third data set to obtain the second result.

7. The image processing method for automatic welding of transmission tower components according to claim 6, characterized in that: Integrating the first result with the second result to obtain an inner-order feature matrix output by the feature generation module, including: broadcasting the second result along the spatial dimension to obtain a third result, and then multiplying the third result by the first result element by element to obtain the inner-order feature matrix output by the feature generation module; or / and, integrating the fifth feature map with the sixth feature map to obtain the first result, including: concatenating the fifth feature map with the sixth feature map to obtain a seventh feature map, and then sequentially subjecting the seventh feature map to a second convolution and a second activation process to obtain the first result; Or / and, integrating the fourth data set with the third data set to obtain the second result, including: adding corresponding elements of the fourth data set and the third data set to obtain a fifth data set, and the fifth data set is successively subjected to a third convolution and a third activation process to obtain the second result.

8. The image processing method for automatic welding of transmission tower components according to claim 7, characterized in that: Before the second-order feature matrix is ​​input into the image reconstruction module, the second-order feature matrix is ​​first processed using the adaptation feature map, and then the second-order feature matrix processed by the adaptation feature map is input into the image reconstruction module.

9. The image processing method for automatic welding of transmission tower components according to claim 8, characterized in that: The adaptation module takes the first data set generated in each second subunit as input, and generates the adaptation feature map after calculation; the adaptation feature map completes the processing of the second-order feature matrix by multiplying the second-order feature matrix element by element.

10. The image processing method for automatic welding of transmission tower components according to claim 9, characterized in that: The internal calculation process of the adaptation module includes: All the first data sets are concatenated to obtain a first adaptation matrix; The first adaptation matrix is ​​sequentially subjected to a fourth convolution and a fourth activation process to obtain a second adaptation matrix; The second adaptation matrix is ​​sequentially processed by the PixelShuffle layer and the fifth activation to obtain a third adaptation matrix; The third adaptation matrix is ​​broadcasted along the channel dimension to obtain the adaptation feature map.

Images