Image optimization method, device and storage medium for tower climbing robot

Abstract

The application discloses an image optimization method and device for a tower climbing robot and a storage medium, and belongs to the technical field of image processing. The optimization method comprises: performing step-by-step feature extraction and information gain on a to-be-processed image to determine a plurality of first representation results and a plurality of second representation results; performing first coupling processing on the plurality of first representation results to determine a first domain connection result from the plurality of first representation results; performing second coupling processing on the plurality of second representation results to determine a second domain connection result from the plurality of second representation results; and determining a target image based on the first domain connection result and the second domain connection result. The application performs step-by-step feature extraction and information gain on the to-be-processed image, so that the neural network focuses on positive information in the image, and the feature correlation between the positive information is strengthened, the quantity and intensity of interference information are squeezed from the negative side, and therefore, a higher image optimization effect is achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an image optimization method, device and storage medium for a tower climbing robot. BACKGROUND

[0002] For manual maintenance and maintenance of power transmission tower, there are often high safety risks, high labor intensity and low work efficiency and many other difficulties. In recent years, with the development of robots and artificial intelligence technologies, the types of robots for power transmission tower climbing on the market are increasingly rich, which greatly improves the automation and intelligent degree of work.

[0003] Most of the current tower climbing robots are equipped with visible light cameras to realize external environment perception. However, the working environment of the tower climbing robot is mostly outdoor high altitude, and faces multiple interferences such as light, weather, device viewing angle, etc. There are often the following problems in the captured images: strong light overexposure, weak light underexposure, rain / mist / debris shielding, noise or distortion, etc., and the image quality needs to be improved. SUMMARY

[0004] Therefore, the application provides an image optimization method, device and storage medium for a tower climbing robot to solve the technical problem of low image quality.

[0005] The technical scheme is as follows: an image optimization method for a tower climbing robot, comprising:

[0006] A plurality of extractors arranged in cascade are used to perform step-by-step feature extraction on a to-be-processed image, each of the extractors determines a corresponding first representation result after calculation, and a plurality of gainers arranged in cascade are used to perform step-by-step information gain on the to-be-processed image, each of the gainers determines a corresponding second representation result after calculation;

[0007] The first representation results are subjected to first coupling processing to determine a first domain connection result from the first representation results;

[0008] The second representation results are subjected to second coupling processing to determine a second domain connection result from the second representation results;

[0009] The target image is determined based on the first domain connection result and the second domain connection result;

[0010] The calculation process of the information gain of the gainer comprises:

[0011] The data input from the front gainers is calculated based on a first convolution unit to determine a domain front feature map;

[0012] fuse the data input from the middle part of the gain device with the domain pre-feature map to determine a domain post-feature map;

[0013] extract the maximum feature value in each channel of the domain pre-feature map and activate it to determine a first gain feature;

[0014] extract the maximum feature value in each channel of the domain post-feature map and activate it to determine a second gain feature;

[0015] extract the maximum feature value in each channel of the data input from the middle part of the gain device and activate it to determine a third gain feature;

[0016] fuse the first gain feature, the second gain feature and the third gain feature with the domain post-feature map to determine the second representation result.

[0017] In some possible implementation manners, the extractor corresponds to the gain device one-to-one.

[0018] In some possible implementation manners, the gain device at the most front end takes the image to be processed as the data input from the front of the gain device, and the other gain devices all take the second representation result determined by the previous gain device as the data input from the front of the gain device; each of the gain devices takes at least one feature map inside the corresponding extractor as the data input from the middle part of the gain device.

[0019] In some possible implementation manners, the calculation process of the feature extraction of the extractor includes:

[0020] based on a 1x1 convolution unit, a kxk convolution unit and a deformable convolution unit, the data input into the extractor is calculated to correspond to determine a first feature map, a second feature map and a third feature map, k is a positive integer greater than 1;

[0021] cross-aggregate the first feature map, the second feature map and the data input into the extractor to determine a fourth feature map;

[0022] fuse the first feature map, the second feature map and the third feature map to determine a fifth feature map; fuse the fourth feature map and the fifth feature map to determine the first representation result;

[0023] The 1x1 convolution unit, the kxk convolution unit and the deformable convolution unit are all provided with an activation function layer.

[0024] In some possible implementation manners, the calculation process of cross-aggregating the first feature map, the second feature map and the data input into the extractor includes:

[0025] after the first feature map is subtracted from the data input into the extractor and activated, a first cross-feature map is determined;

[0026] The second cross feature map is determined by multiplying the second feature map and the data product of the input extractor and activating the product;

[0027] The third cross feature map is determined by fusing the first cross feature map and the second feature map;

[0028] The fourth cross feature map is determined by fusing the second cross feature map and the first feature map;

[0029] The fourth feature map is determined by fusing the third cross feature map, the fourth cross feature map and the data of the input extractor.

[0030] In some possible implementation manners, each of the gainers takes the fourth feature map generated inside the corresponding extractor as the data input from the middle part of the gainer.

[0031] In some possible implementation manners, the calculation process of the second coupling processing on the plurality of second feature results comprises:

[0032] The first connected feature is determined by extracting the maximum feature value in each spatial position of the second feature result and activating the value;

[0033] The second connected feature is determined by splicing the plurality of first connected features;

[0034] The second domain connected result is determined by calculating the second connected feature based on the second convolution unit.

[0035] In some possible implementation manners, the calculation process of the target image reconstructed based on the first domain connected result and the second domain connected result comprises:

[0036] The intermediate feature map is determined by multiplying the first domain connected result and the corresponding element of the second domain connected result and activating the product;

[0037] The optimized target image is determined by calculating the intermediate feature map based on the third convolution unit.

[0038] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method.

[0039] The application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the method when executing the computer program.

[0040] Compared with the prior art, the application has the following beneficial effects:

[0041] The prior art usually focuses on the interference information of negative effects in the image, identifies and filters them by designing methods, in the information filtering process, part of the valuable information is also removed, and the optimization use of the positive information in the image is ignored, resulting in limited image optimization effect. The present application performs feature extraction and information gain on the to-be-processed image step by step, and then determines the target image by coupling a plurality of first feature results and a plurality of second feature results, so that the neural network focuses on the positive information in the image and strengthens the feature correlation between them, and the quantity and intensity of the interference information are squeezed from the opposite side, thereby achieving a more efficient image optimization effect. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The flowchart for determining a plurality of first feature results and a plurality of second feature results by the extractor and the gainer of the present application.

[0043] In the drawings:

[0044] 1 - to-be-processed image, 2 - extractor, 3 - gainer, 4 - first feature result, 5 - second feature result. DETAILED DESCRIPTION

[0045] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0046] The tower climbing robot can autonomously climb the tower, assist or replace manual work to complete the inspection and maintenance tasks. In the inspection process, the robot can carry multiple sensors and detection devices such as high-definition cameras, infrared thermal imagers, etc., which can automatically check the structural integrity and component damage of the tower, and also can be used as a mobile sensing platform to transmit the collected information to the background staff for more comprehensive analysis.

[0047] The image optimization method provided by the embodiment of the present application can be applied to the tower climbing robot body, and can also be applied to a system composed of a robot and a server. At this time, the robot communicates and transmits data with the server through the network. The server can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers.

[0048] Taking the system composed of a robot and a server as an example, the image optimization method provided by the embodiment of the present application can be executed as follows: Figure 1As shown, the original image collected by the camera carried by the robot is taken as the to-be-processed image 1, and then the server obtains the to-be-processed image 1 through the network, performs feature extraction and information gain on the to-be-processed image 1 step by step, determines a plurality of first representation results and a plurality of second representation results, performs first coupling processing on the plurality of first representation results to determine a first domain connection result from the plurality of first representation results, performs second coupling processing on the plurality of second representation results to determine a second domain connection result from the plurality of second representation results, and determines an optimized target image based on the first domain connection result and the second domain connection result, so as to realize denoising, repairing, correcting, adjusting brightness, contrast and saturation of the to-be-processed image 1, and make the image clearer and more coordinated.

[0049] In some embodiments of the present application, a plurality of extractors 2 arranged in cascade are used for step-by-step feature extraction, and the feature map determined by each extractor 2 after calculation is the corresponding first representation result 4; a plurality of gainers 3 arranged in cascade are used for step-by-step information gain, and the feature map determined by each gainer 3 is the corresponding second representation result 5; the extractors and gainers correspond one-to-one. The number of extractors and gainers is not particularly limited in the present application, and the extractors and gainers can be set to 3, 4 or more.

[0050] Based on the above setting, the first extractor 2 takes the to-be-processed image 1 as input, and the other extractors 2 take the first representation result 4 determined by the previous extractor 2 as input. That is, the to-be-processed image 1 is input from the front end of the first extractor 2, the first extractor 2 outputs the first first representation result 4 after calculation, then the first first representation result 4 is input from the front end of the second extractor 2, the second extractor 2 outputs the second first representation result 4 after calculation, and so on, the A-1th first representation result 4 is input from the front end of the Ath extractor 2, and the Ath extractor 2 outputs the Ath first representation result 4 after calculation.

[0051] For the images obtained in the application scenario of the tower climbing robot, there are multiple interference information, and different modes of interference information are superimposed and combined with each other, which can also derive more forms. In the conventional feature extraction mode, the interference information is easy to cause "crosstalk" to the normal information, the obtained features are alienated, and the optimization effect is not complete. Therefore, the multi-scale convolution, deformable convolution and cross-aggregation are used in the extractor to extract features, which has strong deconstruction and solidification effect on feature information, greatly reducing the probability of feature alienation.

[0052] As a specific embodiment of the many embodiments, for a single extractor, the specific calculation process inside it can include the following steps: based on the 1x1 convolution unit, the kxk convolution unit and the deformable convolution unit, respectively, the data input into the extractor is calculated, and the first feature map, the second feature map and the third feature map are determined, k is a positive integer greater than 1; the first feature map, the second feature map and the input data of the extractor are cross-aggregated to determine the fourth feature map; the first feature map, the second feature map and the third feature map are fused to determine the fifth feature map; the fourth feature map and the fifth feature map are fused to determine the first representation result.

[0053] The 1x1 convolution unit and the kxk convolution unit inside are respectively provided with a convolution layer and an activation function layer arranged in cascade, the convolution kernel size of the convolution layer in the 1x1 convolution unit is 1x1, the convolution kernel size of the convolution layer in the kxk convolution unit is kxk, and in order to control the calculation amount of the network model, the value of k is preferably less than or equal to 7. The deformable convolution unit is provided with a deformable convolution layer and an activation function layer arranged in cascade, and the convolution kernel size of the deformable convolution layer is not particularly limited, for example, it can be 3x3. The activation function layer is used to introduce a nonlinear transformation, and give the network the ability to fit complex nonlinear relationships. As an example but not limitation, the activation function layer can be ReLU (Rectified Linear Unit) or Leaky ReLU (Leaky ReLU).

[0054] For each extractor, the data size input into the extractor is equal to the size of the first feature map, the second feature map and the third feature map generated inside it. For example, for the first extractor, the image to be processed is input, if the channel of the image to be processed is C1, the height is H1, and the width is W1, then the channel, height and width of the first feature map, the second feature map and the third feature map generated inside are also C1, H1 and W1 respectively. For the third extractor, the second first representation result is input, if the channel of the second first representation result is C2, the height is H2, and the width is W2, then the channel, height and width of the first feature map, the second feature map and the third feature map generated inside are also C2, H2 and W2 respectively.

[0055] Feature fusion can integrate the features extracted by different layers or different branches of the network to obtain more comprehensive representation information. As a specific embodiment of the many embodiments, the calculation process of cross-aggregating the first feature map, the second feature map and the data input into the extractor includes:

[0056] After the first feature map and the data input into the extractor are activated, for example, the elements in the first feature map can be subtracted from the corresponding elements in the data input into the extractor, to determine the first cross-feature map;

[0057] The second feature map is activated after being multiplied with the data product of the input extractor, and a second cross feature map is determined;

[0058] The first cross feature map and the second feature map are fused to determine a third cross feature map;

[0059] The second cross feature map and the first feature map are fused to determine a fourth cross feature map;

[0060] The third cross feature map, the fourth cross feature map and the data of the input extractor are fused to determine a fourth feature map.

[0061] In the cross aggregation process described above, the activation operation can be performed by using an activation function such as ReLU or LeakyReLU as a general means. The fusion of the first cross feature map and the second feature map, and the fusion of the second cross feature map and the first feature map are not particularly limited, and any existing fusion method can be used to achieve the purpose. For example, the fusion can be performed by using a splicing-convolution-activation method. Similarly, the fusion of the third cross feature map, the fourth cross feature map and the data of the input extractor can be performed by using an existing method, for example, the third cross feature map, the fourth cross feature map and the data of the input extractor can be added element by element and then activated. The fusion of the fourth feature map and the fifth feature map is also an existing technology, and the first representation result can be determined by using a splicing-convolution-activation method.

[0062] In combination with the cross aggregation, as a preferred embodiment in many embodiments, the calculation process of fusing the first feature map, the second feature map and the third feature map includes: multiplying the first feature map and the second feature map element by element, and then activating by using a Tanh function to determine a fusion feature map; multiplying the fusion feature map and the third feature map element by element, and then activating by using a softmax function to determine a fifth feature map. In this way, the neural network model can use multiple features to realize cross verification, strengthen the underlying association between different features, more fully integrate information, and improve the accuracy of feature fusion.

[0063] For each of the gainers 3, the first gainer 3 at the front end takes the to-be-processed image 1 and at least one feature map generated in the corresponding extractor 2 as input, and the other gainers 3 take the second representation result determined by the previous gainer 3 and at least one feature map generated in the corresponding extractor 2 as input. Based on the above process, in some embodiments, the to-be-processed image 1 is input from the front of the first gainer 3 (the gainer 3 at the front end), the fourth feature map generated in the first extractor 2 is input from the middle of the first gainer 3, and the first gainer 3 determines the first second representation result 5 after calculation. Then the first second representation result 5 is input from the front of the second gainer 3, the fourth feature map generated in the second extractor 2 is input from the middle of the second gainer 3, and the second gainer 3 determines the second second representation result 5 after calculation. In this way, the A-1th second representation result 5 is input from the front of the Ath gainer 3, the fourth feature map generated in the Ath extractor 2 is input from the middle of the Ath gainer 3, and the Ath gainer 3 determines the Ath second representation result 5 after calculation.

[0064] As a preferred method in many embodiments, for a certain gainer, the specific calculation process in the gainer can include:

[0065] Based on the calculation of the first convolution unit on the data input from the front gainer, the pre-domain feature map is determined; for the first gainer at the front end, the data input from the front gainer is the to-be-processed image, and for the other gainers, the data input from the front gainer is the second representation result determined by the previous gainer;

[0066] Fusing the corresponding fourth feature map (generated in the extractor) and the pre-domain feature map, the post-domain feature map is determined;

[0067] Extracting the maximum feature value in each channel of the pre-domain feature map and activating it to determine the first gain feature;

[0068] Extracting the maximum feature value in each channel of the post-domain feature map and activating it to determine the second gain feature;

[0069] Extracting the maximum feature value in each channel of the corresponding fourth feature map (generated in the extractor) and activating it to determine the third gain feature;

[0070] Fusing the first gain feature, the second gain feature, and the third gain feature with the post-domain feature map to determine the second representation result.

[0071] As a conventional means for processing the feature map, the maximum feature value in each channel of the feature map is extracted, that is, a global maximum pooling processing is performed on the feature in the spatial direction. For example, a certain feature map channel is C3, the height is H3, and the width is W3. The feature map contains C3 matrices with a size of H3xW3. The maximum feature value in each size H3xW3 matrix is extracted, and then a vector with a length of C3 is obtained.

[0072] As an exemplary embodiment, the fourth feature map and the corresponding domain pre-feature map can be fused by summing the corresponding elements. In some embodiments, the lengths of the first gain feature, the second gain feature, and the third gain feature can be equal. In this case, the fusion of the first gain feature, the second gain feature, and the third gain feature with the post-domain feature map includes: summing and activating the corresponding elements of the first gain feature and the second gain feature to obtain a first integrated gain feature, and then summing and activating the corresponding elements of the first integrated gain feature and the third gain feature to obtain a second integrated gain feature. The length of the second integrated gain feature is equal to the number of channels of the post-domain feature map, and each feature value of the second integrated gain feature is in a one-to-one correspondence with each channel of the post-domain feature map. The elements in each channel of the post-domain feature map are multiplied by the corresponding elements in the second integrated gain feature to achieve fusion, and a second representation result is determined, which realizes the allocation of different gain weight coefficients to each channel of the post-domain feature map through the second integrated gain feature.

[0073] The gainer in the above embodiment can cooperate with the extractor to well strengthen the distribution of the gain feature space boundary, so that the information gain can flow across levels stably and efficiently, which helps the network to systematically integrate scattered source domain features and process domain features to optimize the target domain features. In the above process inside the gainer, the activation operation can be performed by using a ReLU or sigmoid activation function as a conventional means. The first convolution unit is provided with a convolution layer and an activation function layer in cascade, and the activation function layer can be a ReLU or Leaky ReLU function.

[0074] As one of the many embodiments, the calculation process of the first coupling processing on the plurality of first representation results can include: sequentially splicing, convolving, and activating all the first representation results to determine a first domain connection result.

[0075] The calculation process of the second coupling processing on the plurality of second representation results can include: extracting the maximum feature value in each spatial position of each second representation result and activating it to correspondingly determine a plurality of first connection features.

[0076] The plurality of first connection features are spliced to determine a second connection feature.

[0077] The second domain connection result is determined by calculating the second connection feature based on the second convolution unit.

[0078] The convolution layer and the activation function layer are cascaded in the second convolution unit. As a conventional means for processing the feature map, the maximum feature value in each spatial position of the feature map is extracted, that is, global maximum pooling processing is performed on the feature map in the channel direction. For example, a certain feature map has C4 channels, H4 height, and W4 width. The feature map contains H4*W4 vectors with a length of C4. The maximum feature value in each spatial position of the feature map is extracted, that is, the maximum feature value in each vector with a length of C4 is extracted, and then a 1*H4*W4 matrix is obtained.

[0079] The size of the first domain connection result can be equal to the size of the second domain connection result. As an example, the calculation process of reconstructing the target image based on the first domain connection result and the second domain connection result can include: multiplying the corresponding elements of the first domain connection result and the second domain connection result and then activating to determine an intermediate feature map; and calculating the intermediate feature map based on a third convolution unit to determine an optimized target image. The convolution layer and the activation function layer are cascaded in the third convolution unit.

[0080] The above-described process can be implemented as a computer software program. Therefore, another aspect of the present application also provides a computer program product including a computer program carried on a computer readable medium, the computer program including a computer program for executing the above-described method. In such an embodiment, the computer program can be downloaded and installed from the Internet in a communicable manner.

[0081] Another aspect of the present application also provides an electronic device including: a memory for storing one or more programs; and a controller. When the one or more programs are executed by the controller, the above-described image optimization method is performed. As an example, the electronic device can include a central processing unit (CPU) that can perform calculations according to programs stored in a read-only memory (ROM) to complete the method in the above-described embodiment, or perform calculations according to programs in a random access memory (RAM) to complete the method in the above-described embodiment.

[0082] Finally, it should be noted that the above description is merely preferred embodiments of the present application, and those of ordinary skill in the art can make various similar modifications under the inspiration of the present application without departing from the purpose and scope of the present application. Such modifications fall within the protection scope of the present application.

Claims

1. An image optimization method for a tower climbing robot, characterized in that, The method comprises the following steps: a plurality of extractors arranged in cascade are used to perform feature extraction on the image to be processed, each of the extractors determines a corresponding first representation result after calculation, and a plurality of gainers arranged in cascade are used to perform information gain on the image to be processed, each of the gainers determines a corresponding second representation result after calculation; a first coupling process is performed on the plurality of first representation results to determine a first domain connection result from the plurality of first representation results; a second coupling process is performed on the plurality of second representation results to determine a second domain connection result from the plurality of second representation results; a target image is determined based on the first domain connection result and the second domain connection result; the calculation process of the gainers for information gain comprises the following steps: a first convolution unit is used to calculate the data input from the front gainers to determine a domain front feature map; the data input from the middle gainers is fused with the domain front feature map to determine a domain rear feature map; the maximum feature value in each channel of the domain front feature map is extracted and activated to determine a first gain feature; the maximum feature value in each channel of the domain rear feature map is extracted and activated to determine a second gain feature; the maximum feature value in each channel of the data input from the middle gainers is extracted and activated to determine a third gain feature; the first gain feature, the second gain feature and the third gain feature are fused with the domain rear feature map to determine the second representation result; the first gainer at the front end takes the image to be processed as the data input from the front of the gainer, and the other gainers take the second representation result determined by the previous gainer as the data input from the front of the gainer; each of the gainers takes at least one feature map in the corresponding extractor as the data input from the middle of the gainer.

2. The method of claim 1, wherein, The extractors correspond to the gainers one by one.

3. The method of claim 1, wherein, The calculation process of the extractors for feature extraction comprises the following steps: a 1x1 convolution unit, a kxk convolution unit and a deformable convolution unit are used to calculate the data input into the extractor to correspondingly determine a first feature map, a second feature map and a third feature map, k is a positive integer greater than 1; the first feature map and the second feature map are cross-aggregated with the data input into the extractor to determine a fourth feature map; the first feature map, the second feature map and the third feature map are fused to determine a fifth feature map; the fourth feature map and the fifth feature map are fused to determine the first representation result; an activation function layer is arranged in each of the 1x1 convolution unit, the kxk convolution unit and the deformable convolution unit.

4. The method of claim 3, wherein, The calculation process of cross-aggregating the first feature map, the second feature map and the data input into the extractor comprises the following steps: the first feature map is subtracted from the data input into the extractor and then activated to determine a first cross-feature map; the second feature map is multiplied by the data input into the extractor and then activated to determine a second cross-feature map; the first cross-feature map is fused with the second feature map to determine a third cross-feature map; the second cross-feature map is fused with the first feature map to determine a fourth cross-feature map; The third cross feature map and the fourth cross feature map are fused with data of the input extractor to determine the fourth feature map.

5. The method of claim 3, wherein, Each of the gainers takes the fourth feature map generated inside the corresponding extractor as data input from the middle of the gainer.

6. The method of claim 1, wherein, The calculation process of the second coupling processing of the plurality of second feature results includes: The maximum feature value in each spatial position of the second feature result is extracted and activated to determine the first connected feature. The plurality of first connected features are spliced to determine the second connected feature. The second connected feature is calculated based on the second convolution unit to determine the second domain connection result.

7. The method of claim 1, wherein, The calculation process of the target image reconstructed based on the first domain connection result and the second domain connection result includes: The first domain connection result and the second domain connection result are multiplied with corresponding elements and activated to determine the intermediate feature map. The intermediate feature map is calculated based on the third convolution unit to determine the optimized target image.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 7.

Images