Power transmission tower monitoring single image rain removal method based on generative adversarial network

Abstract

The application provides a power transmission tower monitoring single image rain removal method based on a generative adversarial network, which comprises the following steps: obtaining a first target image, and constructing a verification data set, a training data set and a test data set according to the first target image; inputting the training data set into a generative network to obtain a corresponding third target image; constructing a rain removal model according to the generative network and an adversarial network in the case that a loss function value is less than a first threshold value, and calculating a corresponding target signal-to-noise ratio by the rain removal model based on the test data set and the verification data set respectively to obtain a peak signal-to-noise ratio; repeating the inputting step in the case that any target signal-to-noise ratio is less than or equal to a second threshold value until all target signal-to-noise ratios are greater than the second threshold value; and determining the rain removal model as a target model in the case that the second signal-to-noise ratio is greater than the second threshold value, and removing rain by the target model. The method solves the problem of serious distortion in the prior art method for removing rain from a single image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, computer-readable storage medium, and monitoring system for deraining single images of power transmission tower monitoring based on generative adversarial networks. Background Technology

[0002] Online monitoring systems for power transmission towers utilize cameras deployed on the towers to collect images of insulator strings, conductors, hardware, the tower itself, and corridors. These images are then output to a machine vision-based intelligent system to intelligently identify anomalies in the transmission lines and corridors, improving the efficiency of transmission line inspections. However, due to the abundant rainfall in summer, raindrops not only create streaks but also obscure the view, severely impacting the quality of the monitoring images and consequently affecting the accuracy of subsequent anomaly identification. Therefore, researching methods for rain removal from transmission tower monitoring images is of significant importance.

[0003] Rain removal from transmission tower monitoring images can be divided into two categories: single-image-based rain removal methods and multi-image-based rain removal methods. The latter requires continuous acquisition and transmission of multiple images, which results in higher operating costs for mobile communication-based transmission tower monitoring systems and is more suitable for wired communication-based transmission tower monitoring systems. The main methods for removing rain from single images of power transmission tower monitoring include: filtering-based methods, which mainly use spatial or transform domain filtering to eliminate the influence of raindrops on the image. These methods have simple algorithm structures but poor rain removal performance; prior model-based methods, which require modeling the optical properties of the rain and background layers based on prior information to separate them from the image. However, these methods rely heavily on assumptions and manual parameter tuning, and the diverse forms and distributions of rain make it difficult to design suitable models to accurately separate the rain layer from the image in different scenarios, limiting their practical application; and in recent years, a series of deep learning-based methods for removing rain from single images of power transmission tower monitoring have been proposed. These methods directly establish an end-to-end projection between the rain image and its corresponding real background to remove stripes, significantly improving rain removal performance compared to traditional methods, and are applicable to a wider range of scenarios, making them highly practical. However, current deep learning-based deraining algorithms suffer from problems such as blurred image details or loss of texture information, which affect the deraining effect of single images from power transmission tower monitoring. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, computer-readable storage medium, and monitoring system for deraining single images of power transmission tower monitoring based on generative adversarial networks, so as to at least solve the problem of severe distortion in existing methods for deraining single images.

[0005] To achieve the above objectives, according to one aspect of this application, a method for de-raining a single image of a power transmission tower monitoring system based on a generative adversarial network (GAN) is provided, comprising: acquiring a first target image, and constructing a validation dataset, a training dataset, and a test dataset based on the first target image, wherein the first target image is a monitoring image containing a power transmission tower, the validation dataset includes the first target image acquired on a rainy day, and the test dataset includes the first target image acquired on a non-rainy day and the corresponding image after rain enhancement; and an input step, inputting each second target image in the training dataset into a generative network to obtain a corresponding third target image, inputting the third target image into an adversarial network to obtain a loss function value, and adjusting the generative network and the adversarial network based on the loss function value using an optimization algorithm. The network, wherein the generator network is used to perform deraining processing, and the second target image is the first target image acquired on a rainy day; when the loss function value is less than a first threshold, a deraining model is constructed based on the generator network and the adversarial network, and the peak signal-to-noise ratio is calculated based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio; when the target signal-to-noise ratio corresponding to the test dataset and / or the validation dataset is less than or equal to a second threshold, the input steps are repeated until the target signal-to-noise ratio of both the test dataset and the validation dataset is greater than the second threshold; when the second signal-to-noise ratio is greater than the second threshold, the deraining model is determined as the target model, and the first target image acquired on a rainy day is derained using the target model.

[0006] Optionally, the generator network includes a convolutional group module, a first dilated convolutional group module, a second dilated convolutional group module, a multi-scale dilated convolutional module, a spatial attention module, a coding foundation module, a 3×3 convolutional module, and an upsampling module. Each second target image in the training dataset is input into the generator network to obtain a corresponding third target image. This includes: processing the second target image through a first preset number of the 3×3 convolutional modules to obtain a first feature map; processing the first feature map through the first dilated convolutional group module, the multi-scale dilated convolutional module, the spatial attention module, and the convolutional group module to obtain a second feature map; processing the first feature map through the coding foundation module to obtain a third feature map; and processing the third feature map through the coding foundation module. The process involves obtaining a fourth feature map, processing the third feature map using the encoding base module to obtain a fifth feature map, processing the fifth feature map using a second preset number of second dilated convolution modules and the upsampling module, and concatenating it with the fourth feature map to obtain a sixth feature map, processing the sixth feature map using the 3×3 convolution module and the upsampling module, and concatenating it with the third feature map to obtain a seventh feature map, processing the seventh feature map using the 3×3 convolution module and the upsampling module, and concatenating it with the second feature map to obtain an eighth feature map, and finally processing the eighth feature map using the 3×3 convolution module and concatenating it with the second feature map, followed by processing it using the convolution module to obtain the third target image.

[0007] Optionally, the first feature map is processed by the first dilated convolutional module, the multi-scale dilated convolutional module, the spatial attention module, and the convolutional module to obtain the second feature map, including: inputting the first feature map into the first dilated convolutional module to obtain the first target output. Among them, Y i For the first feature map, cov 3,3,3 () represents a dilated convolution with a kernel of 3, a dilation rate of 3, and padding of 3. ReLU() is the activation function. For element-wise addition, Y o The first target output is obtained by inputting the first target output into the multi-scale dilated convolution module; the second target output is then input into the spatial attention module to obtain the third target output. Among them, B o For the third target output, B i The input data for the spatial attention module is S(), the spatial attention submodule is cov1(), which is a 1×1 convolution, and cov 3,1,1 () represents a 3×3 dilated convolution with a dilation rate of 1 and a fill rate of 1. For element-wise multiplication, B o2 satisfy: Among them, cov 3,2,2 () represents a 3×3 dilated convolution with a dilation rate of 2 and a fill rate of 2. o1 satisfy: Among them, cov 3,3,3 () represents a 3×3 dilated convolution with a dilation rate of 3 and a padding rate of 3; the third target output is input into the convolutional group module to obtain the second feature map: Among them, F i For the input data of the convolutional group module, F o For the second feature map, cov3() is a convolution with a kernel of 3.

[0008] Optionally, the multi-scale dilated convolution module includes a 1×1 convolution module, a ReLU function module, a 9×9 convolution module, a batch normalization module, an 11×11 convolution module, a 7×7 convolution module, and a 13×13 convolution module. The first target output is input into the multi-scale dilated convolution module to obtain a second target output, including: sequentially processing the first target output through the 1×1 convolution module, the ReLU function module, the 9×9 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain a fourth target output; sequentially processing the first target output through the 1×1 convolution module, the ReLU function module, the 11×11 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain a fifth target output; and so on. A 1×1 convolution module processes the first target output to obtain a sixth target output; the first target output is then processed sequentially through the 1×1 convolution module, the ReLU function module, the 7×7 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain a seventh target output; the first target output is then processed sequentially through the 1×1 convolution module, the ReLU function module, the 13×13 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain an eighth target output; the fourth target output, the fifth target output, the seventh target output, and the eighth target output are element-wise added to the sixth target output to obtain multiple ninth target outputs; all the ninth target data are concatenated to obtain the second target output.

[0009] Optionally, the first feature map is processed by the encoding base module to obtain the third feature map, including: inputting the first feature map into the encoding base module to obtain the third feature map: D o =Sig(cov 2,2(ReLu(cov3(ReLu(cov3(D o1 ))))));where Sig() is the Sigmoid function, cov 2,2 () represents a 2×2 convolution with a stride of 2, ReLU() is the ReLU activation function, cov3() is a 3×3 convolution, and D o1 satisfy: in, For element-wise multiplication, For element-wise addition, Amp() is the max pooling function, Avp() is the average pooling function, and D... i This is the input data for the encoding base module.

[0010] Optionally, the fifth feature map is processed by a second preset number of the second dilated convolutional group modules and the upsampling module, including: inputting the fifth feature map into the second dilated convolutional group modules: Among them, R i For the fifth feature map, cov 3,2,2 () represents a dilated convolution with a kernel of 3, a dilation rate of 2, and padding of 2. ReLU() is the activation function. For element-wise addition, R o It is the output feature map of the convolutional group module.

[0011] Optionally, the adversarial network includes a ReLU function module and the 3×3 convolution module. Inputting the third target image into the adversarial network to obtain a loss function value includes: processing the third target image sequentially through a third preset number of the 3×3 convolution modules to obtain a ninth feature map; and calculating the loss function value based on the mean squared error of the ninth feature map and the third target output.

[0012] According to another aspect of this application, a rain removal device for a single image of a power transmission tower monitoring system based on a generative adversarial network is provided. The device includes: an acquisition unit, configured to acquire a first target image and construct a verification dataset, a training dataset, and a test dataset based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image acquired on a rainy day, and the test dataset includes the first target image acquired on a non-rainy day and the corresponding rain-treated image. An input unit is configured to perform an input step, inputting each second target image from the training dataset into a generative network to obtain a corresponding third target image, inputting the third target image into an adversarial network to obtain a loss function value, and adjusting the generative network and the adversarial network based on the loss function value using an optimization algorithm. A generator network is used to perform deraining processing, and the second target image is the first target image acquired on a rainy day. A construction unit is used to construct a deraining model based on the generator network and the adversarial network when the loss function value is less than a first threshold, and calculate the peak signal-to-noise ratio (PSNR) of the corresponding target PSNR based on the test dataset and the validation dataset respectively. A repetition unit is used to repeat the input step when the target PSNR of the test dataset and / or the validation dataset is less than or equal to a second threshold, until the target PSNR of both the test dataset and the validation dataset is greater than the second threshold. A determination unit is used to determine the deraining model as the target model when the second PSNR is greater than the second threshold, and perform deraining on the first target image acquired on a rainy day using the target model.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0014] According to another aspect of this application, a monitoring system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any one of the methods described.

[0015] Applying the technical solution of this application, in the above-mentioned method for de-raining a single image of a transmission tower monitoring system based on a generative adversarial network, firstly, a first target image is acquired, and a verification dataset, a training dataset, and a test dataset are constructed based on the first target image. The first target image is a monitoring image containing a transmission tower. The verification dataset includes the first target image acquired on a rainy day, and the test dataset includes the first target image acquired on a non-rainy day and the corresponding image after rain enhancement. Then, in the input step, each second target image in the training dataset is input into the generative network to obtain a corresponding third target image. The third target image is input into the adversarial network to obtain a loss function value, and the generative network and the adversarial network are adjusted based on the loss function value using an optimization algorithm. The network is used to perform deraining processing. The second target image is the first target image acquired on a rainy day. Then, if the loss function value is less than a first threshold, a deraining model is constructed based on the generator network and the adversarial network. The peak signal-to-noise ratio (PSNR) is calculated using the deraining model on the test dataset and the validation dataset to obtain the corresponding target SNR. Then, if the target SNR corresponding to the test dataset and / or the validation dataset is less than or equal to a second threshold, the above input steps are repeated until the target SNR of both the test dataset and the validation dataset is greater than the second threshold. Finally, if the second SNR is greater than the second threshold, the deraining model is determined as the target model, and the first target image acquired on a rainy day is derained using the target model. This application uses a generator network and an adversarial network to construct a region model for image deraining. The generator network is used for region selection, and the adversarial network is used for loss function calculation. Compared to existing technologies, this application's model, while achieving deraining, allows the generator network to better restore image details and textures, solving the problem of severe distortion in existing methods for deraining single images. Attached Figure Description

[0016] Figure 1 A hardware block diagram of a mobile terminal for a single-image rain removal method for power transmission tower monitoring based on generative adversarial networks, provided in an embodiment of this application, is shown.

[0017] Figure 2 A flowchart illustrating a method for de-raining a single image of a power transmission tower monitoring system based on a generative adversarial network, according to an embodiment of this application, is shown.

[0018] Figure 3 A structural diagram of a generative network provided according to an embodiment of this application is shown;

[0019] Figure 4A structural diagram of an improved spatial attention module according to an embodiment of this application is shown;

[0020] Figure 5 A structural diagram of a multi-scale dilated convolution module provided according to an embodiment of this application is shown;

[0021] Figure 6 A structural diagram of an encoding basic module provided according to an embodiment of this application is shown;

[0022] Figure 7 A structural diagram of an adversarial network provided according to an embodiment of this application is shown;

[0023] Figure 8 An example of a surveillance image acquired in rainy weather is shown, according to an embodiment of this application.

[0024] Figure 9 The image shown is a monitoring image of a target model after rain removal, provided according to an embodiment of this application;

[0025] Figure 10 A structural block diagram of a rain removal device for a single image of a power transmission tower monitoring system based on a generative adversarial network, according to an embodiment of this application, is shown.

[0026] The above figures include the following reference numerals:

[0027] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] As described in the background section, existing methods for de-raining single images of transmission tower monitoring based on filtering have poor de-raining effects; methods based on prior models for de-raining single images of transmission tower monitoring are difficult to design suitable models to accurately separate rain layers from images under different scenarios, limiting the practical application of such methods; and deep learning-based algorithms for de-raining single images of transmission tower monitoring suffer from problems such as blurred image details or loss of texture information. To address the problem of severe distortion in existing methods for de-raining single images, embodiments of this application provide a method, apparatus, computer-readable storage medium, and monitoring system for de-raining single images of transmission tower monitoring based on generative adversarial networks.

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a single-image rain removal method for power transmission tower monitoring based on generative adversarial networks, according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] This embodiment provides a method for rain removal from a single image of a power transmission tower monitoring system based on a generative adversarial network, which runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] Figure 2 This is a flowchart of a method for de-raining a single image of a power transmission tower monitoring system based on a generative adversarial network, according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0037] Step S201: Obtain a first target image, and construct a verification dataset, a training dataset, and a test dataset based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image obtained on a rainy day. The test dataset and the training dataset include the first target image obtained on a non-rainy day and the corresponding image after rain processing.

[0038] Specifically, the first target image is obtained by acquiring a single image of the transmission tower monitoring. The first target image is then cropped to a preset size. In one embodiment, the preset size is 512*512. The first target image is then analyzed. The verification dataset is constructed based on the images collected on rainy days. The images collected on non-rainy days are processed to obtain corresponding images. Training and test datasets are constructed by matching the non-rainy images and the images after rain processing one-to-one. In one embodiment, the ratio of the number of images in the training dataset to the number of images in the test dataset is 8:2.

[0039] Step S202, Input Step, input each second target image in the above training dataset into the generator network to obtain the corresponding third target image, input the third target image into the adversarial network to obtain the loss function value, and adjust the generator network and the adversarial network based on the loss function value through an optimization algorithm. The generator network is used to perform rain removal processing. The second target image is the first target image obtained on a rainy day.

[0040] Specifically, a generative network is constructed to perform rain removal processing on a single image of a transmission tower, and an adversarial network is constructed to calculate the loss function and determine whether the input image is a rain-removed image or a truly rain-free image.

[0041] Step S203: If the loss function value is less than the first threshold, construct a rain removal model based on the generator network and the adversarial network, and calculate the peak signal-to-noise ratio based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio.

[0042] Specifically, when the loss function value is less than the first threshold, it is initially determined that the adversarial network and the generator network meet the accuracy requirements. At this time, a deraining model is constructed based on the adversarial network and the generator network, and the target model is tested using the test dataset. That is, the peak signal-to-noise ratio is obtained based on the deviation between the derained image and the image before deraining, which is used to measure the effect of the deraining process.

[0043] Step S204: If the target signal-to-noise ratio corresponding to the above test dataset and / or the above verification dataset is less than or equal to the second threshold, repeat the above input steps until the target signal-to-noise ratio of the above test dataset and the above verification dataset is greater than the second threshold.

[0044] In one embodiment, this application sets up a method to first test the trained rain removal model based on a test dataset, and then perform a verification test when the peak signal-to-noise ratio is greater than 30dB, that is, to test the rain removal model based on a verification dataset.

[0045] Step S205: When the second signal-to-noise ratio is greater than the second threshold, the rain removal model is determined as the target model, and the rain removal is performed on the first target image obtained on a rainy day using the target model.

[0046] Specifically, if both the test dataset and the validation dataset pass the tests, the target model is determined to meet the requirements, and the rain removal model is selected as the target model for region processing.

[0047] Furthermore, this application conducts tests on both the test dataset and the validation dataset. The test dataset is used to test whether the rain removal model meets the accuracy requirements. Since the test dataset and the training dataset are correlated, this application further proposes to use the validation dataset to test the region model, ensuring the accuracy of the region model on an independent dataset and thus ensuring its accuracy in practical applications.

[0048] In this embodiment, firstly, a first target image is acquired, and a verification dataset, a training dataset, and a test dataset are constructed based on the first target image. The first target image is a monitoring image containing a transmission tower. The verification dataset includes the first target image acquired on a rainy day, and the test dataset includes the first target image acquired on a non-rainy day and the corresponding image after rain enhancement. Then, in the input step, each second target image in the training dataset is input into a generator network to obtain a corresponding third target image. The third target image is input into an adversarial network to obtain a loss function value. Based on the loss function value, the generator network and the adversarial network are adjusted using an optimization algorithm. The generator network is used to perform rain removal processing. The image is the first target image acquired on a rainy day. Then, if the loss function value is less than a first threshold, a deraining model is constructed based on the generator network and the adversarial network. The peak signal-to-noise ratio (PSNR) is calculated using the deraining model on both the test dataset and the validation dataset to obtain the corresponding target SNR. Next, if the target SNR of the test dataset and / or the validation dataset is less than or equal to a second threshold, the above input steps are repeated until the target SNR of both the test dataset and the validation dataset is greater than the second threshold. Finally, if the second SNR is greater than the second threshold, the deraining model is determined as the target model, and the first target image acquired on a rainy day is derained using the target model. This application derains images by constructing a region model using a generator network and an adversarial network. Region modeling is performed based on the generator network, and loss function calculation is performed based on the adversarial network. Compared to existing technologies, this application's model, while achieving deraining, allows the generator network to better restore image details and textures, solving the problem of severe distortion in existing methods for deraining single images.

[0049] In order to process the second target image to obtain the third target image, in an optional embodiment, step S202 includes:

[0050] Step S2021: The second target image is processed by a first preset number of the above-mentioned 3×3 convolutional modules to obtain a first feature map;

[0051] In one embodiment, the structure diagram of the generator network of this application is as follows: Figure 3 As shown, the above-mentioned generative network includes a convolutional group module, a first dilated convolutional group module (dilated convolutional group module 1), a second dilated convolutional group module (dilated convolutional group module 2), a multi-scale dilated convolutional module, a spatial attention module (improved spatial attention module), an encoding base module, a 3×3 convolutional module, and an upsampling module.

[0052] Specifically, such as Figure 3 As shown, after the second target image is input into the above-mentioned generation network, two consecutive 3×3 convolutions are first used to perform preliminary feature extraction to obtain feature map T1, which is the first feature map mentioned above.

[0053] Step S2022: The first feature map is processed by the first dilated convolution group module, the multi-scale dilated convolution module, the spatial attention module and the convolution group module to obtain the second feature map.

[0054] Specifically, feature map T1 is sequentially passed through a dilated convolutional group module 1, a multi-scale dilated convolutional module, an improved spatial attention module, and a convolutional group module to obtain feature map T2, which is the second feature map mentioned above.

[0055] Step S2023: The first feature map is processed by the above-mentioned encoding basic module to obtain the third feature map, the third feature map is processed by the above-mentioned encoding basic module to obtain the fourth feature map, and the third feature map is processed by the above-mentioned encoding basic module to obtain the fifth feature map.

[0056] Specifically, feature map T1 is processed by the first basic encoding module to obtain feature map T3, which is the third feature map mentioned above. Feature map T3 is processed by the second basic encoding module to obtain feature map T4, which is the fourth feature map mentioned above. Feature map T4 is processed by the third basic encoding module to obtain feature map T5, which is the fifth feature map mentioned above.

[0057] Step S2024: The fifth feature map is processed by the second preset number of the second dilated convolution group modules and the upsampling module, and then concatenated with the fourth feature map to obtain the sixth feature map.

[0058] Specifically, feature map T5 is processed by six consecutive dilated convolutional modules 2 and an upsampling module, and then concatenated with feature map T4 to obtain feature map T6, which is the sixth feature map mentioned above.

[0059] Step S2025: The sixth feature map is processed by the 3×3 convolution module and the upsampling module, and then concatenated with the third feature map to obtain the seventh feature map.

[0060] Specifically, feature map T6 is convolved with feature map T3 after 3×3 convolution and upsampling to obtain feature map T7, which is the seventh feature map mentioned above.

[0061] Step S2026: The seventh feature map is processed by the 3×3 convolution module and the upsampling module, and then concatenated with the second feature map to obtain the eighth feature map.

[0062] Specifically, feature map T7 is convolved with feature map T2 after 3×3 convolution and upsampling to obtain feature map T8, which is the eighth feature map mentioned above.

[0063] Step S2027: The eighth feature map is processed by the 3×3 convolution module and concatenated with the second feature map. Then, the third target image is obtained by processing it through the convolution group module.

[0064] Specifically, feature map T8 is convolved with feature map T2 after 3×3 convolution, and then processed by the convolution group module to obtain the aforementioned third target image.

[0065] In order to process the first feature map to obtain the second feature map, in an optional implementation, step S2022 includes:

[0066] Step S20221: Input the first feature map into the first dilated convolution module to obtain the first target output:

[0067]

[0068] Among them, Y i For the first feature map mentioned above, cov 3,3,3 () represents a dilated convolution with a kernel of 3, a dilation rate of 3, and padding of 3. ReLU() is the activation function. For element-wise addition, Y o Output the first objective as described above;

[0069] Specifically, the first feature map is input into the dilated convolution module 1, and is processed twice by dilated convolution with a kernel of 3, a dilation rate of 3, and a padding of 3, and by the ReLU activation function. Then, it is further processed by dilated convolution with a kernel of 3, a dilation rate of 3, and a padding of 3, and added element-wise with the initial first feature map to obtain the second feature map.

[0070] Furthermore, this application employs dilated convolution, which offers several advantages: compared to conventional convolution, it increases the receptive field, facilitating the extraction of global feature information. This module utilizes a residual structure, adding features before and after convolution through skip connections, which helps mitigate the vanishing gradient problem. Simultaneously, the residual structure accelerates convergence during training, allowing the model to learn effective features for rain removal more quickly. Skip connections also preserve the initial input features, effectively transmitting information during rain removal and reducing information loss. A convolution with a dilation rate of 3, padding of 3, and a kernel of 3 has the same receptive field as a convolution with a kernel of 7 without increasing the number of parameters. Placing this module at the beginning of the network allows the network to learn the overall image features early on, better understanding the image structure and perceiving the distribution of rain patterns. Furthermore, compared to smaller convolutions, dilated convolutions can cover a wider spatial range from the outset, reducing the possibility of information loss and obtaining richer feature representations.

[0071] Step S20222: Input the first target output into the multi-scale dilated convolution module to obtain the second target output;

[0072] Specifically, the output of the first target is input into a multi-scale dilated convolution module, where features are extracted at different scales and then spliced ​​and fused to achieve more detailed feature capture.

[0073] Step S20223: Input the second target output into the spatial attention module to obtain the third target output.

[0074]

[0075] Among them, B o For the third objective output mentioned above, B i Given the input data for the spatial attention module described above, S() is the spatial attention submodule, cov1() is a 1×1 convolution, and cov... 3,1,1 () represents a 3×3 dilated convolution with a dilation rate of 1 and a fill rate of 1. For element-wise multiplication, B o2 satisfy:

[0076]

[0077] Among them, cov 3,2,2() represents a 3×3 dilated convolution with a dilation rate of 2 and a fill rate of 2. o1 satisfy:

[0078]

[0079] Among them, cov 3,3,3 () represents a 3×3 dilated convolution with a dilation rate of 3 and a padding rate of 3;

[0080] Specifically, the improved spatial attention module structure is as follows: Figure 4 As shown, the second target output is input into the improved spatial attention module, and then processed sequentially through a 1×1 convolution, a 3×3 dilated convolution with a dilation rate of 3 and a padding of 3, and a 1×1 convolution. This is then multiplied element-wise with the result of the second target output processed by the conventional spatial attention module, and finally added element-wise to the first output to obtain the first result. The first result is then processed sequentially through a 1×1 convolution, a 3×3 dilated convolution with a dilation rate of 2 and a padding of 2, and a 1×1 convolution. This is then multiplied element-wise with the result of the second target output processed by the conventional spatial attention module, and finally added element-wise to the first result to obtain the second result. The second result is then processed sequentially through a 1×1 convolution, a 3×3 dilated convolution with a dilation rate of 1 and a padding of 1, and a 1×1 convolution. This is then multiplied element-wise with the result of the second target output processed by the conventional spatial attention module, and finally added element-wise to the second result to obtain the third target output.

[0081] Furthermore, the improved spatial attention module in this application can adjust the weight of features in different regions based on their importance, making the network more focused on feature extraction from areas containing rainwater. This helps the network better focus on regions containing raindrops and patterns in the image, improving its ability to identify and process rainwater areas, thereby effectively removing rainwater interference and highlighting the features in the original image. The combination of the spatial attention module and the dilated convolution module can expand the model's perceptual field while reducing the number of parameters, achieving the fusion of global and local information, improving the model's understanding of the image context, and more effectively guiding rainwater removal, thus improving the accuracy of the rain removal model. The combination of the spatial attention module and the residual structure can compensate for their respective shortcomings. Spatial attention pays more attention to areas containing rainwater, while the residual structure, using skip connections, can promote the flow and transmission of information in the network, reducing information loss during deep propagation and ensuring the integrity of information transmission. The combination of the two allows the model to better understand the structure and content in the image, thereby effectively removing rainwater and improving the quality of image rain removal.

[0082] Step S20224: Input the above-mentioned third target output into the above-mentioned convolutional group module to obtain the above-mentioned second feature map:

[0083]

[0084] Among them, F i For the input data of the above convolutional group module, F o For the second feature map mentioned above, cov3() is a convolution with a kernel of 3.

[0085] Specifically, the third target output is input into the convolutional group module, and after being processed twice by a convolution with a kernel of 3 and a ReLU activation function, it is then processed by a convolution with a kernel of 3 and added element-wise with the third target input to obtain the second feature map.

[0086] In order to obtain the second target output based on the first target output, in an optional implementation, step S20222 includes:

[0087] Step S202221: The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 9×9 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the fourth target output.

[0088] In one embodiment, the multi-scale dilation module includes a 1×1 convolution module, a ReLU function module, a 9×9 convolution module, a batch normalization module, an 11×11 convolution module, a 7×7 convolution module, and a 13×13 convolution module, forming a total of 5 branches, such as... Figure 5 As shown.

[0089] Specifically, in the first branch, the first target output is processed sequentially by the 1×1 convolution module, the ReLU function module, the 9×9 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the fourth target output.

[0090] Step S202222: The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 11×11 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the fifth target output.

[0091] Specifically, in the second branch, the first target output is processed sequentially by the 1×1 convolution module, the ReLU function module, the 11×11 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the fifth target output.

[0092] Step S202223: The first target output is processed by the 1×1 convolution module to obtain the sixth target output;

[0093] Specifically, the third branch processes the first target output using the aforementioned 1×1 convolution module to obtain the sixth target output.

[0094] Step S202224: The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 7×7 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the seventh target output.

[0095] Specifically, in the fourth branch, the first target output is processed sequentially by the 1×1 convolution module, the ReLU function module, the 7×7 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the seventh target output.

[0096] Step S202225: The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 13×13 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the eighth target output.

[0097] Specifically, in the fifth branch, the first target output is processed sequentially by the 1×1 convolution module, the ReLU function module, the 13×13 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the eighth target output.

[0098] Step S202226: Add the fourth target output, the fifth target output, the seventh target output, and the eighth target output to the sixth target output to obtain multiple ninth target outputs;

[0099] Specifically, the output feature maps of the first, second, fourth, and fifth branches are added element-wise to the third branch to obtain feature maps under four different scale convolutions, which are the outputs of the ninth target mentioned above.

[0100] Step S202227: Concatenate all the above-mentioned ninth target data to obtain the above-mentioned second target output.

[0101] Specifically, the above four feature maps are then concatenated to obtain the second target output.

[0102] Furthermore, in the above embodiments, the multi-scale dilated convolution module extracts features using convolution kernels of different sizes at different scales, capturing detailed information at different levels and extracting richer feature information. This helps the model better understand the relationships between different regions in the image and effectively removes interference caused by rainwater. Using large convolution kernels, compared to the commonly used 3×3 convolution, can cover a larger range of pixels, capturing features in a wider area and generating deeper feature information. Simultaneously, large convolution kernels allow each subsequent convolutional layer to obtain broader contextual information, helping the model better perceive the structure between the image background and rainwater distribution. Using only 1×1 convolution branches can extract features of local details, while also reducing computational cost and increasing non-linearity, which helps the network learn features better. Finally, the features extracted by convolution kernels at different scales are added together, achieving feature fusion. This integrates information from different scales, facilitating a better understanding of the structure and texture in the image, resulting in a more natural and accurate rain removal effect.

[0103] In order to obtain the third feature map based on the first feature map, in an optional implementation, step S2023 includes:

[0104] Step S20231: Input the first feature map into the encoding base module to obtain the third feature map.

[0105] D o =Sig(cov 2,2 (ReLu(cov3(ReLu(cov3(D o1 ))))));

[0106] Where Sig() is the Sigmoid function, and cov 2,2 () represents a 2×2 convolution with a stride of 2, ReLU() is the ReLU activation function, cov3() is a 3×3 convolution, and D o1 satisfy:

[0107]

[0108] in, For element-wise multiplication, For element-wise addition, Amp() is the max pooling function, Avp() is the average pooling function, and D... i This is the input data for the above-mentioned basic encoding module.

[0109] Specifically, the basic coding module structure of the coding basic module in this application is as follows: Figure 6As shown, the first feature map is processed by max pooling, 1×1 convolution, ReLU activation function, 1×1 convolution, and Sigmoid function, and then element-wise multiplied with the first feature map to obtain the first branch output. The first feature map is then processed by average pooling, 1×1 convolution, ReLU activation function, 1×1 convolution, and Sigmoid function, and then element-wise multiplied with the first feature map to obtain the second branch output. The first branch output and the second branch output are concatenated to obtain the third branch output. The third branch output is then processed twice by 3×3 convolution and ReLU activation function, and then processed by 2×2 convolution with a stride of 2 and ReLU activation function to obtain the third feature map.

[0110] Furthermore, in the above embodiments, compared with the traditional downsampling module, the basic encoding module of this application adds an attention mechanism and uses 2×2 convolution instead of max pooling to achieve downsampling. Compared with max pooling, 2×2 convolution can learn more subtle feature representations, and parameter sharing during convolution can more smoothly reduce the image size. Therefore, using convolution instead of max pooling can better preserve the details and texture information of the image, avoiding image blurring and distortion while reducing the image size. Introducing an attention mechanism before downsampling allows the model to focus more on image features and reduce the processing of irrelevant information during the deraining process. Among them, global adaptive max pooling automatically performs max pooling operation according to the size of the input feature map, extracting the maximum value in the feature map and retaining the most significant features in the image. This allows the model to focus more on the significant features in the image, selectively retaining the texture and detail information in the image, and improving the clarity of the image after deraining. Adaptive average pooling calculates the average value of features in each region according to the size of the input feature map, averaging the information of each region, allowing the model to focus more comprehensively on the information of each region of the image, achieving a balance of global information, reducing overfitting and helping to improve the grasp of the global structure of the image. By stitching together the feature maps of these two attention mechanisms, diverse information fusion is achieved. The fused feature map simultaneously focuses on the local details and overall structure of the image, integrating the advantages of both to achieve a complementary and synergistic effect, obtaining richer feature representations, reducing artifacts and distortion in derained images, and making the deraining effect more natural and realistic.

[0111] In order to process the fifth feature map mentioned above, in an optional implementation, step S2024 includes:

[0112] Step S20241: Input the fifth feature map into the second dilated convolutional group module.

[0113]

[0114] Among them, R iFor the fifth feature map mentioned above, cov 3,2,2 () represents a dilated convolution with a kernel of 3, a dilation rate of 2, and padding of 2. ReLU() is the activation function. For element-wise addition, R o It is the output feature map of the convolutional group module.

[0115] Specifically, the fifth feature map is input into the dilated convolution module 2, and then processed twice by dilated convolution with a kernel of 3, a dilation rate of 2, and a padding of 2, followed by the ReLU activation function. Then, it is further processed by dilated convolution with a kernel of 3, a dilation rate of 2, and a padding of 2, and then element-wise added to the initial fifth feature map to obtain the target output.

[0116] Furthermore, in the above embodiments, dilated convolution, compared to conventional convolution, increases the receptive field, which is beneficial for extracting global feature information. This module uses a residual structure, propagating gradients through skip connections, which helps alleviate the gradient vanishing problem. Simultaneously, the residual structure is more conducive to optimization, and its faster convergence speed during training allows the model to learn the effective features for rain removal more quickly. Skip connections also preserve the original input features, helping to effectively transmit information during rain removal and improving the model's expressive power. A convolution with a dilation rate of 2, padding of 2, and a kernel of 3 has the same receptive field as a convolution with a kernel of 5 without increasing the number of parameters. This allows for a more global perception of raindrop distribution and image structure, while maintaining image detail resolution and avoiding image blurring and information loss.

[0117] To obtain the aforementioned loss function value, in an optional implementation, step S202 further includes:

[0118] Step S2028: The third target image is processed sequentially through a third preset number of the above-mentioned 3×3 convolution modules to obtain the ninth feature map;

[0119] In one embodiment, the above adversarial network structure is as follows: Figure 7 As shown, the adversarial network includes the ReLU function module and the aforementioned 3×3 convolution module.

[0120] Specifically, this invention designs an improved adversarial network based on the SPA_GAN adversarial network. The improved adversarial network adds a branch containing a 3×3 convolution, which is connected in parallel with the third convolutional layer of the SPA_GAN adversarial network. The output feature map of this branch is fused with the output feature map of the third convolutional layer through element-wise multiplication. This branch has three output channels. The output feature map of this branch is then subjected to mean squared error calculation with the feature map output by the improved spatial attention module in the generator network. The result is used as part of the loss function to measure the discriminator's feature extraction capability. Furthermore, the adversarial network of this application processes the aforementioned third target image using 3×3 convolution and the ReLU function to obtain the ninth feature map.

[0121] Furthermore, the adversarial network of SPA_GAN is also used to distinguish whether the input image is a rain-removed image or a rainless image, which is the second branch of the adversarial network in this application. Based on the 3×3 convolution added in this application and the output of the fourth convolution layer of the SPA_GAN adversarial network, element-wise multiplication is performed, and the multiplication is added to the input 3×3 convolution and classified through a fully connected layer to determine whether the input image is a rain-removed image or a rainless image.

[0122] Step S2029: The loss function value is obtained by calculating the mean square error based on the ninth feature map and the third target output.

[0123] Specifically, the loss function value is obtained by calculating the mean square error based on the ninth feature map and the third target output.

[0124] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the rain removal method for single images of transmission tower monitoring based on generative adversarial networks will be described in detail below with reference to specific embodiments.

[0125] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0126] In one specific embodiment, the surveillance images acquired on rainy days are as follows: Figure 8 As shown, the monitoring image of the target model after rain removal in this application is as follows: Figure 9 As shown, the model of this application can better restore the details and texture information of the image while removing rain from a single image of a transmission tower monitoring system, reduce the distortion caused during the image removal process, and improve the quality of the single image of the transmission tower monitoring system after rain removal.

[0127] This application also provides a rain removal device for a single image of a transmission tower monitoring system based on a generative adversarial network (GAN). It should be noted that this GAN-based single image rain removal device for transmission tower monitoring can be used to execute the GAN-based single image rain removal method for transmission tower monitoring provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0128] The following describes the rain removal device for single-image monitoring of power transmission towers based on generative adversarial networks provided in the embodiments of this application.

[0129] Figure 10 This is a structural block diagram of a rain removal device for a single image of a power transmission tower monitoring system based on a generative adversarial network, according to an embodiment of this application. Figure 10 As shown, the device includes:

[0130] The acquisition unit 10 is used to acquire a first target image and construct a verification dataset, a training dataset and a test dataset based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image acquired on a rainy day. The test dataset and the training dataset include the first target image acquired on a non-rainy day and the corresponding image after rain processing.

[0131] The input unit is used to perform the input step, input each second target image in the training dataset into the generator network to obtain the corresponding third target image, input the third target image into the adversarial network to obtain the loss function value, and adjust the generator network and the adversarial network based on the loss function value through an optimization algorithm. The generator network is used to perform rain removal processing. The second target image is the first target image obtained on a rainy day.

[0132] The construction unit 30 is used to construct a rain removal model based on the generator network and the adversarial network when the loss function value is less than the first threshold, and to calculate the peak signal-to-noise ratio based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio.

[0133] The repeating unit 40 is used to repeat the above input steps when the target signal-to-noise ratio corresponding to the above test dataset and / or the above verification dataset is less than or equal to the second threshold, until the target signal-to-noise ratio of the above test dataset and the above verification dataset is greater than the second threshold.

[0134] The determining unit 50 is used to determine the rain removal model as the target model when the second signal-to-noise ratio is greater than the second threshold, and to remove rain from the first target image acquired on a rainy day using the target model.

[0135] In this embodiment, the acquisition unit acquires a first target image and constructs a verification dataset, a training dataset, and a test dataset based on the first target image. The first target image is a monitoring image containing a transmission tower. The verification dataset includes the first target image acquired on a rainy day. The test dataset includes the first target image acquired on a non-rainy day and the corresponding image after rain enhancement. The input unit performs an input step, inputting each second target image from the training dataset into the generator network to obtain a corresponding third target image. The third target image is then input into the adversarial network to obtain a loss function value. Based on the loss function value, the generator network and the adversarial network are adjusted using an optimization algorithm. The generator network is used to perform rain removal processing. The image is the first target image acquired on a rainy day. The construction unit constructs a deraining model based on the generator network and the adversarial network when the loss function value is less than a first threshold, and calculates the peak signal-to-noise ratio (PSNR) based on the test dataset and the validation dataset respectively to obtain the corresponding target SNR. The repetition unit repeats the above input steps when the target SNR corresponding to the test dataset and / or the validation dataset is less than or equal to a second threshold, until the target SNR of both the test dataset and the validation dataset is greater than the second threshold. The determination unit determines the deraining model as the target model when the second SNR is greater than the second threshold, and uses the target model to derain the first target image acquired on a rainy day. This application derains images by constructing a region model using a generator network and an adversarial network. Regions are constructed based on the generator network, and the loss function is calculated based on the adversarial network. Compared with existing technologies, this application's model, while achieving deraining, allows the generator network to better restore the details and textures of the image, solving the problem of severe distortion in existing methods for deraining single images.

[0136] In order to process the second target image to obtain the third target image, in an optional embodiment, the input unit includes:

[0137] The first processing subunit is used to process the second target image through a first preset number of the above-mentioned 3×3 convolution modules to obtain a first feature map;

[0138] The second processing subunit is used to process the first feature map through the first dilated convolution group module, the multi-scale dilated convolution module, the spatial attention module and the convolution group module to obtain the second feature map.

[0139] The third processing subunit is used to process the first feature map through the above-mentioned encoding basic module to obtain a third feature map, process the third feature map through the above-mentioned encoding basic module to obtain a fourth feature map, and process the third feature map through the above-mentioned encoding basic module to obtain a fifth feature map.

[0140] The fourth processing subunit is used to process the fifth feature map by the second preset number of the second dilated convolution group modules and the upsampling module, and to concatenate it with the fourth feature map to obtain the sixth feature map.

[0141] The fifth processing subunit is used to process the sixth feature map through the 3×3 convolution module and the upsampling module, and then concatenate it with the third feature map to obtain the seventh feature map.

[0142] The sixth processing subunit is used to process the seventh feature map through the 3×3 convolution module and the upsampling module, and then concatenate it with the second feature map to obtain the eighth feature map.

[0143] The seventh processing subunit is used to process the eighth feature map through the 3×3 convolution module and concatenate it with the second feature map, and then process it through the convolution group module to obtain the third target image.

[0144] In order to process the first feature map to obtain the second feature map, in an optional embodiment, the second processing subunit includes:

[0145] The first input module is used to input the first feature map into the first dilated convolution module to obtain the first target output:

[0146]

[0147] Among them, Y i For the first feature map mentioned above, cov 3,3,3 () represents a dilated convolution with a kernel of 3, a dilation rate of 3, and padding of 3. ReLU() is the activation function. For element-wise addition, Y o Output the first objective as described above;

[0148] The second input module is used to input the first target output into the multi-scale dilated convolution module to obtain the second target output.

[0149] The third input module is used to input the second target output into the spatial attention module to obtain the third target output:

[0150]

[0151] Among them, B o For the third objective output mentioned above, B i Given the input data for the spatial attention module described above, S() is the spatial attention submodule, cov1() is a 1×1 convolution, and cov... 3,1,1 () represents a 3×3 dilated convolution with a dilation rate of 1 and a fill rate of 1. For element-wise multiplication, B o2 satisfy:

[0152]

[0153] Among them, cov 3,2,2 () represents a 3×3 dilated convolution with a dilation rate of 2 and a fill rate of 2. o1 satisfy:

[0154]

[0155] Among them, cov 3,3,3 () represents a 3×3 dilated convolution with a dilation rate of 3 and a padding rate of 3;

[0156] The fourth input module is used to input the output of the third target into the convolutional group module to obtain the second feature map.

[0157]

[0158] Among them, F i For the input data of the above convolutional group module, F o For the second feature map mentioned above, cov3() is a convolution with a kernel of 3.

[0159] In order to obtain the second target output based on the first target output, in an optional embodiment, the second input module includes:

[0160] The first processing submodule is used to process the first target output sequentially through the 1×1 convolution module, the ReLU function module, the 9×9 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the fourth target output;

[0161] The second processing submodule is used to process the first target output in sequence through the 1×1 convolution module, the ReLU function module, the 11×11 convolution module, the batch normalization module, the 1×1 convolution module and the ReLU function module to obtain the fifth target output.

[0162] The third processing submodule is used to process the first target output through the 1×1 convolution module to obtain the sixth target output;

[0163] The fourth processing submodule is used to process the first target output in sequence through the 1×1 convolution module, the ReLU function module, the 7×7 convolution module, the batch normalization module, the 1×1 convolution module and the ReLU function module to obtain the seventh target output.

[0164] The fifth processing submodule is used to process the first target output in sequence through the 1×1 convolution module, the ReLU function module, the 13×13 convolution module, the batch normalization module, the 1×1 convolution module and the ReLU function module to obtain the eighth target output.

[0165] The sixth processing submodule is used to add the fourth, fifth, seventh, and eighth target outputs to the sixth target output to obtain multiple ninth target outputs;

[0166] The seventh processing submodule is used to concatenate all the above-mentioned ninth target data to obtain the above-mentioned second target output.

[0167] In order to obtain the third feature map based on the first feature map, in an optional embodiment, the third processing subunit includes:

[0168] The fifth input module is used to input the first feature map into the encoding base module to obtain the third feature map.

[0169] D o =Sig(cov 2,2 (ReLu(cov3(ReLu(cov3(D o1 ))))));

[0170] Where Sig() is the Sigmoid function, and cov 2,2 () represents a 2×2 convolution with a stride of 2, ReLU() is the ReLU activation function, cov3() is a 3×3 convolution, and D o1 satisfy:

[0171]

[0172] in, For element-wise multiplication, For element-wise addition, Amp() is the max pooling function, Avp() is the average pooling function, and D... i This is the input data for the aforementioned basic encoding module.

[0173] In order to process the fifth feature map mentioned above, in an optional embodiment, the fourth processing subunit includes:

[0174] The sixth input module is used to input the fifth feature map mentioned above into the second dilated convolution group module mentioned above:

[0175]

[0176] Among them, R i For the fifth feature map mentioned above, cov 3,2,2 () represents a dilated convolution with a kernel of 3, a dilation rate of 2, and padding of 2. ReLU() is the activation function. For element-wise addition, R o It is the output feature map of the convolutional group module.

[0177] To obtain the aforementioned loss function value, in one optional implementation, the input unit further includes:

[0178] The eighth processing subunit is used to process the third target image by sequentially passing it through a third preset number of the above-mentioned 3×3 convolution modules to obtain the ninth feature map;

[0179] The ninth processing subunit is used to calculate the loss function value by performing mean square error calculation based on the ninth feature map and the third target output.

[0180] The aforementioned rain removal device for single-image monitoring of power transmission towers based on generative adversarial networks includes a processor and a memory. The acquisition unit, input unit, construction unit, repetition unit, and determination unit are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0181] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the accuracy of image deraining.

[0182] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0183] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the above-described method for removing rain from a single image of a power transmission tower based on a generative adversarial network.

[0184] Specifically, the rain removal method for single images of power transmission tower monitoring based on generative adversarial networks includes:

[0185] Step S201: Obtain a first target image, and construct a verification dataset, a training dataset, and a test dataset based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image obtained on a rainy day. The test dataset and the training dataset include the first target image obtained on a non-rainy day and the corresponding image after rain processing.

[0186] Step S202, Input Step, input each second target image in the above training dataset into the generator network to obtain the corresponding third target image, input the third target image into the adversarial network to obtain the loss function value, and adjust the generator network and the adversarial network based on the loss function value through an optimization algorithm. The generator network is used to perform rain removal processing. The second target image is the first target image obtained on a rainy day.

[0187] Step S203: If the loss function value is less than the first threshold, construct a rain removal model based on the generator network and the adversarial network, and calculate the peak signal-to-noise ratio based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio.

[0188] Step S204: If the target signal-to-noise ratio corresponding to the above test dataset and / or the above verification dataset is less than or equal to the second threshold, repeat the above input steps until the target signal-to-noise ratio of the above test dataset and the above verification dataset is greater than the second threshold.

[0189] Step S205: When the second signal-to-noise ratio is greater than the second threshold, the rain removal model is determined as the target model, and the rain removal is performed on the first target image obtained on a rainy day using the target model.

[0190] This invention provides a processor for running a program, wherein the program executes the above-described method for de-raining a single image of a power transmission tower monitoring system based on a generative adversarial network.

[0191] Specifically, the rain removal method for single images of power transmission tower monitoring based on generative adversarial networks includes:

[0192] Step S201: Obtain a first target image, and construct a verification dataset, a training dataset, and a test dataset based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image obtained on a rainy day. The test dataset and the training dataset include the first target image obtained on a non-rainy day and the corresponding image after rain processing.

[0193] Step S202, Input Step, input each second target image in the above training dataset into the generator network to obtain the corresponding third target image, input the third target image into the adversarial network to obtain the loss function value, and adjust the generator network and the adversarial network based on the loss function value through an optimization algorithm. The generator network is used to perform rain removal processing. The second target image is the first target image obtained on a rainy day.

[0194] Step S203: If the loss function value is less than the first threshold, construct a rain removal model based on the generator network and the adversarial network, and calculate the peak signal-to-noise ratio based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio.

[0195] Step S204: If the target signal-to-noise ratio corresponding to the above test dataset and / or the above verification dataset is less than or equal to the second threshold, repeat the above input steps until the target signal-to-noise ratio of the above test dataset and the above verification dataset is greater than the second threshold.

[0196] Step S205: When the second signal-to-noise ratio is greater than the second threshold, the rain removal model is determined as the target model, and the rain removal is performed on the first target image obtained on a rainy day using the target model.

[0197] This invention provides a monitoring system, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0198] Step S201: Obtain a first target image, and construct a verification dataset, a training dataset, and a test dataset based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image obtained on a rainy day. The test dataset and the training dataset include the first target image obtained on a non-rainy day and the corresponding image after rain processing.

[0199] Step S202, Input Step, input each second target image in the above training dataset into the generator network to obtain the corresponding third target image, input the third target image into the adversarial network to obtain the loss function value, and adjust the generator network and the adversarial network based on the loss function value through an optimization algorithm. The generator network is used to perform rain removal processing. The second target image is the first target image obtained on a rainy day.

[0200] Step S203: If the loss function value is less than the first threshold, construct a rain removal model based on the generator network and the adversarial network, and calculate the peak signal-to-noise ratio based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio.

[0201] Step S204: If the target signal-to-noise ratio corresponding to the above test dataset and / or the above verification dataset is less than or equal to the second threshold, repeat the above input steps until the target signal-to-noise ratio of the above test dataset and the above verification dataset is greater than the second threshold.

[0202] Step S205: When the second signal-to-noise ratio is greater than the second threshold, the rain removal model is determined as the target model, and the rain removal is performed on the first target image obtained on a rainy day using the target model.

[0203] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform a program that initializes a method for deraining a single image of a transmission tower monitoring system based on at least a generative adversarial network.

[0204] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0205] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0207] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0209] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0210] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0211] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0212] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0213] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0214] This application discloses a method for de-raining a single image of a power transmission tower monitoring system based on a generative adversarial network. First, a first target image is acquired, and a validation dataset, a training dataset, and a test dataset are constructed based on this first target image. The first target image is a monitoring image containing a power transmission tower. The validation dataset includes the first target image acquired on rainy days, and the test dataset includes the first target image acquired on non-rainy days and the corresponding image after rain enhancement. Then, in the input step, each second target image from the training dataset is input into a generative network to obtain a corresponding third target image. The third target image is then input into the adversarial network to obtain a loss function value. Based on the loss function value, the generative network and the adversarial network are adjusted using an optimization algorithm. The generative network is used to perform rain removal... The rain removal process involves using a generator network and an adversarial network to construct a deraining model. The second target image is the first target image acquired on a rainy day. Then, if the loss function value is less than a first threshold, a deraining model is constructed based on the generator network and the adversarial network. The peak signal-to-noise ratio (PSNR) is calculated using the deraining model on both the test and validation datasets to obtain the corresponding target SNR. Next, if the target SNR of the test and / or validation datasets is less than or equal to a second threshold, the input steps are repeated until the target SNR of both the test and validation datasets is greater than the second threshold. Finally, if the second SNR is greater than the second threshold, the deraining model is determined as the target model, and the first target image acquired on a rainy day is derained using the target model. This application uses a generator network and an adversarial network to construct a region model for image deraining. The generator network is used for region modeling, and the adversarial network is used for loss function calculation. Compared to existing technologies, this application's model, while achieving deraining, allows the generator network to better restore image details and textures, solving the problem of severe distortion in existing methods for deraining single images.

[0215] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for rain removal from a single image of a power transmission tower monitoring system based on generative adversarial networks, characterized in that, include: A first target image is acquired, and a verification dataset, a training dataset, and a test dataset are constructed based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image acquired on a rainy day. The test dataset and the training dataset both include the first target image acquired on a non-rainy day and the corresponding image after rain processing. The input step involves inputting each second target image from the training dataset into the generator network to obtain the corresponding third target image, inputting the third target image into the adversarial network to obtain the loss function value, and adjusting the generator network and the adversarial network based on the loss function value using an optimization algorithm. The generator network is used to perform rain removal processing. The second target image is the first target image obtained on a rainy day. The generator network includes a convolutional group module, a first dilated convolutional group module, a second dilated convolutional group module, a multi-scale dilated convolutional module, a spatial attention module, a coding base module, a 3×3 convolutional module, and an upsampling module. The multi-scale dilated convolutional module includes a 1×1 convolutional module, a ReLU function module, a 9×9 convolutional module, a batch normalization module, an 11×11 convolutional module, a 7×7 convolutional module, and a 13×13 convolutional module. The adversarial network includes a ReLU function module and the 3×3 convolutional module. If the loss function value is less than a first threshold, a rain removal model is constructed based on the generator network and the adversarial network, and the peak signal-to-noise ratio is calculated using the rain removal model based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio. If the target signal-to-noise ratio corresponding to the test dataset and / or the verification dataset is less than or equal to the second threshold, repeat the input step until the target signal-to-noise ratio corresponding to both the test dataset and the verification dataset is greater than the second threshold; If the target signal-to-noise ratio is greater than the second threshold, the rain removal model is determined as the target model, and the rain removal is performed on the first target image acquired on a rainy day using the target model.

2. The method according to claim 1, characterized in that, Each second target image in the training dataset is input into the generator network to obtain the corresponding third target image, including: The second target image is processed by a first preset number of the 3×3 convolution modules to obtain a first feature map; The first feature map is processed by the first dilated convolution group module, the multi-scale dilated convolution module, the spatial attention module, and the convolution group module to obtain the second feature map; The first feature map is processed by the encoding base module to obtain the third feature map, the third feature map is processed by the encoding base module to obtain the fourth feature map, and the fourth feature map is processed by the encoding base module to obtain the fifth feature map. The fifth feature map is processed by the second preset number of second dilated convolution group modules and the upsampling module, and then concatenated with the fourth feature map to obtain the sixth feature map; The sixth feature map is processed by the 3×3 convolution module and the upsampling module, and then concatenated with the third feature map to obtain the seventh feature map; The seventh feature map is processed by the 3×3 convolution module and the upsampling module, and then concatenated with the second feature map to obtain the eighth feature map; The third target image is obtained by processing the eighth feature map through the 3×3 convolution module and concatenating it with the second feature map, and then processing it through the convolution group module.

3. The method according to claim 2, characterized in that, The first feature map is processed by the first dilated convolutional group module, the multi-scale dilated convolutional module, the spatial attention module, and the convolutional group module to obtain the second feature map, including: The first feature map is input into the first dilated convolutional group module to obtain the first target output: ； in, This is the first feature map. This is a dilated convolution with a kernel of 3, a dilation rate of 3, and padding of 3. For activation function, For element addition, Output for the first target; The first target output is input into the multi-scale dilated convolution module to obtain the second target output; The second target output is input into the spatial attention module to obtain the third target output: ； in, The output for the third objective is... The input data for the spatial attention module, Spatial attention submodule, For 1×1 convolution, This is a 3×3 dilated convolution with a dilation rate of 1 and a padding rate of 1. For element-wise multiplication, satisfy: ； in, It is a 3×3 dilated convolution with a dilation rate of 2 and a padding rate of 2. satisfy: ； in, A 3×3 dilated convolution with a dilation rate of 3 and a padding rate of 3; The third target output is input into the convolutional group module to obtain the second feature map: ； in, The input data for the convolutional group module, This is the second feature map. It is a convolution with a kernel of 3.

4. The method according to claim 3, characterized in that, The first target output is input into the multi-scale dilated convolution module to obtain the second target output, including: The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 9×9 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the fourth target output; The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 11×11 convolution module, the batch normalization module, and the ReLU function module to obtain the fifth target output; The first target output is processed by the 1×1 convolution module to obtain the sixth target output; The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 7×7 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the seventh target output; The first target output is processed sequentially through the 1×1 convolution module, the ReLU function module, the 13×13 convolution module, the batch normalization module, the 1×1 convolution module, and the ReLU function module to obtain the eighth target output; Multiple ninth target outputs are obtained by adding the fourth target output, the fifth target output, the seventh target output, and the eighth target output to the sixth target output element by element. The second target output is obtained by concatenating all the outputs of the ninth target.

5. The method according to claim 2, characterized in that, The third feature map is obtained by processing the first feature map through the encoding basic module, including: The first feature map is input into the encoding base module to obtain the third feature map: ； in, For the Sigmoid function, It is a 2×2 convolution with a stride of 2. It is the ReLU activation function. It is a 3×3 convolution. satisfy: in, For element-wise multiplication, For element addition, For max pooling function, For average pooling function, This is the input data for the encoding base module.

6. The method according to claim 2, characterized in that, The fifth feature map is processed by a second preset number of second dilated convolutional modules and the upsampling module, including: The fifth feature map is input into the second dilated convolutional group module: ； in, This is the fifth feature map. This is a dilated convolution with a kernel of 3, a dilation rate of 2, and padding of 2. For activation function, For element addition, It is the output feature map of the convolutional group module.

7. The method according to claim 3, characterized in that, The third target image is input into the adversarial network to obtain the loss function value, including: The third target image is processed sequentially through a third preset number of the 3×3 convolution modules to obtain the ninth feature map; The loss function value is obtained by calculating the mean square error based on the ninth feature map and the third target output.

8. A rain removal device for a single image of a power transmission tower monitoring system based on generative adversarial networks, characterized in that, The device includes: An acquisition unit is used to acquire a first target image and construct a verification dataset, a training dataset, and a test dataset based on the first target image. The first target image is a monitoring image containing a power transmission tower. The verification dataset includes the first target image acquired on a rainy day. The test dataset includes the first target image acquired on a non-rainy day and the corresponding image after rain processing. An input unit is used to perform an input step, inputting each second target image in the training dataset into a generator network to obtain a corresponding third target image, inputting the third target image into an adversarial network to obtain a loss function value, and adjusting the generator network and the adversarial network based on the loss function value using an optimization algorithm. The generator network is used to perform rain removal processing. The second target image is the first target image obtained on a rainy day. The generator network includes a convolutional group module, a first dilated convolutional group module, a second dilated convolutional group module, a multi-scale dilated convolutional module, a spatial attention module, a coding base module, a 3×3 convolutional module, and an upsampling module. The multi-scale dilated convolutional module includes a 1×1 convolutional module, a ReLU function module, a 9×9 convolutional module, a batch normalization module, an 11×11 convolutional module, a 7×7 convolutional module, and a 13×13 convolutional module. The adversarial network includes a ReLU function module and the 3×3 convolutional module. The construction unit is used to construct a rain removal model based on the generator network and the adversarial network when the loss function value is less than a first threshold, and to calculate the peak signal-to-noise ratio based on the test dataset and the validation dataset to obtain the corresponding target signal-to-noise ratio. The repetition unit is used to repeat the input step when the target signal-to-noise ratio corresponding to the test dataset and / or the verification dataset is less than or equal to the second threshold, until the target signal-to-noise ratio corresponding to both the test dataset and the verification dataset is greater than the second threshold; The determining unit is configured to determine the rain removal model as the target model when the target signal-to-noise ratio is greater than the second threshold, and to remove rain from the first target image acquired on a rainy day using the target model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

10. A monitoring system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 7.

Images