Image processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method and device, a storage medium and a program product. The method comprises the steps of obtaining a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing an original power grid inspection image; target detection processing is conducted on the first power grid inspection image, a second power grid inspection image corresponding to the first power grid inspection image is obtained, and the second power grid inspection image comprises a binary mask corresponding to a target area recognized from the first power grid inspection image; the binary mask is processed based on an image reconstruction model obtained through pre-training, a third power grid inspection image is obtained, and the resolution ratio of the third power grid inspection image is higher than that of the first power grid inspection image. According to the method, the resolution of the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of smart grids, and in particular to an image processing method, device, storage medium, and program product. Background Art

[0002] During power grid inspections, drones often need to capture large areas of power transmission lines and related equipment. Due to the long shooting distances and high flight speeds, images often exhibit motion blur, compression noise, or insufficient resolution. These issues result in loss of detail in power facilities, transmission lines, and other key elements, hindering subsequent image analysis and fault detection.

[0003] Existing traditional image enhancement algorithms easily cause image distortion when processing low-resolution images taken by drones.

[0004] Therefore, an image processing method is needed to convert low-resolution images into high-resolution images. Summary of the Invention

[0005] Embodiments of the present application provide an image processing method, device, storage medium, and program product.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Acquire a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing the original power grid inspection image;

[0008] Performing target detection processing on the first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image, wherein the second power grid inspection image includes a binary mask corresponding to the target area identified in the first power grid inspection image;

[0009] The binary mask is processed based on a pre-trained image reconstruction model to obtain a third power grid inspection image. The resolution of the third power grid inspection image is higher than that of the first power grid inspection image.

[0010] Optionally, as in the above method, performing target detection processing on the first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image includes:

[0011] Inputting the first power grid inspection image into a target detection model to obtain a target detection result, wherein the target detection result includes a target area identified from the first power grid inspection image; the target detection model is trained based on the power grid inspection sample images collected by the drone;

[0012] The first power grid inspection image is processed based on an image segmentation algorithm and a target detection result to obtain a binary mask corresponding to the target area.

[0013] Optionally, as in the above method, the binary mask is processed based on a pre-trained image reconstruction model to obtain a third power grid inspection image, including:

[0014] Input the binary mask into the image reconstruction model;

[0015] The generator in the image reconstruction model performs upsampling on the binary mask to generate a third power grid inspection image.

[0016] Optionally, as in the above method, the training process of the image reconstruction model includes:

[0017] Acquire training sample images, where the training sample images include a first training image and a second training image corresponding to the first training image; wherein the resolution of the first training image is lower than that of the second training image;

[0018] A generator included in the image reconstruction model generates a first reconstructed image based on the first training image; a discriminator included in the image reconstruction model outputs a first probability that the first reconstructed image is a real image and a second probability that the second training image is a real image;

[0019] respectively acquiring a first image feature corresponding to the first reconstructed image and a second image feature corresponding to the second training image;

[0020] Determine a first loss function value based on the first image feature and the second image feature, and determine a second loss function value based on the first probability and the second probability;

[0021] The generator is trained based on the first probability and the first loss function value, and the discriminator is trained according to the first loss function value.

[0022] Optionally, as in the above method, the generator includes a fully connected layer, a multi-layer convolutional layer, a residual module, and an output layer; generating a first reconstructed image according to the first training image by the generator included in the image reconstruction model includes:

[0023] The fully connected layer maps the first training image to a high-dimensional image;

[0024] Each convolutional layer extracts third image features corresponding to the high-dimensional image and maps the third image features to the same spatial size as the first training image;

[0025] The residual module connects the third image features extracted by each convolutional layer to obtain the initial feature map;

[0026] The output layer generates the first reconstructed image based on the initial feature map.

[0027] Optionally, as in the above method, generating a first reconstructed image based on the initial feature map includes:

[0028] The initial feature map is mapped to obtain a feature map with the same number of channels as the initial feature map;

[0029] The output layer maps the feature map to a preset pixel interval based on the activation function to generate a first reconstructed image.

[0030] Optionally, as in the above method, the method further includes:

[0031] Obtain the potential code distribution corresponding to the image reconstruction model;

[0032] Determining a third loss function value based on pixel values of the first training image and pixel values of the first reconstructed image, and determining a fourth loss function value based on a potential coding distribution and a standard normal distribution;

[0033] The image reconstruction model is optimized according to the first loss function value, the third loss function value, and the fourth loss function value.

[0034] Optionally, as in the above method, preprocessing the original power grid inspection image includes:

[0035] The original power grid inspection image is processed in sequence with noise removal, image enhancement, region cropping and marking, image normalization and data enhancement.

[0036] In a second aspect, an embodiment of the present application provides an image processing device, comprising:

[0037] An acquisition module is used to acquire a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing the original power grid inspection image;

[0038] a first obtaining module, configured to perform target detection processing on the first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image, wherein the second power grid inspection image includes a binary mask corresponding to the target area identified in the first power grid inspection image;

[0039] The second obtaining module is used to process the binary mask based on the pre-trained image reconstruction model to obtain a third power grid inspection image, where the resolution of the third power grid inspection image is higher than that of the first power grid inspection image.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0041] Memory stores computer-executable instructions;

[0042] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0045] The embodiments of the present application provide an image processing method, device, storage medium, and program product, which obtains a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing the original power grid inspection image; the first power grid inspection image is subjected to target detection processing to obtain a second power grid inspection image corresponding to the first power grid inspection image, wherein the second power grid inspection image includes a binary mask corresponding to the target area identified from the first power grid inspection image; the binary mask is processed based on a pre-trained image reconstruction model to obtain a third power grid inspection image, and the resolution of the third power grid inspection image is higher than that of the first power grid inspection image. Through target detection processing, a second power grid inspection image corresponding to the first power grid inspection image can be obtained, and through the second power grid map, the image reconstruction model is made to focus more on the part that needs to be repaired in resolution, thereby making the resolution of the third power grid inspection image higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A flowchart of an image processing method provided in this application;

[0048] Figure 2 A scene diagram of a drone inspecting a target area provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of a discriminator provided in an embodiment of the present application;

[0050] Figure 4 A jump connection structure diagram provided in an embodiment of the present application;

[0051] Figure 5A flowchart of another image processing method provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the electronic device structure provided in an embodiment of the present application.

[0054] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present application. Rather, they are merely examples of methods and approaches consistent with certain aspects of the present application, as detailed in the appended claims.

[0056] Currently, drones often need to capture large areas of power transmission lines and related equipment during power grid inspections. Due to the long shooting distance and high flight speed, images often exhibit motion blur, compression noise, or insufficient resolution. These issues result in the loss of details of power facilities, transmission lines, and other key elements in the image, affecting subsequent image analysis and fault detection. Existing image enhancement technologies, such as sharpening and denoising, can improve the clarity of certain areas, but their effectiveness in recovering details in complex power grid scenarios is still limited. Especially when high-resolution images are required, traditional methods have difficulty recovering tiny details in power grid images, affecting the accuracy and effectiveness of the detection system.

[0057] The present application proposes an image processing method, which obtains a first power grid inspection map of a target inspection area, and then performs target detection processing on the low-resolution first power grid inspection image to obtain a second inspection map corresponding to the first power grid inspection image. The second power grid inspection map is a binary mask of the target area. The second power grid inspection map is used to focus more on the target area, thereby improving the resolution restoration effect of tiny details of the power grid image. The binary mask is then processed by a pre-trained image reconstruction model to obtain a high-resolution third power grid inspection image.

[0058] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 1 A flowchart of an image processing method provided in this application is shown in FIG. Figure 1 As shown, the method includes:

[0060] S101 , obtaining a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing an original power grid inspection image.

[0061] The target inspection area includes areas with power grid equipment, transmission lines and other important equipment.

[0062] Preprocessing includes noise removal, image enhancement, region cropping and labeling, image normalization, and data enhancement.

[0063] Figure 2 The embodiment of the present application provides a scene diagram of a drone inspecting a target area, such as Figure 2 As shown, the drone inspects the target inspection area and transmits the collected images (i.e., the power grid inspection map) to the system in real time.

[0064] S102 : Perform target detection processing on the first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image, wherein the second power grid inspection image includes a binary mask corresponding to the target area identified from the first power grid inspection image.

[0065] In computer science and image processing, a binary mask is a tool used to selectively manipulate data. It is typically a binary array or matrix of the same size as the target data, where each element is either 0 or 1. The primary function of a binary mask is to selectively retain or ignore portions of the data through bitwise operations.

[0066] Target detection processing may refer to the process of processing the first power grid inspection image to obtain a binary mask corresponding to the target area. Specifically, the first power grid inspection image may be input into a YOLOv5 model, and then pixel-wise segmentation of the YOLOv5 model output may be performed using a segmentation algorithm, such as a Mask Region-Based Convolutional Neural Network (Mask R-CNN) algorithm, to obtain a second power grid inspection image.

[0067] In an embodiment of the present application, target detection processing is performed on a first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image, including:

[0068] The first power grid inspection image is input into a target detection model to obtain a target detection result, which includes a target area identified from the first power grid inspection image; the target detection model is trained based on power grid inspection sample images collected by the drone.

[0069] The first power grid inspection image is processed based on an image segmentation algorithm and a target detection result to obtain a binary mask corresponding to the target area.

[0070] The object detection model can be a YOLOv5 model. This invention uses YOLOv5 as the object detection model because it can provide high-precision detection results and real-time performance in power grid inspection image processing. YOLOv5's structure is based on a convolutional neural network, which can efficiently process input images of power grid equipment and transmission lines and quickly generate corresponding detection boxes and category information.

[0071] The YOLOv5 model consists of a backbone network, a neck network, and a head network. The initial YOLOv5 model is trained using a large number of power grid inspection sample images collected by drones and manually labeled. When the YOLOv5 model's loss function falls below a certain threshold, the model is considered successful. The loss function is used in machine learning and analytical models to measure the difference between the model's predictions and actual results. It is the core optimization objective during model training, helping to guide the adjustment of model parameters to improve prediction accuracy.

[0072] The loss function used by the YOLOv5 model usually includes positioning loss, confidence loss, and category loss. By adding the three functions, when the sum is less than the preset threshold, it can be determined that the YOLOv5 model has converged.

[0073]

[0074] in, is the loss function of the YOLOv5 model, is the positioning loss, is the confidence loss, is the category loss.

[0075] Specifically, YOLOv5 divides the input image (i.e., the first power inspection image) into an S×S grid and predicts a fixed number of bounding boxes, confidence scores, and class probabilities for each grid. Through this process, YOLOv5 can accurately identify power grid equipment, transmission lines, and other target areas (representing other important areas required in specific scenarios). Even in complex power grid environments, it can quickly and accurately locate target areas (such as power grid equipment, transmission lines, and other important areas) and generate detection boxes (i.e., object detection results).

[0076] To further precisely extract the target region, a segmentation algorithm like Mask R-CNN can be used to perform pixel-level segmentation on the target detection results. Mask R-CNN adds a branch to the target detection results to output a binary mask corresponding to the target region, thereby achieving precise target region segmentation. For each detection box, Mask R-CNN generates a binary mask of the same size as the input image (i.e., the target region), accurately representing the target region of the power grid equipment or transmission line.

[0077] The advantage of this setting is that this technical solution can effectively improve the efficiency and accuracy of power grid inspections by combining target detection and image segmentation technology, and has significant technical advantages and application value.

[0078] S103 : Processing the binary mask based on a pre-trained image reconstruction model to obtain a third power grid inspection image, where the resolution of the third power grid inspection image is higher than that of the first power grid inspection image.

[0079] The image reconstruction model is a mathematical model used to reconstruct details of the binary mask to generate a third power grid inspection image with higher resolution than the first power grid inspection image. The binary mask is the portion of the first power grid inspection image that represents power grid equipment and transmission lines.

[0080] In the embodiment of the present application, the process of processing the binary mask based on the pre-trained image reconstruction model to obtain the third power grid inspection image includes:

[0081] The binary mask is input into the image reconstruction model. The generator in the image reconstruction model performs upsampling on the binary mask to generate a third power grid inspection image.

[0082] Among them, upsampling is a technology commonly used in image processing and signal processing, which is mainly used to increase the resolution or size of data.

[0083] The image reconstruction model consists of two parts: a generator and a discriminator. The generator's task is to convert low-resolution power grid images into high-resolution images through upsampling, restoring detailed information within the images, particularly those of power grid equipment, transmission lines, and their surroundings. The discriminator analyzes the authenticity of the generated images and guides the generator's continuous optimization by comparing them with true high-resolution images. Through this adversarial training, the image reconstruction model is able to improve image resolution while maintaining image detail and structural consistency, ensuring accurate restoration of key components of power grid equipment and transmission lines.

[0084] In one example, the image reconstruction model can be an SRGAN (Super-Resolution Generative Adversarial Network) model. The SRGAN model reconstructs the details of the identified power grid equipment and transmission line areas (i.e., binary masks) to generate high-resolution images. The SRGAN model is suitable for repairing blurred or low-resolution power grid inspection images.

[0085] The image containing the binary mask is input into the SRGAN model. The generator in the SRGAN model converts the low-resolution power grid image into a high-resolution image through upsampling, restoring the detailed information in the image, especially the details of the power grid equipment, transmission lines and their surrounding environment (i.e., the binary mask part).

[0086] Specifically, the self-attention mechanism can be used to assign higher weights to areas that require key attention (such as grid equipment, transmission lines, and details of their surroundings), and lower weights to other areas, thereby focusing more on the areas that require key attention.

[0087] Specifically, by introducing a self-attention mechanism, the model can more accurately capture the global information in the power grid image (i.e., the image containing the binary mask), improving the accuracy of image restoration and the quality of detail recovery. This is crucial for image restoration affected by occlusion, low resolution, or other complex environmental factors, further improving the image sharpening effect.

[0088] Query, key, and value matrices: The query matrix Q, key matrix K, and value matrix V are extracted from the input X through a learned linear transformation.

[0089]

[0090] in, 、 and Different weight matrices.

[0091] The attention score is calculated as follows:

[0092]

[0093] in, is the dimension of the key vector, the softmax function ensures that the sum of the attention scores is 1, the query matrix Q, the key matrix, A is the attention score (that is, the attention score of each pixel in the binary mask), and T is the transposed matrix.

[0094] The self-attention mechanism weights the input feature map, enabling the model to prioritize features related to repaired areas during generation. Specifically, the feature map of the input image (i.e., an image containing a binary mask) is mapped into a query matrix Q, a key matrix K, and a value matrix V, and then an attention score A is calculated. This mechanism enables the model to focus on contextual information related to repaired areas of power grid equipment and transmission lines, thereby improving the quality of the generated images.

[0095] The benefit of this setup is that processing binary masks based on a pre-trained image reconstruction model can significantly improve the quality and analysis capabilities of power grid inspection images, providing more reliable data support for power grid maintenance and management.

[0096] In an embodiment of the present application, the training process of the image reconstruction model includes:

[0097] A training sample image is obtained, where the training sample image includes a first training image and a second training image corresponding to the first training image; wherein the resolution of the first training image is lower than that of the second training image.

[0098] A first reconstructed image is generated based on the first training image by a generator included in the image reconstruction model; a first probability of judging the first reconstructed image as a real image and a second probability of judging the second training image as a real image are outputted by a discriminator included in the image reconstruction model.

[0099] A first image feature corresponding to the first reconstructed image and a second image feature corresponding to the second training image are respectively obtained.

[0100] A first loss function value is determined based on the first image feature and the second image feature, and a second loss function value is determined based on the first probability and the second probability.

[0101] The generator is trained based on the first loss function value and the second loss function value, and the discriminator is trained according to the first loss function value.

[0102] Among them, the real image representation itself is a high-resolution image rather than an image generated by the generator.

[0103] The first training image is a low-resolution power grid inspection image, and the second training image is a high-resolution power grid inspection image.

[0104] The generator generates a high-resolution first reconstructed image based on the first training image. The first reconstructed image and the second training image are then input into the discriminator. The discriminator outputs a first probability that the first reconstructed image is a true high-resolution image and a probability that the second training image is a true high-resolution image. The discriminator also feeds back the probability of the generated pseudo-high-resolution image to the generator, which then adjusts its parameters for optimization based on this feedback and the perceptual loss.

[0105] Figure 3 A schematic diagram of the structure of a discriminator provided in an embodiment of the present application is shown in FIG. Figure 3 As shown in the figure, a 128×128 image is input into the discriminator, processed by four convolutional layers (i.e., 4×4 convolution kernels), and then the target area is focused on through the self-attention layer (self-attention mechanism). The result is then output to the convolution layer with a channel of 1 to obtain a true or false result. The true representation is a real high-resolution image, and the false representation is a generated high-resolution image.

[0106] The first loss function can be a perceptual loss function, which is mainly used to analyze the visual quality of the generated image. It uses a pre-trained deep convolutional neural network to extract the features of the real high-resolution image and the generated high-resolution image and calculate the difference between them. The specific steps are:

[0107] Use a layer (such as conv3_3 layer) of the VGG (Visual Geometry Group) network to extract feature representations of the real image (i.e., the second training image) and the generated image (i.e., the first reconstructed image).

[0108] Calculate the mean squared error (MSE) between feature representations:

[0109]

[0110] in, represents the perceived loss, represents the real image features (i.e., the second training image), Represents the features of the generated image (i.e., the first reconstructed image), N is the feature dimension, and in addition, the perceptual loss, adversarial loss, and pixel loss can be weightedly summed to obtain the first loss function, and then the generator is trained according to the first loss function value, and the discriminator is trained according to the first loss function.

[0111] Perceptual loss is used to analyze the quality of the first reconstructed image. When the perceptual loss exceeds a preset threshold, it indicates a significant discrepancy and requires further training of the generator and discriminator. The model training process alternates between training the generator and discriminator, calculating their perceptual losses separately. The Adam optimizer is then used to adjust the learning rate and update the model parameters.

[0112] In another embodiment, for the discriminator, the following adversarial loss is used for training:

[0113]

[0114] in, is the adversarial loss of the generator, z is the random noise of the input, is the probability distribution of noise z, The samples generated by the generator G, The probability that the input to the discriminator D is judged to be real data.

[0115] The second loss function is defined as binary cross entropy loss:

[0116]

[0117] in, represents the real image (i.e. the second training image), represents the generated image (i.e. the first reconstructed image), Express expectations, Represents the generated image expectations, is the probability output of the real image (i.e. the first probability), is the probability output of generating an image (i.e., the second probability).

[0118] The second loss function value can be used to determine whether the generator has been trained. When the difference between the first probability and the second probability is less than the preset threshold, that is, the binary cross entropy loss is less than the preset threshold (a smaller difference), it indicates that the first reconstructed image generated by the generator (that is, the generated high-resolution image) and the second training image (that is, the original high-resolution image) are not much different, and the discriminator cannot recognize them, which means that a trained generator is obtained.

[0119] The advantage of this setting is that this technical solution can effectively improve the quality of low-resolution images through the collaborative work of the generator and discriminator of the generative adversarial network, providing strong technical support for applications that require high-resolution images.

[0120] In an embodiment of the present application, the generator includes a fully connected layer, a multi-layer convolutional layer, a residual module, and an output layer; generating a first reconstructed image based on the first training image by the generator included in the image reconstruction model includes:

[0121] The fully connected layer maps the first training image to a high-dimensional image.

[0122] Each convolutional layer extracts third image features corresponding to the high-dimensional image and maps the third image features to the same spatial size as the first training image.

[0123] The residual module connects the third image features extracted by each convolutional layer to obtain the initial feature map.

[0124] The output layer generates the first reconstructed image based on the initial feature map.

[0125] Among them, the first training image is mapped to a low-dimensional space to obtain a latent vector. After receiving the latent vector, the generator first maps the latent vector to an appropriate high-dimensional feature space through a fully connected layer to obtain a high-dimensional image, which helps to expand the low-dimensional representation of the latent vector into a high-dimensional feature representation suitable for convolution operations.

[0126] The convolution layer extracts the third image features corresponding to the high-dimensional image. Here, the activation function Rectified Linear Unit (ReLU) is used to extract the third image features of the high-dimensional image:

[0127]

[0128] in, is the convolution kernel, is the bias term, is the ReLU activation function, Represents the learning rate, the parameter of the learning rate manually set, Represents the output function value.

[0129] Following the fully connected layers are a series of convolutional transpose layers (i.e., convolutional layers) for feature expansion and upsampling. These convolutional layers, through a series of deconvolution operations, expand the feature maps to the same spatial dimensions as the original image (i.e., the first training image). Simultaneously, a series of ResUp modules implement feature fusion and augmentation. Using skip connections, features from different levels are integrated, increasing the network's expressive power and the richness of image restoration. The concept of skip connections is similar to the design of residual modules. Their primary purpose is to establish direct connections between different levels of the generator, allowing low-level features to be directly transferred to higher levels, thereby enhancing the network's gradient flow and information transfer.

[0130] Figure 4 A jump connection structure diagram provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the output X of the L-1 layer of the neural network can be connected to the output L(x) of the L layer of the neural network.

[0131] The residual module formula is as follows:

[0132]

[0133] in, Represents the residual learning part of the residual module, which usually includes convolutional layers, batch normalization, activation functions, etc. It represents the result of performing residual connection on the output of each layer of neural network.

[0134] The third image features extracted by each convolutional layer are connected through the residual module.

[0135] Finally, the output layer of the generator network is responsible for converting the feature maps after feature expansion and fusion into the final inpainted image. This output layer typically consists of a convolutional layer, which converts the feature maps into feature maps with the same number of channels as the original image (i.e., the first training image). The convolution kernel size is typically 3x3, with a stride of 1 and padding of 1 to ensure that the output image has the same dimensions as the input image. The Tanh activation function is connected below, which maps the pixel values in the output feature map to the range [-1, 1], thereby producing the first reconstructed image.

[0136] The advantage of this setting is that the generator structure achieves high-quality image reconstruction through the synergy of fully connected layers, convolutional layers, residual modules and output layers, and has good application prospects.

[0137] In an embodiment of the present application, generating a first reconstructed image based on the initial feature map includes:

[0138] The initial feature map is mapped to obtain a feature map with the same number of channels as the initial feature map.

[0139] The output layer maps the feature map to a preset pixel interval based on the activation function to generate a first reconstructed image.

[0140] The output layer typically consists of a convolutional layer, which converts the feature map into a feature map with the same number of channels as the original image (i.e., the first training image). The convolution kernel size is typically 3x3, with a stride of 1 and padding of 1 to ensure that the output image has the same dimensions as the input image.

[0141] The activation function is a Tanh activation function, and the preset pixel interval can be: [-1, 1]. The Tanh activation function maps the pixel values in the initial feature map to the range of [-1, 1], thereby obtaining a first reconstructed image.

[0142] The Tanh activation function formula is as follows:

[0143]

[0144] Where x is the pixel value in the initial feature map.

[0145] The advantage of this setting is that the design of the output layer not only improves the quality and stability of the reconstructed image, but also enhances the adaptability and performance of the model in different application scenarios.

[0146] In an embodiment of the present application, the method further includes:

[0147] Get the latent code distribution corresponding to the image reconstruction model.

[0148] A third loss function value is determined based on the pixel values of the first training image and the pixel values of the first reconstructed image, and a fourth loss function value is determined based on the latent code distribution and the standard normal distribution.

[0149] The image reconstruction model is optimized according to the first loss function value, the third loss function value, and the fourth loss function value.

[0150] Among them, the latent code distribution is often associated with generative models (i.e., generators). It involves encoding the data to capture the probability distribution of its latent structure or characteristics.

[0151] The first loss function value is the perceptual loss function.

[0152] The third loss function value is the reconstruction loss function, and the fourth loss function value is the KL divergence loss.

[0153] The reconstruction loss is used to measure the pixel-level difference between the generated image and the target image. It acts between the encoder and the generator to help the generator generate an image as close as possible to the original input image (i.e., the first reconstructed image). Specifically, the MSE loss function is used, and the formula is as follows:

[0154]

[0155] in, is the reconstruction loss, y is the target image (i.e., the second training image), is the generated image (i.e., the first reconstructed image), and N is the number of samples. During model training, we hope to minimize this loss function, prompting the generator to restore the missing parts of the input image to a value as close to the original image as possible. During model training, we hope to minimize this loss function, prompting the generator to restore the missing parts of the input image to a value as close to the original image as possible.

[0156] Finally, the KL divergence loss is calculated as follows:

[0157]

[0158] in, is the divergence loss, is the mean of the latent code, The standard deviation of the latent code is and the standard deviation is respectively, and J is the dimension of the latent code.

[0159] After training, this loss function is minimized and the encoder generates a latent code close to a standard normal distribution.

[0160] The final loss function is the sum of the three:

[0161]

[0162] in, and Weight coefficient, is the reconstruction loss, is the divergence loss, The image reconstruction model is optimized based on the values of the first, third, and fourth loss functions. Specifically, the loss function provides a feedback mechanism for the model, helping it understand its performance in the prediction task. Higher loss values generally indicate that the model needs adjustment, while lower loss values indicate that the model's predictions are more accurate.

[0163] The advantage of this setting is that the model can strike a good balance between reconstructed image quality and model stability, thus performing well in practical applications.

[0164] The embodiments of the present application provide an image processing method, device, storage medium, and program product, which obtains a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing the original power grid inspection image; the first power grid inspection image is subjected to target detection processing to obtain a second power grid inspection image corresponding to the first power grid inspection image, wherein the second power grid inspection image includes a binary mask corresponding to the target area identified from the first power grid inspection image; the binary mask is processed based on a pre-trained image reconstruction model to obtain a third power grid inspection image, and the resolution of the third power grid inspection image is higher than that of the first power grid inspection image. Through target detection processing, a second power grid inspection image corresponding to the first power grid inspection image can be obtained, and through the second power grid map, the image reconstruction model is made to focus more on the part that needs to be repaired in resolution, thereby making the resolution of the third power grid inspection image higher.

[0165] In the above embodiment, preprocessing the original power grid inspection image includes:

[0166] The original power grid inspection image is processed in sequence with noise removal, image enhancement, region cropping and marking, image normalization and data enhancement.

[0167] Noise removal refers to the fact that images of power grid equipment and transmission lines captured by drones are often accompanied by various noises (such as Gaussian noise and salt and pepper noise). This noise can significantly affect image clarity and subsequent feature extraction. Therefore, noise removal is first performed on the input low-resolution image. Various filtering techniques can be used. For example, Gaussian filtering reduces image noise and smoothes the image through convolution. Gaussian filtering effectively reduces high-frequency noise while preserving edge information of power grid equipment and transmission lines. Median filtering is suitable for removing salt and pepper noise, smoothing the image by replacing each pixel value with the median value of its neighboring pixels.

[0168] Image enhancement is crucial for improving image contrast and detail, especially in low-resolution or blurry power grid images. Histogram equalization adjusts the image's brightness distribution to increase contrast, making details of power grid equipment and transmission lines more visible. This technology is particularly effective for images captured in low-light conditions, effectively improving visual quality and laying a solid foundation for subsequent super-resolution reconstruction.

[0169] Region cropping and labeling involves using image processing techniques to crop and label regions of power grid images during processing, ensuring that subsequent processing focuses on key areas of power grid equipment and transmission lines. This process involves first identifying the edges of power grid equipment and transmission lines in the image using methods such as Canny Edge Detection (Canny Edge Detection) to precisely locate the target area. Edge detection effectively separates power grid equipment from the background, improving the efficiency of subsequent recognition and image reconstruction. Next, segmentation algorithms (such as threshold-based segmentation or K-means clustering) are used to further separate the power grid equipment from the background, providing a cleaner input image for super-resolution reconstruction.

[0170] Image normalization refers to normalizing the processed image so that the pixel values are between 0 and 1 in order to improve the training effect of the model. This step helps to accelerate the convergence of the super-resolution model, reduce instability during training, and thus improve the accuracy of image restoration.

[0171] Data augmentation refers to increasing data diversity through data augmentation techniques, as training data may be relatively limited. For example, random rotation, flipping, and scaling of original images can be used to generate more diverse training samples. Furthermore, random noise can be added to the images to enhance the model's robustness, enabling it to perform better when processing real-world power grid inspection images.

[0172] The original power grid inspection image is preprocessed to obtain a first power grid inspection image.

[0173] The benefit of this setup is that through these processing steps, the quality and availability of power grid inspection images are significantly improved, providing a solid foundation for subsequent automated analysis and fault detection.

[0174] Figure 5 A flow chart of another image processing method provided in this application is shown as follows: Figure 5 As shown, the method includes:

[0175] S501: Data preprocessing.

[0176] The data preprocessing method involves acquiring an initial grid inspection map using a drone, then preprocessing the initial map to obtain a first grid inspection map. This preprocessing includes noise removal, image enhancement, region cropping and labeling, image normalization, and data enhancement.

[0177] S502: Region identification and segmentation.

[0178] Among them, the method of region recognition and segmentation is:

[0179] The YOLOv5 model can quickly identify key targets in the first power grid inspection map (such as power grid equipment, transmission lines, and other important areas) and generate corresponding detection frames to help accurately locate the target in the power grid area.

[0180] Use a segmentation algorithm such as the Mask R-CNN algorithm to perform pixel-level segmentation on the detected targets. For each detection box, Mask R-CNN generates a binary mask of the same size as the detection box, accurately representing the target area of the power grid equipment or transmission line, and obtains the second power grid inspection map.

[0181] S503: Super-resolution reconstruction.

[0182] The super-resolution reconstruction method involves using a specific implementation of the SRGAN model in the super-resolution module to reconstruct details of the identified power grid equipment and transmission line areas, generating a third power grid inspection map. This model, combining CNN and GAN, can generate high-resolution images, making it particularly suitable for repairing blurred or low-resolution power grid images during power grid inspections.

[0183] SRGAN consists of two parts: a generator and a discriminator. The generator's task is to convert low-resolution power grid images into high-resolution images through upsampling, restoring detailed information within the images, particularly those of power grid equipment, transmission lines, and their surroundings. The discriminator analyzes the authenticity of the generated images and, by comparing them with true high-resolution images, guides the generator's continuous optimization. Through this adversarial training, SRGAN is able to improve image resolution while maintaining image detail and structural consistency, ensuring accurate restoration of key components of power grid equipment and transmission lines.

[0184] By applying the SRGAN model, it is possible to provide clearer and more accurate grid images in complex power grid environments, especially in images affected by occlusion or low light conditions, providing more refined image input for subsequent image analysis, fault detection, and target recognition, thereby improving the recognition capability and accuracy of the power grid inspection system in drone image processing.

[0185] Among them, the SRGAN model training process is:

[0186] (1) Initialize the generator and discriminator, and load the pre-trained VGG network for perceptual loss calculation.

[0187] (2) Configure optimizers for the generator and discriminator respectively.

[0188] (3) Calculate the losses of the generator and discriminator, including pixel loss, perceptual loss, and adversarial loss.

[0189] (4) Calculate the discriminator loss based on the results of generated and real images and update its parameters.

[0190] (5) Update the generator parameters based on the discriminator feedback and perceptual loss.

[0191] (6) Regularly reduce the learning rate to improve training stability, clip the gradient, and prevent gradient explosion.

[0192] (7) Use the validation set to calculate PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) and track the training progress.

[0193] At the same time, the Adam optimizer is used to train the generator and discriminator of the generative adversarial network. The Adam method is based on the adaptive adjustment of first-order and second-order moment estimates, and provides an effective method for handling non-stationary objectives and sparse gradients. The Adam optimizer was chosen in the experiment mainly because it can efficiently find suitable parameters in high-dimensional parameter space, and has good robustness to learning rate. At the same time, the Adam optimizer shows a relatively stable effect.

[0194] Another image processing method provided by the present application can effectively restore details in long-distance or low-resolution images, clearly presenting details of power facilities and transmission lines. This provides more accurate data support for subsequent fault detection and equipment monitoring.

[0195] Figure 6 This is a schematic diagram of the structure of the image processing device provided by this application, such as Figure 6 As shown, the image processing device 60 provided in this embodiment includes:

[0196] The acquisition module 601 is configured to acquire a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing an original power grid inspection image.

[0197] The first obtaining module 602 is configured to perform target detection processing on the first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image, wherein the second power grid inspection image includes a binary mask corresponding to the target area identified from the first power grid inspection image.

[0198] The second obtaining module 603 is configured to process the binary mask based on a pre-trained image reconstruction model to obtain a third power grid inspection image, where the resolution of the third power grid inspection image is higher than that of the first power grid inspection image.

[0199] In a possible implementation, the acquisition module 601 may be used to:

[0200] The original power grid inspection image is processed in sequence with noise removal, image enhancement, region cropping and marking, image normalization and data enhancement.

[0201] In a possible implementation, the first obtaining module 602 may be configured to:

[0202] The first power grid inspection image is input into a target detection model to obtain a target detection result, which includes a target area identified from the first power grid inspection image; the target detection model is trained based on power grid inspection sample images collected by the drone.

[0203] The first power grid inspection image is processed based on an image segmentation algorithm and a target detection result to obtain a binary mask corresponding to the target area.

[0204] In a possible implementation, the second obtaining module 603 may be configured to:

[0205] The binary mask is input to the image reconstruction model.

[0206] The generator in the image reconstruction model performs upsampling on the binary mask to generate a third power grid inspection image.

[0207] In a possible implementation, the second obtaining module 603 may also be used to:

[0208] A training sample image is obtained, where the training sample image includes a first training image and a second training image corresponding to the first training image; wherein the resolution of the first training image is lower than that of the second training image.

[0209] A first reconstructed image is generated based on the first training image by a generator included in the image reconstruction model; a first probability of judging the first reconstructed image as a real image and a second probability of judging the second training image as a real image are outputted by a discriminator included in the image reconstruction model.

[0210] A first image feature corresponding to the first reconstructed image and a second image feature corresponding to the second training image are respectively obtained.

[0211] A first loss function value is determined based on the first image feature and the second image feature, and a second loss function value is determined based on the first probability and the second probability.

[0212] The generator is trained based on the first loss function value and the second loss function value, and the discriminator is trained according to the first loss function value.

[0213] In a possible implementation, the second obtaining module 603 may also be used to:

[0214] The fully connected layer maps the first training image to a high-dimensional image.

[0215] Each convolutional layer extracts third image features corresponding to the high-dimensional image and maps the third image features to the same spatial size as the first training image.

[0216] The residual module connects the third image features extracted by each convolutional layer to obtain the initial feature map.

[0217] The output layer generates the first reconstructed image based on the initial feature map.

[0218] In a possible implementation, the second obtaining module 603 may also be used to:

[0219] The initial feature map is mapped to obtain a feature map with the same number of channels as the initial feature map.

[0220] The output layer maps the feature map to a preset pixel interval based on the activation function to generate a first reconstructed image.

[0221] In a possible implementation, the second obtaining module 603 may also be used to:

[0222] Get the latent code distribution corresponding to the image reconstruction model.

[0223] A third loss function value is determined based on the pixel values of the first training image and the pixel values of the first reconstructed image, and a fourth loss function value is determined based on the latent code distribution and the standard normal distribution.

[0224] The image reconstruction model is optimized according to the first loss function value, the third loss function value, and the fourth loss function value.

[0225] The present embodiment provides a method that can execute the method provided in the above method embodiment, and its implementation principle and technical effects are similar, so this embodiment will not be described in detail here.

[0226] Figure 7 This is a hardware diagram of the device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0227] During the specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that the at least one processor 701 performs the above method.

[0228] The specific implementation process of the processor 701 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0229] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0230] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0231] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0232] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0233] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0234] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0235] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0236] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via an interface, method, or unit, and may be electrical, mechanical, or otherwise.

[0237] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0238] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0239] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0240] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0241] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first power grid inspection image of a target inspection area; the first power grid inspection image is obtained by preprocessing an original power grid inspection image; performing target detection processing on the first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image, wherein the second power grid inspection image includes a binary mask corresponding to the target area identified in the first power grid inspection image; The binary mask is processed based on a pre-trained image reconstruction model to obtain a third power grid inspection image, where the resolution of the third power grid inspection image is higher than that of the first power grid inspection image.

2. The method according to claim 1, characterized in that The performing target detection processing on the first power grid inspection image to obtain a second power grid inspection image corresponding to the first power grid inspection image includes: Inputting the first power grid inspection image into a target detection model to obtain a target detection result, wherein the target detection result includes a target area identified from the first power grid inspection image; the target detection model is trained based on power grid inspection sample images collected by the drone; The first power grid inspection image is processed based on an image segmentation algorithm and the target detection result to obtain the binary mask corresponding to the target area.

3. The method according to claim 1, characterized in that The process of processing the binary mask based on the pre-trained image reconstruction model to obtain the third power grid inspection image includes: inputting the binary mask into the image reconstruction model; The generator in the image reconstruction model performs upsampling processing on the binary mask to generate the third power grid inspection image.

4. The method according to claim 1, wherein The training process of the image reconstruction model includes: Acquire training sample images, where the training sample images include a first training image and a second training image corresponding to the first training image; wherein the resolution of the first training image is lower than that of the second training image; Generating a first reconstructed image based on the first training image by a generator included in the image reconstruction model; outputting a first probability of judging the first reconstructed image to be a real image and a second probability of judging the second training image to be a real image by a discriminator included in the image reconstruction model; respectively acquiring first image features corresponding to the first reconstructed image and second image features corresponding to the second training image; Determining a first loss function value based on the first image feature and the second image feature, and determining a second loss function value based on the first probability and the second probability; The generator is trained based on the first loss function value and the second loss function value, and the discriminator is trained according to the first loss function value.

5. The method according to claim 4, characterized in that The generator includes a fully connected layer, a multi-layer convolutional layer, a residual module and an output layer; the generator included in the image reconstruction model generates a first reconstructed image according to the first training image, including: The fully connected layer maps the first training image into a high-dimensional image; Each of the convolutional layers extracts a third image feature corresponding to the high-dimensional image, and maps the third image feature to the same spatial size as the first training image; The residual module connects the third image features extracted by each convolutional layer to obtain an initial feature map; The output layer generates the first reconstructed image based on the initial feature map.

6. The method according to claim 5, characterized in that Generating the first reconstructed image based on the initial feature map includes: Mapping the initial feature map to obtain a feature map having the same number of channels as the initial feature map; The output layer maps the feature map to a preset pixel interval based on an activation function to generate the first reconstructed image.

7. The method according to claim 4, characterized in that The method further comprises: Obtaining a potential code distribution corresponding to the image reconstruction model; Determining a third loss function value based on pixel values of the first training image and pixel values of the first reconstructed image, and determining a fourth loss function value based on the latent code distribution and a standard normal distribution; The image reconstruction model is optimized according to the first loss function value, the third loss function value and the fourth loss function value.

8. The method according to any one of claims 1 to 7, characterized in that The preprocessing of the original power grid inspection image includes: The original power grid inspection image is subjected to noise removal processing, image enhancement processing, region cropping and marking processing, image normalization processing and data enhancement processing in sequence.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.

Images