Visible light image processing method and device for power equipment

Abstract

The invention discloses a visible light image processing method and device for power equipment, and the method comprises the steps: collecting visible light images of defects of the power equipment over the years, and generating a first visible light image data set of the defects of the power equipment; freezing parameters of a key layer of the Stable diffusion model, and performing fine adjustment on the pre-trained Stable diffusion model by using an LoRA algorithm to obtain an LoRA diffusion model; expanding the first visible light image data set by using the LoRA diffusion model to obtain a second visible light image data set; training a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image data set; converting actual monitoring video data acquired on site into an image sequence; and based on a trained YOLOv8 target detection model and a ViT defect classification model, identifying and classifying the image sequence. Through the modular design, the training overhead and time are greatly reduced, the effectiveness of data expansion is improved, the model performance is improved, and the generalization ability is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of combining power equipment inspection with artificial intelligence technology, and in particular, relates to a visible light image processing method and device for power equipment. Background Art

[0002] With the rapid development of power systems, the number and complexity of power equipment are increasing. The operating status and safety of power equipment are crucial to the stable operation of power systems. Traditional power equipment inspections mainly rely on manual inspections and infrared imaging technology. These methods have problems such as low detection efficiency, susceptibility to environmental influences, and insufficient detection accuracy. The application of video imaging technology in power equipment defect detection is one of the core technologies for realizing smart power operation and maintenance. It can automatically diagnose faults, provide early warnings, and perform online maintenance on power systems, intelligently identify external defects, thereby optimizing human resource allocation and improving the efficiency of intelligent inspections, effectively supplementing the deficiencies of traditional power transmission and transformation equipment inspection and operation and maintenance models.

[0003] The application of image recognition technology in power operation and maintenance has attracted widespread attention. Automated detection, fault diagnosis and maintenance of power equipment are essential to improving the reliability and efficiency of power systems. The combination of computer vision and deep learning models provides a powerful tool to achieve this goal. Power equipment fault detection is mainly divided into two stages: 1. Target detection stage; 2. Fault classification stage.

[0004] Target detection locates all targets in an image and determines their categories through a model. This task can automatically frame the equipment area or search for the location where the equipment abnormality occurs in the monitoring image of the power transmission and transformation equipment to achieve intelligent fault detection. Traditional target detection algorithms include detection algorithms based on the Histogram of Oriented Gradients (HOG) and the Deformable Part Model (DPM) algorithm. Detection algorithms based on deep learning include the Region-based Convolutional Neural Network (RCNN) and the Faster RCNN algorithm. In addition, in order to speed up the operation of the network, researchers have proposed a one-stage detector represented by SSD and YOLO. In recent years, anchor-free algorithms have become a new hot spot, mainly including DenseBox, CornerNet, ExtremeNet, FSAF, FCOS and FoveaBox.

[0005] Fault classification mainly classifies whether the target equipment identified in the first stage has a fault and what kind of fault it has. Image classification technology is used. Traditional image classification methods include Bayesian method, Fisher classifier, support vector machine, boosting method, etc. The classification theory of these methods is relatively complete, but the ability to extract semantic information inside the image is poor, and the accuracy needs to be improved. The image classification method based on neural network constructs a network structure connected by neurons, obtains internal features of the image during training, and completes the classification task. Early neural networks mainly include back propagation (BP) network, Hopfield neural network and adaptive resonance theory (ART). Although this type of network has a simple structure, convenient training and strong fitting ability, it has problems such as slow learning speed, easy to fall into local extreme values, and gradient disappearance or gradient explosion. In recent years, deep neural networks have performed well in image classification tasks. In 2107, the deep convolutional neural network proposed by Krizhevsky et al. improved the accuracy of neural networks in classification tasks. Residual networks (Resnet), squeeze-excitation networks (Senet), etc. optimize the model architecture in network structure design.

[0006] By monitoring and analyzing equipment images, predictive maintenance and reasonable maintenance plans can be achieved. This not only improves the reliability and life of the equipment, but also reduces maintenance costs and power outage time. However, image recognition technology still faces a series of challenges in the automated detection of power equipment. Factors such as complex background, diverse samples, and image quality may affect recognition accuracy. Summary of the invention

[0007] In view of the defects in the above-mentioned prior art, the present invention provides a visible light image processing method of electric power equipment, comprising the following steps: Step S101, collecting visible light images of defects of the power equipment over the years to generate a first visible light image data set of defects of the power equipment; Step S103: Select a pre-trained Stable diffusion model, ensure that the input and output formats of the Stable diffusion model are compatible with the visible light image data of the defects of the power equipment, freeze the parameters of the key layers of the Stable diffusion model, and use the LoRA algorithm to fine-tune the pre-trained Stable diffusion model to obtain the LoRA diffusion model; Step S105: using the LoRA diffusion model to expand the first visible light image data set to obtain a second visible light image data set; Step S107: training a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image data set; Step S109, converting the actual surveillance video data collected on site into an image sequence; Step S1011: Based on the trained YOLOv8 target detection model and ViT defect classification model, the image sequence is identified and classified.

[0008] It is assumed that the first visible light image dataset is , i=1,…N, where represents the i-th power equipment defect image sample. The goal of the Stable diffusion model is to learn a conditional probability distribution , are the parameters of the pre-trained Stable diffusion model, where represents the original image, and represents the final noise image obtained after T steps of noise diffusion process. The training process of the Stable diffusion model can be expressed as: The noise generation process is ,in is standard Gaussian noise, which means the noise variable It follows a multivariate normal distribution, where represents normal distribution, 0 is the mean vector, that is, the expected value of each element is 0, I represents the covariance unit matrix, is the noise level at step t, t represents the current time step in the noise transfer process, ; The noise transfer process is , which is described as Transfer to The conditional probability distribution of , where I represents the covariance identity matrix; The denoising process is , where T represents the maximum time step of the entire denoising process, and It is a parameterized function learned by the neural network, and its specific form is as follows: , ,in Represents a convolutional neural network structure used to transform the final noisy image The original defect image is predicted The mean and diagonal covariance matrix of ; The loss function is , where E represents the expected function, U{1,T} represents uniform random sampling of an integer from the integer interval [1,T], and the purpose of the loss function is to take the expectation on the training set and random time steps to learn a prediction function that can well reconstruct the original clean image .

[0009] Among them, the parameters of the key layer of the frozen Stable diffusion model are fine-tuned on the pre-trained Stable diffusion model using the LoRA algorithm to obtain the LoRA diffusion model including: Step a: Load the pre-trained Stable diffusion model; Step b: Freeze most of the parameters of the pre-trained model; Step c: insert the LoRA adaptation layer between the key layers and initialize the parameters of these layers; Step d: Use the power equipment defect image dataset to train the LoRA adaptation layer; Step e: After the training is completed, the LoRA adaptation layer and the pre-trained model are merged to obtain the final LoRA diffusion model.

[0010] Wherein, the step b includes: unfreezing the parameters of the U-Net decoder and fine-tuning them through the LoRA adaptation layer.

[0011] Taking the Stable diffusion model as an example, its backbone network is usually composed of multiple convolutional layers, attention layers, etc. These layers can be regarded as key layers. The LoRA adaptation layer can be inserted between these key layers. The specific steps are as follows: Determine where you need to insert the LoRA layer, usually between the key layers in the encoder and decoder parts of the StableDiffusion model.

[0012] Define the structure of the LoRA layer, which consists of two small linear layers, one for calculating the rank-1 update matrix A and the other for calculating B.

[0013] Assuming the weight matrix of the original key layer is W, the output of the LoRA layer is ,in is the scaling factor.

[0014] Initialize the parameters of the LoRA layer. For the matrix A, a normal distribution can be used for initialization, and the standard deviation can be set to 0.02.

[0015] For matrix B, a normal distribution can be used for initialization, and the standard deviation can be set to 0.02.

[0016] Scaling Factor alpha can be initialized to 1.0.

[0017] Freeze the backbone network. During the training process, keep the backbone network weights of the Stable diffusion model unchanged and only update the parameters of the inserted LoRA layer.

[0018] In the Stable diffusion model, the key layers of the encoder and decoder parts usually refer to the following layers: Encoder key layers: InputEncodingBlock, DownsamplingEncodingBlock, TransformerEncoderBlock.

[0019] Decoder key layers: Transformer decoder block (TransformerDecoderBlock), upsampling decoding block (UpsamplingDecoderBlock), output decoding block (OutputDecodingBlock).

[0020] These layers play a key role in the overall architecture of the Stable diffusion model. They are responsible for core functions such as image encoding, feature extraction, spatial attention calculation, and image reconstruction.

[0021] Wherein, the step c comprises: For parameters that need fine-tuning, a low-rank LoRA adaptation layer is inserted into them; For a linear layer , LoRA will decompose it into ,in , , is the parameter of the pre-trained model, r is the low-rank hyperparameter; Randomly initialize the parameters of A and B, and set W0 to the parameters of the pre-trained model.

[0022] Wherein, the step d comprises: Combine the pre-trained model with the LoRA adaptation layer to form new model parameters , and trained on a dataset of defective images of power equipment; During the training process, update the parameters A and B of the LoRA adaptation layer and keep the parameter W0 of the pre-trained model unchanged; The following loss function is minimized during training: [A1], where is the loss function for a specific task, is the first visible light image dataset, Indicates that in the entire data set The loss function of the pair Take the expected average, x represents the input visible light image of the power equipment defect, y is its corresponding label, indicating the fault type of the equipment or other related information; is the output of the model, which is the input image prediction results.

[0023] Wherein, the step S105 further includes: further expanding the sample quantity and diversity of the first visible light image data set by using data enhancement technology; The generated new image is merged with the first visible light image dataset to form a second visible light image dataset.

[0024] Wherein, the step S107 includes: Using the YOLOv8 model as a basis, training is performed on the second visible light image dataset; Optimize the hyperparameters of the YOLOv8 model to achieve the best object detection performance on the second visible light image dataset; After training is completed, save the trained YOLOv8 object detection model.

[0025] Wherein, the step S107 further includes: Prepare training data by using the labeled power equipment defect information in the second visible light image data set as a supervisory signal; Use ViT as the base model and train it on the second visible light image dataset; Optimize the hyperparameters of the ViT model to achieve the best defect classification performance on the second visible light image dataset; After the training is completed, save the trained ViT defect classification model.

[0026] The present invention also proposes a visible light image processing device for electric power equipment, comprising: A historical information collection module, which is used to collect visible light images of defects of the electric power equipment over the years and generate a first visible light image data set of defects of the electric power equipment; A LoRA diffusion model construction module is used to select a pre-trained Stable diffusion model, ensure that the input and output formats of the Stable diffusion model are compatible with the visible light image data of defects in power equipment, freeze the parameters of the key layers of the Stable diffusion model, and use the LoRA algorithm to fine-tune the pre-trained Stable diffusion model to obtain the LoRA diffusion model; A sample expansion module, configured to expand the first visible light image data set using the LoRA diffusion model to obtain a second visible light image data set; A model training module, which is used to train a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image dataset; A field acquisition module, which is used to convert the actual surveillance video data acquired on site into an image sequence; The image processing module is used to identify and classify the image sequence based on the trained YOLOv8 target detection model and ViT defect classification model.

[0027] Compared with the prior art, the present invention has the following advantages: Through LoRA technology, only a small number of additional parameters need to be trained to quickly adapt the pre-trained StableDiffusion model to the task of generating defect images of power equipment, which greatly reduces training costs and time.

[0028] The new images generated using the LoRA diffusion model have good similarity and diversity with the original first visible light image dataset. This helps to train a more powerful YOLOv8 object detection model and ViT defect classification model.

[0029] Training the YOLOv8 and ViT models based on the expanded second visible light image dataset can make full use of richer data, thereby improving the performance of the model in the task of defect identification of power equipment.

[0030] Through LoRA fine-tuning and data expansion, our obtained model has better generalization ability for power equipment defect images and can better adapt to actual application scenarios.

[0031] This method decouples the LoRA diffusion model, YOLOv8 target detection model, and ViT defect classification model, making the entire system scalable and maintainable. In the future, each module can be easily replaced or optimized according to different needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 is a flow chart showing a visible light image processing method for electric power equipment according to an embodiment of the present invention; Figure 2 is a schematic diagram showing detection and positioning of an electric device by an electric device detection model according to an embodiment of the present invention; Figure 3 is a physical diagram showing an insulator according to an embodiment of the present invention; Figure 4 is a schematic diagram showing the pollution of the insulator and the damage of the dial housing according to an embodiment of the present invention; Figure 5 It is a schematic diagram showing insulator contamination, abnormal box door closure, and meter fuzziness according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0035] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0036] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0037] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0038] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0039] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] Embodiment 1 like Figure 1 As shown, the present invention discloses a visible light image processing and evaluation method for electric power equipment, comprising the following steps: Step S101, collecting visible light images of defects of the power equipment over the years to generate a first visible light image data set of defects of the power equipment; Step S103, select a pre-trained Stable diffusion model, ensure that the input and output formats of the Stable diffusion model are compatible with the visible light image data of defects in power equipment, freeze the parameters of the key layers of the Stable diffusion model, and use the LoRA algorithm to fine-tune the pre-trained Stable diffusion model to obtain the LoRA diffusion model.

[0041] The diffusion model is a deep learning framework based on a generative probability model. Its core idea is to simulate a process of slowly reducing noise. It can be used to generate various types of data, such as images, text, audio, etc. The diffusion model learns the distribution of data by constructing a process of "slowly increasing noise" and then reversely "slowly denoising" to generate data.

[0042] This process can be viewed as a gradual transition from a simple noise distribution (such as a Gaussian distribution) to the target data distribution. By establishing the noise transition probability of this gradual process, the diffusion model can learn and approximate complex data distributions.

[0043] The core components of the model include: a noise generator, which produces an initial Gaussian noise sample; The noise shifter predicts the probability of the next noise shift based on the current noise level; A denoising generator that predicts an approximation of the original clean data given the current noise level.

[0044] The training process is as follows: Randomly sample a clean data sample from the training data; Through the noise generator, Gaussian noise is added to the sample to form a series of noise samples; Feed the noise transfer device to learn the noise transfer probability at each step; The final noisy sample is fed into the denoising generator to predict the original clean data.

[0045] The reconstruction error is used to optimize the entire diffusion model.

[0046] When generating new data, the model starts with an initial Gaussian noise and gradually passes through a noise shifter and a denoising generator, slowly removing the noise and eventually generating samples that are close to the target distribution.

[0047] Step S105: using the LoRA diffusion model to expand the first visible light image data set to obtain a second visible light image data set; Step S107: training a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image data set; Step S109, converting the actual surveillance video data collected on site into an image sequence; Step S1011: Based on the trained YOLOv8 target detection model and ViT defect classification model, the image sequence is identified and classified.

[0048] Embodiment 2 The present invention provides a visible light image processing method for electric power equipment, comprising the following steps: Step S101, collecting visible light images of defects of the power equipment over the years to generate a first visible light image data set of defects of the power equipment; Step S103: Select a pre-trained Stable diffusion model, ensure that the input and output formats of the Stable diffusion model are compatible with the visible light image data of the defects of the power equipment, freeze the parameters of the key layers of the Stable diffusion model, and use the LoRA algorithm to fine-tune the pre-trained Stable diffusion model to obtain the LoRA diffusion model; Step S105: using the LoRA diffusion model to expand the first visible light image data set to obtain a second visible light image data set; Step S107: training a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image data set; Step S109, converting the actual surveillance video data collected on site into an image sequence; Step S1011: Based on the trained YOLOv8 target detection model and ViT defect classification model, the image sequence is identified and classified.

[0049] Assume that the first visible light image dataset is , i=1,…N, where represents the i-th power equipment defect image sample. The goal of the Stable diffusion model is to learn a conditional probability distribution , are the parameters of the pre-trained Stable diffusion model, where represents the original image, and represents the final noise image obtained after T steps of noise diffusion process. The training process of the Stable diffusion model can be expressed as: The noise generation process is ,in is standard Gaussian noise, which means the noise variable It follows a multivariate normal distribution, where represents normal distribution, 0 is the mean vector, that is, the expected value of each element is 0, I represents the covariance unit matrix, is the noise level at step t, t represents the current time step in the noise transfer process, ; The noise transfer process is , which is described as Transfer to The conditional probability distribution of , where I represents the covariance identity matrix; The denoising process is , where T represents the maximum time step of the entire denoising process, and It is a parameterized function learned by the neural network, and its specific form is as follows: , ,in Represents a convolutional neural network structure used to transform the final noisy image The original defect image is predicted The mean and diagonal covariance matrix of ; The loss function is , where E represents the expected function, U{1,T} represents uniform random sampling of an integer from the integer interval [1,T], and the purpose of the loss function is to take the expectation on the training set and random time steps to learn a prediction function that can well reconstruct the original clean image .

[0050] Among them, the parameters of the key layer of the frozen Stable diffusion model are fine-tuned on the pre-trained Stable diffusion model using the LoRA algorithm to obtain the LoRA diffusion model including: Step a: Load the pre-trained Stable diffusion model.

[0051] You need to download or load a pre-trained StableDiffusion model, which is usually provided as a PyTorch model. You can use huggingface's transformers library to load pre-trained models. For example, fromtransformersimportStableDiffusionPipeline pipe=StableDiffusionPipeline.from_pretrained("runwayml / stable-diffusion-v1-5").

[0052] Step b: Freeze most of the parameters of the pre-trained model.

[0053] In order to fine-tune the model efficiently, we usually freeze most of the parameters of the pre-trained model and only update the parameters of a few layers. You can use the requires_grad_ attribute of PyTorch to freeze the model parameters. For example, forparaminpipe.model.parameters(): param.requires_grad=False. This can greatly reduce the number of parameters that need to be trained, thereby improving the efficiency of fine-tuning.

[0054] Step c: insert the LoRA adaptation layer between the key layers and initialize the parameters of these layers.

[0055] Among them, LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that inserts low-rank adaptation layers between key layers of the pre-trained model. The parameters of the adaptation layer are updated during the training process to adapt to specific tasks. The LoRA library can be used to insert and initialize these adaptation layers. For example, frompeftimportLoraConfig, get_peft_model lora_config=LoraConfig(r=4, lora_alpha=32, target_modules=[, q,,,k,,,v,], lora_dropout=0.1) pipe=get_peft_model(pipe, lora_config).

[0056] Step d: Use the power equipment defect image dataset to train the LoRA adaptation layer.

[0057] Among them, prepare the power equipment defect image dataset and convert it into a format acceptable to the StableDiffusion model. Use PyTorch's DataLoader to load the dataset and update the parameters of the LoRA adaptation layer in the training loop. For example, optimizer=torch.optim.AdamW(pipe.parameters(),lr=3e-4) forepochinrange(num_epochs): forbatchintrain_dataloader: loss=pipe(batch).loss loss.backward() optimizer.step() optimizer.zero_grad().

[0058] Step e: After the training is completed, the LoRA adaptation layer and the pre-trained model are merged to obtain the final LoRA diffusion model.

[0059] After the training is completed, the LoRA adaptation layer needs to be merged with the pre-trained StableDiffusion model to form the final LoRA diffusion model. You can use the export_onnx function provided by the LoRA library to export the model in ONNX format. For example: pipe = pipe.eval() pipe.export_onnx("lora_diffusion_model.onnx").

[0060] Wherein, the step b includes: unfreezing the parameters of the U-Net decoder and fine-tuning them through the LoRA adaptation layer.

[0061] In the StableDiffusion model, the parameters of the following parts are usually frozen: The Encoder part is responsible for encoding the input image into a potential feature representation, usually using a pre-trained Vision Transformer (ViT) or Convolutional Neural Network (CNN) as the Encoder. The parameters of this part are usually frozen because they have been pre-trained on large-scale datasets and can extract general visual features.

[0062] The U-Net part is responsible for generating the final image from the latent feature representation. The downsampling part of the U-Net (Encoder part) is usually frozen because they have also learned general visual feature extraction capabilities.

[0063] The TimeEmbedding part is responsible for embedding the time step into the feature representation. The parameters of this part are usually frozen because they have learned the time-dependent features during the pre-training process.

[0064] In the Cross-Attention part, the decoder of U-Net uses the Cross-Attention mechanism to fuse visual features and text features. The parameters of this part are often frozen because they have learned the ability of cross-modal interaction during pre-training.

[0065] Usually only the parameters of the U-Net decoder (up-sampling part) are unfrozen and fine-tuned through the LoRA adaptation layer. This can maximize the capabilities of the pre-trained model while being able to quickly adapt to new target tasks.

[0066] Wherein, the step c comprises: For parameters that need fine-tuning, a low-rank LoRA adaptation layer is inserted into them; For a linear layer , LoRA will decompose it into ,in , , is the parameter of the pre-trained model, r is the low-rank hyperparameter; Randomly initialize the parameters of A and B, and set W0 to the parameters of the pre-trained model.

[0067] Wherein, the step d comprises: Combine the pre-trained model with the LoRA adaptation layer to form new model parameters , and trained on a dataset of defective images of power equipment; During the training process, update the parameters A and B of the LoRA adaptation layer and keep the parameter W0 of the pre-trained model unchanged; The following loss function is minimized during training: [A2], where is the loss function for a specific task, is the first visible light image dataset, Indicates that in the entire data set The loss function of the pair Take the expected average, x represents the input visible light image of the power equipment defect, y is its corresponding label, indicating the fault type of the equipment or other related information; is the output of the model, which is the input image prediction results.

[0068] Wherein, the step S105 further includes: further expanding the sample quantity and diversity of the first visible light image data set by using data enhancement technology; The generated new image is merged with the first visible light image dataset to form a second visible light image dataset.

[0069] Wherein, the step S107 includes: Using the YOLOv8 model as a basis, training is performed on the second visible light image dataset; Optimize the hyperparameters of the YOLOv8 model to achieve the best object detection performance on the second visible light image dataset; After training is completed, save the trained YOLOv8 object detection model.

[0070] Wherein, the step S107 further includes: Prepare training data by using the labeled power equipment defect information in the second visible light image data set as a supervisory signal; Use ViT as the base model and train it on the second visible light image dataset; Optimize the hyperparameters of the ViT model to achieve the best defect classification performance on the second visible light image dataset; After the training is completed, save the trained ViT defect classification model.

[0071] like Figure 2 As shown in the figure, the power device detection model detects and locates various power devices. Insulators and box doors of different sizes, distances, and partial obstructions are effectively detected. Taking the insulator as an example, each device is intercepted for unique identification, such as Figure 3 As shown in Figure 2, it is used for subsequent defect classification and target positioning. Figure 4 , 5 As shown, there are schematic diagrams of dirty insulators and damaged dial casing, and dirty insulators, abnormal door closure, and blurred meters.

[0072] Embodiment 3 In another embodiment of the present invention, a diffusion model is used to generate defective / normal samples so as to balance the samples and avoid the overfitting problem caused by sample resampling.

[0073] Among them, the diffusion model is a generative model based on Markov chains. It can be used in visual generation tasks, such as image super-resolution, deblurring, JPEG artifact removal, shadow removal, defogging / haze / rain removal, etc. The diffusion model is divided into a forward (diffusion) process and an inverse process. The forward process gradually adds pixel-by-pixel noise to the image until the image satisfies Gaussian noise; the inverse process reconstructs the image by denoising.

[0074] In the forward diffusion process, noise is gradually added to the image until it becomes pure noise. This process can be represented as a Markov chain:

[0075] in is the training data point, To gradually add noise to the data, are predefined parameters.

[0076] In the reverse generation process, pure noise is gradually denoised to restore the target image:

[0077] in , , .

[0078] The LoRA method is used to fine-tune the diffusion model to generate images of a specific style. During the fine-tuning process, the diffusion model is fine-tuned by LoRA using existing defect samples so that it can generate high-quality defect images. The fine-tuning steps are as follows: Step 2.1. Sample defect image: Step 2.2. Generate time steps: Step 2.3. Add random noise: Step 2.4. Use the model to predict noise: Step 2.5. Calculate the loss function: Sampling and cropping: Oversample or crop small objects to increase the proportion of small objects in the training data and improve the model's attention to small objects.

[0079] Rotation, scaling, and flipping: Perform data augmentation operations such as rotation, scaling, and flipping on small objects to increase the diversity of small objects and improve the generalization ability of the model.

[0080] The Vision Transformer network is used to build a defect classification model for each type of power device. The deep learning algorithm classifies power device defects and optimizes the accuracy and robustness of detection.

[0081] VisionTransformer (ViT) is a Transformer-based image classification model that divides images into fixed-size patches and uses the Transformer network architecture to process these patches as input.

[0082] Step 4.1. Split the input image into fixed-size tiles, each of which has a size of (C, P, P), where C is the number of image channels.

[0083] Step 4.2. Patch Embedding,Each image patch is flattened and embedded through a linear layer (fully connected layer) to generate a vector of fixed dimension.

[0084] Step 4.3. Position encoding. Since the Transformer model itself does not have the ability to process the order of elements in a sequence, it is necessary to add position encoding to the embedding vector of each image block to preserve the spatial information of the image block.

[0085] Step 4.4. Transformer encoding, the embedding vector sequence is fed into a series of Transformer encoder layers. Each encoder layer consists of two main parts: a multi-head self-attention mechanism and a feed-forward neural network. The self-attention mechanism allows the model to understand the relationship between image blocks, while the feed-forward network performs further nonlinear transformations on the features of each image block.

[0086] Step 4.5. Classification, the output of the last Transformer encoder is fed into a classification head, usually a simple feed-forward network, to generate the final classification result. In the ViT model, an additional learnable embedding vector ([CLS] token) is usually used to represent the global information of the entire image, and the output of this embedding vector is used for the classification task.

[0087] Multi-head self-attention mechanism: Calculate the self-attention representation of each tile embedding. , , , W is the linear transformation matrix.

[0088]

[0089] Feedforward network: Each Transformer layer also contains a feedforward network consisting of two fully connected layers and an activation function.

[0090]

[0091] Layer Normalization and Residual Connections: Each sublayer (self-attention mechanism and feed-forward network) is followed by layer normalization and residual connections.

[0092]

[0093] .

[0094] Embodiment 4: The present invention also proposes a visible light image processing device for electric power equipment, which comprises: A historical information collection module, which is used to collect visible light images of defects of the electric power equipment over the years and generate a first visible light image data set of defects of the electric power equipment; A LoRA diffusion model construction module is used to select a pre-trained Stable diffusion model, ensure that the input and output formats of the Stable diffusion model are compatible with the visible light image data of defects in power equipment, freeze the parameters of the key layers of the Stable diffusion model, and use the LoRA algorithm to fine-tune the pre-trained Stable diffusion model to obtain the LoRA diffusion model; A sample expansion module, configured to expand the first visible light image data set using the LoRA diffusion model to obtain a second visible light image data set; A model training module, which is used to train a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image dataset; A field acquisition module, which is used to convert the actual surveillance video data acquired on site into an image sequence; The image processing module is used to identify and classify the image sequence based on the trained YOLOv8 target detection model and ViT defect classification model.

[0095] Embodiment 5 An embodiment of the present disclosure provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method steps described in the above embodiment.

[0096] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0097] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0098] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (AN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0099] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0101] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A visible light image processing method for electric power equipment, characterized in that: The following steps are involved: Step S101, collecting visible light images of defects of the power equipment over the years to generate a first visible light image data set of defects of the power equipment; Step S103, select a pre-trained Stable diffusion model, ensure that the input and output formats of the Stable diffusion model are compatible with the visible light image data of the defects of the power equipment, freeze the parameters of the key layers of the Stable diffusion model, use the LoRA algorithm to fine-tune the pre-trained Stable diffusion model, and obtain the LoRA diffusion model, wherein the key layers include the encoder key layer and the decoder key layer; Step S105: using the LoRA diffusion model to expand the first visible light image data set to obtain a second visible light image data set; Step S107: training a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image data set; Step S109, converting the actual surveillance video data collected on site into an image sequence; Step S1011: Based on the trained YOLOv8 target detection model and ViT defect classification model, the image sequence is identified and classified.

2. The method according to claim 1, characterized in that: Assume that the first visible light image dataset is , i=1,…N, where represents the i-th power equipment defect image sample. The goal of the Stable diffusion model is to learn a conditional probability distribution , are the parameters of the pre-trained Stable diffusion model, where represents the original image, and represents the final noise image obtained after T steps of noise diffusion process. The training process of the Stable diffusion model can be expressed as: The noise generation process is ,in is standard Gaussian noise, which means the noise variable It follows a multivariate normal distribution, where represents normal distribution, 0 is the mean vector, that is, the expected value of each element is 0, I represents the covariance unit matrix, is the noise level at step t, t represents the current time step in the noise transfer process, ; The noise transfer process is , which is described as Transfer to The conditional probability distribution of , where I represents the covariance identity matrix; The denoising process is , where T represents the maximum time step of the entire denoising process, and It is a parameterized function learned by the neural network, and its specific form is as follows: , ,in Represents a convolutional neural network structure used to transform the final noisy image The original defect image is predicted The mean and diagonal covariance matrix of ; The loss function is , where E represents the expected function, U{1,T} represents uniform random sampling of an integer from the integer interval [1,T], and the purpose of the loss function is to take the expectation on the training set and random time steps to learn a prediction function that can well reconstruct the original clean image .

3. The method according to claim 1, characterized in that: The parameters of the key layers of the frozen Stable diffusion model are fine-tuned using the LoRA algorithm to obtain the pre-trained Stable diffusion model, including: Step a: Load the pre-trained Stable diffusion model; Step b, freeze the parameters of the key layers of the pre-trained model; Step c, inserting a LoRA adaptation layer between the key layers, and initializing the parameters of the LoRA adaptation layer; Step d: Use the power equipment defect image dataset to train the LoRA adaptation layer; Step e: After the training is completed, the LoRA adaptation layer and the pre-trained model are merged to obtain the final LoRA diffusion model.

4. The method according to claim 3, characterized in that: The step b also includes: unfreezing the parameters of the U-Net decoder.

5. The method according to claim 4, characterized in that: The step c comprises: For parameters that need fine-tuning, a low-rank LoRA adaptation layer is inserted into them; For a linear layer , LoRA will decompose it into ,in , , is the parameter of the pre-trained model, r is the low-rank hyperparameter; Randomly initialize the parameters of A and B, and set W0 to the parameters of the pre-trained model.

6. The method according to claim 5, characterized in that The step d comprises: Combine the pre-trained model with the LoRA adaptation layer to form new model parameters , and trained on a dataset of defective images of power equipment; During the training process, update the parameters A and B of the LoRA adaptation layer and keep the parameter W0 of the pre-trained model unchanged; The following loss function is minimized during training: [A1], where is the loss function for a specific task, is the first visible light image dataset, Indicates that in the entire data set The loss function of the pair Take the expected average, x represents the input visible light image of the power equipment defect, y is its corresponding label, indicating the fault type of the equipment or other related information; is the output of the model, which is the input image prediction results.

7. The method according to claim 1, characterized in that: The step S105 further includes: further expanding the number and diversity of samples of the first visible light image data set by using data enhancement technology; The generated new image is merged with the first visible light image dataset to form a second visible light image dataset.

8. The method according to claim 1, characterized in that: The step S107 includes: Using the YOLOv8 model as a basis, training is performed on the second visible light image dataset; Optimize the hyperparameters of the YOLOv8 model to achieve the best object detection performance on the second visible light image dataset; After training is completed, save the trained YOLOv8 object detection model.

9. The method according to claim 1, characterized in that: The step S107 further includes: Prepare training data by using the labeled power equipment defect information in the second visible light image data set as a supervisory signal; Use ViT as the base model and train it on the second visible light image dataset; Optimize the hyperparameters of the ViT model to achieve the best defect classification performance on the second visible light image dataset; After the training is completed, save the trained ViT defect classification model.

10. A visible light image processing device for electric power equipment, comprising A historical information collection module, which is used to collect visible light images of defects of the electric power equipment over the years and generate a first visible light image data set of defects of the electric power equipment; A LoRA diffusion model construction module is used to select a pre-trained Stable diffusion model, ensure that the input and output formats of the Stable diffusion model are compatible with the visible light image data of defects in power equipment, freeze the parameters of the key layers of the Stable diffusion model, and use the LoRA algorithm to fine-tune the pre-trained Stable diffusion model to obtain the LoRA diffusion model; A sample expansion module, configured to expand the first visible light image data set using the LoRA diffusion model to obtain a second visible light image data set; A model training module, which is used to train a YOLOv8 target detection model and a ViT defect classification model based on the second visible light image dataset; A field acquisition module, which is used to convert the actual surveillance video data acquired on site into an image sequence; The image processing module is used to identify and classify the image sequence based on the trained YOLOv8 target detection model and ViT defect classification model.

Images