Substation equipment defect detection method and device, model training method and device, and equipment

Abstract

The application relates to a substation equipment defect detection method and device, and a model training method and device. The substation equipment defect detection method comprises the following steps: acquiring a to-be-detected visible light image and a to-be-detected infrared image of a target substation equipment; using a trained defect detection model to perform defect detection on the to-be-detected visible light image and the to-be-detected infrared image, and determining a defect position and a defect category of the target substation equipment; wherein the defect detection model is used for: performing image registration on the to-be-detected visible light image and the to-be-detected infrared image to obtain a registered visible light image and a registered infrared image; performing feature fusion on the registered visible light image and the registered infrared image to obtain a fused image; and determining the defect position and the defect category of the target substation equipment according to the fused image. The method can simultaneously utilize the high definition of the visible light image and the strong penetration of the infrared image, and realize all-weather detection of the substation equipment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, model training method, apparatus and equipment for detecting defects in substation equipment. Background Technology

[0002] A substation is a location in a power system that transforms voltage and current, receives electrical energy, and distributes it. Substations are a crucial component of transmission lines, operating under complex conditions. The equipment within substations has high requirements for insulation and mechanical performance, leading to frequent equipment failures. Therefore, it is essential to inspect and maintain substation equipment to promptly identify faults and defects, determine their location and type, analyze their causes, and eliminate their impact.

[0003] Traditionally, substation fault diagnosis involves taking photographs of the substation and determining the location and type of the defect based on these photographs.

[0004] However, in the operation and maintenance of substations, the photos taken are often analyzed and processed separately, which fails to effectively identify equipment faults and makes it impossible to make comprehensive and incomplete judgments on equipment faults, thus failing to achieve all-weather detection. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, model training method, device, and equipment for detecting substation equipment defects that can achieve all-weather monitoring of substations, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a method for detecting defects in substation equipment. The method includes:

[0007] Acquire the visible light image and infrared image of the target substation equipment to be detected;

[0008] Using a trained defect detection model, defects are detected in visible light images and infrared images to determine the location and type of defects in the target substation equipment.

[0009] The defect detection model is used to: register the visible light image and the infrared image to be detected to obtain the registered visible light image and the registered infrared image; perform feature fusion on the registered visible light image and the registered infrared image to obtain the fused image; and determine the defect location and defect type of the target substation equipment based on the fused image.

[0010] In one embodiment, the defect detection model includes an image registration network, a feature fusion network, and a defect localization and recognition network;

[0011] An image registration network is used to register the visible light image and the infrared image to be detected, resulting in a registered visible light image and a registered infrared image.

[0012] A feature fusion network is used to fuse features of registered visible light images and registered infrared images to obtain a fused image;

[0013] The defect location and identification network is used to determine the location and type of defects in target substation equipment based on the fused images.

[0014] In one embodiment, the image registration network is specifically used for:

[0015] Extract visible light image features from the visible light image to be detected and infrared image features from the infrared image to be detected;

[0016] The first mode of visible light image features is converted into a second mode consistent with infrared image features;

[0017] The registration parameters are determined based on the features of the visible light image and the infrared image.

[0018] The infrared image to be detected is transformed according to the registration parameters to obtain a registered visible light image and a registered infrared image.

[0019] In one embodiment, the defect localization and identification network includes a localization network and a detection network;

[0020] The localization network is used to extract the location information of defects in the fused image, and to crop the fused image according to the location information to obtain defect component blocks;

[0021] The detection network is used to extract feature information of defective component blocks and determine the category of defects.

[0022] Secondly, this application also provides a method for training a defect detection model. This method includes:

[0023] Obtain sample visible light images and sample infrared images of the sample substation equipment; both the sample visible light images and sample infrared images are labeled with the defect location and defect category of the sample substation equipment;

[0024] Using the defect detection model to be trained, defects are detected in the visible light image and infrared image of the sample to obtain the location and type of the sample defects.

[0025] Based on the defect location label and sample defect location, as well as the defect category label and sample defect type, the parameters of the defect detection model to be trained are optimized.

[0026] The defect detection model to be trained is used to: perform image registration on the sample visible light image and the sample infrared image to obtain the sample registered visible light image and the sample registered infrared image; perform feature fusion on the sample registered visible light image and the sample registered infrared image to obtain the sample fused image; and determine the sample defect location and sample defect category of the sample substation equipment based on the sample fused image.

[0027] In one embodiment, the sample infrared image also carries a registration label; before using the defect detection model to be trained to perform defect detection on the sample visible light image and sample infrared image to obtain the sample defect location and sample defect type, the method further includes:

[0028] Using the defect detection model to be trained, extract the visible light image features of the sample visible light image and the infrared image features of the sample infrared image;

[0029] The first mode of the visible light image features of the sample is converted into a second mode that is consistent with the infrared image features of the sample;

[0030] The sample registration parameters are determined based on the visible light image features and infrared image features of the samples.

[0031] Based on the registration labels and sample registration parameters, the parameters of the defect detection model to be trained are optimized.

[0032] Thirdly, this application also provides a substation equipment defect detection device. The device includes:

[0033] The image acquisition module is used to acquire the visible light image and infrared image of the target substation equipment to be detected.

[0034] The image defect detection module uses a trained defect detection model to perform defect detection on the visible light image and the infrared image to be detected, and determines the location and type of defects in the target substation equipment.

[0035] The defect detection model is used to: register the visible light image and the infrared image to be detected to obtain the registered visible light image and the registered infrared image; perform feature fusion on the registered visible light image and the registered infrared image to obtain the fused image; and determine the defect location and defect type of the target substation equipment based on the fused image.

[0036] Fourthly, this application also provides a defect detection model training device. The device includes:

[0037] The sample image acquisition module is used to acquire visible light and infrared images of the sample substation equipment; both the visible light and infrared images of the sample substation equipment are labeled with defect location and defect category.

[0038] The sample defect detection module is used to detect defects in the visible light image and infrared image of the sample using the defect detection model to be trained, and to obtain the location and type of the sample defect.

[0039] The parameter optimization module is used to optimize the parameters of the defect detection model to be trained based on the defect location label and sample defect location, as well as the defect category label and sample defect type.

[0040] The defect detection model to be trained is used to: perform image registration on the sample visible light image and the sample infrared image to obtain the sample registered visible light image and the sample registered infrared image; perform feature fusion on the sample registered visible light image and the sample registered infrared image to obtain the sample fused image; and determine the sample defect location and sample defect category of the sample substation equipment based on the sample fused image.

[0041] Fifthly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described substation equipment defect detection method; or, to implement the steps of the above-described defect detection model training method.

[0042] Sixthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described substation equipment defect detection method; or, implements the steps of the above-described defect detection model training method.

[0043] Seventhly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described substation equipment defect detection method; or, implements the steps of the above-described defect detection model training method.

[0044] The aforementioned substation equipment defect detection method, model training method, device, and equipment first acquire a visible light image and an infrared image of the target substation equipment to be detected. Then, both images are input into a trained defect detection model for defect detection. The defect detection model determines the location and type of defects in the images. Specifically, since the two images are obtained from the same scene in different modalities, image registration can be achieved by transformation between the visible light and infrared images, and the correspondence between their pixels can be determined. The registered visible light and infrared images obtained through image registration can be fused based on their correspondence to obtain a fused image. Target recognition on the fused image can determine the location and type of defects in the fused image. Furthermore, the fused image possesses both the high clarity of visible light and the ability of infrared light to penetrate smoke, ensuring that images acquired in all weather conditions meet the clarity requirements. Compared to traditional methods that use a single photograph for defect identification, this embodiment utilizes the high clarity of visible light images and the ability of infrared images to be captured at night and in smoky conditions, enabling all-weather detection of substation equipment. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the application environment of a substation equipment defect detection method in one embodiment.

[0046] Figure 2 This is a flowchart illustrating a substation equipment defect detection method in one embodiment;

[0047] Figure 3 This is a flowchart illustrating a substation equipment defect detection method in another embodiment;

[0048] Figure 4 This is a flowchart illustrating a defect detection model training method in one embodiment;

[0049] Figure 5 This is a flowchart illustrating a substation equipment defect detection method in another embodiment;

[0050] Figure 6 This is a structural block diagram of a substation equipment defect detection device in one embodiment;

[0051] Figure 7 This is a structural block diagram of a defect detection model training device in one embodiment;

[0052] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The substation equipment defect detection method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed in the cloud or on other network servers. Terminal 102 is used to capture images of substation equipment and transmit the captured images to the server via communication. The server processes the images to determine the location and type of faults in the substation equipment. Terminal 102 can be, but is not limited to, various drones, cameras, video cameras, or other portable imaging terminals, capable of capturing visible light and infrared images of substation equipment. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0055] In one embodiment, such as Figure 2 As shown, a method for detecting defects in substation equipment is provided, which is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0056] S202, acquire the visible light image and infrared image of the target substation equipment to be detected.

[0057] A substation is a location in a power system that transforms voltage and current, receives electrical energy, and distributes it. The target substation equipment refers to the substation equipment whose faults or defects need to be identified; schematically, substation equipment can be electrical equipment.

[0058] Terminal 102 captures images of the target substation equipment, obtaining visible light and infrared images of the equipment to be detected, and transmits them to server 104 via communication. Indicatively, terminal 102 has a visible light imaging module and an infrared imaging module, capable of simultaneously acquiring visible light and infrared images. Alternatively, visible light and infrared images can be captured separately by a visible light camera and a thermal infrared camera, and then transmitted together to server 104 for processing.

[0059] Images acquired using visible light imaging generally have high resolution, enabling high-precision equipment fault identification and detection during the day. However, due to limitations in the imaging principle, ideal imaging is difficult to achieve at night or in smoky conditions, resulting in lower detection and identification efficiency and thus poor all-weather capability. Thermal infrared cameras can acquire thermal radiation information in a scene, making them highly efficient for identifying thermal faults in equipment, and they have good night-time operation and smoke penetration capabilities. However, their image spatial resolution is lower, and target details are relatively blurry, which limits the accuracy of detection and identification.

[0060] S204. Using a trained defect detection model, defects are detected in the visible light image and the infrared image to be detected, determining the location and type of defects in the target substation equipment. Specifically, the defect detection model is used for: image registration of the visible light image and the infrared image to be detected, obtaining registered visible light and registered infrared images; feature fusion of the registered visible light and registered infrared images, obtaining a fused image; and determining the location and type of defects in the target substation equipment based on the fused image.

[0061] The acquired visible light image and infrared image to be detected are input into the trained defect detection model, which then determines the location and type of defects in the target substation equipment. Indicatively, the defect location can be the center point of the defect in the image, as well as the width and height of the defect; the defect type can be damage, corrosion, or cracks, etc.

[0062] Specifically, after the visible light image and the infrared image to be detected are input into the defect detection model, they are first registered to obtain a registered visible light image and a registered infrared image. Alternatively, if only the infrared image to be detected is registered without processing the visible light image, the registered visible light image and the visible light image to be detected can be identical.

[0063] When visible light cameras and infrared cameras capture images of target substation equipment, the resulting visible light and infrared images may have different modes due to differences in resolution, shooting position, and shooting angle. Therefore, registration is necessary to transform the visible light and infrared images and achieve matching. Mode refers to the state under a specific pattern; in this embodiment, it refers to the inconsistency between the mode of the image acquired by the visible light camera and the mode of the image acquired by the infrared camera.

[0064] After the visible light image and the infrared image are registered, the registered visible light image and the registered infrared image can be fused to obtain the fused image. The fused image has the advantage of high resolution of the visible light image, while solving the problem of clarity in low ambient light, so that images acquired in all weather conditions have high resolution.

[0065] The fused image combines the advantages of both visible light and infrared images, achieving the required clarity. By performing feature recognition on the fused image, a predicted bounding box of the defect location can be displayed, thus determining the location of the defect in the target substation equipment. Furthermore, the defect category within the predicted bounding box is determined. Defect location information includes the defect's center point, height, and width. Defect category information includes the defect category and confidence score.

[0066] In this embodiment, a visible light image and an infrared image of the target substation equipment are first acquired. These images are then input into a trained defect detection model for defect detection. The model determines the location and type of defects in the images. Specifically, since the two images are obtained from the same scene in different modalities, they can be registered through transformation to determine the correspondence between their pixels. The registered visible light image and registered infrared image obtained through image registration can then be fused based on their correspondence to obtain a fused image. Target recognition on the fused image can then determine the location and type of defects in the fused image. Furthermore, the fused image possesses both the high definition of visible light and the ability of infrared light to penetrate smoke, ensuring that images acquired in all weather conditions meet the required clarity. Compared to traditional methods that use a single photograph for defect identification, this embodiment utilizes the high definition of visible light images and the ability of infrared images to be captured at night and in smoky conditions, enabling all-weather detection of substation equipment.

[0067] In one embodiment, the defect detection model includes an image registration network, a feature fusion network, and a defect localization and identification network. The image registration network is used to register the visible light image and the infrared image to be detected, resulting in a registered visible light image and a registered infrared image. The feature fusion network is used to fuse the registered visible light image and the registered infrared image, resulting in a fused image. The defect localization and identification network is used to determine the defect location and defect type of the target substation equipment based on the fused image.

[0068] After the visible light image and the infrared image to be detected are input into the defect detection model, they first enter the image registration network. The image registration network registers the two images, resulting in registered visible light and registered infrared images, which are then input into the feature fusion network. The feature fusion network fuses the features of the registered visible light and registered infrared images to obtain a fused image, which is then input into the defect localization and recognition network. The defect localization and recognition network uses the fused image to determine the defect location and defect type.

[0069] In one feasible implementation, the feature fusion network adopts an end-to-end fusion architecture, using DenseNet (Dense Convolutional Network) as the main network. It consists of convolutional layers, pooling layers, activation function layers, batch normalization layers, and corresponding jumper structures. Each layer takes the output of all preceding layers as its input, forming a densely connected structure. By feeding the registered visible light image and the registered infrared image into the trained feature fusion network structure, the fused output, the fused image, can be obtained.

[0070] To illustrate, a short-connect structure can be added to the DenseNet network structure to deepen the number of network layers, avoid gradient vanishing, and enhance the propagation of features between layers.

[0071] Please see Figure 3 Schematic illustration: the visible light image to be detected can be called a visible light image, and the infrared image to be detected can be called an infrared image. The image registration network can be called a multimodal image registration module, the feature fusion network can be called a multimodal image fusion module, and the defect localization and recognition network can be called a fused image detection network. The visible light image and the infrared image are input into the multimodal image registration module for registration, resulting in a registered image. The registered image is then input into the multimodal image fusion module for fusion, resulting in a fused image. The fused image is then input into the fused image detection network to determine the location and type of defects in the fused image, thereby determining the location and type of defects in the target substation equipment.

[0072] In this embodiment, the defect detection model includes an image registration network, a feature fusion network, and a defect localization and recognition network. After the visible light image and the infrared image to be detected are input into the defect detection model, they are first input into the image registration module for image registration. The registered image is then input into the feature fusion network for feature fusion. The fused image is then input into the defect localization and recognition network to finally determine the defect location and defect type.

[0073] In one embodiment, the image registration network is specifically used to: extract visible light image features of the visible light image to be detected and infrared image features of the infrared image to be detected; convert a first mode of the visible light image features into a second mode consistent with the infrared image features; determine registration parameters based on the visible light image features and the infrared image features; and transform the infrared image to be detected according to the registration parameters to obtain a registered visible light image and a registered infrared image.

[0074] The visible light image and the infrared image to be detected belong to different modalities and need to be registered. Specifically, visible light image features and infrared image features of the visible light image and infrared image to be detected are extracted. Using the infrared image as a reference image, the modality of the visible light image is converted to be consistent with that of the infrared image. Then, feature descriptors are used to describe the features of the visible light image and the infrared image. Finally, the features between the two images are matched according to the similarity of the descriptors, and the registration parameters are determined. The registration parameters refer to the geometric transformation method, including translation, rotation, and scaling. Schematically, the extracted image features can be points, lines (edges), or regions (surfaces). After determining the registration parameters, the infrared image to be detected can be transformed according to the registration parameters to obtain the registered visible light image and the registered infrared image.

[0075] In one feasible implementation, a GAN (Generative Adversarial Network) is used in the image registration network for registration. The GAN first extracts image features from the visible light image and the infrared image to be detected, respectively, and then transforms the features of the visible light image into features similar to those of the infrared image, thereby improving the accuracy of subsequent heterogeneous image registration. Illustratively, this GAN can use the U-Net architecture, achieving good image conversion results with a simple and efficient structure. This is because the U-Net network differs from conventional encoder-decoder structures in that it employs a concatenated feature fusion method. Specifically, by adding connections in the corresponding downsampling and upsampling layers, features are concatenated along the channel dimension, fully utilizing low-level information to form a thicker feature set. Downsampling refers to downsampling high-resolution images, while upsampling refers to interpolation or other processing on low-resolution images. Generally, infrared images have low resolution, while visible light images have high resolution.

[0076] Furthermore, using the image transformed by the generative adversarial network as a reference, the spatial transformation module calculates the registration parameters for registering the infrared image to be detected, and obtains the registered infrared image based on the registration parameters. The spatial transformation module (Spatial Transformer Networks, STN) explicitly performs spatial transformation operations on the data within the network, without requiring keypoint calibration, and can adaptively perform spatial transformations and alignments on the data, including translation, scaling, rotation, and other geometric transformations. The registered infrared image is obtained by performing bilinear interpolation and calculations on the infrared image to be detected using the calculated registration parameters. Illustratively, the registered visible light image generated from visible light and the registered image can be fed into the discriminator network to determine whether registration is complete.

[0077] In this embodiment, the image registration network extracts visible light image features and infrared image features, determines registration parameters based on both, and then transforms the infrared image to be detected according to the registration parameters to obtain a registered visible light image and a registered infrared image. After registration, the fusion of the visible light image and the infrared image can be achieved in subsequent steps.

[0078] In one embodiment, the defect localization and identification network includes a localization network and a detection network; the localization network is used to extract the location information of defects in the fused image and crop the fused image according to the location information to obtain defective component blocks; the detection network is used to extract the feature information of the defective component blocks and determine the category of defects.

[0079] The defect location and identification network includes a location network and a detection network. First, the location network determines the location of the defect, and then the detection network determines the category of the defect.

[0080] In one feasible implementation, the defect localization and identification network is a cascaded detection architecture based on convolutional neural networks (CNNs) capable of both locating the defect's position and detecting its type. First, an object detection algorithm is used to locate the component to which the defect is attached. Then, object detection is used again to determine the defect type. This fully leverages the powerful feature extraction and regression capabilities of CNNs.

[0081] Specifically, the defect localization and recognition network consists of two main modules: a defect location network and a defect detection network. The feature extraction base network of the defect localization network is VGG-16 (Visual Geometry Group, 16 layers), which then undergoes operations such as RPN (Region Proposal Network) and ROI (Region of Interest) pooling to detect the location information of the defect in the image. This location information is then mapped back to the input image to obtain component blocks, which are then used as input to the defect category detection network. The base network of the classifier in the defect category detection network is ResNet (Residual Network), which then undergoes RPN and ROI pooling operations to obtain the defect category. The main purpose of introducing RPN is to efficiently compute candidate regions and generate a set of detection boxes that may contain objects in the image.

[0082] In this embodiment, the defect localization and recognition network includes a localization network and a detection network. After the fused image enters the defect localization and recognition network, it first enters the localization network, where the location of the defect is determined. Then, the image containing the defect location information is input to the detection network to determine the defect category, thereby ultimately determining the location and category of the defect.

[0083] In one embodiment, such as Figure 4 As shown, a defect detection model training method is provided, which is then applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0084] S402, acquire sample visible light images and sample infrared images of the sample substation equipment; both the sample visible light images and sample infrared images carry defect location labels and defect category labels of the sample substation equipment.

[0085] S404 uses the defect detection model to be trained to detect defects in the visible light image and infrared image of the sample, and obtains the location and type of the sample defect.

[0086] S406, Based on the defect location label and sample defect location, as well as the defect category label and sample defect type, the parameters of the defect detection model to be trained are optimized. Specifically, the defect detection model to be trained is used for: image registration of the sample visible light image and sample infrared image to obtain a sample registered visible light image and a sample registered infrared image; feature fusion of the sample registered visible light image and sample registered infrared image to obtain a fused image; and determining the sample defect location and sample defect category of the sample substation equipment based on the fused image.

[0087] The sample substation equipment refers to substation equipment with identified defect locations and types, used to train the defect detection model to improve its accuracy. Visible light and infrared cameras capture images of the sample substation equipment, respectively, yielding sample visible light and infrared images. Schematic illustration: the sample visible light and infrared images can be data stored in a data storage module.

[0088] The sample visible light and infrared images contain images with known defect locations and categories, both labeled with defect location and category. These images are input into the defect detection model to be trained, which then determines the defect locations and categories. A loss function is used to optimize the trained defect detection model based on the differences between the defect location labels and the sample defect locations, and based on the differences between the defect category labels and the sample defect categories, resulting in a fully trained defect detection model.

[0089] The defect detection model to be trained includes an image registration network, a feature fusion network, and a defect localization and recognition network. In one feasible training process, the generator's loss function for the image registration network is:

[0090] L a =-logD(I T )-log(1-D(G(I vis )))-μ||I T -G(I vis )||

[0091] The loss function of the discriminator is:

[0092] L g =-logD(G(I) vis ))

[0093] Among them, L a It is the loss function of the generator, L g It is the loss function of the discriminator, I T For infrared images transferred to the visible light feature domain, I vis For the visible light image, D(*) is the output of the discriminator, G(*) is the output of the generator, and μ is the manually set weight parameter.

[0094] For the feature fusion network, the loss function is:

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] in, I represents the loss function, I1 and I2 are the images to be fused, and I... f For the fusion result, For the gradient operator, φ j (*) represents the feature map extracted by the VGG16 network, SSIM is the calculation of structural similarity, and α and β are weight parameters, which are usually set to 5e-5 and 3e3.

[0102] To illustrate, in order to ensure that the fused image retains as much useful information as possible from the source images, a weight block can be introduced during training to evaluate the amount of information contained in the images to be fused. Image patches with more information are assigned higher weights, and vice versa. This weights the various loss terms of the loss function, reasonably preserving useful information from the source images for substation defect identification.

[0103] For the defect localization and identification network, the loss function is defined as:

[0104] L fl =-((1-p) 2 q log(p)+p 2 (1-q)log(1-p))

[0105] Among them, L fl It is the loss function, where p and q represent the classification prediction vector of the candidate sample and the corresponding predicted ground truth value (target label or background label).

[0106] The regression loss function during the training phase is:

[0107]

[0108] Among them, L s1 It is the loss function, x = b t -b p The difference between the predicted value and the true value of each candidate sample predicted by the head network on the feature vector.

[0109] In this embodiment, firstly, visible light images and infrared images of the sample substation equipment, each labeled with a defect location and a defect category, are acquired. Then, these images are input into the defect detection model to be trained, outputting the sample defect location and defect type. The sample defect location and its label are compared, and the sample defect type and its category label are also compared. The defect detection model is then optimized and trained to obtain a fully trained defect detection model.

[0110] In one embodiment, the sample infrared image also carries a registration label; before using the defect detection model to be trained to perform defect detection on the sample visible light image and sample infrared image to obtain the sample defect location and sample defect type, the defect detection model training method further includes: using the defect detection model to be trained to extract the sample visible light image features of the sample visible light image and the sample infrared image features of the sample infrared image; converting the first mode of the sample visible light image features into a second mode consistent with the sample infrared image features; determining the sample registration parameters based on the sample visible light image features and sample infrared image features; and optimizing the parameters of the defect detection model to be trained based on the registration label and the sample registration parameters.

[0111] The sample visible light image and the sample infrared image belong to different modalities and need to be registered. Specifically, the visible light image features of the sample visible light image and the infrared image features of the sample infrared image are extracted. Using the sample infrared image as a reference image, the modality of the visible light image is converted to be consistent with that of the infrared image. Then, feature descriptors are used to describe the features of the visible light image and the infrared image. Finally, the features between the two images are matched according to the similarity of the descriptors, and the sample registration parameters are determined. Based on the sample registration parameters, the sample infrared image is converted to be consistent with the sample visible light image. The registration parameters refer to the geometric transformation method, including translation, rotation, and scaling.

[0112] When the sample infrared image has a registration label, the defect detection model can be optimized based on the difference between the registration label and the sample registration parameters, thereby improving the accuracy of the registration parameters.

[0113] In this embodiment, the sample infrared images are labeled with registration tags. After the defect detection model to be trained determines the sample registration parameters, it is optimized based on the differences between the registration tags and the sample registration parameters, thereby improving the accuracy of the defect detection model in determining the registration parameters.

[0114] In one embodiment, such as Figure 5 As shown, a method for detecting defects in substation equipment is provided, including the following steps:

[0115] S502, acquire sample visible light images and sample infrared images of the sample substation equipment; both the sample visible light images and sample infrared images carry defect location labels and defect category labels of the sample substation equipment.

[0116] S504 uses the defect detection model to be trained to perform defect detection on the visible light image and infrared image of the sample, and obtains the location and type of the sample defect.

[0117] S506, Based on the defect location label and sample defect location, as well as the defect category label and sample defect type, the parameters of the defect detection model to be trained are optimized. Specifically, the defect detection model to be trained is used for: image registration of the sample visible light image and sample infrared image to obtain a sample registered visible light image and a sample registered infrared image; feature fusion of the sample registered visible light image and sample registered infrared image to obtain a fused image; and determining the sample defect location and sample defect category of the sample substation equipment based on the fused image.

[0118] S508, acquire the visible light image and infrared image of the target substation equipment to be detected.

[0119] S510 utilizes a trained defect detection model to perform defect detection on the visible light image and the infrared image to be detected, determining the defect location and defect type of the target substation equipment. Specifically, the defect detection model is used for: image registration of the visible light image and the infrared image to be detected, obtaining registered visible light images and registered infrared images; feature fusion of the registered visible light images and registered infrared images, obtaining a fused image; and determining the defect location and defect type of the target substation equipment based on the fused image.

[0120] In this embodiment, firstly, sample visible light images and sample infrared images of the sample substation equipment, each labeled with a defect location and defect category, are acquired. These images are then input into the defect detection model to be trained to obtain the sample defect location and defect type. Based on the differences in sample defect location and defect location labels, as well as the differences in sample defect type and defect category labels, the defect detection model is trained and optimized to obtain a fully trained defect detection model. After acquiring the target substation equipment's visible light and infrared images to be detected, these images are input into the trained defect detection model to determine the defect location and defect category of the target substation equipment.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0122] Based on the same inventive concept, this application also provides a substation equipment defect detection device for implementing the substation equipment defect detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the substation equipment defect detection device provided below can be found in the limitations of the substation equipment defect detection method described above, and will not be repeated here.

[0123] In one embodiment, such as Figure 6 As shown, a substation equipment defect detection device 600 is provided, including: an image acquisition module 602 and an image defect detection module 604, wherein:

[0124] The image acquisition module 602 is used to acquire the visible light image and the infrared image of the target substation equipment.

[0125] The image defect detection module 604 uses a trained defect detection model to perform defect detection on the visible light image and the infrared image to be detected, determining the defect location and defect type of the target substation equipment. Specifically, the defect detection model is used for: image registration of the visible light image and the infrared image to be detected, obtaining a registered visible light image and a registered infrared image; feature fusion of the registered visible light image and the registered infrared image, obtaining a fused image; and determining the defect location and defect type of the target substation equipment based on the fused image.

[0126] In one embodiment, the defect detection model includes an image registration network, a feature fusion network, and a defect localization and identification network. The image registration network is used to register the visible light image and the infrared image to be detected, resulting in a registered visible light image and a registered infrared image. The feature fusion network is used to fuse the registered visible light image and the registered infrared image, resulting in a fused image. The defect localization and identification network is used to determine the defect location and defect type of the target substation equipment based on the fused image.

[0127] In one embodiment, the image registration network is specifically used to: extract visible light image features of the visible light image to be detected and infrared image features of the infrared image to be detected; convert a first mode of the visible light image features into a second mode consistent with the infrared image features; determine registration parameters based on the visible light image features and the infrared image features; and transform the infrared image to be detected according to the registration parameters to obtain a registered visible light image and a registered infrared image.

[0128] In one embodiment, the defect localization and identification network includes a localization network and a detection network; the localization network is used to extract the location information of defects in the fused image and crop the fused image according to the location information to obtain defective component blocks; the detection network is used to extract the feature information of the defective component blocks and determine the category of defects.

[0129] Each module in the aforementioned substation equipment defect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0130] Based on the same inventive concept, this application also provides a defect detection model training device for implementing the defect detection model training method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more defect detection model training device embodiments provided below can be found in the limitations of the defect detection model training method described above, and will not be repeated here.

[0131] In one embodiment, such as Figure 7 As shown, a defect detection model training device 700 is provided, including: a sample image acquisition module 702, a sample defect detection module 704, and a parameter optimization module 706, wherein:

[0132] The sample image acquisition module 702 is used to acquire sample visible light images and sample infrared images of the sample substation equipment; both the sample visible light images and sample infrared images carry defect location labels and defect category labels of the sample substation equipment.

[0133] The sample defect detection module 704 is used to perform defect detection on the visible light image and infrared image of the sample using the defect detection model to be trained, and to obtain the location and type of the sample defect.

[0134] The parameter optimization module 706 is used to optimize the parameters of the defect detection model to be trained based on the defect location label and sample defect location, as well as the defect category label and sample defect type.

[0135] The defect detection model to be trained is used to: perform image registration on the sample visible light image and the sample infrared image to obtain the sample registered visible light image and the sample registered infrared image; perform feature fusion on the sample registered visible light image and the sample registered infrared image to obtain the sample fused image; and determine the sample defect location and sample defect category of the sample substation equipment based on the sample fused image.

[0136] In one embodiment, the sample infrared image also carries a registration label; the parameter optimization module 706 is further configured to: extract sample visible light image features of the sample visible light image and sample infrared image features of the sample infrared image using the defect detection model to be trained; convert the first mode of the sample visible light image features into a second mode consistent with the sample infrared image features; determine sample registration parameters based on the sample visible light image features and sample infrared image features; and optimize the parameters of the defect detection model to be trained based on the registration label and sample registration parameters.

[0137] Each module in the aforementioned defect detection model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0138] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores visible light and infrared image data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a substation equipment defect detection method or a defect detection model training method.

[0139] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described embodiment of the substation equipment defect detection method; or, to implement the steps in the above-described embodiment of the defect detection model training method.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described embodiment of the substation equipment defect detection method; or, implements the steps in the above-described embodiment of the defect detection model training method.

[0142] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described substation equipment defect detection method embodiment; or, implements the steps in the above-described defect detection model training method embodiment.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting defects in substation equipment, characterized in that, The method includes: Acquire the visible light image and infrared image of the target substation equipment to be detected; Using a trained defect detection model, defects are detected in the visible light image and the infrared image to be detected, thereby determining the location and type of defects in the target substation equipment. The defect detection model includes an image registration network, a feature fusion network, and a defect localization and recognition network. The image registration network comprises a generative adversarial network (GAN) and a spatial transformation module. The GAN extracts visible light image features from the visible light image to be detected and infrared image features from the infrared image to be detected, converting a first mode of the visible light image features into a second mode consistent with the infrared image features. The spatial transformation module determines registration parameters based on the visible light image features and the infrared image features, and transforms the infrared image to be detected according to the registration parameters to obtain a registered visible light image and a registered infrared image. The feature fusion network fuses features from the registered visible light image and the registered infrared image to obtain a fused image. The defect localization and recognition network includes a localization network and a detection network. The localization network extracts the location information of defects in the fused image, determines the defect location of the target substation equipment, and crops the fused image according to the location information to obtain defective component blocks. The detection network extracts feature information from the defective component blocks to determine the defect category of the target substation equipment.

2. A defect detection model training method, characterized in that, The method includes: Acquire sample visible light images and sample infrared images of the sample substation equipment; both the sample visible light images and the sample infrared images carry defect location labels and defect category labels of the sample substation equipment, and the sample infrared images also carry registration labels; Using the generative adversarial network in the image registration network of the defect detection model to be trained, the visible light image features of the sample visible light image and the infrared image features of the sample infrared image are extracted, and the first mode of the visible light image features is converted into a second mode consistent with the infrared image features. Using the spatial transformation module in the image registration network of the defect detection model to be trained, the sample registration parameters are determined based on the visible light image features and the infrared image features of the sample. The infrared image of the sample is then transformed according to the sample registration parameters to obtain the registered visible light image and the registered infrared image. Based on the registration labels and the sample registration parameters, the image registration network in the defect detection model to be trained is optimized. The registered visible light image and the registered infrared image are fused using the feature fusion network in the defect detection model to be trained, resulting in a fused image. The fused image is then subjected to defect detection using the defect localization and recognition network in the defect detection model to obtain the sample defect locations and types in the fused image. The defect localization and recognition network includes a localization network and a detection network. The localization network extracts the location information of defects in the fused image, determines the sample defect locations, and crops the fused image based on the location information to obtain defective component blocks. The detection network extracts the feature information of the defective component blocks and determines the sample defect types in the fused image. Based on the defect location label and the sample defect location of the fused image, as well as the defect category label and the sample defect type of the fused image, the parameters of the feature fusion network and the defect localization and recognition network in the defect detection model to be trained are optimized.

3. A defect detection device for substation equipment, characterized in that, The device includes: The image acquisition module is used to acquire the visible light image and infrared image of the target substation equipment to be detected. The image defect detection module is used to perform defect detection on the visible light image and the infrared image to be detected using a trained defect detection model, and to determine the defect location and defect type of the target substation equipment. The defect detection model includes an image registration network, a feature fusion network, and a defect localization and recognition network. The image registration network comprises a generative adversarial network (GAN) and a spatial transformation module. The GAN extracts visible light image features from the visible light image to be detected and infrared image features from the infrared image to be detected, converting a first mode of the visible light image features into a second mode consistent with the infrared image features. The spatial transformation module determines registration parameters based on the visible light image features and the infrared image features, and transforms the infrared image to be detected according to the registration parameters to obtain a registered visible light image and a registered infrared image. The feature fusion network fuses features from the registered visible light image and the registered infrared image to obtain a fused image. The defect localization and recognition network includes a localization network and a detection network. The localization network extracts the location information of defects in the fused image, determines the defect location of the target substation equipment, and crops the fused image according to the location information to obtain defective component blocks. The detection network extracts feature information from the defective component blocks to determine the defect category of the target substation equipment.

4. A defect detection model training device, characterized in that, The device includes: The sample image acquisition module is used to acquire sample visible light images and sample infrared images of sample substation equipment; both the sample visible light images and the sample infrared images carry defect location labels and defect category labels of the sample substation equipment, and the sample infrared images also carry registration labels; The parameter optimization module is used to extract sample visible light image features of the sample visible light image and sample infrared image features of the sample infrared image using the generative adversarial network in the image registration network of the defect detection model to be trained; to convert the first mode of the sample visible light image features into a second mode consistent with the sample infrared image features; and to determine sample registration parameters based on the sample visible light image features and the sample infrared image features using the spatial transformation module in the image registration network of the defect detection model to be trained, to transform the sample infrared image according to the sample registration parameters to obtain a registered visible light image and a registered infrared image; and to optimize the parameters of the image registration network in the defect detection model to be trained according to the registration label and the sample registration parameters. The sample defect detection module is used to fuse the registered visible light image and the registered infrared image using the feature fusion network in the defect detection model to obtain a fused image; and to detect defects in the fused image using the defect localization and recognition network in the defect detection model to obtain the sample defect location and sample defect type of the fused image. The defect localization and recognition network includes a localization network and a detection network. The localization network is used to extract the location information of defects in the fused image, determine the sample defect location, and crop the fused image according to the location information to obtain defective component blocks. The detection network is used to extract the feature information of the defective component blocks and determine the sample defect type of the fused image. The parameter optimization module is used to optimize the parameters of the feature fusion network and the defect localization and recognition network in the defect detection model to be trained, based on the defect location label and the sample defect location of the fused image, as well as the defect category label and the sample defect type of the fused image.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 2.

Images