Fault detection method, device, system and storage medium

Abstract

The application discloses a fault detection method, device, system and storage medium, and relates to the technical field of power supply, to at least solve the technical problems of high labor cost, low fault troubleshooting efficiency and low power transmission efficiency caused by the way of manually determining faults by inspection personnel in the related art. The method comprises the following steps: extracting first defect features existing in a to-be-detected image by using a first defect detection model to obtain a first feature image; detecting a region position where a defect exists in a fitting image included in the to-be-detected image by using a fitting detection model to obtain a target region where a defect exists in a fitting component in the to-be-detected image; extracting second defect features existing in the target region in the to-be-detected image by using a second defect detection model to obtain a second feature image; the second defect features represent defects on the fitting component; and determining a target fitting component with a fault according to the first feature image and the second feature image.

Description

Technical Field

[0001] This application relates to the field of power supply technology, and in particular to a fault detection method, device, system and storage medium. Background Technology

[0002] Because transmission lines and towers are exposed to the natural environment for extended periods, issues such as line aging, component corrosion, and improper installation are inevitable. Currently, to mitigate the impact of these faults, transmission line inspections primarily rely on manual fault identification by patrol personnel. This fault identification method requires substantial human resources, resulting in high labor costs and low fault-finding efficiency, ultimately leading to low transmission efficiency and disrupting normal power supply for users. Summary of the Invention

[0003] This invention provides a fault detection method, apparatus, system, and storage medium to at least solve the technical problems of high labor costs, low fault diagnosis efficiency, and low power transmission efficiency caused by the reliance on manual fault identification by inspection personnel in related technologies. The technical solution of this invention is as follows:

[0004] According to a first aspect of the present invention, a fault detection method is provided, applied to a power transmission intelligent inspection system; the system includes a power transmission line, a power transmission tower, and hardware components for connecting and fixing the power transmission line and the power transmission tower. The method includes: using a first defect detection model to extract a first defect feature present in an image to be detected, obtaining a first feature image; the first defect feature characterizes fault features on the power transmission line and the power transmission tower; and using a hardware detection model to detect the location of a defective area in a hardware component within a hardware image included in the image to be detected, obtaining a target area in the image to be detected where the hardware component has a defect; using a second defect detection model to extract a second defect feature present in the target area in the image to be detected, obtaining a second feature image; the second defect feature characterizes a defect on the hardware component; and determining the target hardware component with a fault based on the first feature image and the second feature image.

[0005] In one possible implementation, each pixel in the first feature image and the second feature image is associated with a region location in the image to be detected according to a preset sequence; based on the first feature image and the second feature image, a target hardware component with a fault is determined; this includes: extracting feature pixels representing the same region location features from the first feature image and the second feature image respectively, obtaining a first pixel set image and a second pixel set image representing the same region location respectively; wherein the resolution of the first pixel set image is less than the resolution of the second pixel set image; fusing the features in the first pixel set image and the second pixel set image using an upsampling method to obtain a first fused feature image; and fusing the features in the first pixel set image and the second pixel set image using a downsampling method to obtain a second fused feature image; when the defect features indicated in the first fused feature image and the second fused feature image belong to the same hardware component, the same hardware component is determined as the target hardware component.

[0006] In another possible implementation, a first defect feature is extracted from the image to be detected using a first defect detection model to obtain a first feature image, including: adjusting the image to be detected into multiple first images of a first preset size; and extracting the first defect feature from the multiple first images to obtain the first feature image.

[0007] In another possible implementation, a second defect detection model is used to extract second defect features present in the target region of the image to be detected, and a second feature image is obtained. This includes: marking the target regions in the image to be detected according to the size range of multiple different target regions on the image to be detected; adjusting the marked image to be detected into multiple second images of a second preset size; extracting the image containing the second defect features from the marked regions of the multiple second images to obtain the second feature image; the first preset size is larger than the second preset size; and the first feature image is larger than the second feature image.

[0008] In another possible implementation, the system also includes an unmanned aerial vehicle (UAV) with imaging capabilities; the method further includes: acquiring multiple on-site images associated with multiple locations on the power transmission line and transmission tower; the multiple on-site images are acquired from the UAV according to a preset period; performing a deduplication process on the multiple on-site images; the deduplication process specifically includes: identifying at least one on-site image among the multiple on-site images whose image similarity is greater than a similarity threshold, and retaining the on-site image with the highest similarity among the at least one on-site image and deleting the on-site images other than the on-site image with the highest similarity among the at least one on-site image; and determining the multiple on-site images after the deduplication process as images to be detected.

[0009] In another possible implementation, determining the faulty target hardware component based on the first feature image and the second feature image includes: inputting the first feature image and the second feature image into a preset screening model to obtain the faulty target hardware component; the preset screening model is used to classify images including non-defect features and images including defect features in the first feature image and the second feature image respectively, and to remove images with non-defect features and retain images with defect features in the first feature image and the second feature image respectively, and to determine the same hardware component indicated by the images with defect features retained in the first feature image and the second feature image respectively as the target hardware component; wherein, the preset screening model is trained using historical images including non-defect features as positive samples and historical images including defect features as negative samples.

[0010] In another possible implementation, the hardware components include nuts, bolts, studs, screws, washers, pins, helical springs, and threaded sleeves; the first defect detection model is a neural network model trained on multiple sets of first sample images containing various defect categories on transmission lines and transmission towers, and the second defect detection model is a neural network model trained on multiple sets of second sample images containing various defect categories of hardware components.

[0011] According to a second aspect of the present invention, a fault detection device is provided, applied to a power transmission intelligent inspection system; the system includes a power transmission line, a power transmission tower, and hardware components for connecting and fixing the power transmission line and the power transmission tower. The device includes: a first feature extraction unit configured to extract a first defect feature present in an image to be detected using a first defect detection model to obtain a first feature image; the first defect feature characterizes a fault feature on the power transmission line and the power transmission tower; and to detect the location of a defective area in a hardware component in a hardware image included in the image to be detected using a hardware detection model to obtain a target area in the image to be detected where the hardware component has a defect; a second feature extraction unit configured to extract a second defect feature present in the target area in the image to be detected using a second defect detection model to obtain a second feature image; the second defect feature characterizes a defect on the hardware component; and a fault determination unit configured to determine a target hardware component with a fault based on the first feature image and the second feature image.

[0012] In one possible implementation, each pixel in the first feature image and the second feature image is associated with a region location in the image to be detected according to a preset sequence; the fault determination unit is specifically configured to: extract feature pixels representing the same region location features in the first feature image and the second feature image respectively, to obtain a first pixel set image and a second pixel set image representing the same region location respectively; wherein the resolution of the first pixel set image is less than the resolution of the second pixel set image; using an upsampling method, the features in the first pixel set image and the second pixel set image are fused to obtain a first fused feature image; and using a downsampling method, the features in the first pixel set image and the second pixel set image are fused to obtain a second fused feature image; when the defect features indicated in the first fused feature image and the second fused feature image belong to the same hardware component, the same hardware component is identified as the target hardware component.

[0013] In another possible implementation, the first feature extraction unit is specifically configured to: adjust the image to be detected into multiple first images of a first preset size; extract the first defect feature from the multiple first images to obtain a first feature image.

[0014] In another possible implementation, the second feature extraction unit is specifically configured to: mark the target regions in the image to be detected according to the size range of multiple different target regions on the image to be detected; adjust the image to be detected after region marking to multiple second images of second preset sizes; extract the image including the second defect feature in the marked regions of the multiple second images to obtain the second feature image; the first preset size is larger than the second preset size; the first feature image is larger than the second feature image.

[0015] In another possible implementation, the system also includes an unmanned aerial vehicle (UAV) with a shooting function; the device further includes: an acquisition unit configured to acquire multiple on-site images associated with multiple locations on the power transmission line and transmission tower; the multiple on-site images are acquired from the UAV at a preset period; the multiple on-site images are subjected to deduplication; the deduplication specifically includes: identifying at least one on-site image among the multiple on-site images whose image similarity is greater than a similarity threshold, and retaining the on-site image with the highest similarity among the at least one on-site image and deleting the on-site images other than the on-site image with the highest similarity among the at least one on-site image; and determining the multiple on-site images after deduplication as images to be detected.

[0016] In another possible implementation, the fault determination unit is specifically configured to: input the first feature image and the second feature image into a preset screening model to obtain the target hardware component with a fault; the preset screening model is used to classify the images including non-defect features and the images including defect features in the first feature image and the second feature image respectively, and to remove the images with non-defect features and retain the images with defect features in the first feature image and the second feature image respectively, and to determine the same hardware component indicated by the images with defect features retained in the first feature image and the second feature image respectively as the target hardware component; wherein, the preset screening model is trained with historical images including non-defect features as positive samples and historical images including defect features as negative samples.

[0017] In another possible implementation, the hardware components include nuts, bolts, studs, screws, washers, pins, helical springs, and threaded sleeves; the first defect detection model is a neural network model trained on multiple sets of first sample images containing various defect categories on transmission lines and transmission towers, and the second defect detection model is a neural network model trained on multiple sets of second sample images containing various defect categories of hardware components.

[0018] According to a third aspect of the present invention, a fault detection device is provided, which is configured to perform a fault detection method as described in the first aspect and any possible implementation thereof.

[0019] According to a fourth aspect of the present invention, a computer device is provided, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement a fault detection method as described in the first aspect and any possible implementation thereof.

[0020] According to a fifth aspect of the present invention, a power transmission intelligent inspection system is provided; the system includes a power transmission line, a power transmission tower, and hardware components for connecting and fixing the power transmission line and the power transmission tower, so as to realize the fault detection method as described in the first aspect and any possible implementation thereof.

[0021] According to a sixth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of a computer device, the computer device is able to perform a fault detection method as described in the first aspect and any possible implementation thereof.

[0022] According to a seventh aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions, which, when executed on a computer device, cause the computer device to perform the fault detection method of the first aspect and any possible implementation thereof.

[0023] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects: A first defect detection model is used to detect and extract all fault features on transmission lines and towers in the image to be inspected, so as to initially determine the overall fault features from the image. Simultaneously, based on a hardware detection model, the target areas where hardware components in the image to be inspected have defects are determined, and a second defect detection model is used to detect and extract defects in hardware components in each target area of ​​the image to be inspected, so as to achieve specialized identification, detection, and extraction of specific defect features in local areas. Therefore, by combining the general defect features and specific component defect features included in the two defect feature images of the above two different defect types, the defects existing in all hardware components can be identified more accurately, so as to more accurately detect the target hardware components with faults. Furthermore, the above different types of models can intelligently process a large number of images to be inspected in batches, improving the detection efficiency of faulty hardware components, reducing high labor costs, and improving fault diagnosis efficiency and power transmission efficiency.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0026] Figure 1 This is a schematic block diagram of a power transmission intelligent inspection system according to an exemplary embodiment;

[0027] Figure 2 This is a flowchart of a fault detection method according to an exemplary embodiment. Figure 1 ;

[0028] Figure 3 This is a flowchart of a fault detection method according to an exemplary embodiment. Figure 2 ;

[0029] Figure 4 This is a flowchart of a fault detection method according to an exemplary embodiment. Figure 3 ;

[0030] Figure 5 This is a flowchart of a fault detection method according to an exemplary embodiment. Figure 4 ;

[0031] Figure 6 This is a schematic block diagram illustrating a fault detection device according to an exemplary embodiment;

[0032] Figure 7 This is a schematic diagram of a fault detection device according to an exemplary embodiment. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] Before providing a detailed description of the fault detection method provided in the embodiments of this application, let's briefly introduce the application scenarios and implementation architecture involved in the embodiments of this application.

[0036] First, the application scenarios of the fault detection method of this application are explained as follows.

[0037] Because transmission lines and towers are exposed to the natural environment for extended periods, issues such as line aging, component corrosion, and improper installation are inevitable. Currently, to mitigate the impact of these faults, transmission line inspections primarily rely on manual fault identification by patrol personnel. This fault identification method requires substantial human resources, resulting in high labor costs and low fault-finding efficiency, ultimately leading to low transmission efficiency and disrupting normal power supply for users.

[0038] Furthermore, there is currently no method for automatically identifying whether the installation of hardware components in power transmission systems is standardized. Specifically, there is currently no effective method for detecting and identifying improperly installed nuts, and managers cannot promptly and accurately determine whether nut installation is standardized and take appropriate action. Therefore, it is necessary to provide an effective method for detecting and judging the standardization of nut installation during intelligent inspection of power transmission lines.

[0039] To address the aforementioned issues, this application provides a fault detection method. A first defect detection model is used to detect and extract fault features on the transmission line in the image to be inspected, thereby initially determining the overall fault characteristics from the image. Simultaneously, based on a hardware detection model, target areas with defects in hardware components in the image to be inspected are identified. A second defect detection model is then used to detect and extract defects in hardware components in each target area of ​​the image to be inspected, achieving specialized identification, detection, and extraction of specific defect features in local areas. Therefore, by combining the defect feature images of these two different defect types, defects present in all hardware components can be identified more accurately, leading to more precise detection of faulty target hardware components. Furthermore, these different models can intelligently batch process a large number of images to be inspected, improving the detection efficiency of faulty hardware components, reducing high labor costs, and improving fault diagnosis and transmission efficiency.

[0040] Secondly, the implementation framework of the fault detection method of this application is described as follows.

[0041] like Figure 1 As shown, the intelligent power transmission inspection system includes a power transmission line 11, a power transmission tower 12, hardware components 13, and an unmanned aerial vehicle (UAV) 14. The hardware components 13 are used to connect and fix the power transmission line 11 and the power transmission tower 12, while the UAV 14 is used to capture images of the power transmission line 11, the power transmission tower 12, and the hardware components 13 at the site to obtain images to be inspected.

[0042] The aforementioned hardware components include nuts, bolts, studs, screws, washers, pins, coil springs, and threaded sleeves.

[0043] In some embodiments, the unmanned aerial vehicle 14 is also referred to as a drone or drone equipment. When the drone equipment captures images of the scene and uses image recognition methods to detect defects such as improper nut installation, it sends an alarm message to remind management personnel to handle the situation promptly.

[0044] The fault detection method provided in this application can be applied to a power transmission intelligent inspection system using the above-described implementation architecture. For ease of understanding, the fault detection method provided in this application will be described in detail below with reference to the accompanying drawings.

[0045] Figure 2 This is a flowchart illustrating a fault detection method according to an exemplary embodiment, such as... Figure 2 As shown, the fault detection method includes the following steps.

[0046] S11, using the first defect detection model, extract the first defect features present in the image to be detected to obtain the first feature image.

[0047] The first defect feature characterizes the fault characteristics on transmission lines and transmission towers.

[0048] In some implementations, the defects characterized by the first defect feature described above include: insulator faults, conductor breakage faults, tower damage, tower tilting, insulation material aging, and foreign object interference.

[0049] In other embodiments, the first defect detection model described above is used to identify and extract first defect features included in the input image. The first defect detection model may be a neural network model trained based on multiple sets of first sample images containing defects of various defect categories present on transmission lines and transmission towers.

[0050] S12, using the hardware inspection model, detect the location of the defective area of ​​the hardware component in the hardware image included in the image to be inspected, and obtain the target area of ​​the defective hardware component in the image to be inspected.

[0051] The aforementioned hardware inspection model is used to detect, identify, and locate defects, damage, dimensional deviations, or foreign objects in hardware components in input images using computer vision and deep learning technologies, and to mark the areas where defects are located.

[0052] In one implementation, the target area is the area where the number of defective hardware components exceeds a preset number or where the number of defective hardware components within the preset number is the largest. It is understood that the target area is a key hardware area where the frequency of improper installation of hardware components is high (i.e., the frequency of improper installation of hardware components is higher than a preset frequency).

[0053] S13, using the second defect detection model, extract the second defect features existing in the target region of the image to be detected, and obtain the second feature image.

[0054] The second defect feature characterizes defects on hardware components.

[0055] In some embodiments, the aforementioned second defect features include: excessive tightness of the fitting components, misalignment of the fitting components during installation, damage to the fitting components, disordered installation sequence between different fitting components, mismatch of fitting component models, and large installation errors of the fitting components.

[0056] Specifically, the second defect characteristic may include the following fault characteristics: the nut is installed too tight or too loose, resulting in too small or too large a nut gap; the nut is installed in reverse; the thread is scratched; the thread specification is mismatched: using mismatched nuts and bolts, or the nut and bolt thread specifications do not match, will result in an unstable connection and easy loosening; the nut is over-tightened; no washer is used; the nut is contaminated during use.

[0057] In one implementation, the second defect detection model is a neural network model trained on multiple sets of second sample images that contain various types of defects in hardware components.

[0058] To preserve the accuracy of the second defect feature extraction, the aforementioned second detection model can employ a multi-scale region target detection strategy to achieve defect target detection in "multi-scale regions".

[0059] S14, Based on the first feature image and the second feature image, determine the target hardware component with the fault.

[0060] The above-mentioned identification of faulty hardware components based on two different dimensions—the global defect features (i.e., the first defect feature) included in the first feature image and the local specific defect features (i.e., the second defect feature) included in the second feature image—is more accurate.

[0061] Through the above implementation method, the first defect detection model is used to detect and extract all fault features on the transmission lines and transmission towers in the image to be inspected, so as to initially determine the overall fault features from the image to be inspected. At the same time, based on the hardware detection model, the target areas where hardware components in the image to be inspected have defects are determined, and then the second defect detection model is used to detect and extract the defects of hardware components in each target area in the image to be inspected, so as to achieve the specialized identification, detection and extraction of specific defect features in local areas.

[0062] Therefore, by combining the general defect features and specific component defect features included in the two defect feature images of the two different defect types mentioned above, defects existing in all hardware components can be identified more accurately, thereby more precisely detecting target hardware components with faults. Furthermore, the aforementioned different types of models can intelligently batch process a large number of images to be inspected, improving the detection efficiency of faulty hardware components, reducing high labor costs, and improving fault diagnosis efficiency and power transmission efficiency.

[0063] As a refinement and extension of the specific implementation of the above embodiments, in order to fully illustrate the specific implementation process of this embodiment, this application provides some other implementation methods for fault detection methods.

[0064] As a method for acquiring images to be detected, multiple on-site images associated with multiple locations on transmission lines and transmission towers are obtained. To avoid duplicate detection of images with high similarity, the multiple on-site images are screened for duplicates, and the multiple on-site images after the screening process are determined as the images to be detected, so as to ensure the detection efficiency of the images to be detected.

[0065] The aforementioned images were acquired from the unmanned aerial vehicle according to a preset cycle.

[0066] The specific process of the above screening and deduplication is as follows: identify at least one on-site image among multiple on-site images whose image similarity is greater than the similarity threshold; and retain the on-site image with the highest similarity among the at least one on-site images, and delete the on-site images other than the on-site image with the highest similarity among the at least one on-site images.

[0067] In one implementation, to facilitate the extraction of common features from the first feature image and the second feature image, each pixel in the first and second feature images is associated with its region location in the image to be detected according to a preset sequence. Based on this implementation, combined with... Figure 2 like Figure 3 As shown, step S14 above can be implemented in the following steps.

[0068] S141, extract the feature pixels representing the location features of the same region in the first feature image and the second feature image respectively, and obtain the first pixel set map and the second pixel set map representing the location of the same region respectively.

[0069] The resolution of the first pixel set is smaller than that of the second pixel set.

[0070] S142, using an upsampling method, the features in the first pixel point set map and the second pixel point set map are fused to obtain a first fused feature map; and using a downsampling method, the features in the first pixel point set map and the second pixel point set map are fused to obtain a second fused feature map.

[0071] S143, when the defect features indicated in the first fusion feature map and the second fusion feature map belong to the same hardware component, the same hardware component is identified as the target hardware component.

[0072] In another implementation, to ensure the speed and accuracy of identifying the target hardware component, it can be based on the above... Figure 3 The pre-defined screening model for classification, constructed and trained according to the implementation principle shown, performs the above step S14.

[0073] Specifically, the first feature image and the second feature image are input into a preset screening model to obtain the target hardware component with the fault.

[0074] The aforementioned preset screening model is used to classify images including non-defect features and images including defect features in the first feature image and the second feature image, respectively, and to remove images with non-defect features and retain images with defect features in the first feature image and the second feature image, and to determine the same hardware component indicated by the images with defect features retained in the first feature image and the second feature image as the target hardware component.

[0075] The preset screening model is trained using historical images containing non-defect features as positive samples and historical images containing defect features as negative samples.

[0076] The aforementioned first defect detection model is designed for multiple defect categories on transmission lines and towers (i.e., including various types of hardware defects). Therefore, this first defect detection model can learn the features of various defect types. Consequently, the training data for this first defect detection model has a wide range of categories, and the image input size is relatively large.

[0077] The second defect detection model targets defects in hardware components with non-standard layout and installation, as well as key types of hardware components. This second model focuses more on learning the defect features specific to each category of the hardware component. Therefore, the training data for this second defect detection model is more targeted, and the image input size is smaller.

[0078] Optional, combined Figure 2 like Figure 4 As shown, step S11 above can be implemented in the following steps.

[0079] S111, adjust the image to be detected into a first image of multiple first preset sizes.

[0080] S112, extract the first defect feature from the multiple first images to obtain the first feature image.

[0081] Optional, combined Figure 2 like Figure 5 As shown, step S13 above can be implemented in the following steps.

[0082] S131, mark the target regions in the image to be detected according to the size range of multiple different target regions on the image to be detected.

[0083] S132, adjust the region-marked image to be detected into a second image of multiple second preset sizes.

[0084] It is understood that there are multiple images to be detected. In one embodiment, each of the multiple images to be detected is scaled to adjust it into multiple second images of a second preset size.

[0085] In another implementation, in order to ensure that local specific characteristics are more prominent, the image to be detected after region marking is further segmented into multiple second images of a second preset size.

[0086] S133, extract the second defect features from the marked regions of multiple second images to obtain the second feature image.

[0087] In order to match the characteristics of the first defect detection model for recognizing large-sized, low-resolution images and the second defect detection model for recognizing small-sized, high-resolution images, the first preset size is set to be larger than the second preset size, so that the first feature image is larger than the second feature image.

[0088] In one specific implementation, the fault detection process described in the above embodiments is implemented using a YOLOv7 target detection network architecture. The specific implementation process is described below. The YOLOv7 architecture mainly consists of three parts: Backbone, FPN, and YOLO Head.

[0089] Firstly, the backbone is the main feature extraction network of YOLOv7. The input image undergoes feature extraction within this backbone network, and the extracted features are called feature layers, which are the feature sets of the input image. In the backbone, this application employs three feature layers for the next stage of network construction; these three feature layers are also called effective feature layers. These three feature layers correspond to the first defect detection model, the hardware detection model, and the second defect detection model, respectively.

[0090] Secondly, FPN is an enhanced feature extraction network of YOLOv7. The three effective feature layers obtained in the backbone are fused in this part to combine feature information at different scales. In the FPN part, the effective feature layers that have already been obtained are used to continue extracting features. In addition, this application not only upsamples the above features to achieve feature fusion, but also downsamples the above features again to achieve feature fusion.

[0091] Third, the YOLO Head is the classifier and regressor of YOLOv7. Through the Backbone and FPN, three enhanced effective feature layers can be obtained. Each feature layer has width, height, and number of channels. At this time, the feature map can be regarded as a set of feature points. The work done by the YOLO Head is to judge the feature points and determine whether the prior box on the feature point has a corresponding hardware component.

[0092] Fourth, the detection results of non-standard installation defects in hardware components (i.e., the first feature image and the second feature image) are summarized and fed into a specially designed classification model. This classification model is used to eliminate image features of hardware components that are installed normally, focusing only on those hardware components with non-standard installation defects. Based on this step, accurate verification of non-standard installation defects in hardware components is achieved, significantly reducing the false detection rate.

[0093] The classification network described above can be the Vision Transformer (ViT), an image classification model based on a self-attention mechanism. First, the input image is divided into fixed-size image patches. Each image patch undergoes a linear transformation to obtain a corresponding embedding vector. These embedding vectors serve as the input to the model. To preserve the positional information of pixels in the image, ViT introduces positional embeddings. Positional embeddings are information added to the input embedding vectors to represent the position of each embedding vector within the image.

[0094] Furthermore, these embedding vectors are processed through a multi-layered Transformer Encoder, each layer including a self-attention mechanism and a feedforward neural network. The self-attention mechanism is used to capture the relationships between different positions in the input sequence. The feedforward neural network is used to perform a non-linear transformation on the features at each position.

[0095] The multiple Transformer Encoder layers mentioned above are stacked to form a deep network, which enables the model to learn more complex feature representations. Global average pooling is then used to merge the features at each location into a global feature vector.

[0096] Furthermore, the feature vectors after global average pooling are input into a fully connected layer to generate the final classification result of the model. This allows for the removal of normally installed hardware components, enabling the verification of defects caused by non-standard installation of hardware components and reducing the false detection rate.

[0097] To achieve the above functions, the fault detection device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art will readily recognize that, based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] This application embodiment also provides a method such as Figure 6 The fault detection device shown is applied to a power transmission intelligent inspection system; the system includes transmission lines, transmission towers, and hardware components for connecting and fixing the transmission lines and transmission towers. The device includes: a first feature extraction unit 61, a second feature extraction unit 62, a fault determination unit 63, and an acquisition unit 64.

[0099] The first feature extraction unit 61 is configured to use a first defect detection model to extract a first defect feature present in the image to be detected, thereby obtaining a first feature image; the first defect feature represents the fault features on the transmission line and transmission tower; and, using a fitting detection model, to detect the location of the defective area of ​​the fitting component in the fitting image included in the image to be detected, thereby obtaining the target area of ​​the defective fitting component in the image to be detected.

[0100] The second feature extraction unit 62 is configured to use the second defect detection model to extract the second defect features present in the target area of ​​the image to be detected, and obtain the second feature image; the second defect features characterize the defects on the hardware component.

[0101] The fault determination unit 63 is configured to determine the target hardware component with a fault based on the first feature image and the second feature image.

[0102] In one possible implementation, each pixel in the first feature image and the second feature image is associated with a region location in the image to be detected according to a preset sequence. The fault determination unit 63 is specifically configured to: extract feature pixels representing the same region location features from the first feature image and the second feature image respectively, obtaining a first pixel set image and a second pixel set image representing the same region location respectively; wherein the resolution of the first pixel set image is less than the resolution of the second pixel set image; fuse the features in the first pixel set image and the second pixel set image using an upsampling method to obtain a first fused feature image; and fuse the features in the first pixel set image and the second pixel set image using a downsampling method to obtain a second fused feature image; when the defect features indicated in the first fused feature image and the second fused feature image belong to the same hardware component, the same hardware component is identified as the target hardware component.

[0103] In another possible implementation, the first feature extraction unit 61 is specifically configured to: adjust the image to be detected into a plurality of first images of a first preset size; extract the first defect feature from the plurality of first images to obtain a first feature image.

[0104] In another possible implementation, the second feature extraction unit 62 is specifically configured to: mark the target regions in the image to be detected according to the size range of multiple different target regions on the image to be detected; adjust the image to be detected after region marking to multiple second images of second preset sizes; extract the image including the second defect feature in the marked region of the multiple second images to obtain the second feature image; the first preset size is larger than the second preset size; the first feature image is larger than the second feature image.

[0105] In another possible implementation, the system further includes an unmanned aerial vehicle (UAV) with a shooting function; the device also includes: an acquisition unit 64 configured to acquire multiple on-site images associated with multiple location areas on the power transmission line and transmission tower; the multiple on-site images are acquired from the UAV at a preset period; the multiple on-site images are subjected to deduplication; the deduplication specifically includes: determining at least one on-site image among the multiple on-site images whose image similarity is greater than a similarity threshold, and retaining the on-site image with the highest similarity among the at least one on-site image and deleting the on-site images other than the on-site image with the highest similarity among the at least one on-site image; and determining the multiple on-site images after deduplication as images to be detected.

[0106] In another possible implementation, the fault determination unit 63 is specifically configured to: input the first feature image and the second feature image into a preset screening model to obtain the target hardware component with a fault; the preset screening model is used to classify the images including non-defect features and the images including defect features in the first feature image and the second feature image respectively, and to remove the images with non-defect features and retain the images with defect features in the first feature image and the second feature image respectively, and to determine the same hardware component indicated by the images with defect features retained in the first feature image and the second feature image respectively as the target hardware component; wherein, the preset screening model is trained with historical images including non-defect features as positive samples and historical images including defect features as negative samples.

[0107] In another possible implementation, the hardware components include nuts, bolts, studs, screws, washers, pins, helical springs, and threaded sleeves; the first defect detection model is a neural network model trained on multiple sets of first sample images containing various defect categories on transmission lines and transmission towers, and the second defect detection model is a neural network model trained on multiple sets of second sample images containing various defect categories of hardware components.

[0108] Regarding the apparatus in the above embodiments, the specific manner in which each unit module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0109] Figure 7 This is a schematic diagram of a fault detection device provided in this application. Figure 7 The fault detection device 70 may include at least one processor 701 and a memory 703 for storing processor-executable instructions. The processor 701 is configured to execute the instructions in the memory 703 to implement the fault detection method in the following embodiments.

[0110] In addition, the fault detection device 70 may also include a communication bus 702, at least one communication interface 704, an input device 706, and an output device 705.

[0111] The processor 701 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0112] The communication bus 702 may include a path for transmitting information between the aforementioned components.

[0113] The communication interface 704 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0114] Input device 706 is used to receive input signals and output device 705 is used to output signals.

[0115] The memory 703 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0116] The memory 703 stores instructions for executing the scheme of this application, and the processor 701 controls the execution. The processor 701 executes the instructions stored in the memory 703 to implement the functions of the method of this application.

[0117] In a specific implementation, as one example, the processor 701 may include one or more CPUs, for example... Figure 7CPU0 and CPU1 in the CPU.

[0118] In a specific implementation, as one example, the fault detection device 70 may include multiple processors, such as... Figure 7 Processors 701 and 707 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0119] The fault detection equipment, such as Figure 7 The diagram includes a processor 701 and a memory 703 for storing executable instructions of the processor 701; wherein the processor 701 is configured to execute executable instructions to implement the fault detection method as described in any of the possible embodiments above. And it can achieve the same technical effect, so to avoid repetition, it will not be described again here.

[0120] This application also provides a power transmission intelligent inspection system, which includes a power transmission line, a power transmission tower, and hardware components for connecting and fixing the power transmission line and the power transmission tower. The system is configured to perform a fault detection method as described in any of the possible embodiments above. Since it achieves the same technical effect, it will not be repeated here to avoid repetition.

[0121] This application also provides a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of a control device or control apparatus, enables the control device or control apparatus to perform a fault detection method as described in any of the possible embodiments above. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0122] This application also provides a computer program product, including a computer program or instructions, which are executed by a processor as a fault detection method according to any of the possible implementations described above. It achieves the same technical effects, and to avoid repetition, will not be described again here.

[0123] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0124] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A fault detection method, characterized in that, Applications in intelligent power transmission inspection systems; The system includes a transmission line, a transmission tower, and hardware components for connecting and fixing the transmission line and the transmission tower; the method includes: Adjust the image to be detected into multiple first images of preset sizes; First feature images are obtained by extracting first defect features from multiple first images; the first defect features characterize the fault features on the transmission line and the transmission tower; and, using a hardware inspection model, the location of defective areas in hardware components in the hardware images included in the image to be inspected is detected to obtain the target area of ​​defective hardware components in the image to be inspected; the target area is a key hardware area where the frequency of improper installation of hardware components is higher than a preset frequency. The target regions in the image to be detected are marked according to the size range of multiple different target regions on the image to be detected; The image to be detected after region marking is adjusted into multiple second images of second preset sizes; The second defect feature is extracted from the marked regions of multiple second images to obtain a second feature image; the first preset size is larger than the second preset size; the first feature image is larger than the second feature image; the second defect feature characterizes a defect on the hardware component; Based on the first feature image and the second feature image, the target hardware component with the fault is identified; The step of determining the faulty target hardware component based on the first feature image and the second feature image includes: when the defects indicated in the first feature image and the second feature image both point to the same hardware component, the hardware component is determined as the faulty target hardware component.

2. The method according to claim 1, characterized in that, The system also includes an unmanned aerial vehicle with a shooting function; the method further includes: Multiple on-site images associated with multiple locations on the power transmission line and the power transmission tower are acquired; the multiple on-site images are acquired from the unmanned aerial vehicle at a preset period. The multiple on-site images are subjected to screening and deduplication. The screening and deduplication process specifically includes: identifying at least one on-site image among the multiple on-site images whose image similarity is greater than a similarity threshold, and retaining the on-site image with the highest similarity among the at least one on-site image and deleting the on-site images other than the on-site image with the highest similarity among the at least one on-site image. Multiple on-site images after weight screening were identified as the images to be tested.

3. The method according to claim 1, characterized in that, The step of determining the faulty target hardware component based on the first feature image and the second feature image includes: The first feature image and the second feature image are input into a preset screening model to obtain the target hardware component with a fault; the preset screening model is used to classify the images including non-defect features and the images including defect features in the first feature image and the second feature image respectively, and to remove the images with non-defect features in the first feature image and the second feature image and retain the images with defect features respectively, and to determine the same hardware component indicated by the images with defect features retained in the first feature image and the second feature image respectively as the target hardware component; The preset screening model is trained using historical images containing non-defect features as positive samples and historical images containing defect features as negative samples.

4. The method according to any one of claims 1 to 3, characterized in that, The hardware components include nuts, bolts, studs, screws, washers, pins, helical springs, and threaded sleeves; the first defect detection model is a neural network model trained on multiple sets of first sample images containing various defect categories on the transmission line and the transmission tower, and the second defect detection model is a neural network model trained on multiple sets of second sample images containing various defect categories of hardware components.

5. A fault detection device, characterized in that, Applications in intelligent power transmission inspection systems; The system includes a transmission line, a transmission tower, and hardware components for connecting and fixing the transmission line and the transmission tower; the device includes: The first feature extraction unit is configured as follows: The image to be detected is adjusted to multiple first images of a first preset size; images including first defect features are extracted from the multiple first images to obtain first feature images; the first defect features characterize the fault features on the transmission line and the transmission tower; and, using a hardware inspection model, the location of defective areas in the hardware images included in the image to be detected is detected to obtain target areas where defective hardware components exist in the image to be detected; the target areas are key hardware areas where the frequency of improper installation of hardware components is higher than a preset frequency. The second feature extraction unit is configured to mark the target regions in the image to be detected according to multiple different target regions within the size range of the image to be detected; adjust the marked image to be detected into multiple second images of preset sizes; extract images including second defect features from the marked regions of the multiple second images to obtain a second feature image; the first preset size is larger than the second preset size; the first feature image is larger than the second feature image; the second defect feature characterizes a defect on the hardware component; The fault determination unit is configured to determine the target hardware component with a fault based on the first feature image and the second feature image. Specifically, the fault determination unit is configured to determine the hardware component with a fault as such when the defects indicated in the first feature image and the second feature image both point to the same hardware component.

6. A power transmission intelligent inspection system, characterized in that, The system includes a transmission line, a transmission tower, and hardware components for connecting and fixing the transmission line and the transmission tower, and the system is configured to perform the fault detection method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by a processor, they enable the fault detection method as described in any one of claims 1-4 to be performed.

Images