Power transmission line fault detection method and device, computer equipment, computer readable storage medium and computer program product

Abstract

The invention relates to a power transmission line fault detection method and device, computer equipment, a computer readable storage medium and a computer program product, and relates to the technical field of power grids. The method comprises the following steps: acquiring a power transmission line image of a power transmission line and a fault detection task for the power transmission line image; inputting the power transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the power transmission line image and text features of the fault detection task; performing fusion processing on the image features and the text features through a fault detection model to obtain cross-modal features of the power transmission line image; and based on the fault detection model and the cross-modal features, determining a target area associated with the fault detection task in the power transmission line image, and taking the target area as a fault detection result of the power transmission line image under the fault detection task. By adopting the method, the fault detection accuracy of the power transmission line can be improved.

Description

Technical Field

[0001] The present application relates to the field of power grid technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for detecting power transmission line faults. Background Art

[0002] As the scale of power grids continues to expand and their complexity increases, their safe operation faces unprecedented challenges.

[0003] In the related art, transmission lines are usually inspected manually to detect faults in the transmission lines, thereby ensuring the safe operation of the power grid.

[0004] However, manual inspection methods are difficult to ensure the comprehensiveness and timeliness of transmission line fault detection, which can easily lead to false detection and missed detection of faults, thereby resulting in a low accuracy rate of transmission line fault detection. Summary of the invention

[0005] Based on this, it is necessary to provide a transmission line fault detection method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy in order to address the technical problem of low accuracy in the above-mentioned transmission line fault detection.

[0006] In a first aspect, the present application provides a method for detecting a power transmission line fault, comprising:

[0007] Acquire a transmission line image of a transmission line and a fault detection task for the transmission line image;

[0008] Inputting the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task;

[0009] Obtaining cross-modal features of the power transmission line image by fusing the image features and the text features through the fault detection model;

[0010] Based on the fault detection model and the cross-modal features, a target area associated with the fault detection task is determined in the power transmission line image as a fault detection result of the power transmission line image under the fault detection task.

[0011] In one of the embodiments, the image feature comprises a feature map of the power transmission line image, and the feature map comprises a plurality of regions;

[0012] The fusing of the image features and the text features by the fault detection model to obtain the cross-modal features of the transmission line image includes:

[0013] Based on the fault detection model and the feature map, obtaining a first feature of each region;

[0014] Based on the fault detection model and the first feature of each region, obtain the second feature of each region; and based on the fault detection model, the first feature of each region and the text feature, obtain the third feature of each region;

[0015] The first feature, the second feature and the third feature of each area are fused through the fault detection model to obtain cross-modal features of the transmission line image.

[0016] In one of the embodiments, the image features further include region features of each region;

[0017] The obtaining of a first feature of each region based on the fault detection model and the feature map includes:

[0018] Determining a first cross-attention layer between each region and the power transmission line image according to the first cross-attention layer in the fault detection model, the regional features of each region and the feature map;

[0019] Based on the first cross-attention of each region, the first cross-attention feature of each region is extracted from the feature map as the first feature of each region.

[0020] In one embodiment, obtaining the second feature of each region based on the fault detection model and the first feature of each region includes:

[0021] Determining the self-attention of each region according to the self-attention layer in the fault detection model and the first feature of each region;

[0022] Based on the self-attention of each region, the first feature of each region is enhanced to obtain the self-attention feature of each region as the second feature of each region.

[0023] In one embodiment, obtaining the third feature of each region based on the fault detection model, the first feature of each region and the text feature includes:

[0024] Determining a second cross attention between each region and the fault detection task according to the second cross attention layer in the fault detection model, the first feature of each region, and the text feature;

[0025] Based on the second cross-attention of each region, the second cross-attention feature of each region is extracted from the text feature as the third feature of each region.

[0026] In one of the embodiments, the power transmission line fault targeted by the fault detection task is at least one type;

[0027] Acquire a plurality of sample fault images of the power transmission line; each sample fault image contains at least one type of power transmission line fault;

[0028] The power transmission line faults present in each sample fault image are labeled to obtain a plurality of labeled sample fault images, and a plurality of sample fault detection tasks are constructed for the plurality of labeled sample fault images; each sample fault detection task targets at least one type of power transmission line fault;

[0029] Based on the multiple labeled sample fault images and the multiple sample fault detection tasks, the fault detection model to be trained is trained until the loss value of the fault detection model is less than a preset loss value, and the fault detection model corresponding to the loss value less than the preset loss value is used as the pre-trained fault detection model;

[0030] Among them, the sample fault detection task includes a single-target sample fault detection task for a single type of transmission line fault, and a multi-target sample fault detection task for multiple types of transmission line faults; the loss value includes at least a single-target loss value, a multi-target loss value and a no-target loss value; the single-target loss value is the loss value of the fault detection model under the single-target sample fault detection task, the multi-target loss value is the loss value of the fault detection model under the multi-target sample fault detection task, and the no-target loss value is the loss value of the fault detection model when the corresponding transmission line fault targeted by the sample fault detection task does not exist in the labeled sample fault image.

[0031] In a second aspect, the present application also provides a transmission line fault detection device, comprising:

[0032] An information acquisition module, used to acquire a transmission line image of a transmission line and a fault detection task for the transmission line image;

[0033] A feature extraction module, used for inputting the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task;

[0034] A feature fusion module, used for fusing the image features and the text features through the fault detection model to obtain cross-modal features of the transmission line image;

[0035] A fault detection module is used to determine a target area associated with the fault detection task in the power transmission line image based on the fault detection model and the cross-modal features as a fault detection result of the power transmission line image under the fault detection task.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Acquire a transmission line image of a transmission line and a fault detection task for the transmission line image;

[0038] Inputting the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task;

[0039] Obtaining cross-modal features of the power transmission line image by fusing the image features and the text features through the fault detection model;

[0040] Based on the fault detection model and the cross-modal features, a target area associated with the fault detection task is determined in the power transmission line image as a fault detection result of the power transmission line image under the fault detection task.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] Acquire a transmission line image of a transmission line and a fault detection task for the transmission line image;

[0043] Inputting the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task;

[0044] Obtaining cross-modal features of the power transmission line image by fusing the image features and the text features through the fault detection model;

[0045] Based on the fault detection model and the cross-modal features, a target area associated with the fault detection task is determined in the power transmission line image as a fault detection result of the power transmission line image under the fault detection task.

[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0047] Acquire a transmission line image of a transmission line and a fault detection task for the transmission line image;

[0048] Inputting the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task;

[0049] Obtaining cross-modal features of the power transmission line image by fusing the image features and the text features through the fault detection model;

[0050] Based on the fault detection model and the cross-modal features, a target area associated with the fault detection task is determined in the power transmission line image as a fault detection result of the power transmission line image under the fault detection task.

[0051] The above-mentioned transmission line fault detection method, device, computer equipment, computer-readable storage medium and computer program product first obtain a transmission line image of the transmission line and a fault detection task for the transmission line image; then, the transmission line image and the fault detection task are input into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task; then, the image features and text features are fused by the fault detection model to obtain cross-modal features of the transmission line image; finally, based on the fault detection model and the cross-modal features, a target area associated with the fault detection task is determined in the transmission line image as the fault detection result of the transmission line image under the fault detection task. In this way, through the transmission line image and the fault detection task for the transmission line image, the image features of the transmission line image and the text features of the fault detection task can be obtained. Based on the fusion processing of the image features and the text features, the cross-modal features of the transmission line image can be obtained. Based on the cross-modal features of the transmission line image, the target area associated with the fault detection task can be determined from the transmission line image, thereby realizing the fault detection of the transmission line. The above process no longer relies on manual inspections for fault detection, thereby improving the accuracy of transmission line fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] FIG1 is a schematic diagram of a flow chart of a method for detecting a power transmission line fault in one embodiment;

[0054] FIG2 is a flow chart showing steps of obtaining cross-modal features of a power transmission line image by fusing image features and text features through a fault detection model in one embodiment;

[0055] FIG3 is a flow chart of steps for obtaining a first feature of each region based on a fault detection model and a feature map in one embodiment;

[0056] FIG4 is a flow chart of steps for obtaining a second feature of each region based on a fault detection model and a first feature of each region in one embodiment;

[0057] FIG5 is a flowchart of steps for obtaining a third feature of each region based on a fault detection model, a first feature of each region, and a text feature in one embodiment;

[0058] FIG6 is a flow chart of steps for training a fault detection model in one embodiment;

[0059] FIG7 is a schematic flow chart of a method for detecting a power transmission line fault in another embodiment;

[0060] FIG8 is a schematic diagram of a flow chart of a method for detecting defects in a power transmission line based on a multimodal model in one embodiment;

[0061] FIG9 is a schematic diagram of a relationship modeling module in one embodiment;

[0062] FIG10 is a block diagram of a transmission line fault detection device according to an embodiment;

[0063] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0066] In one embodiment, as shown in FIG1 , a method for detecting a power transmission line fault is provided. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a server and a terminal, and can be implemented through the interaction between the server and the terminal; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. In this embodiment, the method includes the following steps:

[0067] Step S102: acquiring a transmission line image of the transmission line and a fault detection task for the transmission line image.

[0068] The transmission line image may be acquired by an image acquisition device installed near the transmission line, or by an inspection device (such as a drone) equipped with an image acquisition device.

[0069] The fault detection task is a text used to describe the detection requirements for the power transmission line faults existing in the power transmission line image.

[0070] Among them, transmission line failure refers to abnormal conditions of transmission lines, specifically including defects in transmission line components and hidden dangers of transmission line components; transmission line components include at least glass insulators, composite insulators, chains, suspension wire clamps, long clips, short clips, hardware triangles, hardware discs, shock-proof hammers and cotter pins; transmission line failures include at least self-explosion of glass insulators, dirty glass insulators, damaged composite insulator sheds, rusted hardware, skewed hardware, damaged shock-proof hammers, displacement and touching of shock-proof hammers, missing cotter pins, bird nests and honeycombs.

[0071] Specifically, when the inspection personnel need to perform fault inspection on a certain power transmission line image, they input the power transmission line image and the fault inspection task text into the server.

[0072] For example, if the inspector needs to detect whether there is a transmission line fault such as a broken anti-shock hammer and a skewed hardware in a transmission line image, the transmission line image and the fault detection task text "broken anti-shock hammer, skewed hardware" are input into the server.

[0073] Step S104: input the power transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the power transmission line image and text features of the fault detection task.

[0074] The server is equipped with a pre-trained fault detection model, which is a neural network model used to determine the target area associated with the transmission line fault in the transmission line image. The fault detection model mainly includes an encoder layer, an attention layer, and a decoder layer.

[0075] Specifically, the server inputs the transmission line image and the fault detection task into the encoder layer of the fault detection model, and the encoder encodes the transmission line image and the fault detection task to obtain the image features of the transmission line image and the text features of the fault detection task respectively.

[0076] In a specific application, the encoder layer includes an image encoder and a text encoder; the server inputs the transmission line image into the image encoder, and obtains a feature map of the transmission line as the image feature of the transmission line through encoding processing of the transmission line image by the image encoder; the server inputs the fault detection task text into the text encoder, and obtains the text feature of the fault detection task through encoding processing of the fault detection task text by the text encoder.

[0077] In practical applications, the image encoder uses Swin Transformer (a deep learning architecture based on the Transformer model) based on a sliding window mechanism with a hierarchical design (downsampling layer); the sliding window operation includes non-overlapping local windows and overlapping cross windows. Swin Transformer limits the attention calculation to one window, which can introduce the locality of the convolution operation of CNN (Convolutional Neural Network) on the one hand, and greatly save the amount of calculation on the other hand. Through the hierarchical design of downsampling, the receptive field can be gradually increased, so that the attention mechanism can also pay attention to global features.

[0078] The text encoder uses BERT (Bidirectional Encoder Representations from Transformers), which is a pre-trained language model based on Transformer, driven by two main tasks: masked language model and next sentence prediction. In the masked language model task, BERT randomly masks some words in the input sentence, and then trains the model to predict these masked words. In the next sentence prediction task, BERT trains the model to predict whether two sentences are continuous. Through these two tasks, BERT can learn rich language representations. Furthermore, the lightest BERT model used in this application has a 12-layer Transformer encoder module, 12-head attention, and a 768-dimensional hidden layer.

[0079] Step S106, fusing the image features and the text features through the fault detection model to obtain cross-modal features of the transmission line image.

[0080] Specifically, the server inputs the image features and text features into the attention layer of the fault detection model, and fuses the image features and text features based on the attention mechanism to obtain the cross-modal features of the transmission line image.

[0081] In specific applications, the attention layer can be multiple attention layers, such as self-attention layer, cross-attention layer, etc.

[0082] Step S108 , based on the fault detection model and the cross-modal features, a target area associated with the fault detection task is determined in the power transmission line image as a fault detection result of the power transmission line image under the fault detection task.

[0083] Specifically, the server inputs the cross-modal features of the transmission line image into the decoding layer of the fault detection model, and determines the target area associated with the fault detection task in the transmission line image through the decoding layer's decoding of the cross-modal features, for example, the area where the transmission line fault targeted by the fault detection task exists, thereby obtaining the fault detection result of the transmission line image under the fault detection task.

[0084] For example, assuming that the fault detection task text is "the anti-shock hammer is broken and the hardware is crooked", the decoding layer decodes the cross-modal features, determines the areas where "the anti-shock hammer is broken" and "the hardware is crooked" exist in the transmission line image, and outputs the determined areas as the target areas associated with the fault detection task "the anti-shock hammer is broken and the hardware is crooked", and obtains the fault detection result of the transmission line image under the fault detection task "the anti-shock hammer is broken and the hardware is crooked".

[0085] In the above-mentioned transmission line fault detection method, first, the server obtains a transmission line image of the transmission line and a fault detection task for the transmission line image; then, the server inputs the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task; then, the server fuses the image features and text features through the fault detection model to obtain cross-modal features of the transmission line image; finally, based on the fault detection model and the cross-modal features, the server determines the target area associated with the fault detection task in the transmission line image as the fault detection result of the transmission line image under the fault detection task. In this way, through the transmission line image and the fault detection task for the transmission line image, the image features of the transmission line image and the text features of the fault detection task can be obtained. Based on the fusion processing of the image features and the text features, the cross-modal features of the transmission line image can be obtained. Based on the cross-modal features of the transmission line image, the target area associated with the fault detection task can be determined from the transmission line image, thereby realizing the fault detection of the transmission line. The above process no longer relies on manual inspections for fault detection, thereby improving the accuracy of transmission line fault detection.

[0086] In an exemplary embodiment, the image feature includes a feature map of a power transmission line image, and the feature map includes a plurality of regions.

[0087] When the image encoder encodes the power transmission line image, it obtains a feature map of the power transmission line image and divides the feature map into multiple regions.

[0088] As shown in FIG. 2 , the above step S106, in which the image features and the text features are fused by the fault detection model to obtain the cross-modal features of the transmission line image, specifically includes the following steps:

[0089] Step S202: obtaining a first feature of each region based on the fault detection model and the feature map.

[0090] Step S204, based on the fault detection model and the first feature of each region, obtain the second feature of each region, and based on the fault detection model, the first feature of each region and the text feature, obtain the third feature of each region.

[0091] Step S206: The first feature, the second feature, and the third feature of each region are fused by the fault detection model to obtain a cross-modal feature of the transmission line image.

[0092] Specifically, the attention layer includes three layers; the first attention layer is based on the feature map to obtain the first feature of each area of ​​the feature map, the second attention layer is based on the first feature of each area, and further obtains the second feature of each area. The third attention layer is parallel to the second attention layer, and obtains the third feature of each area based on the first feature and text feature of each area. Finally, the fault detection model also includes a multi-layer perceptron, and the first feature, second feature and third feature of each area are weighted fused by the multi-layer perceptron to obtain the cross-modal features of the transmission line image.

[0093] In this embodiment, the server can obtain the first feature, the second feature and the third feature of each area of ​​the feature map based on the multi-layer attention layer, the feature map as the image feature and the text feature, and based on the fusion processing of the first feature, the second feature and the third feature, it can realize the trans-membrane fusion of the image and the text and obtain the trans-membrane feature of the transmission line image.

[0094] In an exemplary embodiment, the image features further include region features of each region.

[0095] Among them, when the image encoder divides the feature map into multiple regions, it also generates regional features for each region.

[0096] As shown in FIG3 , the above step S202 obtains the first feature of each region based on the fault detection model and the feature map, and specifically includes the following steps:

[0097] Step S302, determining the first cross attention between each region and the power transmission line image according to the first cross attention layer in the fault detection model, the regional features and the feature map of each region.

[0098] Step S304, based on the first cross-attention of each region, extract the first cross-attention feature of each region from the feature map as the first feature of each region.

[0099] Among them, the first attention layer is the first cross-attention layer; the first feature of each region is the first cross-attention feature between the region and the power transmission line image.

[0100] Specifically, as shown in Formula 1, for each region, the first cross-attention layer first calculates the attention map between the regional features and feature maps of the region, and obtains the first cross-attention between the region and the power line image:

[0101] (Formula 1)

[0102] in, is the attention map corresponding to the region, that is, the first cross attention corresponding to the region, indicating its corresponding spatial area in the feature map; is the characteristic vector of the region; is the feature map; is the first learnable parameter; is the GELU activation function.

[0103] Then, as shown in Formula 2, the first criss-cross attention layer is based on the first criss-cross attention of the region , extract the first cross-attention feature of the region from the feature map, and get the first feature of the region :

[0104] (Formula 2)

[0105] in, It is the first characteristic of the region; is the second learnable parameter.

[0106] In this embodiment, the server can extract the first feature of each area from the feature map by cross-attention between the area of ​​the feature map and the feature map of the transmission line, thereby better capturing the relationship between the local feature map and the overall feature map.

[0107] In an exemplary embodiment, as shown in FIG4 , in the above step S204, based on the fault detection model and the first features of each region, the second feature of each region is obtained, which specifically includes the following steps:

[0108] Step S402, determining the self-attention of each region according to the self-attention layer in the fault detection model and the first feature of each region.

[0109] Step S404: enhance the first feature of each region based on the self-attention of each region to obtain the self-attention feature of each region as the second feature of each region.

[0110] Among them, the second attention layer is the self-attention layer; the second feature of each region is the self-attention feature of the region.

[0111] Specifically, the self-attention layer first calculates the attention matrix corresponding to each region based on the first feature of each region to obtain the self-attention of each region; then, for each region, the first feature of the region is enhanced based on the self-attention of the region to obtain the self-attention feature of the region and the second feature of the region. .

[0112] In this embodiment, the server can enhance the first feature of each region through self-attention of the region of the feature map, thereby obtaining the second feature of each region, thereby better capturing the relationship between the local parts of the feature map.

[0113] In an exemplary embodiment, as shown in FIG5 , in the above step S204, based on the fault detection model, the first feature of each region and the text feature, the third feature of each region is obtained, which specifically includes the following steps:

[0114] Step S502, determining the second cross attention between each region and the fault detection task according to the second cross attention layer in the fault detection model, the first feature of each region and the text feature.

[0115] Step S504, based on the second cross attention of each region, extract the second cross attention feature of each region from the text feature as the third feature of each region.

[0116] Among them, the third attention layer is the second cross attention layer; the third feature of each region is the second cross attention between the region and the fault detection task.

[0117] Specifically, as shown in Formula 3, for each region, the second cross-attention layer first calculates the regional features of the region and text features The second cross attention :

[0118] (Formula 3)

[0119] in, is the second cross attention corresponding to the region; is the characteristic vector of the region; is the text feature; is the third learnable parameter, is the fourth learnable parameter; is the GELU activation function.

[0120] Then, as shown in Formula 4, the second criss-cross attention layer is based on the second criss-cross attention of the region , from the text features Extract the second cross-attention feature of the area and get the third feature of the area:

[0121] (Formula 4)

[0122] in, It is the third characteristic of the region.

[0123] In this embodiment, the server can extract the third feature of each region from the text feature through cross-attention between the region of the feature map and the text feature, thereby better capturing the relationship between the local feature map and the fault detection task.

[0124] Based on the above-mentioned first cross-attention layer, self-attention layer and second cross-attention layer, the complex interaction between the transmission line image and the fault detection task text can be better modeled, thereby improving the expression and understanding ability of the fault detection model.

[0125] In an exemplary embodiment, the fault detection task targets at least one type of transmission line fault; for example, the fault detection task may be to detect the self-explosion of glass insulators in a transmission line image, or to detect both the self-explosion of glass insulators and the contamination of glass insulators.

[0126] As shown in FIG6 , the power transmission line fault detection method provided in the present application further includes the following steps for training the pre-trained fault detection model in any of the above embodiments:

[0127] Step S602: Acquire multiple sample fault images of the power transmission line.

[0128] Step S604 , annotating the power transmission line faults in each sample fault image to obtain a plurality of annotated sample fault images, and constructing a plurality of sample fault detection tasks for the plurality of annotated sample fault images.

[0129] Step S606, based on multiple labeled sample fault images and multiple sample fault detection tasks, the fault detection model to be trained is trained until the loss value of the fault detection model is less than the preset loss value, and the corresponding fault detection model with a loss value less than the preset loss value is used as the pre-trained fault detection model.

[0130] There is at least one type of power transmission line fault in each sample fault image; for example, there may be only one type of power transmission line fault in the sample fault image, or there may be multiple types of power transmission line faults at the same time.

[0131] Each sample fault detection task targets at least one type of transmission line fault.

[0132] The sample fault detection task is a text used to describe the detection requirements for power transmission line faults present in sample images in the power transmission line fault dataset.

[0133] Among them, the sample fault detection task includes a single-target sample fault detection task for a single type of transmission line fault, and a multi-target sample fault detection task for multiple types of transmission line faults.

[0134] Among them, the loss values ​​include at least single-target loss values, multi-target loss values ​​and no-target loss values; the single-target loss value is the loss value of the fault detection model under the single-target sample fault detection task, the multi-target loss value is the loss value of the fault detection model under the multi-target sample fault detection task, and the no-target loss value is the loss value of the fault detection model when the corresponding sample fault detection task does not exist in the labeled sample fault image.

[0135] Specifically, the server first collects multiple fault images of the power transmission line based on the drone, and at least one type of power transmission line fault exists in these fault images.

[0136] Then, the server expands multiple fault images to obtain more fault images and obtain multiple sample fault images. In practical applications, the server uses data enhancement methods such as changing the brightness, saturation, contrast of image colors, changing the scale of images, cropping, scaling, horizontal and vertical flipping, translation transformation, radiation transformation, Gaussian noise, 0 pixel filling, etc. of random images to increase the number of fault images and the diversity of fault images.

[0137] Next, the server marks the transmission line faults present in multiple sample fault images to obtain multiple marked sample fault images. In practical applications, the server needs to mark the defects of the transmission line components present in each sample fault image. The transmission line components targeted by this application include at least glass insulators, composite insulators, chains, suspension wire clamps, long clips, short clips, hardware triangles, hardware discs, shockproof hammers and cotter pins. The targeted defects include at least self-explosion of glass insulators, dirty glass insulators, damaged composite insulator sheds, rusted hardware, skewed hardware, damaged shockproof hammers, displacement and contact of shockproof hammers, missing cotter pins, bird nests and honeycombs.

[0138] Next, the server constructs multiple sample fault detection tasks for multiple labeled sample fault images. In actual applications, the server constructs multiple sample fault detection task texts.

[0139] Finally, the server iteratively trains the fault detection model to be trained based on multiple labeled sample fault images and multiple sample fault detection tasks until the loss value of the fault detection model is less than the preset loss value, and uses the corresponding fault detection model with a loss value less than the preset loss value as the pre-trained fault detection model.

[0140] In practical applications, "the loss value of the fault detection model is less than the preset loss value" can mean that the single-target loss value, the multi-target loss value and the no-target loss value are all less than their corresponding preset loss values, or the fusion loss value of the single-target loss value, the multi-target loss value and the no-target loss value is less than the preset loss value.

[0141] In this embodiment, the server trains the fault detection model to be trained through sample images and sample fault detection tasks, and can obtain a pre-trained fault detection model. Based on the pre-trained fault detection model, the server can no longer rely on manual inspections for fault detection, thereby improving the accuracy of fault detection of transmission lines.

[0142] In an exemplary embodiment, if the fault detection model cannot determine the target area associated with the fault detection task in the transmission line image based on the cross-modal features, then the fault detection model outputs prompt information indicating that the transmission line fault targeted by the fault detection task does not exist in the transmission line image as the fault detection result of the transmission line image under the fault detection task.

[0143] In an exemplary embodiment, as shown in FIG. 7 , another method for detecting a power transmission line fault is provided, which is described by taking the method applied to a server as an example, and includes the following steps:

[0144] Step S702: Acquire multiple sample fault images of the power transmission line.

[0145] Step S704, annotate the power transmission line faults in each sample fault image to obtain multiple annotated sample fault images, and construct multiple single-target sample fault detection tasks and multiple multi-target sample fault detection tasks for the multiple annotated sample fault images.

[0146] Step S706, based on multiple labeled sample fault images and multiple sample fault detection tasks, the fault detection model to be trained is trained until the single-target loss value, multi-target loss value and no-target loss value of the fault detection model are all less than their corresponding preset loss values, thereby obtaining a pre-trained fault detection model.

[0147] Step S708: acquiring a transmission line image of the transmission line and a fault detection task for the transmission line image.

[0148] Step S710, encoding the transmission line image through the image encoder in the fault detection model to obtain a feature map of the transmission line image and regional features of each region in the feature map, and encoding the fault detection task through the text encoder in the pre-trained fault detection model to obtain text features of the fault detection task.

[0149] Step S712, according to the first cross-attention layer in the fault detection model, the regional features of each region and the feature map, determine the first cross-attention between each region and the transmission line image, and based on the first cross-attention of each region, extract the first cross-attention feature of each region from the feature map.

[0150] Step S714, according to the self-attention layer in the fault detection model and the first feature of each region, the self-attention of each region is determined, and the first feature of each region is enhanced based on the self-attention of each region to obtain the self-attention feature of each region.

[0151] Step S716, determines the second cross-attention between each region and the fault detection task according to the second cross-attention layer in the fault detection model, the first feature of each region and the text feature, and extracts the second cross-attention feature of each region from the text feature based on the second cross-attention of each region.

[0152] Step S718, fusing the first cross-attention features, the self-attention features, and the second cross-attention features of each region to obtain the cross-modal features of the power transmission line image.

[0153] Step S720 , through the decoding process of the cross-modal features by the decoder layer in the fault detection model, a target area associated with the fault detection task is determined in the power transmission line image as a fault detection result of the power transmission line image under the fault detection task.

[0154] In this embodiment, through the transmission line image and the fault detection task for the transmission line image, the image features of the transmission line image and the text features of the fault detection task can be obtained. Based on the fusion processing of the image features and the text features, the cross-modal features of the transmission line image can be obtained. Based on the cross-modal features of the transmission line image, the target area associated with the fault detection task can be determined from the transmission line image, thereby realizing the fault detection of the transmission line. The above process no longer relies on manual inspections for fault detection, thereby improving the accuracy of fault detection of the transmission line.

[0155] In order to more clearly illustrate the power transmission line fault detection method provided by the embodiment of the present application, the power transmission line fault detection method is specifically described below with a specific embodiment, but it should be understood that the embodiment of the present application is not limited thereto. As shown in FIG8 , in one of the exemplary embodiments, the present application also provides a power transmission line defect detection method based on a multimodal model, which specifically includes the following contents:

[0156] First, the transmission line defect images are collected, and the transmission line defect dataset based on multi-objective expressions is constructed. Then, the fusion module of the image features and text features of the multimodal model is improved, and the relational modeling network is added. Finally, the image and text are input into the multimodal model for training, so as to understand the input expression and locate the transmission line defect target described by it in the image. It has certain practical value in the intelligent inspection of transmission lines.

[0157] Among them, referring expression comprehension (REC), as a multimodal task combining vision and language, aims to understand the content of the input referring expression and locate the target object described by it in the image. It is a task that is focused on in both computer vision and natural language processing. Traditional referring expression comprehension does not consider non-target expressions, that is, natural expression sentences that do not have a matching object in the image. This means that errors may occur if the target described by the sentence does not exist in the input image. This embodiment constructs a transmission line defect dataset based on multi-target expressions, which contains 50,000 expressions, including 45,000 multi-target expressions and 5,000 non-target expressions.

[0158] The multimodal model includes a backbone network for image feature extraction, a backbone network for text feature extraction, a module for fusing image features and text features and modeling relationships, and a cross-modal decoder for detection box refinement. For each pair of (image, text) inputs, the image feature extraction backbone network is first used to extract the image feature map, and the text feature of the text is extracted using the text feature extraction backbone network. These two features are input into the relationship modeling module for cross-modal feature fusion. After obtaining the cross-modal features, they are input into the last decoder layer to output the object box related to the text in the image.

[0159] As shown in Figure 9, the relational modeling module consists of a Region-Image Cross Attention (RIA) module and a Region-Language Cross Attention (RLA) module. The RIA module flexibly collects regional features of images, while the RLA module captures the relationship between regions and the dependency between regions and text. Through these two modules, the complex interaction between images and text can be better modeled, and the ability of multimodal models to understand reference expressions can be improved.

[0160] In addition, in order to objectively evaluate the detection effect of the improved multimodal model on the transmission line defect test set, the present invention selects accuracy and recall as objective evaluation indicators. The target detection based on the multimodal model proposed in this embodiment has an accuracy of 75.5% and a recall of 78.7% on the transmission line defect test set, while the accuracy of the classic YOLOv5 model (a target detection model) is 78.5% and the recall is 80.1%. Although the detection accuracy of this embodiment is not as good as the commonly used YOLOv5 detection algorithm, it can realize multimodal input of images and texts, and can locate multiple related targets described in the referential expression in the image, which has certain practical value in the intelligent inspection of transmission lines.

[0161] In this embodiment, based on the multimodal input of textual reference expressions and images, multiple related targets described in the reference expressions can be located in the image. This is an exploration of the multimodal model in transmission line defect detection and has certain practical value in the intelligent inspection of transmission lines.

[0162] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiment of the present application also provides a transmission line fault detection device for implementing the above-mentioned transmission line fault detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the transmission line fault detection device provided below can refer to the limitations of the transmission line fault detection method above, and will not be repeated here.

[0164] In an exemplary embodiment, as shown in FIG10 , a transmission line fault detection device is provided, comprising: an information acquisition module 1002, a feature extraction module 1004, a feature fusion module 1006 and a fault detection module 1008, wherein:

[0165] The information acquisition module 1002 is used to acquire a transmission line image of the transmission line and a fault detection task for the transmission line image.

[0166] The feature extraction module 1004 is used to input the power transmission line image and the fault detection task into the pre-trained fault detection model to obtain the image features of the power transmission line image and the text features of the fault detection task.

[0167] The feature fusion module 1006 is used to obtain cross-modal features of the power transmission line image by fusing image features and text features through a fault detection model.

[0168] The fault detection module 1008 is used to determine a target area associated with the fault detection task in the power transmission line image based on the fault detection model and the cross-modal features as a fault detection result of the power transmission line image under the fault detection task.

[0169] In an exemplary embodiment, the image feature comprises a feature map of the power line image, and the feature map comprises a plurality of regions.

[0170] The feature fusion module 1006 is also used to obtain the first feature of each area based on the fault detection model and the feature map; obtain the second feature of each area based on the fault detection model and the first feature of each area; and obtain the third feature of each area based on the fault detection model, the first feature of each area and the text feature; and obtain the cross-modal features of the transmission line image by fusing the first feature, the second feature and the third feature of each area through the fault detection model.

[0171] In an exemplary embodiment, the image features further include region features of each region.

[0172] The feature fusion module 1006 is also used to determine the first cross-attention between each region and the transmission line image according to the first cross-attention layer in the fault detection model, the regional features of each region and the feature map; based on the first cross-attention of each region, the first cross-attention feature of each region is extracted from the feature map as the first feature of each region.

[0173] In an exemplary embodiment, the feature fusion module 1006 is also used to determine the self-attention of each region based on the self-attention layer in the fault detection model and the first feature of each region; based on the self-attention of each region, the first feature of each region is enhanced to obtain the self-attention feature of each region as the second feature of each region.

[0174] In an exemplary embodiment, the feature fusion module 1006 is also used to determine the second cross-attention between each region and the fault detection task based on the second cross-attention layer in the fault detection model, the first feature of each region and the text feature; based on the second cross-attention of each region, the second cross-attention feature of each region is extracted from the text feature as the third feature of each region.

[0175] In an exemplary embodiment, the fault detection task targets at least one type of power transmission line fault.

[0176] The power transmission line fault detection device also includes a model training module, which is used to obtain multiple sample fault images of the power transmission line; each sample fault image has at least one type of power transmission line fault; the power transmission line fault existing in each sample fault image is labeled to obtain multiple labeled sample fault images, and multiple sample fault detection tasks are constructed for the multiple labeled sample fault images; based on the multiple labeled sample fault images and the multiple sample fault detection tasks, the fault detection model to be trained is trained until the loss value of the fault detection model is less than the preset loss value, and the corresponding fault detection model with the loss value less than the preset loss value is used as the pre-trained The obtained fault detection model; wherein the sample fault detection task includes a single-target sample fault detection task for a single type of transmission line fault, and a multi-target sample fault detection task for multiple types of transmission line faults; the loss value includes at least a single-target loss value, a multi-target loss value and a no-target loss value; the single-target loss value is the loss value of the fault detection model under the single-target sample fault detection task, the multi-target loss value is the loss value of the fault detection model under the multi-target sample fault detection task, and the no-target loss value is the loss value of the fault detection model when the corresponding sample fault detection task does not exist in the labeled sample fault image.

[0177] Each module in the above-mentioned power transmission line fault detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0178] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store images and fault data of the transmission line. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting a transmission line fault is implemented.

[0179] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0181] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0182] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0183] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting a power transmission line fault, characterized in that: The method comprises: Acquire a transmission line image of a transmission line and a fault detection task for the transmission line image; Inputting the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task; Obtaining cross-modal features of the power transmission line image by fusing the image features and the text features through the fault detection model; Based on the fault detection model and the cross-modal features, a target area associated with the fault detection task is determined in the power transmission line image as a fault detection result of the power transmission line image under the fault detection task.

2. The method according to claim 1, characterized in that The image feature comprises a feature map of the power transmission line image, wherein the feature map comprises a plurality of regions; The fusing of the image features and the text features by the fault detection model to obtain the cross-modal features of the transmission line image includes: Based on the fault detection model and the feature map, obtaining a first feature of each region; Based on the fault detection model and the first feature of each region, obtain the second feature of each region; and based on the fault detection model, the first feature of each region and the text feature, obtain the third feature of each region; The first feature, the second feature and the third feature of each area are fused through the fault detection model to obtain cross-modal features of the power transmission line image.

3. The method according to claim 2, characterized in that The image features also include regional features of each region; The obtaining of a first feature of each region based on the fault detection model and the feature map includes: Determining a first cross-attention layer between each region and the power transmission line image according to the first cross-attention layer in the fault detection model, the regional features of each region and the feature map; Based on the first cross-attention of each region, the first cross-attention feature of each region is extracted from the feature map as the first feature of each region.

4. The method according to claim 2, characterized in that: The obtaining the second feature of each region based on the fault detection model and the first feature of each region includes: Determining the self-attention of each region according to the self-attention layer in the fault detection model and the first feature of each region; Based on the self-attention of each region, the first feature of each region is enhanced to obtain the self-attention feature of each region as the second feature of each region.

5. The method according to claim 2, characterized in that: The obtaining, based on the fault detection model, the first feature of each region and the text feature, the third feature of each region comprises: Determining a second cross attention between each region and the fault detection task according to the second cross attention layer in the fault detection model, the first feature of each region, and the text feature; Based on the second cross-attention of each region, the second cross-attention feature of each region is extracted from the text feature as the third feature of each region.

6. The method according to any one of claims 1 to 5, characterized in that: The power transmission line fault targeted by the fault detection task is at least one type; The pre-trained fault detection model is trained in the following way: Acquire a plurality of sample fault images of the power transmission line; each sample fault image contains at least one type of power transmission line fault; Annotate the power transmission line faults in each sample fault image to obtain a plurality of annotated sample fault images, and construct a plurality of sample fault detection tasks for the plurality of annotated sample fault images; Based on the multiple labeled sample fault images and the multiple sample fault detection tasks, the fault detection model to be trained is trained until the loss value of the fault detection model is less than a preset loss value, and the fault detection model corresponding to the loss value less than the preset loss value is used as the pre-trained fault detection model; The sample fault detection task includes a single-target sample fault detection task for a single type of transmission line fault and a multi-target sample fault detection task for multiple types of transmission line faults; the loss value includes at least a single-target loss value, a multi-target loss value and a non-target loss value; The single-target loss value is the loss value of the fault detection model under the single-target sample fault detection task, the multi-target loss value is the loss value of the fault detection model under the multi-target sample fault detection task, and the non-target loss value is the loss value of the fault detection model when there is no corresponding transmission line fault targeted by the sample fault detection task in the labeled sample fault image.

7. A transmission line fault detection device, characterized in that: The device comprises: An information acquisition module, used to acquire a transmission line image of a transmission line and a fault detection task for the transmission line image; A feature extraction module, used for inputting the transmission line image and the fault detection task into a pre-trained fault detection model to obtain image features of the transmission line image and text features of the fault detection task; A feature fusion module, used for fusing the image features and the text features through the fault detection model to obtain cross-modal features of the transmission line image; A fault detection module is used to determine a target area associated with the fault detection task in the power transmission line image based on the fault detection model and the cross-modal features as a fault detection result of the power transmission line image under the fault detection task.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images