A power transmission line hidden danger detection method

Abstract

The application provides a power transmission line hidden danger detection method, which combines image and sensor data to construct a power transmission line hidden danger detection model based on visual language mode. Image preprocessing and image encoder are used to extract hidden danger features in the image, and text description information is combined for training. Combined with the autoregressive language model, the model can generate inference results based on visual features and sensor data, providing intelligent decision support for operation and maintenance personnel. In particular, when combined with images obtained by unmanned aerial vehicles and pin sensor data, the model can not only identify current hidden dangers, but also infer potential abnormal situations, achieving comprehensive detection and prediction of hidden dangers. This method significantly improves the accuracy and robustness of hidden danger identification in complex environments such as wind disasters and rainfall, providing efficient and intelligent support for power inspection and operation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of power transmission line detection, and particularly relates to a power transmission line hidden danger detection method. BACKGROUND

[0002] With the development of modern power networks, power transmission lines, as an important part of the power system, bear the key task of power transmission. In order to ensure the safe and stable operation of the power system, the detection and maintenance of hidden dangers of power transmission lines have become an important link to ensure power supply. However, in the process of inspection of power transmission lines, many challenges are often faced, especially in adverse weather conditions such as wind disasters, the identification and positioning of hidden dangers become more difficult.

[0003] Common hidden dangers of power transmission lines include broken strands of conductors, foreign objects (such as floating objects) on the conductors, and loose hardware, etc. Broken strands of conductors can cause conductor breakage, and in severe cases can cause large-scale power outages; floating objects on the conductors, especially in wind disaster weather, can cause changes in the electrical performance of the conductors or damage to power equipment; loose hardware can cause the line connection to be not firm, affecting the safety of power transmission. Traditional manual inspection methods rely on visual inspection by workers or the use of portable detection equipment, which is low in efficiency and high in cost, and in adverse weather conditions, the difficulty and risk of manual inspection are greatly increased.

[0004] In adverse weather conditions such as wind disasters, the detection of hidden dangers of power transmission lines faces more complex challenges. The increase in wind speed and the change in precipitation can seriously affect the stability of image quality during inspection. Strong winds can cause the conductor to vibrate violently, affecting the clarity of the image; at the same time, rain and snow weather can cause the image to be blurred, and even block part of the appearance of the conductor or hardware, increasing the difficulty of hidden danger identification. In addition, traditional image processing methods are difficult to effectively cope with these dynamic changes, resulting in reduced recognition accuracy and making it difficult to accurately detect in complex environments.

[0005] Currently, the detection technology for hidden dangers of power transmission lines mainly includes manual inspection, unmanned aerial vehicle inspection, and automatic detection methods based on image processing and sensor data. Manual inspection has the problems of long cycle, low efficiency, high cost, and difficulty in operation in adverse weather; unmanned aerial vehicle inspection can improve the inspection efficiency, but is limited by image transmission, weather influence and the limitations of recognition algorithms, and it is difficult to achieve efficient recognition in complex environments. While the automatic method based on image processing and sensor data has made some progress in some scenarios, it still faces the problems of low image quality, tedious data processing, etc. in complex environments such as wind disasters, rain and snow. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the prior art and provide a power transmission line hidden danger detection method.

[0007] The object of the present application can be realized by the following technical solutions:

[0008] The present application provides a power transmission line hidden danger detection method, comprising:

[0009] Obtain hidden danger images of the power transmission line, the hidden danger images comprising broken strand pictures, conductor foreign object pictures, and loose pin pictures;

[0010] Obtain text data corresponding to the hidden danger images, the text data comprising text descriptions of the broken strand pictures, text descriptions of the conductor foreign object pictures, and text descriptions of the loose pin pictures from pin sensors;

[0011] Construct a power transmission line hidden danger detection model based on the hidden danger images and the text data, the power transmission line hidden danger detection model comprising a visual encoder, a text encoder, a classification head, and a language model;

[0012] Obtain image data of the power transmission line taken by a drone;

[0013] Input the obtained image data into the visual encoder of the power transmission line hidden danger detection model, input the visual features output by the visual encoder into the language model, and output text data corresponding to the obtained image data through the language model;

[0014] Determine whether the power transmission line has hidden dangers and the corresponding hidden danger types and descriptions according to the output text data.

[0015] Further, the construction of the power transmission line hidden danger detection model based on the hidden danger images and the text data specifically comprises:

[0016] Preprocess the hidden danger images, the preprocessing comprising image normalization, data enhancement, denoising, and deblurring operations;

[0017] Encode the preprocessed hidden danger images through the visual encoder to extract first visual features of each hidden danger image;

[0018] Encode the text data through the text encoder to extract first text features of each text description;

[0019] Train the image encoder and the text encoder according to the first visual features of each hidden danger image and the first text features of each text description through a contrastive learning method, and update the parameters of the image encoder and the text encoder through a first loss function;

[0020] The extracted first visual feature and the first text feature are input into a classification head, and the probability of matching each feature is output, and the image encoder, the text encoder and the classification head are updated in parameters through a second loss function according to the output probability;

[0021] After the image encoder and the text encoder are trained, the preprocessed hidden danger images are encoded by the trained visual encoder, and the second visual features of the hidden danger images are extracted;

[0022] The text data are encoded by the trained text encoder, and the second text features of the text descriptions are extracted;

[0023] The language model receives the second visual features extracted by the visual encoder, takes the second text features as a target, and updates the parameters of the language model based on a third loss function to obtain a trained language model.

[0024] Further, the encoding of the preprocessed hidden danger images by the visual encoder and the extraction of the first visual features of the hidden danger images specifically include:

[0025] The hidden danger images are divided into a plurality of image patches of a fixed size;

[0026] Each image patch is unfolded into a one-dimensional vector, and the position encoding is performed thereon;

[0027] The one-dimensional vectors of the image patches and the position encoding thereof are input into the visual encoder, and the visual encoder adopts a network structure based on a self-attention mechanism and uses a plurality of layers of self-attention mechanisms to capture long-range dependency relationships between different regions of the image;

[0028] The similarity between each patch and other patches is calculated by the self-attention mechanism of the visual encoder, and the representation of the one-dimensional vector of each image patch is adjusted according to the similarity;

[0029] The adjusted one-dimensional vector of the image patch is further processed by a feedforward neural network, and the feedforward neural network includes a nonlinear activation function and layer normalization;

[0030] The processed one-dimensional vector features of the image patch are fused to obtain the first visual features of the hidden danger images.

[0031] Further, the encoding of the text data by the text encoder and the extraction of the first text features of the text descriptions specifically include:

[0032] The input text data are subjected to word segmentation processing, and the text data are divided into a plurality of subword units;

[0033] Each subword unit is mapped to a corresponding word vector representation;

[0034] The word vector representation is input into a text encoder, which is a language model BERT based on a bidirectional attention mechanism;

[0035] The context dependency of each subword unit is calculated by the language model BERT, and the word vector representation of each subword is updated according to the context information;

[0036] The updated subword representation is combined into the overall text feature representation of the text through a pooling operation or an aggregation strategy, obtaining the first text feature of each text description.

[0037] Further, the image encoder and the text encoder are trained according to the first visual feature of each hidden danger image and the first text feature of each text description through contrastive learning, specifically including:

[0038] The first visual feature of the hidden danger image and the first text feature of the text data are paired to form positive samples and negative samples, the positive samples include image and its corresponding text description matching pair in semantics, and the negative samples include image and its corresponding text description without actual semantic matching pair;

[0039] The similarity between the image feature v z and the text feature v T is calculated, and the image encoder and the text encoder are optimized by the similarity and the first loss function.

[0040] Further, the first loss function is:

[0041]

[0042] Wherein, is the first loss function, N B is the data size of a training batch, τ is a learnable coefficient, <·,·> represents dot product, is the image feature of the i-th hidden danger image in the training data, is the text feature corresponding to the i-th hidden danger image in the training data.

[0043] Further, the first visual feature extracted and the first text feature are input into a classification head, and the probability of matching each feature is output, and the image encoder, the text encoder and the classification head are updated according to the output probability through a second loss function, specifically including:

[0044] The first visual feature of the hidden danger image extracted by the visual encoder and the text feature of the first text description extracted by the text encoder The input classification head is a binary classification network module used to output the probability p of whether a first visual feature matches a first text feature. v ;

[0045] The classification head processes the first visual feature v of the input. z and the first text feature v T Output a probability p v , which represents the degree of matching between the image and the text in the semantic space. If the image and the text match semantically, the output probability is close to 1; if they do not match, the output probability is close to 0.

[0046] During training, positive samples are pairs where the image matches its corresponding text description, with a label y = 1; negative samples are pairs where the image does not match the text description, with a label y = 0.

[0047] Based on the output matching probability p v The difference between the actual label y and the actual label y is used to update the parameters of the image encoder, text encoder, and classification head through a second loss function.

[0048] Furthermore, the second loss function is:

[0049]

[0050] in, For the second loss function, Let be the mathematical expectation.

[0051] Furthermore, the language model is an autoregressive language model, LLaMa.

[0052] The language model receives the second visual features extracted by the visual encoder, and uses the second text features as the target. Based on the third loss function, the language model's parameters are updated, specifically including:

[0053] Input the second visual feature into the language model, given the visual feature v z As input, the language model outputs a conditional probability distribution. The conditional probability distribution It describes the probability of a word at each position in a text sequence, representing the probability given previously generated text. and visual features v z In the case of generating the word at position m in the text, The conditional probability, based on the conditional probability distribution Calculate the third loss function and update the language model based on the calculated third loss function.

[0054] Furthermore, the third loss function is:

[0055]

[0056] wherein, is a third loss function, is a mathematical expectation, denotes all text features before the predicted position m, denotes the conditional probability of the word at position m in the generated text given the previously generated text z and the image feature v .

[0057] Compared with the prior art, the present application has the following advantages:

[0058] (1) The present application first uses the method of contrastive learning to ensure that the visual encoder and the text encoder can find the corresponding relationship in the semantic space by training the feature matching of the two. Specifically, each image feature and its corresponding text description are semantically matched one by one, and the goal is to train the visual features extracted by the visual encoder and the text features extracted by the text encoder to have higher similarity in high-dimensional space. Through the training of feature matching, it is ensured that there is a high degree of consistency between the visual features output from the visual encoder and the text features output from the text encoder. Specifically, the visual features and the text features will be aligned according to their respective characteristics. This alignment not only achieves a surface level (such as matching the labels of images and text), but also finds a contrast relationship in deep semantics, so that the model can understand the essential connection between images and text. The trained visual encoder and text encoder respectively output high-quality visual features and text features. The visual features are used as the input of the language model, while the text features are used as the target (i.e. the label) of the language model. In this way, the language model can learn to infer the relevant text description from the visual features, further optimizing its generation ability to enable it to understand and generate natural language descriptions highly related to visual information. This process enables the language model to recognize visual features in image data and generate hazard description text that conforms to the actual scene when it encounters image data.

[0059] (2) The present application adopts a language model based on an autoregressive mechanism (such as LLaMa), which combines images and text features and generates and reasons text through an autoregressive manner. This technical means enables the system not only to recognize hazards in images, but also to generate text descriptions related to the hazards, achieving anomaly prediction. For example, after receiving image data of a loose pin, the model not only recognizes the problem of loose pins, but also generates the corresponding prediction text (such as "loose pin") through reasoning. This anomaly prediction capability enhances the intelligence of the system, enabling hazard detection not only to recognize the current state, but also to provide early warning for potential problems.

[0060] (3)The present application realizes real-time hidden danger detection by designing a lightweight multi-modal perception model and deploying it on the side of the unmanned aerial vehicle. Compared with the traditional cloud computing method, the end-side reasoning scheme of the present application not only greatly reduces the consumption of computing resources, but also avoids data transmission delay, thereby improving the real-time performance and inspection efficiency of hidden danger detection. Especially during the inspection process, the model can identify and feedback hidden danger information in real time, providing more timely decision support for power operation personnel.

[0061] (4)The present application introduces multi-modal contrast learning technology, which aligns the features of images and texts in a shared space through contrast learning methods. Specifically, the image encoder and the text encoder are trained jointly to ensure that the image and the text are consistent in semantics. During training, the contrast between positive and negative samples optimizes the semantic matching degree of images and texts. Multi-modal contrast learning not only improves the representation ability of image and text features, but also enhances the robustness of the model, so that it can still maintain high recognition accuracy in complex environments such as wind disasters and heavy rain.

[0062] (5)The visual encoder of the present application adopts a network structure based on ViT, which captures the long-term dependence between different regions of the image through self-attention mechanism, and can effectively extract hidden danger features in the image. Compared with traditional convolutional neural networks, ViT divides the image into several fixed-size patches and enhances the global information processing ability of the image through self-attention mechanism, so that it can still accurately identify hidden dangers in complex situations such as conductor vibration and image blur. Through the introduction of this self-attention mechanism, the image recognition ability of the present application is significantly improved, especially suitable for complex scenes in power line inspection. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a method flowchart of the present application;

[0064] Figure 2 is a power line hidden danger detection model diagram of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0066] Example 1:

[0067] The application provides a power transmission line hidden danger detection method, as shown in the accompanying drawings, comprising: Figure 1

[0068] Obtaining hidden danger images of the power transmission line, the hidden danger images comprising broken strand pictures, wire foreign matter pictures, and loose pin pictures;

[0069] Obtaining text data corresponding to the hidden danger images, the text data comprising text descriptions of the broken strand pictures, text descriptions of the wire foreign matter pictures, and text descriptions of the loose pin pictures from pin sensors;

[0070] Constructing a power transmission line hidden danger detection model based on the hidden danger images and the text data, as shown in the accompanying drawings, the power transmission line hidden danger detection model comprising a visual encoder, a text encoder, a classification head, and a language model; Figure 2

[0071] Obtaining image data of the power transmission line taken by a drone;

[0072] Inputting the obtained image data into the visual encoder of the power transmission line hidden danger detection model, and outputting text data corresponding to the obtained image data through the language model;

[0073] Determining whether the power transmission line has hidden dangers and corresponding hidden danger types and hidden danger descriptions according to the output text data.

[0074] Constructing the power transmission line hidden danger detection model based on the hidden danger images and the text data, specifically comprising:

[0075] Preprocessing the hidden danger images, the preprocessing comprising image normalization, data enhancement, denoising, and deblurring operations;

[0076] Encoding the preprocessed hidden danger images through the visual encoder to extract visual features of each hidden danger image;

[0077] Encoding the text data through the text encoder to extract text features of each text description;

[0078] Training the image encoder and the text encoder according to the visual features of each hidden danger image and the text features of each text description through a contrastive learning method, and updating parameters of the image encoder and the text encoder through a first loss function;

[0079] Inputting the extracted visual features and text features into the classification head to output probabilities of matching each feature, and updating parameters of the image encoder, the text encoder, and the classification head through a second loss function according to the output probabilities;

[0080] Inputting the extracted visual features and text features into the language model, and updating parameters of the language model through a third loss function.

[0081] ​​The preprocessed hidden danger image is encoded by a visual encoder to extract visual features of each hidden danger image, specifically including:

[0082] The hidden danger image is divided into a plurality of image patches of fixed size;

[0083] Each image patch is unfolded into a one-dimensional vector, and the position encoding is performed thereon;

[0084] The one-dimensional vector of each image patch and the position encoding thereof are input into the visual encoder, the visual encoder adopts a network structure based on a self-attention mechanism, and uses a plurality of layers of self-attention mechanisms to capture long-range dependency relationships between different regions of the image;

[0085] The similarity between each patch and other patches is calculated by the self-attention mechanism of the visual encoder, and the representation of the one-dimensional vector of each image patch is adjusted according to the similarity;

[0086] The adjusted one-dimensional vector of the image patch is further processed by a feedforward neural network, the feedforward neural network including a nonlinear activation function and layer normalization;

[0087] The processed one-dimensional vector features of the image patch are fused to obtain visual features of each hidden danger image.

[0088] The text data is encoded by a text encoder to extract text features of each text description, specifically including:

[0089] The input text data is processed by word segmentation to divide the text data into a plurality of subword units;

[0090] Each subword unit is mapped to a corresponding word vector representation;

[0091] The word vector representation is input into the text encoder, and the text encoder is a language model BERT based on a bidirectional attention mechanism;

[0092] The context dependency relationship of each subword unit is calculated by the language model BERT, and the word vector representation of each subword is updated according to the context information;

[0093] The updated subword representation is combined into an overall text feature representation of the text by a pooling operation or an aggregation strategy to obtain text features of each text description.

[0094] The image encoder and the text encoder are trained according to the visual features of each hidden danger image and the text features of each text description by a contrastive learning manner, specifically including:

[0095] The visual features of the hidden danger images are paired with the text features of the text data to form positive samples and negative samples, the positive samples include pairs of images and corresponding text descriptions that are semantically matched, and the negative samples include pairs of images and corresponding text descriptions that are not semantically matched;

[0096] The image features v of the positive samples and the negative samples are calculated z and the text features v T The similarity between the image features v and the text features v is calculated, and the image encoder and the text encoder are optimized by the similarity and a first loss function.

[0097] The first loss function is:

[0098]

[0099] wherein, is the first loss function, N B is the size of a training batch of data, τ is a learnable coefficient, and <·,·> represents dot product, is the image feature of the i-th hidden danger image in the training data, is the text feature corresponding to the i-th hidden danger image in the training data.

[0100] The extracted visual features and text features are input into a classification head to output probabilities of matching of the respective features, and the image encoder, the text encoder, and the classification head are updated in parameters by a second loss function according to the output probabilities, specifically including:

[0101] The hidden danger image visual features v extracted by the visual encoder and the text features v of the text descriptions extracted by the text encoder are input into the classification head, the classification head is a binary classification network module, and is used to output a probability p v whether the visual features and the text features match or not.

[0102] The classification head processes the input visual features v z and the text features v T to output a probability p v , which represents the matching degree of the image and the text in the semantic space, if the image and the text are semantically matched, the output probability is close to 1, and if they are not matched, the output probability is close to 0.

[0103] In the training process, the positive samples are pairs of images and corresponding text descriptions that are matched, and the labels y thereof are 1, and the negative samples are pairs of images and text descriptions that are not matched, and the labels y thereof are 0.

[0104] According to the output matching probability p vThe difference between the actual label y and the predicted label y is encoded by a second loss function to update the parameters of the image encoder, the text encoder, and the classification head.

[0105] The second loss function is:

[0106]

[0107] wherein, is the second loss function, is the mathematical expectation.

[0108] The language model is a language model LLaMa based on an autoregressive mechanism.

[0109] The third loss function is:

[0110]

[0111] wherein, is the third loss function, is the mathematical expectation, represents all text features before the prediction position m, represents the conditional probability of the word at position m in the generated text given the previously generated text and the image feature v z .

[0112] Embodiment 2:

[0113] The parts not mentioned in this embodiment are the same as in Embodiment 1.

[0114] This embodiment demonstrates a power line hidden danger detection method based on multi-modal data fusion and deep learning technology through specific implementation. This method combines image enhancement technology in wind disaster environment, multi-modal fusion of image and text data, and language model reasoning based on autoregressive mechanism, and finally realizes efficient and accurate detection of power line hidden dangers.

[0115] In this embodiment, first, the power line images taken by the unmanned aerial vehicle and the text data obtained from the line fitting sensor are used as input data. The input images include broken strand pictures, wire foreign object pictures, and pin loose pictures, while the text data includes corresponding text descriptions of these images and displacement information from the pin sensor. After preprocessing, these input data are input into different modules of the model for processing.

[0116] ​The image data is first preprocessed, including image normalization, data enhancement, denoising, and deblurring. The preprocessed image data is sent to the image encoder, which is based on the ViT (Vision Transformer) structure and processes the image data through a self-attention mechanism. ViT divides the image into several fixed-size patches, each of which is expanded into a one-dimensional vector and input into the encoder with position encoding. Through the self-attention mechanism, each patch in the image can exchange information with other patches, capturing long-range dependencies and enhancing the global understanding of the image. This structure is particularly suitable for handling complex vibrations, blurring, or occlusion of the lead in the image, effectively improving the extraction of image features and providing more accurate feature representation for subsequent hazard identification.

[0117] The text data input together with the image data is processed by the text encoder. The text data is first segmented into tokens, and each token is mapped to a corresponding vector space and then extracted by a BERT-based language model based on a bidirectional attention mechanism. The BERT model learns the context relationship between words in the text to extract semantic information and generate feature representation of the text. In this embodiment, the text data mainly includes text descriptions corresponding to the image and displacement descriptions of the sensor data, which provides additional information that cannot be directly obtained from the image data. By combining the text information of the image and sensor data, the model can analyze the hazards from multiple dimensions, improving the comprehensiveness and accuracy of hazard detection.

[0118] The output features of the image encoder and the text encoder are sent to the classification head for feature matching. The classification head uses a binary classification network module to output the probability of matching the image and text features. The parameters of the classification head, image encoder, and text encoder are updated based on the output probability and a second loss function, where the second loss function is:

[0119]

[0120] where, is the second loss function, is the mathematical expectation.

[0121] The main purpose of setting the classification head and the second loss function in the present application is to optimize the output features of the image encoder, so that it can be more accurately aligned with the text description, thereby improving the expression ability of the image features. The role of the classification head is to match the image features extracted by the image encoder with the text description to determine whether they are semantically consistent. Through this matching process, the model can learn how to correspond the hidden danger features in the image to the corresponding text description during the training process, and the matching degree between the image features and the text features is calculated through the second loss function to optimize the parameters of the image encoder, so that it can better extract the key features related to the hidden danger from the image. This process effectively improves the recognition accuracy of the image encoder in practical applications, especially in harsh environments such as wind disasters, it can better handle the problems of image blur, noise or occlusion, and improve the accuracy and robustness of hidden danger identification.

[0122] At the same time, the system further optimizes the semantic alignment between image and text features through multi-modal contrastive learning method. In the training process, the positive sample is the matching pair of image and its corresponding text description, and the negative sample is the pair of image and irrelevant text. By calculating the contrast loss between positive and negative samples, the image encoder and the text encoder can jointly learn more accurate feature representation, ensuring that the semantics between image and text remain consistent, thereby improving the accuracy of hidden danger identification.

[0123] Multi-modal contrastive learning aims to jointly learn shared feature representations of different modalities (such as images, text, etc.) through contrastive learning methods. This method can learn effective semantic alignment between multiple modalities, so that information from different modalities can be compared and matched in the same representation space.

[0124] Positive and negative samples: In multi-modal contrastive learning, positive samples refer to corresponding paired data between different modalities (such as an image and its correct text description), and negative samples refer to those that do not match (such as an image and an unrelated text description).

[0125] Contrastive loss (Contrastive Loss): The goal of contrastive learning is to optimize a loss function so that the representation of positive samples is as close as possible, while the representation of negative samples is as far apart as possible. The contrastive loss function is commonly InfoNCE Loss, which maximizes the discrimination between positive and negative samples to ensure that relevant information from different modalities is more closely connected in the shared embedding space.

[0126] The corresponding features extracted by the image encoder and the text encoder are represented as v z and v T , then the contrastive loss includes image-text contrastive loss and text-image contrastive loss, and the first loss function is as follows:

[0127]

[0128] wherein, is the first loss function, N B is the size of a training batch, τ is a learnable coefficient, <·,·> denotes the dot product, is the image feature of the i-th hidden danger image in the training data, is the text feature corresponding to the i-th hidden danger image in the training data.

[0129] When the image and text features are extracted, they are input into a language model based on the autoregressive mechanism (such as LLaMa) for inference. The training of text generation is carried out through the autoregressive text prediction loss function, and the third loss function is as follows:

[0130]

[0131] wherein, is the third loss function, is the mathematical expectation, denotes all text features before the prediction position m, denotes the conditional probability of the word at position m in the generated text given the previously generated text z and the image feature v .

[0132] The autoregressive language model generates inference results based on visual features and text features. For example, by inputting a pin image, the model can predict the result ("detection result: loose pin") through causal inference. This technical means can provide more intelligent decision support for maintenance personnel, not only identifying the current hidden danger, but also inferring and predicting potential abnormal situations.

[0133] In addition, the transmission line hidden danger detection model can also combine the images taken by the unmanned aerial vehicle and the data of the pin sensor (such as the displacement, loosening, etc. of the pin sensor data), to further enrich the description information of the hidden danger. Through the fusion of multi-modal data, the model can utilize both the visual features in the image and the text data provided by the pin sensor, so as to obtain more accurate and comprehensive hidden danger description. This not only improves the accuracy of hidden danger identification, but also enhances the adaptability and robustness of the model, especially in the face of complex environments or variable power lines, it can provide more comprehensive hidden danger detection and prediction support for maintenance personnel.

[0134] Finally, after the multi-modal learning and inference process, the system can accurately identify and classify the hidden danger types in the image, such as wire breakage, hanging objects, loose pins, etc., and output the text description and positioning information of the hidden danger in real time. All processing is completed on the UAV side, reducing data transmission and computing delay, improving the real-time performance and efficiency of the system.

[0135] Through the implementation of the above technical features, the application can significantly improve the detection accuracy of hidden dangers in adverse environments such as wind disasters, reduce the blind area of traditional manual inspection, enhance the intelligent level of the system, and reduce the computing cost and delay through end-side inference technology.

[0136] The above functions, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application or the part of the application that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various program code storage media.

[0137] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto. Any skilled person in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the application, and these modifications or replacements should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A method for detecting hidden dangers in power transmission lines, characterized in that, include: Acquire images of potential hazards in power transmission lines, including images of broken strands, foreign objects in conductors, and loose pins; Obtain the text data corresponding to the hazard images. The text data includes text descriptions of broken strand images, foreign object images of wires, and text descriptions of loose pin images from pin sensors. A power transmission line hazard detection model is constructed based on hazard images and text data. The power transmission line hazard detection model includes a visual encoder, a text encoder, a classification head, and a language model. Acquire image data of power transmission lines taken by drones; The acquired image data is input into the visual encoder of the power transmission line hidden danger detection model, the visual features output by the visual encoder are input into the language model, and the text data corresponding to the acquired image data is output through the language model. Determine whether there are hidden dangers in the transmission line and the corresponding types and descriptions of the hidden dangers based on the output text data; The transmission line hazard detection model constructed based on hazard images and text data specifically includes: The image of the potential hazard is preprocessed, including image normalization, data augmentation, denoising and deblurring operations. The preprocessed hazard images are encoded using a visual encoder, and the first visual features of each hazard image are extracted. The text data is encoded using a text encoder to extract the first text features of each text description; The image encoder and text encoder are trained by contrastive learning based on the first visual features of each potential hazard image and the first text features of each text description. The parameters of the image encoder and text encoder are updated by the first loss function. The extracted first visual features and first text features are input into the classification head, and the probability of matching each feature is output. Based on the output probability, the parameters of the image encoder, text encoder and classification head are updated using the second loss function. After the image encoder and text encoder are trained, the preprocessed hazard images are encoded by the trained visual encoder to extract the second visual features of each hazard image. The text data is encoded by the trained text encoder, and the second text features of each text description are extracted. The language model receives the second visual features extracted by the visual encoder, uses the second text features as the target, and updates the parameters of the language model based on the third loss function to obtain the trained language model.

2. The method for detecting hidden dangers in transmission lines according to claim 1, characterized in that, The process of encoding the preprocessed hazard images using a visual encoder and extracting the first visual features of each hazard image specifically includes: The potential hazard image is divided into several image patches of fixed size; Each image patch is unfolded into a one-dimensional vector and its position is encoded; The one-dimensional vector and its position encoding of each image patch are input into the visual encoder, which adopts a network structure based on a self-attention mechanism and uses a multi-layer self-attention mechanism to capture the long-range dependencies between different regions of the image. The similarity between each patch and other patches is calculated using the self-attention mechanism of the visual encoder, and the one-dimensional vector representation of each image patch is adjusted according to the similarity. The adjusted image patch one-dimensional vector is further processed by a feedforward neural network, which includes a nonlinear activation function and layer normalization. The first visual features of each potential hazard image are obtained by fusing the one-dimensional vector features of the processed image patches.

3. The method for detecting hidden dangers in transmission lines according to claim 1, characterized in that, The process of encoding text data using a text encoder and extracting the first text features of each text description specifically includes: The input text data is segmented into several sub-word units. Each sub-word unit is mapped to its corresponding word vector representation; Word vectors are input into a text encoder, which is a BERT language model based on a bidirectional attention mechanism. The contextual dependencies of each word unit are calculated using the BERT language model, and the word vector representation of each word is updated based on the contextual information. The updated sub-word representations are merged into the overall text feature representation of the text through pooling operations or aggregation strategies, thus obtaining the first text feature of each text description.

4. The method for detecting hidden dangers in transmission lines according to claim 1, characterized in that, The method of training the image encoder and text encoder by contrastive learning based on the first visual features of each potential hazard image and the first text features of each text description specifically includes: The first visual features of the potential hazard image are paired with the first text features of the text data to form positive samples and negative samples. The positive samples include pairings where the image and its corresponding text description are semantically matched, and the negative samples include pairings where there is no actual semantic match between the image and its corresponding text description. Calculate the image features of positive and negative samples Text features The similarity between the two is used to optimize the image encoder and text encoder using the similarity and a first loss function.

5. The method for detecting hidden dangers in transmission lines according to claim 4, characterized in that, The first loss function is: in, For the first loss function, The size of a training batch of data. For learnable coefficients, Dot product, The training data i Image features of a potential hazard image, For the training data, the first i Text features corresponding to each potential hazard image.

6. The method for detecting hidden dangers in transmission lines according to claim 1, characterized in that, The process involves inputting the extracted first visual features and first text features into the classification head, outputting the probability of matching each feature, and updating the parameters of the image encoder, text encoder, and classification head using a second loss function based on the output probabilities. Specifically, this includes: The first visual features of the hazard image extracted by the visual encoder and the first text description text features extracted by the text encoder The input classification head is a binary classification network module used to output the probability of whether a first visual feature matches a first text feature. ; The classification head processes the first visual features of the input. and first text features Output a probability , which represents the degree of matching between the image and the text in the semantic space. If the image and the text match semantically, the output probability is close to 1; if they do not match, the output probability is close to 0. During training, positive samples are pairs of images that match their corresponding text descriptions, and their labels are... Negative samples are pairs where the image and text description do not match, and their labels are... ; Based on the output matching probability With actual label The differences between them are used to update the parameters of the image encoder, text encoder, and classification head through a second loss function.

7. The method for detecting hidden dangers in transmission lines according to claim 6, characterized in that, The second loss function is: ， in, For the second loss function, Let be the mathematical expectation.

8. The method for detecting hidden dangers in transmission lines according to claim 1, characterized in that, The language model is an autoregressive language model, LLaMa. The language model receives the second visual features extracted by the visual encoder, and uses the second text features as the target. Based on the third loss function, the language model's parameters are updated, specifically including: Input the second visual features into the language model, given the visual features As input, the language model outputs a conditional probability distribution. The conditional probability distribution It describes the probability of a word at each position in a text sequence, representing the probability given previously generated text. and visual features In the case of generating the word at position m in the text, The conditional probability, based on the conditional probability distribution Calculate the third loss function and update the language model based on the calculated third loss function.

9. A method for detecting hidden dangers in transmission lines according to claim 8, characterized in that, The third loss function is: ， in, For the third loss function, For mathematical expectation, Indicates the predicted location All previous text features, Indicates that given previously generated text and image features In the case of generating the word at position m in the text, The conditional probability.

Images