Distribution line equipment defect grading identification method based on open set target detection

Abstract

The application discloses a distribution network line equipment defect grading identification method based on open set target detection, and relates to the field of image processing.The method comprises the following steps: acquiring image coding features and label language coding features of distribution network line equipment, combining a bidirectional two-domain self-attention mechanism and decoding technology, and obtaining fused image decoding features and fused language coding features; extracting and generating secondary language coding features based on the label language coding features; performing two-stage similarity calculation and matching on the fused image decoding features, the fused language coding features and the secondary language coding features respectively, and obtaining a defect grading determination result of the distribution network line equipment; and performing credibility evaluation and decision explanation on the defect grading determination result based on a causal diagram and a feature attribution game technology.The application effectively enhances the representation capability of a model for complex backgrounds and various defects, improves the recognition accuracy and robustness, and effectively solves the black box problem of a deep learning model.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a method for graded identification of defects in power distribution network line equipment based on open set target detection. Background Technology

[0002] With the continuous expansion of power system scale and the improvement of intelligence level, the operational safety and reliability of distribution network line equipment have received increasing attention. Power grid inspection, as a key link in ensuring stable power grid operation, has traditionally relied mainly on manual inspection and visible light image capture, which suffers from low efficiency, high risk, strong subjectivity, and difficulty in achieving full coverage. Especially in complex environments, high-altitude areas, and remote regions, manual inspection is not only costly but also poses safety hazards. In recent years, intelligent inspection equipment such as drones and robots have been gradually promoted and applied, accumulating a large amount of image and video data. How to efficiently and accurately automatically identify equipment defects and determine their severity from massive amounts of visual data has become a core issue that urgently needs to be addressed in the intelligent operation and maintenance of power systems.

[0003] Over the years, computer vision-based defect detection methods have been increasingly applied in the field of power line inspection, primarily including traditional image processing algorithms and deep learning-based target detection methods. Traditional methods, such as edge detection and template matching, are sensitive to changes in ambient lighting and angle, have poor generalization ability, and struggle to handle complex backgrounds and multi-scale targets in real-world scenarios. While deep learning models based on the closed-set assumption, such as YOLO and Faster R-CNN, perform well in identifying specific defects, they cannot effectively identify defect categories not present in the training set, lack the ability to perceive unknown defects, and are even less capable of achieving fine-grained classification of defect severity. Furthermore, existing methods often rely on a single visual modality and lack multimodal information fusion mechanisms, resulting in insufficient semantic understanding of defects, and significant room for improvement in recognition accuracy and robustness.

[0004] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0005] To address the problems in related technologies, this invention proposes a method for graded identification of defects in distribution network line equipment based on open set target detection, in order to overcome the aforementioned technical problems existing in the existing related technologies.

[0006] Therefore, the specific technical solution adopted by the present invention is as follows:

[0007] A method for graded identification of defects in distribution network line equipment based on open set target detection includes:

[0008] Image encoding features and tag language encoding features of distribution network line equipment are obtained, and combined with bidirectional two-domain self-attention mechanism and decoding technology, fused image decoding features and fused language encoding features are obtained; based on the tag language encoding features, secondary language encoding features are extracted and generated.

[0009] By performing two-level similarity calculation and matching between the fused image decoding features and the fused language coding features and the secondary language coding features, the defect classification judgment result of the distribution network line equipment is obtained.

[0010] Based on causal graph and feature attribution game theory, the credibility assessment and decision interpretation of defect classification results are performed.

[0011] Furthermore, prior to obtaining the image encoding features and tag language encoding features of the distribution network line equipment, the following steps are included:

[0012] The original images and defect label texts of the distribution network line equipment are preprocessed and standardized to obtain standardized images, negative sample labels and positive sample hierarchical labels.

[0013] Based on the actual situation of the standardized images, labels are applied to the standardized images using the cascading labeling method.

[0014] Furthermore, the random sampling of negative sample labels includes:

[0015] Configure a neural policy network and output a probability distribution based on the encoding of the current positive sample label to represent the value of the negative sample;

[0016] Based on the probability distribution, negative samples are dynamically extracted and combined with the current positive samples to train the deep learning model.

[0017] After the training cycle, the performance change of the deep learning model is used as a reward signal to feed back to the reinforcement learning agent; the neural policy network parameters are updated using the policy gradient algorithm to maximize the expected reward.

[0018] Repeated iterations enable the neural policy network to learn intelligent sampling strategies, adaptively selecting negative samples for different positive samples and assigning corresponding labels.

[0019] Furthermore, obtaining the image encoding features and tag language encoding features of the distribution network line equipment includes:

[0020] A deep learning model was selected as the training model for object detection, and multi-level visual features of the standardized images were extracted.

[0021] By introducing a self-attention mechanism into a deep learning model, multi-level visual features are fused to obtain image coding features;

[0022] Using a language inference model, the tokenized defect tag sequence is converted into tag language encoded features.

[0023] Furthermore, image encoding features and tag language encoding features of distribution network line equipment are obtained, and combined with a bidirectional two-domain self-attention mechanism and decoding technology, fused image decoding features and fused language encoding features are obtained, including:

[0024] A bidirectional two-domain self-attention module is constructed; cross-modal information fusion is performed by inputting image coding features and label language coding features into the bidirectional two-domain self-attention module, and the fused image coding features and fused language coding features are output.

[0025] The query vector, which combines the fused image coding features with the projection of the image coding features, is input into the decoder, and the fused image decoding features are output.

[0026] Furthermore, by inputting image coding features and labeled language coding features into the bidirectional two-domain self-attention module, cross-modal information fusion is performed, and the fused image coding features and fused language coding features are output, including:

[0027] Image encoding features and label language encoding features are mapped to a multi-head attention space through a fully connected layer;

[0028] Attention weights for visual enhancement and language enhancement are calculated separately using matrix multiplication, and the attention weights are then normalized to a probability distribution.

[0029] Based on attention weights, visual and linguistic features are updated bidirectionally, and the fused image coding features and fused linguistic coding features are output through mapping of fully connected layers.

[0030] Furthermore, based on the tag language encoding features, the extracted and generated second-level language encoding features include:

[0031] Based on the number of tag tokens, secondary language encoding features are extracted from the tag language encoding features, and the size of the secondary language encoding features is adjusted; all extracted secondary language encoding features are then concatenated.

[0032] Furthermore, by performing two-level similarity calculations and matching between the fused image decoding features and the fused language coding features and the secondary language coding features, the defect classification judgment results of the distribution network line equipment are obtained, including:

[0033] The similarity between the fused image decoding features and the fused language coding features is calculated to generate a multimodal feature similarity matrix;

[0034] The similarity between the fused image decoding features and the secondary language coding features is calculated to generate a secondary multimodal feature similarity matrix;

[0035] Based on the multimodal feature similarity matrix and the second-level multimodal feature similarity matrix, and using the combinatorial optimization algorithm to match positive and negative samples, a cross-entropy loss function is constructed according to the positive and negative samples to train a defect classification and identification model, so as to identify the subject of the defect and the defect level, and obtain the defect classification and judgment result of the distribution network line equipment.

[0036] Furthermore, based on causal graphs and feature attribution game theory, the credibility assessment and decision interpretation of the defect classification results include:

[0037] A causal graph is constructed with visual features as the cause and defect type and level as the effect. Based on knowledge of the power defect domain, the variable nodes of the causal graph are configured. Causal edges are constructed according to the causal relationships between variables to form a directed acyclic graph.

[0038] A multi-agent attribution game framework is constructed, and the fused image decoding features are divided into several feature segments according to the spatial dimension to generate corresponding visual agents.

[0039] The embedding vectors of each token in the tag sequence participating in similarity calculation are used as language agents;

[0040] An approximation algorithm based on Monte Carlo sampling is used to distribute the calculation of Shapley attribution values ​​for the final prediction results for each visual agent and language agent.

[0041] Based on the Shapley attribution value, select image regions corresponding to several visual agents and text tokens corresponding to language agents as attribution consensus evidence.

[0042] The attribution consensus evidence is matched with the rules of the directed acyclic graph, and the credibility of the defect classification results is evaluated and the decision is interpreted by verifying the logical consistency between the attribution consensus evidence and the rules in the causal graph.

[0043] Furthermore, the formula for calculating the Shapley attribution value is as follows:

[0044] ;

[0045] In the formula, This is the Shapley attribution value;

[0046] For the set of all intelligent agents, for Without intelligent agents a subset of The total number of agents;

[0047] For subsets The model's output value is obtained when the agent in the model is used as input. For marginal contribution.

[0048] The beneficial effects of this invention are as follows:

[0049] 1. A multimodal fusion method based on vision-language bidirectional attention mechanism is adopted to fully explore the deep correlation between image features and text semantics, effectively enhance the model's ability to represent complex backgrounds and diverse defects, and significantly improve recognition accuracy and robustness.

[0050] 2. A cascaded label design and a two-level language feature pool construction mechanism are proposed to simultaneously realize defect subject identification and severity classification in a unified model, overcoming the limitation of traditional methods that cannot achieve fine classification, and improving the efficiency of defect management and decision support level.

[0051] 3. By introducing an open set recognition framework and a negative sample strategy, the model is able to identify defects that have not been seen during the training phase, which significantly improves the generalization and practical value of the system in real-world scenarios and reduces the risk of missed detections and false detections.

[0052] 4. This invention overcomes the uncertainty of single attribution methods and improves the credibility and interpretability of defect classification identification results. It treats the features of different modalities as independent intelligent agents, calculates their contribution through Shapley value in game theory, and finally performs collaborative verification with the causal graph model, effectively solving the black box problem of deep learning models, enabling maintenance personnel to understand and trust the automated inspection results, and improving the reliability of system output. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a method for graded identification of defects in distribution network line equipment based on open set target detection according to an embodiment of the present invention;

[0055] Figure 2 This is a flowchart illustrating the random sampling of negative sample labels in the distribution network line equipment defect classification and identification method based on open set target detection according to an embodiment of the present invention.

[0056] Figure 3 This is a flowchart illustrating the process of obtaining fused image decoding features and fused language coding features in the distribution network line equipment defect classification identification method based on open set target detection according to an embodiment of the present invention.

[0057] Figure 4This is a flowchart of the process of evaluating the credibility of the defect classification judgment result and interpreting the decision in the defect classification identification method of distribution network line equipment based on open set target detection according to an embodiment of the present invention.

[0058] Figure 5 This is a flowchart according to an embodiment of the present invention. Detailed Implementation

[0059] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0060] According to an embodiment of the present invention, a method for graded identification of defects in distribution network line equipment based on open set target detection is provided, which uses a multimodal cascaded detection method to complete the graded identification of defects in distribution network line equipment.

[0061] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figures 1-5 As shown, the method for graded identification of defects in distribution network line equipment based on open set target detection according to an embodiment of the present invention includes:

[0062] S1. Obtain the image encoding features and tag language encoding features of the distribution network line equipment, and combine the bidirectional two-domain self-attention mechanism and decoding technology to obtain the fused image decoding features and fused language encoding features; based on the tag language encoding features, extract and generate secondary language encoding features.

[0063] S2. By performing two-level similarity calculation and matching between the fused image decoding features and the fused language coding features and the secondary language coding features, the defect classification judgment result of the distribution network line equipment is obtained.

[0064] S3. Based on causal graph and feature attribution game technology, the credibility assessment and decision interpretation of defect classification results are carried out.

[0065] In one embodiment, the process before obtaining the image encoding features and tag language encoding features of the distribution network line equipment includes:

[0066] The original images and defect label texts of the distribution network line equipment are preprocessed and standardized to obtain standardized images, negative sample labels, and positive sample hierarchical labels. Based on the actual situation of the standardized images, the standardized images are labeled using the cascading label annotation method.

[0067] In one embodiment, the random sampling of negative sample labels includes:

[0068] Configure a neural policy network and output a probability distribution based on the encoding of the current positive sample label to represent the value of negative samples; dynamically extract negative samples according to the probability distribution and combine them with the current positive samples to train the deep learning model; after each training cycle, use the performance change of the deep learning model as a reward signal to feed back to the reinforcement learning agent; update the parameters of the neural policy network using a policy gradient algorithm to maximize the expected reward; repeat the iteration to enable the neural policy network to learn an intelligent sampling strategy, adaptively select negative samples for different positive samples, and assign corresponding labels.

[0069] In one embodiment, obtaining the image encoding features and tag language encoding features of the distribution network line equipment includes:

[0070] A deep learning model was selected as the training model for object detection, and multi-level visual features of standardized images were extracted. By introducing a self-attention mechanism into the deep learning model, the multi-level visual features were fused to obtain image encoding features. A language inference model was used to convert the tokenized defect label sequence into label language encoding features.

[0071] In one embodiment, the image encoding features and tag language encoding features of the distribution network line equipment are obtained, and combined with a bidirectional two-domain self-attention mechanism and decoding technology, the fused image decoding features and fused language encoding features are obtained, including:

[0072] A bidirectional two-domain self-attention module is constructed. By inputting image coding features and label language coding features into the bidirectional two-domain self-attention module, cross-modal information fusion is performed, and fused image coding features and fused language coding features are output. The query vector projected from the fused image coding features and image coding features is input into the decoder, and fused image decoding features are output.

[0073] In one embodiment, cross-modal information fusion is performed by inputting image coding features and labeled language coding features into a bidirectional two-domain self-attention module, and the fused image coding features and fused language coding features are output, including:

[0074] Image coding features and labeled language coding features are mapped to a multi-head attention space through a fully connected layer; attention weights for visual enhancement and language enhancement are calculated separately through matrix multiplication, and the attention weights are normalized to a probability distribution; based on the attention weights, visual and language features are updated bidirectionally, and the fused image coding features and fused language coding features are output through the mapping of the fully connected layer.

[0075] In one embodiment, extracting and generating second-level language encoding features based on tag language encoding features includes:

[0076] Based on the number of tag tokens, secondary language encoding features are extracted from the tag language encoding features, and the size of the secondary language encoding features is adjusted; all extracted secondary language encoding features are then concatenated.

[0077] In one embodiment, by performing two-level similarity calculation and matching between fused image decoding features and fused language coding features and secondary language coding features respectively, the defect classification judgment result of the distribution network line equipment is obtained, including:

[0078] The similarity between the fused image decoding features and the fused language coding features is calculated to generate a multimodal feature similarity matrix. The similarity between the fused image decoding features and the secondary language coding features is calculated to generate a secondary multimodal feature similarity matrix. Based on the multimodal feature similarity matrix and the secondary multimodal feature similarity matrix, a combinatorial optimization algorithm is used to match positive and negative samples. A cross-entropy loss function is constructed based on the positive and negative samples to train a defect classification and identification model, so as to identify the subject and level of the defect and obtain the defect classification judgment result of the distribution network line equipment.

[0079] In one embodiment, the credibility assessment and decision interpretation of the defect classification results based on causal graph and feature attribution game techniques include:

[0080] A causal graph is constructed with visual features as the cause and defect type and level as the effect. Based on knowledge of the power defect domain, variable nodes are configured in the causal graph. Causal edges are constructed according to the causal relationships between variables, forming a directed acyclic graph (DAG). A multi-agent attribution game framework is constructed, and the fused image decoding features are divided into several feature segments according to spatial dimensions, generating corresponding visual agents. The embedding vectors of each token in the label sequence participating in similarity calculation are used as language agents. Based on an approximation algorithm using Monte Carlo sampling, the Shapley attribution values ​​for the final prediction result are calculated in a distributed manner for each visual agent and language agent. Based on the Shapley attribution values, image regions corresponding to several visual agents and text tokens corresponding to several language agents are selected as attribution consensus evidence. The attribution consensus evidence is matched with the rules of the DAG, and the credibility of the defect classification judgment result is evaluated and the decision is interpreted by verifying the logical consistency between the attribution consensus evidence and the rules in the causal graph.

[0081] In one embodiment, the Shapley attribution value is calculated as follows:

[0082] ;

[0083] In the formula, This is the Shapley attribution value; For the set of all intelligent agents, for Without intelligent agents a subset of The total number of agents; For subsets The model's output value is obtained when the agent in the model is used as input. The change in model output after adding a subset S to agent i is the marginal contribution.

[0084] To facilitate understanding of the above technical solutions of the present invention, the working principle of the present invention in actual process will be described in detail below.

[0085] A method for graded identification of defects in distribution network line equipment based on open set target detection includes: S1: preprocessing and standardization of distribution network line equipment image and tag data; S2: parallel extraction of image coding features and tag language coding features, and further generation of secondary language coding features based on language coding features; S3: inputting image coding features and language coding features into a bidirectional two-domain self-attention module for multimodal deep fusion, outputting fused image coding features and query, which are then decoded by a decoder to output fused image decoding features; S4: determining the type and severity of defects by calculating and matching the two-level similarity between fused image decoding features and fused language coding features / secondary language coding features; and introducing an interpretability decision module based on causal graph model and feature attribution game technology to further improve the interpretability of the model.

[0086] like Figure 2 As shown, it specifically includes the following:

[0087] Step 1: Preprocess the raw images of power distribution line equipment collected on-site, such as insulators and clamps. First, uniformly crop or scale the images to a fixed size of 1024×1024 pixels to eliminate scale differences caused by factors such as equipment distance and shooting angle, providing standardized input for subsequent deep network processing. Standardize the defect label text, construct a label pool, store all defect-related labels in the pool, and randomly select 15 labels as negative sample labels to enhance the model's ability to distinguish between defects / non-defects and different defect levels. Simultaneously, clean and segment the labels containing positive sample hierarchical labels and the 15 negative sample labels randomly selected from the label pool, and use the BERT tokenizer to convert them into an encoding format that the model can recognize, generating text input for feature extraction. The BERT tokenizer is a core component used to convert input text into a format that the model can process. A token refers to the smallest basic unit that the input data is divided into and that the model can process.

[0088] Taking insulator damage as an example, a cascaded labeling method is introduced in the label design. Based on the extent of insulator damage, the damage is categorized as: general, severe, and critical. Therefore, the label design for this example would be: Insulator damaged, Insulator damaged critically. Simultaneously, negative sample labels are introduced. This invention designs a label pool, storing all labels in the pool and randomly selecting 15 labels as negative sample labels.

[0089] A hierarchical structure is defined for each type of defect requiring graded identification, comprising two layers. The first-level label represents the defect itself, i.e., what the defect is, such as insulator damage. The second-level label represents the specific severity of the defect, i.e., the degree of severity, such as moderate insulator damage, severe insulator damage, or critical insulator damage. When labeling images, the labeler needs to label both types of labels simultaneously based on the actual condition of the defect in the image. For example, if image A shows a slightly damaged insulator, the labeling result would be: [Insulator damaged, moderate insulator damage]. When preparing training data, a training sample will consist of an image and its corresponding cascaded label pair.

[0090] This invention proposes that randomly selected negative samples, if significantly different from current positive samples, can be easily distinguished by the model, failing to provide effective learning signals. This allows the model to proactively and intelligently select more valuable negative samples, thereby significantly improving training efficiency and model performance.

[0091] For random sampling, the first step is to design a neural policy network π, which receives the current state (encoding of positive sample labels) and outputs a probability distribution covering the entire negative sample label pool. A higher probability indicates that the negative sample is considered more valuable and should be sampled. The second step, for each training batch, dynamically samples negative samples based on the probability distribution output by the policy network π (e.g., sampling 15 samples based on probability). The current positive samples and the dynamically sampled negative samples are used to form training data for the regular training of the DINO model. After one or several training epochs, the model performance is evaluated on a reserved validation set. Performance changes, such as a decrease in the loss function, are fed back as a reward signal to the RL agent. The parameters of the policy network π are updated using a policy gradient algorithm to maximize the expected reward. If selecting certain negative samples significantly improves model performance, the probability of selecting these negative samples will increase when similar positive samples are seen again; conversely, the probability will decrease. DINO is a visual model based on a self-supervised learning framework. The RL agent is a reinforcement learning agent. Repeat step two, iteratively optimizing the policy network π to gradually learn an intelligent sampling strategy: adaptively selecting the negative samples that best challenge the current model and provide the most information gain for different positive samples.

[0092] Step 2: Use DINO as the training model for object detection and ResNet as the model backbone to extract multi-level visual features of the normalized image, namely: 128×128×256, 64×64×512, and 32×32×1024. Introduce a self-attention mechanism in the DINO Encoder module to fuse the three-level features into an image encoder feature with a size of 21504×256, so as to achieve effective integration of information at different scales.

[0093] Using BERT as the language inference model, the tokenized defect label sequence is fed into the Encoder in BERT to generate language encoded features of size N×256. Where N represents the number of tokenized tags; simultaneously, in order to further construct a semantic representation that can be used for hierarchical recognition, based on the number of tag tokens... Extracting second-level language coding features of size S×256 from language coding features The second-level language encoding features are reshaped into features of size 1×256 using a fully connected layer.

[0094] ;

[0095] In the formula, This is a second-level language encoding feature. To find based on tags The corresponding language features in the text, with a size of , Determined by the number of tokens after tokenization. For language encoding features, This is a list containing both positive and negative sample labels.

[0096] Since this invention uses 15 negative samples and 2 positive sample labels, the above operation needs to be performed 17 times, and all features... Ultimately, this forms the second-level language coding features. :

[0097] ;

[0098] In the formula, For feature concatenation, 17 1×256 features are concatenated into a second-level language coding feature. (Dimensions are 17×256) As a fully connected layer, Reshape into a 1×256 feature. Reshape is a remodeling process.

[0099] Step 3: Input the image encoder features and the language encoder features into the bidirectional two-domain self-attention module designed in this invention to perform deep interactive fusion of cross-modal information, and output the fused image coding features and the fused language coding features respectively.

[0100] The input features are mapped to a multi-head attention space through a fully connected layer, where the image encoder features are projected as a query vector and the language encoder features are projected as a key vector; simultaneously, the image encoder features are projected as a visual feature vector. The language encoder features are projected into language feature vectors. :

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] in, For image encoder features, For fully connected projection layers, For language encoder features, For fully connected projection layers, For fully connected projection layers, It is a fully connected projection layer.

[0106] The bidirectional attention weights for visual and linguistic enhancements are calculated separately using matrix multiplication, and the attention weights are normalized to a probability distribution using the Softmax function. The query is then incorporated into visual and linguistic features respectively, so that the attention weights focus on their respective modal information.

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] in, This represents attention weights guided by linguistic information. Represents the query vector. For language feature vectors, Represents the transpose matrix of the key. Representing matrix multiplication, the Softmax function converts the weights into a probability distribution. This represents attention weights guided by visual information. This represents a visual feature vector.

[0112] Bidirectional updates of visual and linguistic features are performed based on attention weights, and fused image-coded features are output through further mapping via fully connected layers. With fusion language coding features :

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] in, , For different fully connected projections; fuse the query vector with image-encoded features. The input is fed into the decoder, which decodes the image to obtain a fused image decoding feature with a size of 900×256, providing core multimodal features for subsequent defect identification and classification.

[0118] Step 4: Addressing the need for defect classification in distribution network defect detection, this invention designs a multimodal cascaded detection method. This method provides the model with the ability to classify defects, achieving defect subject identification and severity classification sequentially through two levels of cascaded loss, wherein:

[0119] First-level cascaded loss construction: Similarity calculation is performed on the fused image decoding features and language encoding features to generate a 900×N multimodal feature similarity matrix, where, The dataset contains target defect labels, such as insulator damage, and 15 negative sample labels (excluding descriptions of defect severity), tokenized into linguistic features. The fused image decoding feature size is 900×256, and the linguistic encoding feature size is N×256. Positive and negative samples are matched using the Hungarian method, i.e., the index corresponding to the example is found in the query. The model identifies the index corresponding to insulator damage and uses this as the positive sample index, with all others being negative samples. A cross-entropy loss function is constructed based on the positive and negative samples to train the model for identifying the presence of a specific type of defect, such as insulator damage. The Hungarian method is a combinatorial optimization algorithm. The trained model refers to the entire multimodal cascaded defect classification model, which is a complete network architecture undergoing end-to-end training, including a visual encoder, a language encoder, a bidirectional attention fusion module, and a final decision layer for similarity matching and classification.

[0120] ;

[0121] in, The cross-entropy loss function; For sample labels; To predict probabilities; The total number of samples.

[0122] Second-level cascaded loss construction: Similarity calculation is performed on the fused image decoding features and second-level language coding features to generate a second-level multimodal feature similarity matrix of size 900×B. Simultaneously, the second-level language coding features contain positive and negative labels for: moderate insulator damage, severe insulator damage, and critical insulator damage. Therefore, the second-level multimodal feature similarity matrix contains similarities between the image and the classification labels. Positive and negative samples are matched using the Hungarian method, and a cross-entropy loss function is constructed based on these samples to train the model to complete the classification and identification of defect severity, such as identifying severe insulator damage. The size of the second-level language coding features is B×256, where B contains the defect classification label and negative sample labels; in this invention, B=17.

[0123] Overcoming the uncertainty of single attribution methods, the credibility and interpretability of defect classification identification results are further improved; features of different modalities are treated as independent intelligent agents, their contribution is calculated through Shapley value in game theory, and finally verified in collaboration with causal graph model, effectively solving the black box problem of deep learning model, enabling operation and maintenance personnel to understand and trust the automated inspection results, and improving the reliability of system output.

[0124] Step 5: Introduce an interpretability decision module based on causal graph model and feature attribution game technology to further improve the interpretability of the model.

[0125] A cause-and-effect graph is constructed with visual features as the cause and defect type and level as the effect. Based on knowledge of the power defect domain, key variable nodes are defined, where:

[0126] Visual feature nodes represent key visual characteristics, extracted from the fused image features, such as crack length, contamination area, and color anomalies. Defect type nodes represent the type of defect, i.e., the primary identification result, such as insulator damage. Defect level nodes represent the severity, i.e., the secondary identification result, such as general, severe, or critical. Causal edges are constructed by defining the causal relationships between variables, forming a directed acyclic graph. For example: crack length - insulator damage, crack length + contamination area - critical. This means that crack length is the cause of insulator damage, while crack length and contamination area together determine whether the level is critical.

[0127] A multi-agent attribution game framework is constructed. The multimodal feature set upon which the model's decisions depend is defined as multiple agents participating in the game, wherein the fused image decoding feature map is divided according to the spatial dimension. Feature fragments The token embedding vectors of the tag sequences involved in similarity calculation As language-based intelligent agents, the common goal of these agents is to collaboratively allocate contributions to the model's final decision, such as insulator breakage emergencies.

[0128] An approximation algorithm based on Monte Carlo sampling is used to compute computations for each agent in a distributed manner. The Shapley attribution value for the final prediction result The Shapley value fairly measures the intelligence of the agent. The average marginal contribution across all possible collaborative alliances, for all The calculation process is expressed as follows:

[0129] ;

[0130] in, For the set of all intelligent agents, for Without intelligent agents a subset of The total number of agents. For subsets The output value of the model when the agent in the model is used as input.

[0131] After calculating the Shapley values ​​for all agents, the image regions corresponding to the top K visual agents and the text tokens corresponding to the top R language agents with the highest values ​​are selected as the output of consensus-based attribution evidence verified by game theory.

[0132] The attribution consensus evidence reached through the above game is matched with the causal graph rules constructed in the first step to check whether the attribution evidence supports the preconditions of the causal graph rules. By verifying the logical consistency between the attribution consensus and the rules defined in the causal graph, the credibility of the model's output is evaluated and the decision is explained.

[0133] For example, the fused attribution evidence, such as crack length = 35 pixels and oil stain area = 40 cm², will be used to... 2 The total contribution of critical keywords is 0.85. Substituting this into the cause-and-effect graph of power defects: If the evidence conforms to the rules of the cause-and-effect graph, such as crack length > 30 pixels and oil stain area > 30 cm², then... 2 →If the defect level is critical, the judgment result is highly reliable; if the evidence contradicts the causal rules, such as a crack length of 25 pixels being judged as critical, a low confidence warning is triggered, and manual review is recommended.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for graded identification of defects in distribution network line equipment based on open set target detection, characterized in that, include: The image encoding features and tag language encoding features of the distribution network line equipment are obtained, and combined with the bidirectional two-domain self-attention mechanism and decoding technology, the fused image decoding features and fused language encoding features are obtained. Based on the tag language encoding features, extract and generate second-level language encoding features; By performing two-level similarity calculations and matching between fused image decoding features and fused language coding features and secondary language coding features, the defect classification judgment results of distribution network line equipment are obtained, specifically including: Based on the multimodal feature similarity matrix and the second-level multimodal feature similarity matrix, and using the combinatorial optimization algorithm to match positive and negative samples; constructing a cross-entropy loss function based on the positive and negative samples, training a defect classification and identification model to identify the main body and level of defects, and obtaining the defect classification and determination results of distribution network line equipment; Based on causal graph and feature attribution game theory techniques, the credibility assessment and decision interpretation of defect classification results are performed, specifically including: A causal graph is constructed with visual features as the cause and defect type and level as the effect. Based on knowledge of the power defect domain, variable nodes are configured in the causal graph. Causal edges are constructed according to the causal relationships between variables, forming a directed acyclic graph (DAG). A multi-agent attribution game framework is constructed, and the fused image decoding features are divided into several feature segments according to spatial dimensions, generating corresponding visual agents. The embedding vectors of each token in the label sequence participating in similarity calculation are used as language agents. Based on an approximation algorithm using Monte Carlo sampling, the Shapley attribution values ​​for the final prediction result are calculated in a distributed manner for each visual agent and language agent. Based on the Shapley attribution values, image regions corresponding to several visual agents and text tokens corresponding to several language agents are selected as attribution consensus evidence. The attribution consensus evidence is matched with the rules of the DAG, and the credibility of the defect classification judgment result is evaluated and the decision is interpreted by verifying the logical consistency between the attribution consensus evidence and the rules in the causal graph. The formula for calculating Shapley's attribution value is: ； In the formula, This is the Shapley attribution value; For the set of all intelligent agents, for Without intelligent agents a subset of The total number of agents; For subsets The model's output value is obtained when the agent in the model is used as input. For marginal contribution.

2. The method for graded identification of defects in distribution network line equipment based on open set target detection according to claim 1, characterized in that, Before acquiring the image encoding features and tag language encoding features of the distribution network line equipment, the following steps are included: The original images and defect label texts of the distribution network line equipment are preprocessed and standardized to obtain standardized images, negative sample labels and positive sample hierarchical labels. Based on the actual situation of the standardized images, labels are applied to the standardized images using the cascading labeling method.

3. The method for graded identification of defects in distribution network line equipment based on open set target detection according to claim 2, characterized in that, The random sampling of negative sample labels includes: Configure a neural policy network and output a probability distribution based on the encoding of the current positive sample label to represent the value of the negative sample; Based on the probability distribution, negative samples are dynamically extracted and combined with the current positive samples to train the deep learning model. After the training cycle, the performance change of the deep learning model is used as a reward signal to feed back to the reinforcement learning agent; the neural policy network parameters are updated using the policy gradient algorithm to maximize the expected reward. Repeated iterations enable the neural policy network to learn intelligent sampling strategies, adaptively selecting negative samples for different positive samples and assigning corresponding labels.

4. The method for graded identification of defects in distribution network line equipment based on open set target detection according to claim 2, characterized in that, The acquisition of image encoding features and tag language encoding features of distribution network line equipment includes: A deep learning model was selected as the training model for object detection, and multi-level visual features of the standardized images were extracted. By introducing a self-attention mechanism into a deep learning model, multi-level visual features are fused to obtain image coding features; Using a language inference model, the tokenized defect tag sequence is converted into tag language encoded features.

5. The method for graded identification of defects in distribution network line equipment based on open set target detection according to claim 1, characterized in that, The process of acquiring image encoding features and tag language encoding features of distribution network line equipment, and combining them with a bidirectional two-domain self-attention mechanism and decoding technology to obtain fused image decoding features and fused language encoding features includes: A bidirectional two-domain self-attention module is constructed; cross-modal information fusion is performed by inputting image coding features and label language coding features into the bidirectional two-domain self-attention module, and the fused image coding features and fused language coding features are output. The query vector, which combines the fused image coding features with the projection of the image coding features, is input into the decoder, and the fused image decoding features are output.

6. The method for graded identification of defects in distribution network line equipment based on open set target detection according to claim 5, characterized in that, The process of inputting image coding features and tag language coding features into a bidirectional two-domain self-attention module, performing cross-modal information fusion, and outputting fused image coding features and fused language coding features includes: Image encoding features and label language encoding features are mapped to a multi-head attention space through a fully connected layer; Attention weights for visual enhancement and language enhancement are calculated separately using matrix multiplication, and the attention weights are then normalized to a probability distribution. Based on attention weights, visual and linguistic features are updated bidirectionally, and the fused image coding features and fused linguistic coding features are output through mapping of fully connected layers.

7. The method for graded identification of defects in distribution network line equipment based on open set target detection according to claim 1, characterized in that, The extraction and generation of second-level language encoding features based on tag-based language encoding features includes: Based on the number of tag tokens, secondary language encoding features are extracted from the tag language encoding features, and the size of the secondary language encoding features is adjusted; all extracted secondary language encoding features are then concatenated.

8. The method for graded identification of defects in distribution network line equipment based on open set target detection according to claim 1, characterized in that, The method of obtaining the defect classification judgment result of distribution network line equipment by performing two-level similarity calculation and matching between fused image decoding features and fused language coding features and secondary language coding features respectively also includes: The similarity between the fused image decoding features and the fused language coding features is calculated to generate a multimodal feature similarity matrix; The similarity between the fused image decoding features and the secondary language coding features is calculated to generate a secondary multimodal feature similarity matrix.

Images