Electric power defect multi-mode identification method and system fused with spatial reasoning learning
By using multimodal feature alignment and spatial reasoning learning of graph neural networks, the problem of false alarms and missed alarms in power defect identification technology under complex scenarios is solved, and high-precision identification and robust discrimination of composite defects are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- STATE GRID HEBEI ELECTRIC POWER CO LTD
- Filing Date
- 2026-01-12
- Publication Date
- 2026-05-05
AI Technical Summary
Existing power defect identification technologies have high false alarm and false negative rates in complex scenarios, struggle to identify complex defects, and lack the ability to understand the spatial relationship between the target and the environment.
By constructing a multimodal feature alignment model, visual and textual features are extracted, a spatial relationship graph is established, and spatial reasoning learning is performed using graph neural networks to generate enhanced features for defect identification and interference filtering.
It improves the recognition accuracy in complex scenarios, reduces the false alarm rate, enhances the ability to identify complex defects, and improves the system's anti-interference and scenario adaptability.
Smart Images
Figure CN121980342A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of power system automation technology, and in particular to a method and system for multimodal identification of power defects that integrates spatial reasoning learning. Background Technology
[0002] Regular inspections of power transmission lines are a crucial foundation for ensuring the safe and stable operation of power systems. With the deep integration of drone technology and artificial intelligence algorithms, computer vision-based intelligent inspections are gradually replacing traditional manual inspections and becoming the mainstream model in the industry. Early intelligent recognition technologies mainly relied on single-modal vision models such as convolutional neural networks, focusing on feature extraction and detection of preset targets such as insulator damage, bird nests, or foreign objects on conductors. However, they were usually limited by closed-set settings, only able to identify predefined specific defect categories, and lacked the ability to understand unstructured scenes and complex text descriptions. In complex conditions with strong background interference or where defect targets are highly similar in appearance to normal components, false alarms and missed alarms are prone to occur.
[0003] Current intelligent power defect identification technologies primarily rely on visual feature matching models, using single-modal methods such as convolutional neural networks to detect and classify pre-defined targets such as insulator damage, bird nests, or foreign objects on conductors. While these methods have achieved some success in structured scenarios, the identification mechanism is essentially based on closed-set matching of visual features such as appearance, texture, color, and shape, without modeling the interaction relationships and spatial causal logic between targets. In real-world, complex power inspection scenarios, defect determination often depends not only on the target itself but also on its spatial relationships and interaction patterns with surrounding equipment.
[0004] For example, the reason why bird nests pose a hazard is that they are entangled or built on power lines or towers, rather than simply because their visual shape resembles that of birds; similarly, the danger of hanging foreign objects comes from their contact or hanging relationship with power lines.
[0005] Existing methods cannot understand the spatial topology and causal constraints of such spaces and can only make apparent similarity judgments. This leads to a significant increase in false positive and false negative rates in scenarios with visual similarity interference, complex background superposition, and multi-target interaction, and the ability to identify complex and hidden defects is seriously insufficient. Summary of the Invention
[0006] This invention provides a method and system for multimodal identification of power defects that integrates spatial reasoning learning, which solves the problem that current defect identification technologies are insufficient in identifying complex defects.
[0007] In a first aspect, the present invention provides a multimodal power defect identification method integrating spatial reasoning learning. The method includes: acquiring power inspection images and text description data; extracting visual features and text semantic features based on the power inspection images and text description data, and obtaining multimodal features through cross-modal alignment and semantic mapping; identifying multiple target entities based on the multimodal features, and constructing a spatial relationship graph with target entities as nodes and spatial relationships between target entities as edges; using a graph neural network to perform spatial reasoning learning on the spatial relationship graph to generate enhanced features that integrate geometric topology and spatial constraint relationships between targets; and performing defect category identification and interference target filtering based on the enhanced features and the spatial relationship graph to obtain the power defect identification result.
[0008] Secondly, the present invention provides a power defect multimodal identification device integrating spatial reasoning learning. The device includes: a communication module for acquiring power inspection images and text description data; a processing module for extracting visual features and text semantic features based on the power inspection images and text description data, and obtaining multimodal features through cross-modal alignment and semantic mapping; identifying multiple target entities based on the multimodal features, and constructing a spatial relationship graph with target entities as nodes and spatial relationships between target entities as edges; using a graph neural network to perform spatial reasoning learning on the spatial relationship graph to generate enhanced features that integrate geometric topology and spatial constraint relationships between targets; and performing defect category identification and interference target filtering based on the enhanced features and the spatial relationship graph to obtain the power defect identification result.
[0009] Thirdly, embodiments of the present invention provide a power defect multimodal identification system that integrates spatial reasoning learning. The system includes an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor is used to call and run the computer program stored in the memory to perform the steps of the method as described in the first aspect and any possible implementation thereof.
[0010] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the method as described in the first aspect and any possible implementation thereof.
[0011] This invention provides a multimodal power defect identification method and system that integrates spatial reasoning learning. By modeling target entities and their spatial relationships as a graph structure, the model can explicitly learn the geometric topology and interactions between devices in a power scenario, enhancing its understanding of complex spatial logic and improving the identification accuracy of defect types that rely on relative position determination. Subsequently, the message passing mechanism of a graph neural network is used to reason about the spatial relationship graph, generating enhanced features that integrate the geometric topology and spatial constraints between targets. This improves the discrimination ability under visually similar target interference, enhancing anti-interference capability and scene adaptability. Finally, in the identification stage, interference filtering is performed using the spatial relationship graph to filter out false defect targets, reducing the false alarm rate. This solves the problem of insufficient identification capability for composite defects in current defect identification technologies and improves the identification capability for composite defects in power equipment lines. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a flowchart illustrating a multimodal power defect identification method that integrates spatial reasoning learning, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a power defect multimodal identification technology architecture that integrates spatial reasoning, provided by an embodiment of the present invention. Figure 3 This is a schematic diagram of an adaptive text-image alignment encoding process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a spatial reasoning principle based on a graph neural network provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of logic verification and interference filtering provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a power defect multimodal identification device that integrates spatial reasoning learning, provided in an embodiment of the present invention. Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.
[0015] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.
[0016] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include other steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or device.
[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.
[0018] As described in the background section, the development of image-text multimodal large model technology has provided a new technological paradigm for the field of power line inspection. By pre-training on massive image-text data pairs, a cross-modal semantic alignment mechanism between image visual features and natural language text has been established, enabling the model to locate and analyze image content based on text instructions. Some research attempts to transfer general multimodal models to the power field through fine-tuning techniques to improve their adaptability in special scenarios.
[0019] While existing technologies have made some progress in image-text semantic alignment, significant technical bottlenecks remain when applied to power defect identification tasks that demand a high level of understanding of spatial relationships. First, existing multimodal models generally lack explicit spatial reasoning capabilities. Their core mechanisms are mostly based on attention mechanisms to learn statistical co-occurrence relationships between visual regions and text words, rather than precise geometric location logic in the physical world. These models struggle to accurately resolve logical relationships involving complex spatial constraints, such as an insulator located below a crossarm or foreign objects wrapped around the line surface, resulting in an inability to effectively utilize the location information of the target and its environment to aid in judgment. Second, these models are highly susceptible to interference from complex backgrounds and visually similar objects. Power inspection scenarios often involve complex natural environmental elements, and some non-defective objects are visually highly similar to defective objects. For example, the difference between a bird roosting and a bird's nest lies essentially in their spatial contact with power components. Models lacking spatial reasoning capabilities often classify based solely on visual similarity, making it difficult to distinguish between potential hazards and non-hazards when semantics are similar. Finally, existing models for image-text alignment mostly remain at the level of coarse-grained entity semantic matching, making it difficult to respond to complex inspection commands that include fine-grained directional descriptions, resulting in insufficient positioning accuracy when dealing with complex defect identification.
[0020] In summary, existing single-modal recognition methods and general multimodal models are insufficient to meet the stringent requirements of complex spatial logic reasoning in power line inspection, exhibiting significant shortcomings in recognition accuracy and scene adaptability. Therefore, there is an urgent need to develop an intelligent recognition method that can explicitly enhance the model's spatial reasoning capabilities, enabling a leap from simple target detection to scene understanding based on spatial relationship analysis.
[0021] To address the issues of low accuracy in power defect identification models, such as a lack of understanding of the spatial relationship between targets and the environment, weak logical reasoning ability, and susceptibility to interference from complex backgrounds and visually similar targets, this invention proposes a multimodal power defect identification method and system that integrates spatial reasoning learning. By introducing an explicit spatial relationship modeling mechanism, the model is expanded from simple visual feature extraction to scene-level logical reasoning, achieving high-precision identification and robust discrimination of power defects under complex operating conditions.
[0022] This invention provides a multimodal power defect identification method that integrates spatial reasoning learning, comprising steps one to three.
[0023] Step 1: Construct a domain-adaptive multimodal feature alignment model. This involves acquiring image data and corresponding text description data from the power inspection domain, and building a two-stream network architecture consisting of a visual encoder and a text encoder. The visual encoder extracts high-dimensional visual features from the images, while the text encoder extracts semantic features from the text descriptions. The two-stream network is then fine-tuned and trained on a power domain image-text dataset. A contrastive learning mechanism maps visual and semantic features to the same shared vector space, achieving semantic alignment between visual objects and textual concepts in the power domain.
[0024] Step Two: Construct a Spatial Reasoning Network Based on Spatial Relationship Modeling. A spatial reasoning refinement module is established to receive the aligned image-text features obtained in Step One; a spatial relationship dataset containing spatial orientation descriptions and truth labels is constructed, with samples covering various topological relationships between target objects and reference objects; this spatial relationship dataset is used to train the spatial reasoning refinement module, enabling it to explicitly model the spatial positional relationships and interaction logic between target entities in the image, and output enhanced feature representations that incorporate spatial geometric information.
[0025] Step 3: Perform defect identification and judgment based on spatial logic reasoning. The enhanced feature representations mentioned above are input into the defect identification network. The network combines the learned spatial relationship priors with visual features to identify the defect category and determine the potential hazards of the target object. During the judgment process, the network filters out visually similar but spatially logically inconsistent interfering targets based on the spatial reasoning results, and outputs the final power defect identification result.
[0026] Furthermore, in step two, the spatial reasoning refinement module adopts a graph neural network structure, models the target entities in the image as graph nodes, models the spatial positional relationships between targets as graph edges, and aggregates local and global contextual spatial information through a graph message passing mechanism, thereby enhancing the model's ability to represent complex spatial topological structures.
[0027] Furthermore, in step two, the spatial relationship dataset is constructed in the form of positive and negative sample pairs: for the same image, a description consistent with the actual spatial relationship is generated as a positive sample, and a description that violates the actual spatial relationship is generated as a negative sample. The spatial discrimination ability of the model is trained by binary classification or ranking loss function to improve its sensitivity to fine-grained spatial constraints.
[0028] Furthermore, in step three, the interference target filtering mechanism is as follows: when the identification network detects that a target object meets a specific defect category in terms of appearance features, but its corresponding spatial reasoning result does not meet the predefined spatial constraints of the defect category, the target is judged as a non-defect or background interference, thereby reducing the false alarm rate in complex scenarios.
[0029] This invention also provides a multimodal power defect recognition system integrating spatial reasoning learning, comprising: a multimodal feature extraction module for extracting visual features and semantic features of text descriptions from power inspection images and performing cross-modal feature alignment; a spatial reasoning learning module for receiving the aligned features, using a graph neural network to explicitly model and reason about the spatial topological relationships between targets, and generating enhanced features containing spatial logical information; and a defect recognition decision module for classifying defects based on the enhanced features, filtering visual interference targets in conjunction with the spatial reasoning results, and outputting power defect recognition results.
[0030] Furthermore, this invention can enhance spatial perception capabilities: by introducing a dedicated spatial reasoning learning stage, the model overcomes the limitations of relying solely on visual texture features, enabling explicit modeling and understanding of the relative positions and topological connections between targets, thus improving the deep perception and analysis level of complex power scenarios; improve anti-interference performance: relying on an explicit spatial logic reasoning mechanism, the model can effectively distinguish target objects that are visually similar but have different attributes, such as accurately distinguishing between normally perched birds and illegally built bird nests, significantly reducing the false alarm rate in complex background environments, and significantly improving the robustness of the system in high-noise scenarios; improve recognition accuracy: by adopting a multi-stage progressive training strategy, while ensuring that the model masters basic image-text semantic alignment capabilities, it further strengthens its ability to model fine spatial structures and relational constraints, enabling the model to exhibit higher recognition accuracy and scene adaptability when facing complex and hidden power defects.
[0031] like Figure 1 As shown, this embodiment of the invention provides a multimodal power defect identification method that integrates spatial reasoning learning. The method includes steps S101-S105.
[0032] S101. Obtain power inspection images and text description data.
[0033] S102. Based on power inspection images and text description data, extract visual features and text semantic features, and obtain multimodal features through cross-modal alignment and semantic mapping.
[0034] As one possible implementation, step S102 can be specifically implemented as steps S1021-S1023.
[0035] S1021. A visual encoder is used to encode the power inspection images in blocks and extract high-dimensional visual feature sequences.
[0036] S1022. Use a text encoder to segment and encode the text description data, and extract the text semantic feature sequence.
[0037] S1023. Input the high-dimensional visual feature sequence and the text semantic feature sequence into the multimodal feature fusion module, perform feature interaction through learnable query vectors, and use the cross-attention mechanism to perform cross-modal feature fusion, mapping the visual and text features to the same shared semantic space to obtain aligned multimodal features.
[0038] like Figure 2 As shown, this invention provides a schematic diagram of a multimodal power defect identification technology architecture that integrates spatial reasoning. The first stage: image-text encoding and adaptive feature alignment. This invention transforms the input heterogeneous multimodal data into a unified, semantically aligned feature representation. The image-text adaptive alignment encoding process is as follows... Figure 3 As shown. This invention first receives a pair of heterogeneous input data, including inspection images. and related text descriptions : ; in, It is the height of the image. It is the width of the image. This is the number of channels in the image, typically 3. The inspected image is fed into a visual encoder, preferably using a Visual Transformer (ViT) architecture. The image is first segmented into fixed-size patches, then linearly mapped and positionally encoded before being input into the encoder, which outputs a series of visual feature sequences: ; in, It is the sequence of visual features output by the visual encoder. It is the number of image blocks into which the image is divided. This is the dimension of visual features. Simultaneously, the text description is fed into a Transformer-based text encoder to generate a feature sequence containing semantic information about the text. ; in, It is the text feature sequence output by the text encoder. It is the number of tokens after the text has been segmented. It is a dimension of text features.
[0039] To achieve feature alignment between visual and textual data, the two unimodal features mentioned above are fed into a multimodal feature fusion module (e.g., ...). Figure 2 (As shown in the middle section), this module preferably uses a Q-Former structure. This structure initializes a set of learnable query vectors: ; in, It is a set of learnable query vectors. It is the number of query vectors. This refers to the dimension of the query vector. Through a multi-layered cross-attention mechanism, the query vector... With visual features To enable interaction while integrating text features The final output is a set of compact and semantically aligned multimodal features. .
[0040] S103. Based on multimodal features, identify multiple target entities and construct a spatial relationship graph with target entities as nodes and spatial relationships between target entities as edges.
[0041] As one possible implementation, step S103 can be specifically implemented as steps S1031-S1034.
[0042] S1031. Input the multimodal features into the target detection network to predict the bounding boxes and local visual features of multiple target entities in the power inspection image.
[0043] S1032. Construct an initial graph using each target entity as a graph node and the local visual features of each target entity as an initial feature vector.
[0044] S1033. On the initial graph, based on the spatial relative positions of each target entity, establish connecting edges between nodes to obtain a secondary graph.
[0045] S1034. Based on the geometric relationships between the bounding boxes of multiple target entities, assign initial weights representing the spatial relationship type to each edge in the quadratic graph to obtain a structured spatial relationship graph.
[0046] In some embodiments, the spatial relationship type includes one or more of the following: up and down, left and right, contact, and entanglement.
[0047] S104. A graph neural network is used to perform spatial reasoning learning on the spatial relationship graph to generate enhanced features that integrate the geometric topology and spatial constraint relationships between targets.
[0048] As one possible implementation, step S104 can be specifically implemented as steps S1041-S1045.
[0049] S1041. Extract the visual feature vector of each node and the spatial relationship type encoding of each edge from the spatial relationship graph to form a node-edge feature pair.
[0050] S1042. Input the node-edge feature pairs into the graph attention network, where the visual feature vector is used as the initial node embedding and the spatial relation type encoding is used as the relation bias in the attention calculation.
[0051] S1043. In each layer of graph reasoning in the graph attention network, a relation-aware attention mechanism is adopted to adjust the attention weights between nodes according to edge features, so as to realize message transmission guided by spatial relations.
[0052] S1044. Use the global semantic vector in the text semantic features as the context condition for graph reasoning, and fuse it with the neighbor features of the node in each layer node update.
[0053] S1045. After multiple rounds of iterative updates, the final feature vector of each node is extracted as an enhanced feature that integrates the global spatial context and local topological relationships.
[0054] For example, the second stage of this invention is spatial reasoning learning and feature enhancement. This invention, through explicit graph structure modeling, endows the model with the ability to reason about the spatial relationships between objects in a scene, based on the spatial reasoning principle of graph neural networks, as follows: Figure 4 As shown. Alignment features output from the first stage. First, the data is fed into a Transformer-based object detection network, preferably a DETR structure in this embodiment. This network includes a feature extraction backbone and a set of object query vectors. The network directly predicts the set of bounding boxes for all potential objects in the image. and its corresponding local visual feature vector .
[0055] ; Based on the detection results, the system constructs a spatial relationship graph. , where each vertex This corresponds to a detected entity. For example... Figure 4 As shown in the upper part, the detected entities include the bird's nest as the central node. Wires as neighboring nodes and insulators as neighboring nodes The edges in the diagram represent the initial spatial adjacency between targets. For example, a bird's nest is located above a conductor, and the conductor is connected to an insulator.
[0056] Constructed graph The data is fed into the graph neural network inference module; in this embodiment, a graph attention network architecture is preferably used. To incorporate prior knowledge from the text, this embodiment introduces a text guidance mechanism during graph message passing. In the... In the next iteration of reasoning, the node The feature update process is as follows: ; in: It is a node In the Layer update characteristics; It is a non-linear activation function, ReLU. It is the set of neighboring nodes of node $i$; It is the first The learnable weight matrix of the layer; Indicates feature concatenation operation; From text features The global semantic vector CLS token extracted from it is injected into the reasoning process as a global context condition. The importance weights are calculated by the attention mechanism, representing the importance of nodes. For nodes The degree of spatial impact.
[0057] like Figure 4 As shown, this module uses specially constructed triplet spatial relationship data, such as <insulator, bottom, crossarm>, for supervised training to force the network to learn geometric logic. After layer-by-layer iterative inference, the module finally outputs a set of enhanced features. Each feature incorporates precise spatial relationship information between the target and other targets in the scene.
[0058] S105. Based on the enhanced features and spatial relationship diagram, perform defect category identification and interference target filtering to obtain power defect identification results.
[0059] As one possible implementation, step S105 can be specifically implemented as steps S1051-S1056.
[0060] S1051. Input the enhanced features of each target entity into the multi-head self-attention layer and aggregate them to generate a scene-level global context vector.
[0061] S1052. The scene-level global context vector is concatenated and fused with the enhanced features of each target entity to obtain a fused vector.
[0062] S1053. Input the fused vector into the classifier to predict the defect category and obtain the prediction result for each target entity.
[0063] In some embodiments, the prediction results include the presence of a defect and the type of defect.
[0064] S1054. Based on the prediction results, for each target entity with defects, verify whether the spatial relationship conforms to the preset spatial logic constraints of the defect type, and obtain the verification results of each entity with defects.
[0065] S1055. If the verification result does not conform to the preset spatial logic constraints, it is determined to be an interference target and filtered out to obtain the real defect entity.
[0066] S1056. Based on real defect entities and the defect category and spatial location information of each real defect entity, generate power defect identification results.
[0067] For example, the third stage in this invention is logic verification and defect determination. This invention utilizes enhanced features incorporating spatial logic for final decision-making and interference filtering. The logical flow of logic verification and interference filtering is as follows: Figure 5 As shown. Enhanced feature set The data is then fed into the defect identification and filtering module. First, to obtain scene-level global discrimination information, a multi-head self-attention layer is used to aggregate the information from all nodes, generating a scene context vector. : ; Subsequently, The data is fed into a multilayer perceptron classifier for final decision-making. Since the input features have been cleaned by a spatial reasoning network, the classifier can not only identify defect categories but also perform implicit logical checks.
[0068] Reference Figure 5 For example, when the classifier detects that the visual features of an object resemble a hanging object, but its spatial inference features indicate that the object is above a guideline and conforms to the geometric topology of a bird standing in a standing posture, the model will activate a logical filtering mechanism. Based on the prior knowledge that objects not wrapped around a guideline are generally not potential hazards, it will classify the object as background interference that is not a defective target. Conversely, if the object is wrapped around a guideline, it will be classified as a potential hazard.
[0069] Finally, the present invention outputs the recognition result. It includes defect category labels and precise spatial location descriptions, enabling a deep understanding and accurate identification of power scenarios.
[0070] This invention provides a multimodal power defect identification method that integrates spatial reasoning learning. By modeling target entities and their spatial relationships as a graph structure, the model can explicitly learn the geometric topology and interactions between devices in a power scenario, enhancing its understanding of complex spatial logic and improving the identification accuracy of defect types that rely on relative position determination. Then, the message passing mechanism of a graph neural network is used to reason about the spatial relationship graph, generating enhanced features that integrate the geometric topology and spatial constraints between targets. This improves the discrimination ability under visually similar target interference, enhancing anti-interference capability and scene adaptability. Finally, in the identification stage, interference filtering is performed using the spatial relationship graph to filter out false defect targets, reducing the false alarm rate. This solves the problem of insufficient identification capability for composite defects in current defect identification technologies and improves the identification capability for composite defects in power equipment lines.
[0071] Optionally, the power defect multimodal identification method integrating spatial reasoning learning provided in this embodiment of the invention further includes steps S201-S204 after step S105.
[0072] S201. Based on the power defect identification results, determine the risk level of power equipment and provide handling recommendations.
[0073] S202. Based on the power defect identification results, determine the target inspection image related to the real defect entity from the power inspection images.
[0074] S203. Based on the defect type, spatial location information, confidence level, risk level and handling suggestions in the power defect identification results, as well as the target inspection image, generate a structured inspection report.
[0075] S204. After adding timestamps and equipment identifiers to the structured inspection report, store it in the inspection database.
[0076] Thus, this invention can generate structured inspection reports based on defect identification results, achieving automatic conversion from raw identification data to standardized, archiveable management information. By associating multi-dimensional information such as defect type, spatial location, risk level, and handling suggestions, and adding timestamps and equipment identifiers, this invention not only establishes a standardized inspection record system for easy historical tracing and statistical analysis, but also improves the informatization and automation level of defect management, providing a complete and reliable data foundation for subsequent maintenance decisions and accountability.
[0077] Optionally, the power defect multimodal identification method integrating spatial reasoning learning provided in this embodiment of the invention further includes steps S301-S303 after step S105.
[0078] S301. Based on the power defect identification results, determine the risk level of power equipment.
[0079] S302. Set push priority based on risk level.
[0080] High-risk defects are pushed in real time, while medium-risk and low-risk defects are pushed in batches at scheduled times.
[0081] S303. Based on the push priority, send inspection data packets to the inspection management platform.
[0082] In some embodiments, the inspection data package includes power defect identification results and structured inspection reports.
[0083] Thus, this invention achieves differentiated and intelligent responses to defect alarms by setting push priorities based on risk levels. Real-time push notifications for high-risk defects ensure timely handling of urgent hidden dangers, while batch, timed push notifications for medium- and low-risk defects optimize system communication load and human resource allocation. This mechanism effectively balances the timeliness of inspection responses with resource utilization efficiency, enhancing the collaborative capabilities and emergency response effectiveness of the power operation and maintenance system in closed-loop defect management.
[0084] Optionally, the power defect multimodal identification method integrating spatial reasoning learning provided in this embodiment of the invention further includes steps S401-S404 after step S105.
[0085] S401. Extract key spatial relationship triples based on the defect type and spatial location information in the power defect identification results.
[0086] In some embodiments, the key spatial relationship triple includes a subject, a relationship, and an object.
[0087] S402. Compare the key spatial relationship triples with the historical spatial relationship knowledge base to determine the comparison results.
[0088] S403. If the comparison result is a new relation pattern, then add the key spatial relation triple to the historical spatial relation knowledge base to obtain the updated knowledge base.
[0089] S404. Based on the updated knowledge base, adjust the weight distribution of samples in the spatial relationship dataset to enhance relational reasoning during model training.
[0090] Thus, this invention establishes a dynamic update mechanism for the spatial relationship knowledge base based on recognition results, enabling the model to continuously learn and evolve in real-world scenarios. By extracting key spatial relationship triples and comparing them with the historical knowledge base, the system can automatically discover and accumulate new relationship patterns, thereby adjusting the weights of training samples to enhance relationship reasoning capabilities. This approach significantly improves the model's adaptability and generalization ability to complex and novel scenarios, driving the recognition system towards self-evolving and scalable intelligent development.
[0091] Optionally, the power defect multimodal identification method integrating spatial reasoning learning provided in this embodiment of the invention further includes steps S501-S503.
[0092] S501. Record key intermediate data during the reasoning process.
[0093] In some embodiments, key intermediate data include object detection results, spatial relationship graph structure, node attention weights, classification confidence, and logical filtering decision paths.
[0094] S502. Associate the intermediate data with the power defect identification results to generate a structured inference log.
[0095] S503. Store the structured inference log to the log database.
[0096] Thus, this invention provides complete traceability and interpretability support for the identification process by recording key intermediate data throughout the entire reasoning chain and generating structured logs. The logs cover all stages from object detection and relationship modeling to decision filtering, facilitating auditing and problem identification, and providing a data foundation for model optimization, algorithm parameter tuning, and causal analysis. This mechanism enhances the transparency and credibility of the system in actual deployment, aligning with the development trend of auditable and interpretable artificial intelligence systems in key areas.
[0097] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0098] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.
[0099] Figure 6 A schematic diagram of a power defect multimodal identification device integrating spatial reasoning learning, provided by an embodiment of the present invention, is shown. The identification device 600 includes a communication module 601 and a processing module 602.
[0100] The communication module 601 is used to acquire power inspection images and text description data.
[0101] The processing module 602 is used to extract visual features and textual semantic features based on power inspection images and textual description data, and obtain multimodal features through cross-modal alignment and semantic mapping; based on the multimodal features, it identifies multiple target entities and constructs a spatial relationship graph with target entities as nodes and spatial relationships between target entities as edges; it uses a graph neural network to perform spatial reasoning learning on the spatial relationship graph to generate enhanced features that integrate the geometric topology and spatial constraint relationships between targets; based on the enhanced features and the spatial relationship graph, it performs defect category identification and interference target filtering to obtain power defect identification results.
[0102] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 700 includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, it implements the steps in the above-described method embodiments. Alternatively, when the processor 701 executes the computer program 703, it implements the functions of each module / unit in the above-described device embodiments.
[0103] For example, the computer program 703 may be divided into one or more modules / units, which are stored in the memory 702 and executed by the processor 701 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 703 in the electronic device 700.
[0104] The processor 701 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0105] The memory 702 can be an internal storage unit of the electronic device 700, such as a hard disk or memory of the electronic device 700. The memory 702 can also be an external storage device of the electronic device 700, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 700. Furthermore, the memory 702 can include both internal and external storage units of the electronic device 700. The memory 702 is used to store the computer program and other programs and data required by the terminal. The memory 702 can also be used to temporarily store data that has been output or will be output.
[0106] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A multimodal power defect identification method integrating spatial reasoning learning, characterized in that, include: Acquire power line inspection images and text descriptions; Based on the power inspection images and text description data, visual features and text semantic features are extracted, and multimodal features are obtained through cross-modal alignment and semantic mapping. Based on the multimodal features, multiple target entities are identified, and a spatial relationship graph is constructed with the target entities as nodes and the spatial relationships between the target entities as edges. A graph neural network is used to perform spatial reasoning learning on the spatial relationship graph to generate enhanced features that integrate the geometric topology and spatial constraint relationships between targets; Based on the enhanced features and spatial relationship diagram, defect category identification and interference target filtering are performed to obtain power defect identification results.
2. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, Based on the power inspection images and text description data, visual features and text semantic features are extracted, and multimodal features are obtained through cross-modal alignment and semantic mapping, including: A visual encoder is used to encode power inspection images in blocks and extract high-dimensional visual feature sequences. A text encoder is used to segment and encode the text description data, and the semantic feature sequence of the text is extracted. The high-dimensional visual feature sequence and the text semantic feature sequence are input into the multimodal feature fusion module. Feature interaction is performed through learnable query vectors, and cross-modal feature fusion is performed using a cross-attention mechanism. Visual and text features are mapped to the same shared semantic space to obtain aligned multimodal features.
3. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, The step of identifying multiple target entities based on the multimodal features and constructing a spatial relationship graph with the target entities as nodes and the spatial relationships between the target entities as edges includes: The multimodal features are input into the target detection network to predict the bounding boxes and local visual features of multiple target entities in the power inspection image; An initial graph is constructed using each target entity as a graph node and the local visual features of each target entity as an initial feature vector. On the initial graph, by combining the spatial relative positions of each target entity, connecting edges between nodes are established to obtain a secondary graph; Based on the geometric relationships between the bounding boxes of multiple target entities, each edge in the quadratic graph is assigned an initial weight representing the spatial relationship type to obtain a structured spatial relationship graph; the spatial relationship type includes one or more of the following: up and down, left and right, contact, and entanglement.
4. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, The step of employing a graph neural network to perform spatial reasoning learning on the spatial relationship graph generates enhanced features that integrate the geometric topology and spatial constraint relationships between targets, including: The visual feature vector of each node and the spatial relationship type encoding of each edge are extracted from the spatial relationship graph to form a node-edge feature pair; The node-edge feature pairs are input into the graph attention network, where the visual feature vector is used as the initial node embedding and the spatial relation type encoding is used as the relation bias in the attention calculation. In each layer of the graph inference process of the graph attention network, a relation-aware attention mechanism is adopted to adjust the attention weights between nodes according to edge features, so as to realize message passing guided by spatial relations. The global semantic vector in the text semantic features is used as the context condition for graph reasoning and fused with the neighbor features of the node in each layer node update. After multiple rounds of iterative updates, the final feature vector of each node is extracted as an enhanced feature that integrates global spatial context and local topological relationships.
5. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, The step of identifying defect categories and filtering interference targets based on the enhanced features and spatial relationship map to obtain power defect identification results includes: The enhanced features of each target entity are input into a multi-head self-attention layer and aggregated to generate a scene-level global context vector; The scene-level global context vector is concatenated and fused with the enhanced features of each target entity to obtain a fused vector; The fused vector is input into a classifier to predict the defect category, resulting in a prediction for each target entity. The prediction result includes whether a defect exists and the defect category. Based on the prediction results, for each target entity containing defective entities, the spatial relationships are checked to see if they conform to the preset spatial logic constraints of the defect type, and the check results of each defective entity are obtained. If the verification result does not conform to the preset spatial logic constraints, it is determined to be an interference target and filtered out to obtain the real defective entity; Based on the actual defect entities, as well as the defect category and spatial location information of each actual defect entity, the power defect identification result is generated.
6. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, After obtaining the power defect identification result by performing defect category identification and interference target filtering based on the enhanced features and spatial relationship map, the method further includes: Based on the power defect identification results, the risk level of the power equipment and handling recommendations are determined; Based on the power defect identification results, target inspection images related to the actual defect entities are determined from the power inspection images; Based on the defect type, spatial location information, confidence level information, risk level and handling suggestions in the power defect identification results, and the target inspection image, a structured inspection report is generated. The structured inspection report is stored in the inspection database after adding a timestamp and equipment identifier.
7. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, After obtaining the power defect identification result by performing defect category identification and interference target filtering based on the enhanced features and spatial relationship map, the method further includes: Based on the power defect identification results, the risk level of the power equipment is determined; Based on the risk level, push priority is set; high-risk defects are pushed in real time, while medium-risk and low-risk defects are pushed in batches at regular intervals. Based on the push priority, an inspection data packet is sent to the inspection management platform. The inspection data packet includes the power defect identification results and the structured inspection report.
8. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, After obtaining the power defect identification result by performing defect category identification and interference target filtering based on the enhanced features and spatial relationship map, the method further includes: Based on the defect type and spatial location information in the power defect identification results, key spatial relationship triples are extracted. The key spatial relationship triples include subject, relationship and object. The key spatial relationship triples are compared with the historical spatial relationship knowledge base to determine the comparison results; If the comparison result is a new relation pattern, then the key spatial relation triples are added to the historical spatial relation knowledge base to obtain an updated knowledge base; Based on the updated knowledge base, the weight distribution of samples in the spatial relation dataset is adjusted to enhance relational reasoning during model training.
9. The power defect multimodal identification method integrating spatial reasoning learning according to claim 1, characterized in that, The method further includes: During the inference process, key intermediate data is recorded, including target detection results, spatial relationship graph structure, node attention weights, classification confidence, and logical filtering decision paths. The intermediate data is correlated with the power defect identification results to generate a structured inference log; The structured inference log is stored in the log database.
10. A multimodal power defect identification system integrating spatial reasoning learning, characterized in that, The system includes an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor is configured to invoke and run the computer program stored in the memory to perform the method as described in any one of claims 1 to 9.