A method and system for constructing cross-media datasets based on scene parsing

Abstract

This invention discloses a method and system for constructing a cross-media dataset based on scene parsing, belonging to the field of transformer oil leakage detection technology. The method includes: acquiring cross-media prior knowledge for transformer oil leakage detection; constructing a transformer oil leakage scene and semantically fusing the cross-media prior knowledge with the transformer oil leakage scene; parsing the semantically fused transformer oil leakage scene to generate transformer inspection images and corresponding text descriptions; and combining these with manually annotated transformer inspection data to generate a cross-media dataset. This invention achieves efficient construction of a transformer inspection cross-media dataset, significantly enhances the completeness of the dataset in complex scenarios, avoids data bias caused by single-media data, and improves the accuracy and robustness of transformer oil leakage detection.

Description

Technical Field

[0001] This invention relates to the field of transformer oil leakage detection technology, and more specifically, to a method and system for constructing a cross-media dataset based on scene parsing. Background Technology

[0002] Transformer oil, as the insulation and cooling medium of oil-immersed power transformers, is an important guarantee for the safe and stable operation of transformers. Oil leakage during operation is the main cause of transformer failures, and in severe cases, it can lead to breakdown accidents, resulting in large-scale power outages or even grid collapse.

[0003] In recent years, drones, high-definition cameras, and ground-based intelligent robots have become routine inspection methods for intelligent transformer inspection, acquiring massive amounts of aerial photographs. Building on this, deep learning-based visual inspection technology is increasingly widely used in power systems. However, transformers have complex structures (including the transformer body, oil conservator, bushings, tap changers, radiators, and relays, among other key components), and even the same component can vary in size, color, and constituent parts (e.g., bushings). Accurate detection of these components presents certain challenges. Furthermore, transformer oil leaks occur randomly, with early leaks being small in area and varied in shape, presenting unique characteristics compared to visual inspection of other power equipment. Feature extraction from transformer oil leak images is particularly difficult under severe interference (shadows and water mist). Moreover, the complex background of inspection images, significant occlusion between components, random occurrence points, and the small, varied, and indistinct appearance of early leaks further complicate the representation of defect features in transformer oil leak images, a recognized bottleneck problem. Meanwhile, single image data cannot guarantee the accuracy and completeness of the dataset, necessitating the integration and fusion of textual information. This places more comprehensive data type requirements on the construction of transformer oil leakage datasets. The reason for this is that the image features of power transformer oil leakage are difficult to represent, limiting transformer oil leakage detection methods that rely solely on image information processing. The integration and fusion of other media (such as text) information is required. Therefore, constructing a cross-media dataset for power transformer oil leakage detection that includes both images and text, and then performing cross-media multi-knowledge representation for transformer oil leakage detection on this basis to enhance the fusion of multiple knowledge in power transformer oil leakage detection, is urgent and a crucial prerequisite for achieving efficient transformer oil leakage detection.

[0004] Although the construction of cross-media datasets is receiving increasing attention, and scene parsing based on them has been applied to image-text description generation tasks, research on the construction of cross-media datasets for transformer oil leakage detection is still lacking. How to acquire effective visual knowledge, integrate textual knowledge for scene parsing, and conduct research on the construction of cross-media datasets for transformer oil leakage detection is an important prerequisite for successful transformer oil leakage detection. Summary of the Invention

[0005] To address the above problems, this invention provides a method for constructing a cross-media dataset based on scene parsing, comprising the following steps:

[0006] Acquire cross-media prior knowledge for transformer oil leakage detection;

[0007] A transformer oil leakage scenario is constructed, and cross-media prior knowledge is semantically fused with the transformer oil leakage scenario;

[0008] The semantically fused transformer oil leakage scenario is analyzed to generate transformer inspection images and corresponding text descriptions. These images are then combined with manually annotated transformer inspection data to generate a cross-media dataset.

[0009] Preferably, in the process of acquiring cross-media prior knowledge, visual spatial prior knowledge is acquired by combining graph neural networks with inspection images, and textual prior knowledge of transformer oil leakage is acquired by a Transformer encoder combined with an attention mechanism.

[0010] Obtain cross-media prior knowledge through visual spatial prior knowledge and textual prior knowledge.

[0011] Preferably, in the process of acquiring visual spatial prior knowledge, based on graph neural networks, visual spatial prior knowledge is acquired by using three types of spatial structure information: the intersection-overlap ratio between target boxes, the center point distance, and spatial geometric attributes. Among them, the intersection-overlap ratio between target boxes is used to reflect the occlusion relationship between key components of the transformer, the center point distance is used as a supplement to the spatial relationship of non-overlapping target boxes, and the spatial geometric attributes are used to represent the feature that there are large differences in the size of different key components of the power transformer.

[0012] Preferably, in the process of acquiring prior knowledge of the text, textual information about transformer components is acquired, and word embedding is used for analysis. The textual information is input into the Transformer text encoder, and word vectors are generated through the text position encoding layer and the multi-head attention layer, thereby generating prior knowledge of the text.

[0013] Preferably, in the process of constructing the transformer oil leakage scenario, the correlation between transformer target nodes is inferred through class distribution, and the target nodes of the inspection image are integrated by using the weight adaptive allocation method to construct the transformer oil leakage scenario.

[0014] Preferably, in the process of obtaining correlation, given the initial class distribution P of the target transformer node. o For all associated nodes To score, The class distribution representing the target node i, The class distribution of neighboring nodes j of target node i is represented by an asymmetric function. Perform correlation calculation

[0015] Preferably, during the process of integrating target nodes in the inspection image, firstly, the relationship between target node i and its neighboring node j is denoted as z. i ,use Representation; then, adaptive weights Automatic updates are made based on the relationships between target nodes; finally, z is used. i Forming scene layout V layout The scenario analysis was completed, and then the transformer oil leakage detection scenario model was completed.

[0016] Preferably, in the process of acquiring transformer inspection images, based on the generated scene layout, a diffusion model is used to generate transformer inspection images at an ever-increasing spatial layout scale. Specifically, a Markov chain is used to gradually add noise to the data x at each time step t. t In order to obtain the posterior probability q(x) 1:T |x0), x 1:T This represents the input data;

[0017] By training the probability distribution p(x) t-1 |x t ), and traverse backward along the Markov chain to generate a new transformer inspection image.

[0018] Preferably, in the process of acquiring the text description information corresponding to the transformer inspection image, the text is decoded based on the Transformer decoder that introduces local attention and channel attention, and the text description information is generated through a dual scoring mechanism of local and channel. Specifically, the extracted scene layout features are input into the text decoder, and after passing through a multilayer perceptron, local scoring and channel scoring, the text description corresponding to the image is completed.

[0019] This invention provides a cross-media dataset construction system based on scene parsing, comprising:

[0020] The data acquisition module is used to acquire cross-media prior knowledge for transformer oil leakage detection;

[0021] The scenario building module is used to construct a transformer oil leakage scenario, and semantically fuses cross-media prior knowledge with the transformer oil leakage scenario;

[0022] The dataset generation module is used to analyze the transformer oil leakage scene after semantic fusion, generate transformer inspection images and corresponding text description information, and generate a cross-media dataset by combining it with manually annotated transformer inspection data.

[0023] The present invention discloses the following technical effects:

[0024] Unlike previous single-image media datasets, the transformer oil leakage cross-media dataset construction technology proposed in this invention, based on scene parsing, enables the efficient construction of transformer inspection cross-media datasets, significantly enhances the completeness of datasets in complex scenarios, avoids the data bias caused by single-media data, and improves the accuracy and robustness of transformer oil leakage detection. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is the cross-media prior knowledge acquisition described in this invention;

[0027] Figure 2 This invention relates to the knowledge fusion-based scene semantic parsing.

[0028] Figure 3 This invention relates to the generation of the cross-media dataset of leaked oil.

[0029] Figure 4 This is the text description information generation described in this invention;

[0030] Figure 5 This is a schematic diagram of the method described in this invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0032] like Figure 1-5 As shown, this invention addresses the problem of limited data types and lack of cross-media datasets in transformer oil leakage detection. It proposes a cross-media dataset construction method based on scene parsing, which acquires visual and textual prior knowledge in transformer oil leakage detection, completes scene parsing based on knowledge fusion, and generates a cross-media dataset for transformer oil leakage detection through scene parsing, thus laying the foundation for solving the problem of transformer oil leakage detection.

[0033] 1. Acquisition of cross-media prior knowledge:

[0034] Currently, transformer oil leakage detection relies solely on inspection image data, which fails to adequately consider factors such as occlusion caused by the transformer's structural environment, random occurrence points, variable appearance, indistinct features, and interference from shadows and water mist. Therefore, it is difficult to guarantee the effectiveness and completeness of the oil leakage dataset. To address this, this invention employs a Transformer encoder with an attention mechanism and a graph neural network to extract textual and visual spatial prior knowledge from images, such as... Figure 1 As shown.

[0035] To obtain prior textual knowledge of transformer components, a word embedding method is used for analysis. Professional documents such as substation inspection reports and maintenance specifications, as well as power monographs and papers, are input into the Transformer text encoder. Word vectors are generated through the text position encoding layer and the multi-head attention layer, thereby generating prior textual knowledge.

[0036] For spatial knowledge extraction, a graph neural network sensitive to graph structure is used to represent it. To construct spatial structure knowledge of the component dataset from multiple perspectives, this invention utilizes three types of spatial structure information: the intersection-over-union ratio (IoU) between target boxes, the distance between center points, and spatial geometric attributes. Specifically, the IoU reflects the occlusion relationship between key transformer components; the center point distance supplements the spatial relationship of non-overlapping target boxes; and the spatial geometric attributes represent the significant differences in size between different key components of the power transformer.

[0037] This invention provides a method for acquiring visual spatial prior knowledge by combining graph neural networks with inspection images, and a method for acquiring textual prior knowledge of transformer oil leakage by utilizing power monographs and transformer inspection reports and a Transformer encoder with an attention mechanism, thereby completing the acquisition of cross-media prior knowledge in transformer oil leakage detection.

[0038] 2. Scene semantic parsing based on knowledge fusion:

[0039] Transformer oil leakage images involve complex component structures, making leakage detection difficult and inaccurate. Therefore, this invention uses scene parsing to omit unimportant visual information from the image while retaining its structural information and main semantic elements. Compared to methods using image visual element retrieval, scene parsing reduces the computational load of the constructed model and improves its inherent interpretability. The prior knowledge and visual appearance knowledge obtained in the previous step are input into the scene parsing module to complete the scene parsing of the transformer inspection image, as follows: Figure 2 As shown.

[0040] This invention utilizes class distribution to infer the correlation between transformer target nodes, given an initial class distribution P of the transformer target nodes. o For all associated nodes To score, The class distribution representing the target node i, The class distribution of neighboring nodes j of node i is represented by an asymmetric function. To save computational resources, we consider using this function. Perform correlation calculation

[0041] This invention utilizes an adaptive weight allocation method to integrate target nodes in inspection images, thereby completing the modeling of a transformer oil leakage detection scenario. The adaptive weight allocation module dynamically allocates weights to the target node relationships used for inference. In this way, both basic visual details (size, shape, etc. of oil stains) and abstract textual knowledge (e.g., oil leakage has occurred on the bushing) can be inferred. The relationship between target node i and its adjacent node j is denoted as z. i ,use Representation; then, adaptive weights Automatic updates are made based on the relationships between target nodes. Finally, z is used. i Forming scene layout V layout Complete the scene analysis.

[0042] This invention utilizes a method that effectively integrates cross-media prior knowledge through a transformer scene semantic parsing module, and employs class distribution association inference and weight adaptive allocation methods to construct a transformer oil leakage scene, thereby completing the parsing of a transformer oil leakage scene based on cross-media knowledge fusion.

[0043] 3. Construction of a cross-media dataset for transformer oil leakage detection:

[0044] Images are generated based on scene analysis, along with corresponding text descriptions. Using the generated scene layout as a foundation, a diffusion model is employed to generate images at progressively increasing spatial scales. The diffusion model is a latent variable model that uses a Markov chain to gradually add noise to the data x at each time step t. t To obtain the posterior probability q(x) 1:T |x0), where x 1:T This represents the input data. The probability distribution p(x) is trained using this method. t-1 |x t ), Traversing backwards along the Markov chain, a new transformer inspection image is generated, such as... Figure 3 As shown.

[0045] The generation of text data corresponding to the image will incorporate a Transformer decoder with both local and channel attention for text decoding. Text features will be generated through a dual scoring mechanism of local and channel attention, such as... Figure 4 As shown, the extracted scene layout features are input into the text decoder, and after passing through a multilayer perceptron, local scoring, and channel scoring, the corresponding text description of the image is completed.

[0046] The generated transformer inspection images and corresponding text descriptions are organically combined with the previously manually labeled transformer inspection data to finally complete the construction of a cross-media dataset for transformer oil leakage.

[0047] This invention provides a method for acquiring scene layout and processing the layout based on a diffusion model to generate image data, and a method for generating text data using a multilayer perceptron. The generated image-text pairs are combined with previously manually labeled data to complete the construction of a cross-media dataset for transformer oil leakage.

[0048] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0049] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0050] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for constructing a cross-media dataset based on scene parsing, characterized in that, Includes the following steps: Acquire cross-media prior knowledge for transformer oil leakage detection; A transformer oil leakage scenario is constructed, and the cross-media prior knowledge is semantically fused with the transformer oil leakage scenario, specifically as follows: In the process of constructing the transformer oil leakage scenario, the correlation between transformer target nodes is inferred through class distribution, and the target node integration of inspection images is completed by using the weight adaptive allocation method to construct the transformer oil leakage scenario. In the process of obtaining correlation, given the initial class distribution of the target nodes of the transformer... For all associated nodes To score, The class distribution representing the target node i, The class distribution of neighboring nodes j of target node i is represented by an asymmetric function. Perform correlation calculation ; In the process of integrating target nodes in the inspection image, firstly, for target node i and its neighboring nodes... The relationship between them is denoted as ,use Representation; then, adaptive weights Automatic updates are made based on the relationships between target nodes; finally, [the process is utilized]. Forming scene layout The scene analysis was completed, and then the transformer oil leakage detection scene modeling was completed; The semantically fused transformer oil leakage scene is analyzed to generate transformer inspection images and corresponding text descriptions. These images are then combined with manually annotated transformer inspection data to generate a cross-media dataset, specifically: In acquiring transformer inspection images, based on the generated scene layout, a diffusion model is used to generate the transformer inspection images at an ever-increasing spatial layout scale. Specifically, a Markov chain is used to gradually add noise to the data at each time step t. In order to obtain the posterior probability , This represents the input data; Through training probability distribution Traverse the Markov chain backward to generate a new transformer inspection image; In the process of acquiring the text description information corresponding to the transformer inspection image, the text is decoded based on the Transformer decoder that introduces local attention and channel attention. The text description information is generated through a dual scoring mechanism of local and channel. The extracted scene layout features are input into the text decoder, and after passing through a multilayer perceptron, local scoring and channel scoring, the text description corresponding to the image is completed.

2. The method for constructing a cross-media dataset based on scene parsing according to claim 1, characterized in that: In the process of acquiring cross-media prior knowledge, visual spatial prior knowledge is obtained by combining graph neural networks with inspection images, and textual prior knowledge of transformer oil leakage is obtained by combining an attention mechanism with a Transformer encoder. The cross-media prior knowledge is obtained through the visual spatial prior knowledge and the textual prior knowledge.

3. The method for constructing a cross-media dataset based on scene parsing according to claim 2, characterized in that: In the process of acquiring visual spatial prior knowledge, based on the graph neural network, the visual spatial prior knowledge is acquired by using three spatial structure information: the intersection-over-union ratio between target boxes, the center point distance, and spatial geometric attributes. Among them, the intersection-over-union ratio between target boxes is used to reflect the occlusion relationship between key components of the transformer, the center point distance is used as a supplement to the spatial relationship of non-overlapping target boxes, and the spatial geometric attributes are used to represent the feature that there are large differences in the size of different key components of the power transformer.

4. The method for constructing a cross-media dataset based on scene parsing according to claim 3, characterized in that: In the process of acquiring prior knowledge of the text, textual information about transformer components is obtained and analyzed using word embedding. The textual information is input into the Transformer text encoder, and word vectors are generated through the text position encoding layer and the multi-head attention layer, thereby generating prior knowledge of the text.

5. A cross-media dataset construction system based on scene parsing, applied to the cross-media dataset construction method based on scene parsing as described in any one of claims 1-4, characterized in that, include: The data acquisition module is used to acquire cross-media prior knowledge for transformer oil leakage detection; The scenario building module is used to construct a transformer oil leakage scenario and semantically fuse the cross-media prior knowledge with the transformer oil leakage scenario. The dataset generation module is used to analyze the transformer oil leakage scene after semantic fusion, generate transformer inspection images and corresponding text description information, and generate a cross-media dataset by combining it with manually annotated transformer inspection data.

Images