Substation defect identification method and system

Abstract

The invention provides a transformer substation defect identification method and system, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring text information and image information of a transformer substation defect; inputting the text information and the image information of the transformer substation defect into a trained transformer substation defect identification model; wherein the transformer substation defect identification model comprises a visual encoder of a Transformers architecture and a large language model decoder; analyzing a defect type of the transformer substation corresponding to the text information and the image information of the transformer substation defect by using the transformer substation defect identification model, and outputting an analysis process; and determining and outputting a transformer substation defect type corresponding to the text information and the image information of the transformer substation defect. Text information such as substation point location information and the like can be fully utilized, the substation defect type can be output, and the thinking process for recognizing the substation defects can also be output; the method is high in adaptability and recognition rate for various scenes.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and substation technology, and specifically relates to a substation defect identification method, system, electronic equipment and computer-readable storage medium. Background Art

[0002] Substation defect detection is an indispensable component of power system operation and maintenance. It not only ensures stable operation, extends equipment life, and reduces maintenance costs, but also effectively prevents accidents, ensuring personal and equipment safety, and providing crucial support for smart grid construction and economic and social development.

[0003] The rapid development of computer vision technology has ushered in new solutions for substation defect detection and identification. Computer vision technology can automatically process and analyze large-scale substation image data in an objective and consistent manner, significantly improving the accuracy of defect detection. Compared with traditional manual inspections, computer vision technology not only effectively reduces human resource investment and time costs, but also significantly improves the efficiency and speed of defect detection. Through automated identification processes, it can quickly screen out defective areas and promptly identify potential problems. Object detection technology is a key technology in the field of computer vision. Existing technologies have disclosed solutions for applying object detection technology to substation defect identification. Currently, mainstream substation defect object detection algorithms can be divided into two categories: one-stage object detection algorithms and two-stage object detection algorithms. The YOLO series, including YOLOv5, YOLOv8, and YOLOv10, is the primary one-stage object detection algorithm, while the Faster R-CNN series is a representative two-stage object detection algorithm. Although these mainstream algorithms have solved the problem of substation defect identification to a certain extent, the following disadvantages still exist in some special scenarios: 1) For scenarios with defect targets of the same type and many deformation variations (such as hanging suspended objects, oil leakage, damaged covers, etc.), the defect identification accuracy is not high due to the different shapes of the defect targets and the difficulty in unifying the features; and for scenarios with smaller defect targets, such as cigarette butts in substations, the defect identification accuracy is not high due to the small target size; 2) For scenarios where defect targets are easily confused with background objects, such as bird nests and trees in the background, round dials and other round parts on the transformer body, there is a high false alarm rate; 3) The current substation defect target detection algorithm is a single-modal recognition, and there is a problem that text information such as substation location information cannot be used.

[0004] Therefore, there is an urgent need for a substation defect identification method with high recognition rate and wide application scenarios. Summary of the Invention

[0005] The present invention provides a substation defect identification method, system, electronic equipment and computer-readable storage medium, which are used to overcome at least one technical problem existing in the prior art.

[0006] To achieve the above-mentioned objectives, the present invention provides a substation defect identification method, comprising: obtaining text information and image information of a substation defect; inputting the text information and image information of the substation defect into a trained substation defect identification model; wherein the substation defect identification model is trained using a substation defect dataset through reinforcement learning; the substation defect identification model includes a visual encoder with a Transformers architecture and a large language model decoder; using the substation defect identification model to analyze the substation defect type corresponding to the text information and image information of the substation defect, and outputting the analysis process; determining and outputting the substation defect type corresponding to the text information and image information of the substation defect.

[0007] Further, the preferred method is that the method for obtaining the reward function of the reinforcement learning includes determining the position overlap reward value through the intersection-and-union ratio of the position detection box and the Ground truth detection box in the prediction result of the substation defect recognition model, determining the image semantic feature reward value through the semantic similarity value of the detection box cropped image and the Ground truth detection box cropped image in the model prediction result, and determining the text semantic feature similarity reward value through the text semantic feature similarity of the detection box name and the Groundtruth detection box name in the model prediction result; determining the reward function according to the position overlap reward value, the image semantic feature reward value and the text semantic feature similarity reward value.

[0008] Furthermore, a preferred method is that the substation defect text is substation site location information; wherein the substation site location information includes components corresponding to the substation site location and defect types corresponding to each component.

[0009] Furthermore, the preferred method also includes using the visual encoder to process the image information of the substation defect to obtain a first preset number of feature maps; the first preset number is equal to the image block number threshold; using the visual encoder to extract feature vectors for each feature map and obtain a feature sequence for each feature map; and obtaining the feature vector of the image information of the substation defect according to the feature sequence of each feature map.

[0010] Further, a preferred method is that the method of using the visual encoder to process the image information of the substation defect to obtain a first preset number of feature maps includes: determining an adaptive aspect ratio according to a preset image block number threshold; based on the adaptive aspect ratio, determining a matching aspect ratio according to the actual resolution and actual aspect ratio of the image information of the substation defect; slicing the image information of the substation defect according to the matching aspect ratio to obtain a second preset number of image blocks; wherein the second preset number = image block number threshold - 1; inputting the second preset number of image blocks and the global thumbnail of the image information of the substation defect into the visual encoder to obtain a first preset number of feature maps; wherein the global thumbnail of the image information of the substation defect is obtained after the image information of the substation defect is processed according to the pixels of the image block.

[0011] Furthermore, a preferred method is that the method of inputting the preset number of image blocks and the global thumbnail of the image information of the substation defect into the visual encoder to obtain a first preset number of feature maps includes: the text information of the substation defect is processed by multiple layers of self-attention layers to obtain a text feature vector containing global context information; the global thumbnail of the image information of the substation defect is processed by the visual encoder and the self-attention layer to obtain a thumbnail feature vector; the image block is processed by the visual encoder to obtain an image block map feature vector; the text feature vector and the image block map feature vector are fused and linearly projected to obtain an image block map vector that matches the dimension of the large language model decoder; the text feature vector and the image block map vector are used to obtain gated weighted image features; the text feature vector and the corresponding gated weighted image features are used to perform cross-attention calculation to obtain the final weighted features of the image block; the final weighted features of the image block, the text feature vector and the feature vector of the thumbnail are fused to obtain a feature map of a preset image block number threshold.

[0012] Furthermore, a preferred method is that the substation defect dataset includes substation defect text, substation defect image and substation defect label.

[0013] In order to solve the above problems, the present invention also provides a substation defect identification system, which uses the above-mentioned substation defect identification method to perform substation defect identification. The system includes: an information acquisition unit, which is used to obtain text information and image information of substation defects; an analysis unit, which is used to input the text information and image information of the substation defects into a trained substation defect identification model; wherein the substation defect identification model is trained using a substation defect dataset through reinforcement learning; the substation defect identification model includes a visual encoder with a Transformers architecture and a large language model decoder; the substation defect identification model is used to analyze the substation defect type corresponding to the text information and image information of the substation defect, and output the analysis process; the defect identification unit is used to determine and output the substation defect type corresponding to the text information and image information of the substation defect.

[0014] In order to solve the above problems, the present invention also provides an electronic device, which includes a memory, a processor, and a substation defect identification program stored in the memory and executable on the processor. When the substation defect identification program is executed by the processor, the steps in the above-mentioned substation defect identification method are implemented.

[0015] In order to solve the above problem, the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned substation defect identification method when executed by a processor.

[0016] The present invention provides a substation defect identification method, system, electronic device, and computer-readable storage medium. The substation defect identification model of the present invention is composed of a visual encoder based on a Transformers architecture and a large language model decoder. The substation defect identification model is trained using reinforcement learning. Unlike traditional large language models, the present invention does not require annotation of the thought process, only the answer. The present invention can fully utilize text information such as substation location information to output not only the substation defect type, but also the thought process for identifying substation defects. The present invention exhibits a high recognition rate for scenes with defect targets with many deformations of the same defect type, scenes with small defect targets, and scenes where defect targets are easily confused with background objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic flow chart of a method for identifying substation defects provided by an embodiment of the present invention.

[0019] Figure 2 A schematic diagram of the network structure of a substation defect identification model provided by one embodiment of the present invention.

[0020] Figure 3 A schematic diagram of the structure of a visual encoder with a Transformers architecture provided by one embodiment of the present invention.

[0021] Figure 4 A schematic diagram of the image processing principle provided by an embodiment of the present invention.

[0022] Figure 5 A data flow diagram from an input graph to a gated attention mechanism is provided for one embodiment of the present invention.

[0023] Figure 6 A flowchart of a reward function provided by one embodiment of the present invention.

[0024] Figure 7 A schematic diagram of modules of a substation defect identification system provided by one embodiment of the present invention.

[0025] Figure 8 A schematic diagram of the internal structure of an electronic device for implementing a substation defect identification method provided by an embodiment of the present invention.

[0026] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0027] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] The following is a clear and detailed description of the technical solutions in the embodiments of the present application, with reference to the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, the term "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of the features. In addition, in the description of the embodiments of this application, "plurality" means two or more than two.

[0030] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify and measure targets, and further performs graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies. The substation defect recognition model in the embodiment of the present application utilizes computer vision technology.

[0032] Large models typically refer to deep learning models with a large number of parameters, such as GPT, large-scale language models (LLMs), or image models. Their training typically relies on large amounts of data and computing resources, optimizing model parameters through supervised learning (such as training with labeled data) or self-supervised learning (such as predicting the next word in a text).

[0033] Reinforcement Learning (RL) is a paradigm of machine learning, the core of which is the interaction process between the agent and the environment: the agent selects an action based on the current state of the environment, the environment provides rewards, and continuously optimizes the policy to maximize the long-term cumulative rewards. In the specific implementation process, reinforcement learning can be used to train a large model to optimize certain behaviors or strategies of the large model, or use the large model as an agent or environment agent in reinforcement learning. The substation defect recognition model of the present invention is composed of a visual encoder of the Transformers architecture and a large language model decoder. The substation defect recognition model is trained by reinforcement learning. Unlike traditional large language models, the present invention does not need to label the thinking process, only the answer.

[0034] The contextual attention (CotAttention) mechanism is a neural network model for visual question answering tasks in multimodal scenarios. It is often used for text sequence modeling tasks in the field of natural language processing and has strong model capabilities when processing large-scale data sets.

[0035] Example 1

[0036] Figure 1 The overall description of the substation defect identification method is given. Figure 1 A schematic flow chart of a method for identifying substation defects according to an embodiment of the present invention; the method may be executed by a system, which may be implemented by software and / or hardware.

[0037] like Figure 1 As shown, in this embodiment, the substation defect identification method includes steps S110 to S140.

[0038] S110: Obtain text information and image information of substation defects.

[0039] Computer vision technology is used to obtain image information of substation defects, that is, images taken at the substation location. The State Grid has clear specifications for substation design, such as the "330kV~750kV Intelligent Substation Design Specification" (GB / T51071-2014) and the "110(66)kV~220kV Intelligent Substation Design Specification" (GB / T 51072-2014); point information usually refers to the name, definition and sequence list of telemetry and telesignaling information collected by various equipment in the substation. This information needs to be described in accordance with the specifications to ensure the standardization and integrity of the monitoring information. The point table (information point table) in the substation automation system is generated and verified through standardized templates to ensure that it complies with national standards and industry requirements.

[0040] Compared to existing substation defect target detection algorithms that use single-modal recognition, resulting in the inability to utilize textual information such as substation location information, the substation defect text is substation location information. This substation location information includes components corresponding to the substation location and the defect type corresponding to each component. For example, [38118_Main transformer area No. 1 main transformer bay No. 1 oil-immersed transformer body No. 1 main transformer respirator 01_SXJ0018_20241030063832.jpg respirator silicone discoloration, respirator silicone cylinder damage, respirator oil seal damage, oil level status oil seal abnormality, respirator alarm, oil seal alarm]; specifically, this substation location information indicates that the primary component of concern at this location is the respirator, which is prone to several defects, including respirator silicone discoloration, respirator silicone cylinder damage, respirator oil seal damage, and oil level status oil seal abnormality.

[0041] S120. Input the text information and image information of the substation defect into a trained substation defect recognition model; wherein the substation defect recognition model is trained using a substation defect dataset through reinforcement learning; the substation defect recognition model includes a visual encoder and a large language model decoder of a Transformers architecture; the substation defect recognition model is used to analyze the substation defect type corresponding to the text information and image information of the substation defect, and output the analysis process.

[0042] It should be noted that the substation defect recognition model is trained using reinforcement learning based on the substation defect dataset to obtain the trained substation defect recognition model; unlike the existing large language model, the thinking process does not require human annotation; after the model is trained by reinforcement learning, the substation defect recognition model is autonomously generated in the reasoning stage.

[0043] The substation defect dataset includes substation defect text, substation defect image, and substation defect label, i.e., a <text, image, label> triple. The substation defect text includes substation location information and instruction words. Specifically, the first part is the preset location information of the substation location, which is used to describe the main components of the location and the defects that are likely to occur. For example,

[0044] [38118_Main Transformer Area No. 1 Main Transformer Bay No. 1 Oil-Immersed Transformer Body No. 1 Main Transformer Breather 01_SXJ0018_20241030063832.jpg Breather Silicone Discoloration, Breather Silicone Cylinder Damage, Breather Oil Seal Damage, Oil Level Status Oil Seal Abnormality, Breather Alarm, Oil Seal Alarm] indicates that the primary component of concern at this location is the breather, which is prone to defects such as breather silicone discoloration, breather silicone cylinder damage, breather oil seal damage, and oil level status oil seal abnormality. The second part is the instruction word, for example, instruction: The image is the original image taken at this location; the label is the actual defect type in the image, and the coordinates of the defect box location are: upper left corner coordinates: [xmin, ymin], lower right corner coordinates: [xmax, ymax].

[0045] Figure 2 This is a schematic diagram of the network structure of a substation defect identification model provided by an embodiment of the present invention. Figure 2 As shown, the substation defect recognition model of the present invention includes a visual encoder, which is responsible for encoding the input image and extracting the feature vector; a large language model decoder, which is responsible for decoding the combined feature vector and finally generating the reasoning process, the location of the defect, and the defect type. In the specific implementation process, as an example, the visual encoder is a Vision Transformer (ViT) model with a parameter volume of 600M; the decoder is a qwen2-7B LLM decoder. Specifically, as Figure 2 As shown, the input image path includes Figure 1 、 Figure 2 and Figure 3 .Will Figure 1 、 Figure 2 and Figure 3 The input visual encoder is used to encode and extract feature vectors; the LLM decoder decodes the combined feature vectors and finally generates the reasoning process, defect location, and defect type.

[0046] Figure 3 This is a schematic diagram of the structure of a visual encoder of the Transformers architecture provided by one embodiment of the present invention. Figure 3As shown, the network structure of the visual encoder can be a classic VIT. Specifically, the Vision Transformer (ViT) is essentially a model that applies the Transformer architecture to image processing. In ViT, the input image is first segmented into a series of non-overlapping blocks, with common block resolutions such as 16×16 or 32×32. These image blocks are then fed into a linear embedding layer for embedding. On this basis, the model adds position embedding to preserve the spatial position information of the image blocks. These embedded image block sequences are then fed into the Transformer encoder layer for feature extraction and information exchange. To obtain a global representation of the entire image, a special [CLS] marker is added to the beginning of the sequence. After processing by the encoder layer, the final hidden state can be used for subsequent tasks. For example, a linear classification head can be added on top of the final hidden state to perform image classification tasks. In the specific implementation process, S1201, use the visual encoder to process the image information of the substation defect to obtain a first preset number of feature maps; the first preset number is equal to the image block number threshold; S1202, use the visual encoder to extract feature vectors for each feature map and obtain a feature sequence of each feature map; S1204, obtain the feature vector of the image information of the substation defect according to the feature sequence of each feature map.

[0047] In a specific embodiment, in order to take into account images of different resolutions, so that the model can simultaneously extract global overall features and local detail features, the present invention uses multi-patch dynamic aspect ratio technology to process the input image before using the visual encoder for processing. Figure 4 This is a schematic diagram of the image processing principle provided by one embodiment of the present invention. Figure 4As shown, the method of using the visual encoder to process the image information of the substation defect to obtain a first preset number of feature maps includes: S1211, determining the adaptive aspect ratio according to a preset image block number threshold; S1212, based on the adaptive aspect ratio, determining the matching aspect ratio according to the actual resolution and actual aspect ratio of the image information of the substation defect; S1213, slicing the image information of the substation defect according to the matching aspect ratio to obtain a second preset number of image blocks; wherein the second preset number = image block number threshold - 1; S1214, inputting the second preset number of image blocks and the global thumbnail of the image information of the substation defect into the visual encoder to obtain a first preset number of feature maps; wherein the global thumbnail of the image information of the substation defect is obtained after the image information of the substation defect is processed according to the pixels of the image block. Specifically, exemplarily, when the image block number threshold is 7, 1) the maximum number of patches is preset: max_patches, the image will be divided into multiple 448x448 patches, but the number of patches will be guaranteed not to exceed max_patch. As follows Figure 3 In the example, max_patches is set to 7. 2) According to the maximum number of patches, the aspect ratio that can be adapted is given (length * width <= max_patch-1). Figure 4 The lower left part is the aspect ratio that can be adapted when max_patches=7. 3) According to the original image resolution and aspect ratio, the best matching aspect ratio is given ( Figure 4 2:3), and resize the image into multiple 448x448 patches ( Figure 4 In the example, the image is first resized from [800, 1300] to [448x2=896, 448x3=1344], and then sliced into 2x3=6 448x 448 patches. 4) Finally, the original image is resized to 448x448 to obtain a thumbnail, which is placed after the patch to obtain the image with a dynamic aspect ratio. After the above images are passed through the VIT visual encoder, max_patch fixed-size feature maps are obtained for subsequent feature fusion. In other words, VIT is used to extract feature vectors for each patch image. The feature extraction of each of the seven patches is independent, which is equivalent to obtaining a sequence for each patch image. The beginning of the sequence is the vector corresponding to the [CLS] tag, which is generally used as the feature vector for the entire image.

[0048] In a specific embodiment, in order to improve the multimodal understanding ability of the large language model. The present invention uses the Gated X-Attention mechanism. The gating mechanism is usually used to control the flow of information. When processing complex tasks, it can decide which information should pass and which information should be suppressed. In Gated X-Attention, the gating layer weights the calculation results of the cross attention to ensure that only important information can be passed to the subsequent network layer. This mechanism greatly improves the performance of the large language model of the present invention in multimodal understanding tasks. Figure 5 The data flow diagram from the input graph to the gated attention mechanism provided by one embodiment of the present invention. Figure 5 As shown in the figure, the images processed by the multi-patch dynamic aspect ratio technology, namely 6 448x 448 patch images and 1 thumbnail, will obtain 7 fixed-size feature maps after passing through the VIT visual encoder for subsequent feature fusion. Specifically, the image feature maps of 6 448x 448 patches and 1 thumbnail are downsampled and processed by a multi-layer perceptron. It should be noted that the VIT model outputs the feature vectors of 6 patches and 1 thumbnail, which are feature vectors with relatively large dimensions. Downsampling and MLP can be regarded as part of the encoder. The feature vectors of thumbnails and patches are obtained after downsampling and MLP processing. The feature vector dimensions at this time are relatively low, which is convenient for calculation. As shown in the figure Figure 5 As shown in the figure, the cross-attention module is primarily used in the decoder. Its input consists of two parts: an embedding vector fused from the thumbnail and text information, and a vector fused from the encoded patch embedding vector and the tag text vector representing the patch's location information. The output of the cross-attention module is a sequence vector fused from the image and text. Specifically, the thumbnail vector and the text embedding vector are element-wise added at corresponding locations to create the fused vector. The encoded patch embedding vector and the tag text vector representing the patch's location information are layered together to create the fused vector. The tag text vector representing the patch's location information is a randomly initialized, learnable vector using a neural network.

[0049] The method of inputting the preset number of image blocks and the global thumbnail of the image information of the substation defect into the visual encoder to obtain a first preset number of feature maps includes: S1221, the text information of the substation defect is processed by multiple self-attention layers to obtain a text feature vector containing global context information; S1222, the global thumbnail of the image information of the substation defect is processed by the visual encoder and the self-attention layer to obtain a thumbnail feature vector; S1223, the image block is processed by the visual encoder to obtain an image block map feature vector; S1224, the text feature vector and the image block map feature vector are fused and linearly projected to obtain an image block map vector that matches the dimension of the large language model decoder; S1225, the text feature vector and the image block map vector are used to obtain a gated weighted image feature; S1226, the text feature vector and the corresponding gated weighted image feature are used to perform cross-attention calculation to obtain the final weighted feature of the image block; S1227, the final weighted feature of the image block, the text feature vector and the feature vector of the thumbnail are fused to obtain a feature map of a preset image block number threshold.

[0050] For example, the text embedding vector in the text vector, the thumbnail vector and the patch vector in the visual vector are first combined with the gated cross attention mechanism, and finally the large language model decodes the output sequence as follows:

[0051] First, the processing of text vectors includes:

[0052] 1) Text embedding vector: The text input is first tokenized and embedded into a high-dimensional space to generate the initial text token embedding:

[0053] T=[t1,t2,t3,......,tn]

[0054] Where tn is the embedding vector of the nth text token.

[0055] 2) Self-Attention Processing:

[0056] The text token embedding vector T is processed by multiple layers of Self-Attention layers to generate text features containing global context information:

[0057] T′=Self-Attention(T)

[0058] These features will be used for subsequent multimodal fusion.

[0059] 3) Enter the gated cross attention mechanism module

[0060] The text feature T′ after Self-Attention processing is input as Query to Gated X-

[0061] In the Attention unit.

[0062] Secondly, the processing of visual vectors includes:

[0063] 1) The global thumbnail obtains the feature vector G of the thumbnail through the visual encoder and self-attention.

[0064] 2) The patch image is processed by the visual encoder to generate a patch image feature vector, which is then fused with the text feature vector representing the patch image location to obtain the patch image vector P. P passes through a linear projection layer to adjust the dimension to match the hidden layer dimension of the language model, generating a vector P′:

[0065] Finally, the fusion process in Gated X-Attention includes:

[0066] The gating unit dynamically calculates the weight of each P′ based on the current text context feature T′:

[0067] g k =σ(w g T'+b g )

[0068] Among them, g k is the gating weight of the kth patch graph vector, w g and b g are the weights and biases of the gated unit, and σ is the Sigmoid activation function, which limits the output value to the range [0, 1].

[0069] The gate weight is g k The gated weighted image feature P′gated is used to calculate the patch vector P′

[0070] P′gated=g k ⊙P ′ ,

[0071] where ⊙ represents element-wise multiplication.

[0072] Finally, T′ is used as the query to perform cross-attention calculation with the gated weighted image feature P′gated (as the key and value):

[0073] C=Attention(Q=T ′ ,K=P ′ gated, V=P ′ gated)

[0074] Among them, C is the final weighted feature of the patch graph, which is combined with the text context feature T ′ , the feature vector G of the thumbnail is fused and then enters the feedforward neural network for decoding to generate the final output sequence.

[0075] It should be noted that Attention is a cross-attention mechanism, and the calculation formula is:

[0076]

[0077] In terms of the training of the substation defect recognition model, the present invention introduces GRPO reinforcement learning into substation defect detection, and proposes a reward function LS-reward based on the combination of position and semantics.

[0078] GRPO reinforcement learning is a cost-effective reinforcement learning method that can greatly reduce the GPU computing resources required for training.

[0079] The method for obtaining the reward function of the reinforcement learning includes: S1231, determining a position overlap reward value by the intersection-over-union ratio of the position detection frame and the ground truth detection frame in the prediction result of the substation defect recognition model, determining an image semantic feature reward value by the semantic similarity value of the detection frame cropped image and the ground truth detection frame cropped image in the model prediction result, and determining a text semantic feature similarity reward value by the text semantic feature similarity of the detection frame name in the model prediction result and the ground truth detection frame name; S1232, determining the reward function according to the position overlap reward value, the image semantic feature reward value, and the text semantic feature similarity reward value. Specifically, LS-reward is a rule-based reward function that generates a weighted reward value by combining position overlap, text semantic feature similarity, and image semantic feature similarity, and optimizes the objective function according to the final reward value. Figure 6 A flowchart of a reward function according to an embodiment of the present invention is shown in FIG. Figure 6As shown in the figure, the position overlap reward value is obtained by calculating the IOU value of all detection boxes in the prediction results of the model and the Ground truth detection box. For a certain Ground truth detection box, if the maximum IOU value of all detection boxes in the prediction results and the calculated value exceeds 0.5, then the position overlap reward value is the maximum IOU value, otherwise the position overlap reward value is 0; the image semantic feature reward value is obtained by calculating the semantic similarity value of all detection box cropped images in the prediction results and the Ground truth detection box cropped image. For a certain Ground truth detection box cropped image, if the maximum similarity value of the semantic features of all detection box cropped images in the prediction results and the semantic features of the Ground truth detection box cropped image exceeds 0.8, then the image semantic feature reward value is the maximum similarity value, otherwise the image semantic feature reward value is 0 (wherein, the image semantic features are extracted by visiontransformer); for a certain Ground truth detection box, if the position overlap reward value or the image semantic feature reward value is greater than 0.5, then calculate the Ground The text semantic feature similarity between the truth detection box name and all detection box names in the prediction results. If the maximum text semantic feature similarity is greater than 0.8, then the similarity value is the text semantic feature reward value (it should be noted that text semantic features can be extracted through the BERT model, and the text semantic feature similarity and image semantic feature similarity both use the cosine similarity of the vector). Finally, for a ground truth detection box, the reward value is determined by the following formula:

[0080] Reward value = position overlap reward value * 0.5 + image semantic feature reward value * 0.25 + text semantic feature reward value * 0.25.

[0081] S130: Determine and output the substation defect type corresponding to the text information and image information of the substation defect.

[0082] For example, the complete reasoning process of the sample data is as follows:

[0083]

[0084]

[0085] It should be noted that in the above examples, "input" represents input text information; "instruction" represents input instruction words; "output" represents the model output, including the autonomously generated reasoning process and final answer; and "image" represents the image path.

[0086] In summary, the present invention proposes a substation defect recognition model based on a multimodal architecture, which is composed of a visual encoder and a large language model decoder of a Transformer architecture, and is trained using a reinforcement learning method. Compared with the traditional large language model, the present invention only needs to mark the final answer, without marking the thinking process, thereby significantly reducing the labeling cost and complexity. The substation defect recognition model of the present invention combines the visual encoder and the large language model decoder. The model can make full use of text data such as the point information of the substation, not only accurately output the defect type, but also generate the thinking process of identifying the defect, providing a more comprehensive basis for defect diagnosis. The substation defect recognition model of the present invention has a high recognition rate and performs well in a variety of complex scenarios, including: scenarios where the defect target has large deformation and the same type of defect has diverse morphologies; scenarios where the defect target is small in size; scenarios where the defect target is easily confused with the background, etc.; for some defect types existing in the prior art (such as broken respirator silicone barrels, abnormal oil level status, etc.), the number of samples is too small to train an effective model through traditional methods. The present invention significantly improves the recognition effect of these rare defect types through reinforcement learning and multimodal fusion, solving the technical bottleneck caused by sample scarcity. In summary, based on the architecture of a large multimodal model, the present invention can identify real defects in substations by analogy, and show stronger detection and generalization capabilities for newly emerging defects of unknown morphology.

[0087] like Figure 7 As shown, the present invention provides a substation defect identification system 700, which can be installed in an electronic device. Depending on the functionality implemented, the substation defect identification system 700 may include an information acquisition unit 710, an analysis unit 720, and a defect identification unit 730. A unit in the present invention, also known as a module, is a series of computer program segments that can be executed by an electronic device processor and perform a fixed function. These modules are stored in the electronic device's memory.

[0088] In this embodiment, the functions of each module / unit are as follows:

[0089] Information acquisition unit 710, for obtaining text information and image information of substation defects.

[0090] An analysis unit 720 is configured to input the text information and image information of the substation defect into a trained substation defect recognition model; wherein the substation defect recognition model is trained using a substation defect dataset through reinforcement learning; the substation defect recognition model includes a visual encoder with a Transformers architecture and a large language model decoder; and the substation defect recognition model is used to analyze the substation defect type corresponding to the text information and image information of the substation defect, and output the analysis process.

[0091] The defect identification unit 730 is configured to determine and output the substation defect type corresponding to the text information and image information of the substation defect.

[0092] The substation defect identification system 700 of the present invention, by combining a visual encoder and a large language model decoder, can fully utilize textual data such as substation location information. It can not only accurately output defect types but also generate the thought process for identifying defects, providing a more comprehensive basis for defect diagnosis. Based on the architecture of a multimodal vision-language large model, the present invention can identify real defects in substations by analogy and demonstrate enhanced detection and generalization capabilities for newly emerging defects of unknown morphology.

[0093] like Figure 8 As shown, the present invention also provides an electronic device 1 for a substation defect identification method.

[0094] The electronic device 1 may include a processor 10, a memory 11, and a bus. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a substation defect identification program 12. The memory 11 may include both an internal storage unit of the substation defect identification system and an external storage device. The memory 11 may be used not only to store application software and various data, such as the code for the substation defect identification program, but also to temporarily store data that has been or will be output.

[0095] The electronic device 1 may include a processor 10, a memory 11, and a bus. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a substation defect identification program 12. The memory 11 may include both an internal storage unit of the substation defect identification system and an external storage device. The memory 11 may be used not only to store application software and various data, such as the code for the substation defect identification program, but also to temporarily store data that has been or will be output.

[0096] The memory 11 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1. Furthermore, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed in the electronic device 1, such as the code of the substation defect identification program, but also to temporarily store data that has been output or is to be output.

[0097] In some embodiments, the processor 10 may be comprised of an integrated circuit, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (control unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and circuits. It executes programs or modules stored in the memory 11 (e.g., a substation defect identification program) and accesses data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0098] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11 and at least one processor 10, etc.

[0099] Figure 8 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 8The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0100] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management system, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management system. The power source may further include any components such as one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may further include a variety of sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be described in detail here.

[0101] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0102] Optionally, the electronic device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0103] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0104] The substation defect recognition program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve: obtaining text information and image information of the substation defect; inputting the text information and image information of the substation defect into a trained substation defect recognition model; wherein the substation defect recognition model is trained using a substation defect dataset through reinforcement learning; the substation defect recognition model includes a visual encoder and a large language model decoder of a Transformers architecture; using the substation defect recognition model to analyze the substation defect type corresponding to the text information and image information of the substation defect, and output the analysis process; determining and outputting the substation defect type corresponding to the text information and image information of the substation defect.

[0105] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiment is not repeated here. Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or system that can carry the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0106] An embodiment of the present invention also provides a computer-readable storage medium, which can be non-volatile or volatile, and stores a computer program. When the computer program is executed by a processor, it implements: obtaining text information and image information of substation defects; inputting the text information and image information of the substation defects into a trained substation defect recognition model; wherein the substation defect recognition model is trained using a substation defect dataset through reinforcement learning; the substation defect recognition model includes a visual encoder and a large language model decoder of a Transformers architecture; using the substation defect recognition model to analyze the substation defect type corresponding to the text information and image information of the substation defect, and output the analysis process; determining and outputting the substation defect type corresponding to the text information and image information of the substation defect.

[0107] Specifically, the specific implementation method when the computer program is executed by the processor can refer to the description of the relevant steps in the substation defect identification method in the embodiment, and will not be repeated here.

[0108] In the several embodiments provided herein, it should be understood that the disclosed devices, systems, and methods may be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0109] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0110] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0111] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0112] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0113] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems recited in a system claim may also be implemented by a single unit or system through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for identifying substation defects, characterized in that: include, Obtain text and image information of substation defects; Inputting the text information and image information of the substation defect into a trained substation defect recognition model; wherein the substation defect recognition model is trained using a substation defect dataset through reinforcement learning; the substation defect recognition model includes a visual encoder based on a Transformers architecture and a large language model decoder; Analyzing the defect type of the substation corresponding to the text information and image information of the substation defect using the substation defect recognition model, and outputting the analysis process; According to the analysis process, the substation defect type corresponding to the text information and image information of the substation defect is determined and output.

2. The substation defect identification method according to claim 1, characterized in that: The method for obtaining the reward function of the reinforcement learning includes: Determine the position overlap reward value by the intersection-and-union ratio of the position detection frame and the ground truth detection frame in the prediction result of the substation defect recognition model, determine the image semantic feature reward value by the semantic similarity value of the detection frame cropped image in the model prediction result and the ground truth detection frame cropped image, and determine the text semantic feature similarity reward value by the text semantic feature similarity of the detection frame name in the model prediction result and the ground truth detection frame name; The reward function is determined according to the position overlap reward value, the image semantic feature reward value and the text semantic feature similarity reward value.

3. The substation defect identification method according to claim 1, characterized in that: The substation defect text is substation location information; wherein, the substation location information includes components corresponding to the substation location and defect types corresponding to each component.

4. The substation defect identification method according to claim 1, characterized in that: Before the step of analyzing the substation defect type corresponding to the text information and image information of the substation defect using the substation defect recognition model and outputting the analysis process, the method further includes processing the image information of the substation defect using the visual encoder to obtain a first preset number of feature maps; the first preset number is equal to the image block number threshold; Using the visual encoder to extract feature vectors for each feature map and obtain a feature sequence for each feature map; A feature vector of the image information of the substation defect is obtained according to the feature sequence of each feature graph.

5. The substation defect identification method according to claim 4, characterized in that: The method of using the visual encoder to process the image information of the substation defect to obtain a first preset number of feature maps includes: Determining the adapted aspect ratio according to a preset threshold value of the number of image blocks; Based on the adapted aspect ratio, determining a matching aspect ratio according to an actual resolution and an actual aspect ratio of the image information of the substation defect; Slicing the image information of the substation defect according to the matching aspect ratio to obtain a second preset number of image blocks; wherein the second preset number = the image block number threshold - 1; The second preset number of image blocks and the global thumbnail of the image information of the substation defect are input into the visual encoder to obtain the first preset number of feature maps; wherein, the global thumbnail of the image information of the substation defect is obtained after processing the image information of the substation defect according to the pixels of the image blocks.

6. The substation defect identification method according to claim 5, characterized in that: The method of inputting the preset number of image blocks and the global thumbnail of the image information of the substation defect into the visual encoder to obtain a first preset number of feature maps includes: The text information of the substation defect is processed by multiple self-attention layers to obtain a text feature vector containing global context information; the global thumbnail of the image information of the substation defect is processed by the visual encoder and the self-attention layer to obtain a thumbnail feature vector; the image block is processed by the visual encoder to obtain an image block feature vector; The text feature vector and the image block feature vector are fused and linearly projected to obtain an image block vector that matches the dimension of the large language model decoder; the text feature vector and the image block vector are used to obtain a gated weighted image feature; the text feature vector and the corresponding gated weighted image feature are used to perform a cross-attention calculation to obtain the final weighted feature of the image block; The final weighted features of the image block, the text feature vector, and the feature vector of the thumbnail are fused to obtain a feature map of a preset image block quantity threshold.

7. The substation defect identification method according to claim 1, characterized in that: The substation defect dataset includes substation defect text, substation defect image and substation defect label.

8. A substation defect identification system, characterized in that: The substation defect identification method according to any one of claims 1 to 7 is used to identify substation defects, and the system includes: An information acquisition unit, used to acquire text information and image information of substation defects; an analysis unit, configured to input the text information and image information of the substation defect into a trained substation defect recognition model; wherein the substation defect recognition model is trained using a substation defect dataset through reinforcement learning; the substation defect recognition model includes a visual encoder with a Transformers architecture and a large language model decoder; and use the substation defect recognition model to analyze the substation defect type corresponding to the text information and image information of the substation defect, and output an analysis process; The defect recognition unit is used to determine and output the substation defect type corresponding to the text information and image information of the substation defect.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a substation defect identification program stored in the memory and executable on the processor. When the substation defect identification program is executed by the processor, the steps of the substation defect identification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the substation defect identification method according to any one of claims 1 to 7 is implemented.

Images