Electric power industry image sample labeling method based on artificial intelligence
By adding a module-level attention mechanism to the Transformer architecture, the problem of inaccurate annotation of power line images is solved, achieving more accurate and richer annotation results and improving the ability to extract image and text features.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- STATE GRID ANHUI ELECTRIC POWER CO LTD ELECTRIC POWER SCI RES INST
- Filing Date
- 2025-12-17
- Publication Date
- 2026-04-24
AI Technical Summary
Existing methods for labeling power line images collected by drones suffer from inaccurate labeling and insufficient information.
We employ a Transformer-based self-attention mechanism, which saves and fuses information of different levels by adding module-level attention, and injects it into the decoder module by module during the encoding and decoding stages to improve the accuracy of image annotation.
It improves the accuracy and information richness of image annotation, enhances the ability of deep neural networks to extract image and text features, and generates more accurate annotation text.
Smart Images
Figure CN121921605A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image annotation technology, and in particular to an artificial intelligence-based method for annotating image samples in the power industry. Background Technology
[0002] Automatic image annotation refers to the process of automatically adding textual feature information reflecting the visual content of an image using machine learning methods. The basic idea is to automatically learn the potential associations or mappings between the semantic concept space and the visual feature space using already labeled image sets or other available information, thus adding textual keywords to unknown images. Through automatic image annotation technology, image information problems can be transformed into text information processing problems, which are technically more mature.
[0003] Currently, power line inspections mostly rely on drones to collect images of the lines, which are then processed to produce inspection results. However, current methods for annotating drone-collected images suffer from inaccurate annotations and insufficient information.
[0004] Based on this, this application proposes an artificial intelligence-based image sample annotation method for the power industry. Summary of the Invention
[0005] This invention provides an AI-based image sample annotation method for the power industry. By incorporating module-level attention to preserve information with varying degrees of fusion, and then injecting the fused information into the decoder module by module, the annotation effect is improved.
[0006] According to one aspect of this disclosure, an artificial intelligence-based image sample annotation method for the power industry is provided, the method comprising: Acquire images of power lines captured by drones; The original image is input into the object detection network to obtain a fixed-length visual feature vector; where each feature vector represents an entity in the image with different semantics, including: safety warning signs, equipment identification, line and connection identification, and safe operation instructions; In object detection networks: the position and category of candidate boxes for each candidate region in the image are obtained by object detectors, feature maps of regional image features are extracted by convolutional networks, and multiple feature vectors are obtained by region feature cropping methods; The multiple feature vectors are input into the encoder module. During the attention encoding stage, the self-attention operation is used to assign weights to the visual feature vector group. The encoder's output is input into the fusion unit: the number of vector groups after adding modal attention is N, the same as the number of decoders and encoders in the model. After obtaining the i-th modal attention vector group, if i ≠ 1, the current vector group is fused with the (i-1)-th vector group using additive attention. The output of the fusion unit is input into the decoder: the output of the fusion unit is used as the key and value of the multi-head attention and input into N decoders. After N intermodal information fusions, the decoder obtains a vector group dt. Then, the softmax operation is used to obtain the probability yt of the generated labeled word in the dataset dictionary at the current time step, and outputs a natural language sequence.
[0007] In one possible implementation, the original image is input into an object detection network to obtain a fixed-length visual feature vector, including: Use convolutional networks to extract features from the original image; The feature vector is represented using one-hot encoding, and then a matrix transformation is performed to transform the feature vector into a target vector of fixed length.
[0008] In one possible implementation, the feature vectors are weighted in the encoder through a self-attention operation: first, the input vector group is assigned to Q, K, and V, then the three vector group matrices are divided into h=8 parts, and after dot product attention calculation, they are concatenated to obtain the weight parameter matrix.
[0009] In one possible implementation, the outputs of the first i encoders in the modal attention self-attention encoding stage are X1,...,Xi (i=1,...,N), a total of i vector groups, representing visual information with different degrees of self-attention. The specific calculation process of inter-module attention is shown in Equation (1). After concatenating the vectors in the first i self-encoding vector groups, a multi-head attention layer operation is performed, and then after passing through a fully connected layer, the vector group Mi with added modal attention is obtained. (1) Among them, X cat For cascading operation X cat The feature vector set W after = Cat(Xi,...,Xi) m ∈R idmodel × dmodel For fully connected layer parameters, the number of heads h for multi-head attention is set to 8.
[0010] In one possible implementation, the decoder comprises three sub-modules. The cross-modal information Fi is input into the second multi-head attention for information fusion. The calculation process of the second multi-head attention in the i-th decoder module is as shown in equation (2): (2) Among them, F iLet W be the fused information vector of the input, y be the current input information of the decoder, and W be the input information vector of the decoder. de y is the initial input y of the fully connected layer parameter decoder t-1 The supervision information is masked, which is the encoding vector of the first t-1 words in the image annotation after word embedding. Its dimension is the same as the visual encoding vector. After N intermodal information fusions, the decoder obtains a vector group dt, and then uses a softmax operation to obtain the probability yt of the generated labeled word in the dataset dictionary at the current time, as shown in equation (3): (3) Among them W d ∈ Rdmodel×ω ω represents the parameters of the fully connected layer, I represents the dictionary size of the dataset used, and I represents the original image.
[0011] Compared with the prior art, the beneficial effects of the present invention are: This application presents an image sample annotation method that achieves significant performance improvements across various tasks, including image annotation, within a self-attention framework represented by Transformer. The self-attention mechanism employed by Transformer can simultaneously operate on both the encoding and decoding stages of the image annotation process, further enhancing the feature extraction capabilities of deep neural networks for images and text, thus enabling the model to generate accurate annotation text. However, existing Transformers do not fully utilize multi-stage semantic information in image annotation tasks. From the perspective of multi-stage information fusion, this paper proposes a deep neural network architecture for image annotation based on Transformer. Building upon the use of self-attention, architectural improvements are made specifically for image annotation tasks. By incorporating module-level attention to preserve information with varying degrees of fusion, and injecting the fused information into the decoder module by module, the annotation performance is improved.
[0012] As an intermediate structure in the traditional "encoder-decoder" architecture, the fusion unit has brought objective performance improvements to image annotation tasks. The fusion unit can be said to simultaneously assist the information processing of both the encoder and decoder. It incorporates inter-module attention at the encoding end to extract higher-order semantic information, and implements phased information injection at the decoding end to improve inter-modal information fusion, jointly enhancing the performance of models based on self-attention architectures. Attached Figure Description
[0013] Figure 1 This diagram illustrates a flowchart of an image sample annotation method for the power industry based on artificial intelligence, according to an embodiment of this disclosure. Detailed Implementation
[0014] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.
[0015] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.
[0016] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.
[0017] An AI-based image sample annotation method for the power industry, the method comprising: Acquire images of power lines captured by drones; The original image is input into the object detection network to obtain a fixed-length visual feature vector; where each feature vector represents an entity in the image with different semantic meanings, including: Safety warning signs: Red indicates prohibition (e.g., "No Smoking"); Yellow indicates warning (e.g., "High Voltage Danger"); Blue is a mandatory instruction (e.g., "Protective Equipment Must Be Worn"); Green indicates the location of safety facilities (e.g., fire extinguisher signs). Equipment identification includes the equipment name (e.g., transformer, circuit breaker); displaying parameter information (rated voltage / current, model and specifications); and indicating the installation location (e.g., "cabinet color is gray"). Line and connection markings, using different line types to distinguish high-voltage / low-voltage lines; marking the incoming and outgoing line methods (top in, bottom out / side in, side out); marking the cable specifications (e.g., "YJV-10kV-3×185" indicates cross-linked cable, 10kV voltage, 3 cores, 185mm² cross-section).
[0018] Safe operation guidelines: Emergency evacuation route markings (blue background with white text and directional arrows); Equipment maintenance point markings (e.g., "Regularly check transmission devices").
[0019] Labeling example: High-voltage equipment: marked with yellow and black stripes, indicating "High Voltage Danger"; Cable wells: Use dashed boxes to indicate dimensions and mark "fireproof sealing"; Distribution cabinet: Label with cabinet number, serial number, and warning messages such as "Do not close". For example, "Sorry, there was a small problem, please try again later".
[0020] In object detection networks: the position and category of candidate boxes for each candidate region in the image are obtained by object detectors, feature maps of regional image features are extracted by convolutional networks, and multiple feature vectors are obtained by region feature cropping methods; The image feature vectors obtained by the object detection network are input into the encoder module of the Transformer. Before being fed into the first encoder module, the feature vectors representing different semantic information in the image are first converted to fixed-length dimensions through word embedding. In the encoder module, the embedded vector groups containing visual information are sequentially input into two sub-modules: a multi-head attention mechanism and a feedforward network. Each sub-module includes one residual connection and layer normalization operation.
[0021] The self-attention mechanism in Transformers assigns new weights to a set of visual vectors. The input vector set is linearly transformed and weighted into three matrices Q, K, and V: query, key, and value. Specifically, ScaledDot-Product Attention is used to calculate the similarity between these matrices, thus adding self-attention weights to the visual information. During the encoding phase, the self-attention operation ensures that each feature vector representing a different region of the image is weighted in conjunction with all other feature vectors. Intuitively, the addition of self-attention allows the computer to better infer the semantic information contained in the image and acquire higher-order visual features.
[0022] In the original Transformer architecture, both the encoding and decoding stages consist of several identical attention modules (typically set to a hyperparameter of 6). This application optimizes the original Transformer architecture. The original Transformer architecture, after N=6 encoder modules, calculates cross-attention by inputting the same information into the encoding module and the input information into the decoding module. This application, with the same number of encoders and decoders N, uses the weighted output of the first n encoders as the input of the nth decoder. The annotation method in this application provides the decoder with multi-stage encoded visual information, achieving multi-stage semantic information fusion. Experiments on the large-scale dataset MSCOCO show good results.
[0023] The multiple feature vectors are input into the encoder module. During the attention encoding stage, the self-attention operation is used to assign weights to the visual feature vector group. The encoder's output is input into the fusion unit: the number of vector groups after adding modal attention is N, the same as the number of decoders and encoders in the model. After obtaining the i-th modal attention vector group, if i ≠ 1, the current vector group is fused with the (i-1)-th vector group using additive attention, as shown in the following formula: First, the additive attention of the i-th and (i-1)-th module attention vectors is calculated, and then this is used as a weight vector set to be applied to the visual vector currently incorporating modal attention. Here, W1, W2, and W3 are all parameters of the fully connected layer. In this way, the model obtains N fusion feature vector sets F1,...,FN containing high-order semantic information that have been incorporated with self-attention and module attention. These can be seen as representing different levels of computer understanding of the image.
[0024] The output of the fusion unit is input into the decoder: the output of the fusion unit is used as the key and value of the multi-head attention and input into N decoders. After N intermodal information fusions, the decoder obtains a vector group dt. Then, the softmax operation is used to obtain the probability yt of the generated labeled word in the dataset dictionary at the current time step, and outputs a natural language sequence.
[0025] In one possible implementation, the original image is input into an object detection network to obtain a fixed-length visual feature vector, including: Use convolutional networks to extract features from the original image; The feature vector is represented using one-hot encoding, and then a matrix transformation is performed to transform the feature vector into a target vector of fixed length.
[0026] In one possible implementation, the feature vectors are weighted in the encoder through a self-attention operation: first, the input vector group is assigned to Q, K, and V, then the three vector group matrices are divided into h=8 parts, and after dot product attention calculation, they are concatenated to obtain the weight parameter matrix.
[0027] In one possible implementation, the outputs of the first i encoders in the modal attention self-attention encoding stage are X1,...,Xi (i=1,...,N), a total of i vector groups, representing visual information with different degrees of self-attention. The specific calculation process of inter-module attention is shown in Equation (1). After concatenating the vectors in the first i self-encoding vector groups, a multi-head attention layer operation is performed, and then after passing through a fully connected layer, the vector group Mi with added modal attention is obtained. (1) Among them, X cat For cascading operation X cat The feature vector set W after = Cat(Xi,...,Xi) m ∈R idmodel × dmodelFor fully connected layer parameters, the number of heads h for multi-head attention is set to 8.
[0028] In one possible implementation, the decoder comprises three sub-modules. The cross-modal information Fi is input into the second multi-head attention for information fusion. The calculation process of the second multi-head attention in the i-th decoder module is as shown in equation (2): (2) Among them, F i Let W be the fused information vector of the input, y be the current input information of the decoder, and W be the input information vector of the decoder. de y is the initial input y of the fully connected layer parameter decoder t-1 The supervision information is masked, which is the encoding vector of the first t-1 words in the image annotation after word embedding. Its dimension is the same as the visual encoding vector. After N intermodal information fusions, the decoder obtains a vector group dt, and then uses a softmax operation to obtain the probability yt of the generated labeled word in the dataset dictionary at the current time, as shown in equation (3): (3) Among them W d ∈ Rdmodel×ω ω represents the parameters of the fully connected layer, I represents the dictionary size of the dataset used, and I represents the original image.
[0029] Compared with the prior art, the beneficial effects of the present invention are: This application presents an image sample annotation method that achieves significant performance improvements across various tasks, including image annotation, within a self-attention framework represented by Transformer. The self-attention mechanism employed by Transformer can simultaneously operate on both the encoding and decoding stages of the image annotation process, further enhancing the feature extraction capabilities of deep neural networks for images and text, thus enabling the model to generate accurate annotation text. However, existing Transformers do not fully utilize multi-stage semantic information in image annotation tasks. From the perspective of multi-stage information fusion, this paper proposes a deep neural network architecture for image annotation based on Transformer. Building upon the use of self-attention, architectural improvements are made specifically for image annotation tasks. By incorporating module-level attention to preserve information with varying degrees of fusion, and injecting the fused information into the decoder module by module, the annotation performance is improved.
[0030] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0031] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. An image sample annotation method for the power industry based on artificial intelligence, characterized in that, The method includes: Acquire images of power lines captured by drones; The original image is input into the object detection network to obtain a fixed-length visual feature vector; where each feature vector represents an entity in the image with different semantics, including: safety warning signs, equipment identification, line and connection identification, and safe operation instructions; In object detection networks: the position and category of candidate boxes for each candidate region in the image are obtained by an object detector, feature maps of regional image features are extracted by a convolutional network, and multiple feature vectors are obtained by a region feature cropping method; The multiple feature vectors are input into the encoder module. During the attention encoding stage, the self-attention operation is used to assign weights to the visual feature vector group. The encoder's output is input into the fusion unit: the number of vector groups after adding modal attention is N, the same as the number of decoders and encoders in the model. After obtaining the i-th modal attention vector group, if i ≠ 1, the current vector group is fused with the (i-1)-th vector group using additive attention. The output of the fusion unit is input into the decoder: the output of the fusion unit is used as the key and value of the multi-head attention and input into N decoders. After N intermodal information fusions, the decoder obtains a vector group dt. Then, the softmax operation is used to obtain the probability yt of the generated labeled word in the dataset dictionary at the current time step, and outputs a natural language sequence.
2. The image sample annotation method for the power industry based on artificial intelligence according to claim 1, characterized in that, The original image is input into the object detection network to obtain a fixed-length visual feature vector, including: Use convolutional networks to extract features from the original image; The feature vector is represented using one-hot encoding, and then a matrix transformation is performed to transform the feature vector into a target vector of fixed length.
3. The image sample annotation method for the power industry based on artificial intelligence according to claim 1, characterized in that, In the encoder, the feature vectors are weighted through self-attention operation: first, the input vector group is assigned to Q, K, and V, then the three vector group matrices are divided into h=8 parts, and after dot product attention calculation, they are concatenated to obtain the weight parameter matrix.
4. The image sample annotation method for the power industry based on artificial intelligence according to claim 1, characterized in that, The outputs of the first i encoders in the modal attention self-attention encoding stage are X1,...,Xi (i=1,...,N), a total of i vector groups, representing visual information with different degrees of self-attention. The specific calculation process of inter-module attention is shown in the following formula (1). After concatenating the vectors in the first i self-encoding vector groups, a multi-head attention layer operation is performed, and then after passing through a fully connected layer, the vector group Mi with added modal attention is obtained. (1) Among them, X cat For cascading operation X cat The feature vector set W after = Cat(Xi,...,Xi) m ∈R idmodel× dmodel For fully connected layer parameters, the number of heads h for multi-head attention is set to 8.
5. The image sample annotation method for the power industry based on artificial intelligence according to claim 1, characterized in that, The decoder consists of three sub-modules. The cross-modal information Fi is input into the second multi-head attention for information fusion. The calculation process of the second multi-head attention in the i-th decoder module is as shown in equation (2): (2) Among them, F i Let W be the fused information vector of the input, y be the current input information of the decoder, and W be the input information vector of the decoder. de y is the initial input y of the fully connected layer parameter decoder t-1 The supervision information is masked, which is the encoding vector of the first t-1 words in the image annotation after word embedding. Its dimension is the same as the visual encoding vector. After N intermodal information fusions, the decoder obtains a vector group dt, and then uses a softmax operation to obtain the probability yt of the generated labeled word in the dataset dictionary at the current time, as shown in equation (3): (3) Among them W d ∈ Rdmodel×ω ω represents the parameters of the fully connected layer, I represents the dictionary size of the dataset used, and I represents the original image.