Power equipment image abstract generation method and device
The image characteristics of power equipment are extracted through Darknet and capsule networks, and combined with the multi-scale relationship generation network of Transformer, BERT and CLIP models, the problem of scarcity of training samples and high computational cost in the generation of power equipment image summary is solved, and high precision and efficiency are achieved. Image summary generation improves the accuracy of power fault diagnosis and system management security.
Patent Information
- Application Number
- CN202411089406.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
AI Technical Summary
The existing deep learning and image summary generation technology based on generative adversarial networks have problems such as scarcity of training samples and changes in view angles caused by large image deviations, high computational costs and low generation feature accuracy in the generation of power equipment, making it difficult to adapt to power. The complex structure and texture of the image affects the sensitive identification and diagnosis of abnormal states.
The Darknet model and capsule network are used to extract the key structural features of the power equipment image and input them into the pre-trained Transformer model. Combining BERT and CLIP models, a multi-scale relationship generation network is built to perform image-text feature matching and optimization to generate high-quality image summary.
It significantly improves the accuracy of power fault diagnosis by 30%, reduces false alarm rate, optimizes operation and maintenance decisions, and provides strong technical support for the intelligent management and safe operation of the power system.
Smart Images

Figure CN118968517A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image analysis, and in particular to a method and device for generating a summary of an image of electric power equipment. Background Art
[0002] Image summarization is an important technology in the field of computer vision and natural language processing. It aims to automatically extract core information from images and present it to users in a concise and clear form, such as text descriptions, key frame extraction, thumbnails, or key areas marked in the image. This technology is widely used in many fields, such as news media quickly generating eye-catching image summaries, social media platforms automatically generating descriptions of posts to improve interactivity, e-commerce websites adding descriptions to product images to improve search efficiency, and quickly browsing key events in monitoring systems. Image summarization not only improves the speed of information processing, but also enhances accessibility for users from different backgrounds, especially for groups with limited vision.
[0003] The basic methods for implementing image summarization are diverse and constantly evolving. Early techniques may rely on object detection algorithms to identify salient objects in an image and build summaries based on the location and features of these objects. With the development of deep learning, especially the combination of convolutional neural networks (CNN) and recurrent neural networks (RNN), the system can more accurately extract image features and generate natural language descriptions. In this process, the introduction of the attention mechanism has become a major breakthrough, which allows the model to focus on the most critical parts of the image when generating descriptions, thereby improving the relevance and accuracy of the summary.
[0004] In addition, multimodal learning frameworks are also used for image summary generation. By considering both image and text data, the model is able to establish a deeper connection between image understanding and text generation, producing a description that is more in line with the image content. In recent years, generative adversarial networks (GANs) have also begun to play a role in this field, especially in generating high-quality, visually continuous and expressive image summaries.
[0005] The above are the basic methods for generating image summaries. In practical applications, they usually need to be combined with specific algorithms to meet the requirements of the task.
[0006] Image summary generation technology based on deep learning mainly uses deep learning models, especially convolutional neural networks (CNNs) and recurrent neural networks (RNNs) or more advanced long short-term memory networks (LSTMs) and gated recurrent units (GRUs), to automatically extract key features from the original image and generate image summaries. Image summaries can be image classification labels, bounding box annotations of key areas, or text descriptions of image content. Convolutional neural networks (CNNs): play a central role in this process. They learn local features and patterns in images through multiple convolutional layers and gradually abstract high-level semantic information. This enables the model to understand objects, scenes, etc. in the image. Recurrent neural networks (RNNs) and their variants: RNNs are particularly important when generating image summaries, especially when it comes to sequence generation tasks (such as converting image content into text descriptions). These models are able to process sequence data and remember previous information at each time step, making them very suitable for generating continuous text descriptions.
[0007] However, this type of technology is highly dependent on a large amount of labeled data for training to learn the mapping relationship from image to text. The performance of the model is directly limited by the quality and diversity of the data. In view of the particularity and diversity of power equipment, traditional deep learning models often require a large number of labeled samples to ensure high recognition accuracy, which is difficult to achieve in actual application scenarios, especially in the case of small samples. The generalization ability of the model is limited, which is easy to cause recognition errors.
[0008] Image summary generation technology based on generative adversarial networks focuses on extracting the core content of images through creative synthesis, similar to artistic visual narratives. In the field of news editing, this may be reflected in refining complex news scene photos into summary images that can convey key information and are highly attractive. The core of this technology lies in building a sophisticated adversarial system, including an innovative generator and a discerning discriminator, which co-evolve to push the boundaries of image summary quality. Generator component: Playing the role of a creative artist, the generator uses deep learning architecture, especially convolutional neural networks (CNNs) and other advanced modules to transform random noise or simplified image representations into image summaries that appear to be derived from the real world. It not only captures the essence of the subject of the original image, but also adds artistic interpretation through the synthesis process to ensure that the summary is both faithful to the original and expressive. Discriminator component: Equivalent to a strict connoisseur, the discriminator is trained to distinguish between real image fragments and summaries created by the generator. Through continuous evaluation and feedback, it forces the generator to improve the realism and information relevance of its work, ensuring that the generated summary is not only visually attractive, but also closely corresponds to the original image in terms of content. Adversarial training mechanism: While the generator tries to deceive the discriminator, the discriminator is also constantly learning how to distinguish between real and synthetic images more accurately. This dynamic process drives the quality of the image summaries generated by the generator to the point where it is difficult to distinguish between the real and the fake. Through iterative optimization, the generated summaries not only maintain a high degree of relevance to the source image, but also incorporate creative visual expressions, making them suitable for scenarios that require a high degree of generalization and beautification, such as social media summaries, news highlight generation, and other applications.
[0009] However, the inherent training instability and pattern collapse risks of GANs still exist, which may cause the generated image summaries to be distorted or inaccurate in some cases, affecting the reliability of diagnosis and analysis; also, when dealing with complex structures and textures of power equipment, GANs may have difficulty accurately capturing key details due to the lack of domain-specific optimization, affecting the sensitive identification of abnormal conditions; third, compared with deep learning methods that directly perform feature description, the GANs generation process is more complex and computationally expensive, which is not conducive to the real-time monitoring and rapid response needs of power systems.
[0010] In summary, the deep learning image content recognition and description generation method and the image summary generation technology based on generative adversarial networks (GANs) each have their own shortcomings and deficiencies in the generation of power image summaries. The deep learning image content recognition and description generation method cannot adapt to the situation where there are few power image samples and cannot fully train the prediction model. The image summary generation technology based on generative adversarial networks (GANs) has the problem of low accuracy of extracted power image features and high cost.
[0011] Considering the problems of scarce training samples and large image deviation caused by perspective changes faced by power image summary generation, the above two algorithms have certain limitations when processing power image summary generation. Summary of the invention
[0012] In order to overcome the above defects, the present invention proposes a method and device for generating a summary of an electric power equipment image.
[0013] In a first aspect, a method for generating a summary of an electric power equipment image is provided, the method comprising:
[0014] The power equipment image is sequentially passed through the Darknet model and the capsule network to extract the key structural features corresponding to the power equipment image;
[0015] The key structural features are used as input of a pre-trained Transformer model to obtain summary information corresponding to the power equipment image output by the pre-trained Transformer model.
[0016] Preferably, the mathematical model of the capsule network is as follows:
[0017]
[0018] In the above formula, v j is the squash activation function value corresponding to the output of the j-th digital capsule layer, s j is the output of the j-th digital capsule layer, j∈[1,J], and J is the total number of digital capsule layers.
[0019] Furthermore, the output of the j-th digital capsule layer is as follows:
[0020]
[0021] In the above formula, is the transformation amount of the j-th digital capsule layer corresponding to the i-th 1×8-dimensional two-dimensional feature, c ij is the coupling coefficient matrix between the i-th 1×8-dimensional two-dimensional feature and the j-th digital capsule layer, i∈[1,I], I is the number of 1×8-dimensional two-dimensional features extracted by the Darknet model.
[0022] Furthermore, the transformation amount of the j-th digital capsule layer corresponding to the i-th 1×8-dimensional two-dimensional feature is as follows:
[0023]
[0024] In the above formula, u i Extract the i-th 1×8-dimensional 2D feature of the image for the Darknet model, w abIt is a 8×16 dimensional transformation matrix.
[0025] Preferably, the pooling layer in the capsule network is replaced by a convolution with a step size of 2.
[0026] Preferably, the training process of the pre-trained Transformer model includes:
[0027] Pass the selected power equipment image through the Darknet model and the capsule network in sequence to extract the key structural features corresponding to the selected power equipment image;
[0028] The summary information corresponding to the selected power equipment image is passed through a BERT model to extract text features corresponding to the selected power equipment image;
[0029] Using the key structural features and text features as inputs of a multi-scale relationship generation network to obtain image-text feature matching pairs;
[0030] Passing the image-text feature matching pair through the CLIP model to generate image-text feature matching pair positive samples and image-text feature matching pair negative samples;
[0031] Constructing training data using the image-text feature matching pair positive samples and the image-text feature matching pair negative samples;
[0032] The Transformer model is trained using the training data to obtain the pre-trained Transformer model.
[0033] In a second aspect, a device for generating a summary of an electric power equipment image is provided, the device for generating a summary of an electric power equipment image comprising:
[0034] A feature extraction module is used to sequentially pass the power equipment image through the Darknet model and the capsule network to extract key structural features corresponding to the power equipment image;
[0035] The analysis module is used to use the key structural features as input of a pre-trained Transformer model to obtain summary information corresponding to the power equipment image output by the pre-trained Transformer model.
[0036] In a third aspect, a computer device is provided, comprising: one or more processors;
[0037] The processor is used to store one or more programs;
[0038] When the one or more programs are executed by the one or more processors, the method for generating a summary of an electric power equipment image is implemented.
[0039] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed, the method for generating a summary of an electric power equipment image is implemented.
[0040] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:
[0041] The present invention relates to the field of image analysis technology, and specifically provides a method and device for generating a summary of an electric power equipment image, comprising: passing the electric power equipment image through a Darknet model and a capsule network in sequence to extract key structural features corresponding to the electric power equipment image; using the key structural features as input to a pre-trained Transformer model to obtain summary information corresponding to the electric power equipment image output by the pre-trained Transformer model. The technical solution provided by the present invention, through fine joint optimization, not only deeply reveals the inherent laws of power failures, but also greatly enhances the generation quality and diagnostic efficiency of summaries. In the training stage of the Transformer model, specifically:
[0042] First, the capsule network is used to refine the key structural information of the power image, and the BERT technology is used to deeply analyze the related text and transform it into an information vector containing rich context. Subsequently, a multi-scale relationship generation network and matrix model are adopted to intelligently connect image features and text vectors, and a customized feature matching database for power equipment is constructed. This strategy especially strengthens the generalization performance of the model in the case of summary generation in the context of fewer power images. In the final stage, leveraging the sequence processing advantages of the Transformer model, deep learning is performed on the integrated "image-text" data. Through fine joint optimization, it not only deeply reveals the inherent laws of power failures, but also greatly enhances the generation quality and diagnostic efficiency of summaries. Practice has proved that it can significantly improve the diagnostic accuracy by 30%, while reducing false alarms and optimizing operation and maintenance decisions, providing strong technical support for the intelligent management and safe operation of power systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a schematic flow chart of the main steps of the method for generating a summary of an electric power equipment image according to an embodiment of the present invention;
[0044] Figure 2 Schematic diagram of the Darknet model and capsule network principle of an embodiment of the present invention;
[0045] Figure 3 2 is a schematic diagram of the Clip model principle of an embodiment of the present invention. DETAILED DESCRIPTION
[0046] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.
[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Example 1
[0049] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a method for generating a summary of an electric power equipment image according to an embodiment of the present invention. Figure 1 As shown, the method for generating a summary of an electric power equipment image in an embodiment of the present invention mainly includes the following steps:
[0050] Step S101: The power equipment image is sequentially passed through the Darknet model and the capsule network to extract key structural features corresponding to the power equipment image;
[0051] Step S102: using the key structural features as input of a pre-trained Transformer model to obtain summary information corresponding to the power equipment image output by the pre-trained Transformer model.
[0052] In this embodiment, the capsule network principle is as follows Figure 2 As shown in Figure 1, current deep learning models generally require a sufficient amount of data to achieve good results, and the classification effect on complex small sample data sets is often not ideal. Therefore, the present invention proposes a fusion model (such as Figure 1 As shown in Figure 1, this model first extracts effective features of the image through the Darknet model to solve the overfitting problem that the capsule network retains too many useless details of the image; then the extracted effective features are further processed by the capsule network, and the features are represented as 1×8 two-dimensional vectors, and then nonlinearly processed by the squash activation function, and the modulus of the output vector v is compressed to 0~1. The modulus of v represents the probability of belonging to the corresponding category. The vector itself represents the position, posture and other information of the feature, and the amount of information is richer. The mathematical model of the capsule network is as follows:
[0053]
[0054] In the above formula, v j is the squash activation function value corresponding to the output of the j-th digital capsule layer, s jis the output of the j-th digital capsule layer, j∈[1,J], and J is the total number of digital capsule layers.
[0055] The output of the j-th digital capsule layer is as follows:
[0056]
[0057] In the above formula, is the transformation amount of the j-th digital capsule layer corresponding to the i-th 1×8-dimensional two-dimensional feature, c ij is the coupling coefficient matrix between the i-th 1×8-dimensional two-dimensional feature and the j-th digital capsule layer, i∈[1,I], I is the number of 1×8-dimensional two-dimensional features extracted by the Darknet model.
[0058] The transformation amount of the j-th digital capsule layer corresponding to the i-th 1×8-dimensional two-dimensional feature is as follows:
[0059]
[0060] In the above formula, u i Extract the i-th 1×8-dimensional 2D feature of the image for the Darknet model, w ab It is a 8×16 dimensional transformation matrix.
[0061] This model uses capsule networks to vectorize features to obtain multi-view spatial information of effective features, allowing the model to use less data to achieve higher classification accuracy. In addition, the model as a whole uses Darknet convolution with a step size of 2 instead of pooling to avoid the information loss problem caused by pooling.
[0062] In this embodiment, the training process of the pre-trained Transformer model includes:
[0063] Pass the selected power equipment image through the Darknet model and the capsule network in sequence to extract the key structural features corresponding to the selected power equipment image;
[0064] The summary information corresponding to the selected power equipment image is passed through a BERT model to extract text features corresponding to the selected power equipment image;
[0065] Using the key structural features and text features as inputs of a multi-scale relationship generation network to obtain image-text feature matching pairs;
[0066] Passing the image-text feature matching pair through the CLIP model to generate image-text feature matching pair positive samples and image-text feature matching pair negative samples;
[0067] Constructing training data using the image-text feature matching pair positive samples and the image-text feature matching pair negative samples;
[0068] The Transformer model is trained using the training data to obtain the pre-trained Transformer model.
[0069] The core of the multi-scale relationship generation network framework is to cleverly utilize the multi-scale feature vectors of power images and texts to capture and match the rich information between the two through differentiated processing methods.
[0070] In the method of the present invention, multi-scale feature maps play a key role: larger-scale feature maps are rich in macroscopic visual structures and contextual information, suitable for deep networks to capture the extensive associations between them; on the contrary, smaller-scale feature maps focus on details and uniqueness, suitable for being refined by shallower networks to distinguish subtle differences. Based on this cognition, the multi-scale relation generation network (MSRGN) constructed by the present invention can systematically learn and bridge these multi-dimensional feature differences.
[0071] The multi-scale relation generation network consists of a series of convolutional blocks with cascade connections. To achieve the purpose of hierarchical relation learning, larger scales of multi-scale feature map pairs are used for relation learning using higher-level networks, which contain more convolutional blocks of the multi-scale relation generation network. Since the multi-scale relation generation network is mainly composed of multiple convolutional blocks in series, and the number of convolutional blocks decreases as the network level decreases, it is reasonable that the lower-level network with fewer convolutional blocks can share parameters with some convolutional blocks of the high-level network. Afterwards, these scales are combined at the top layer of the multi-scale relation generation network.
[0072] The multi-scale relationship generation network is able to learn the relationship of larger-scale feature map pairs through higher-level metric learning. The lower level has a shallower metric learning network and the higher level has a deeper learning network. The parameters are shared to reduce the complexity of the model. The multi-scale relationship generation network is based on a feed-forward convolutional network, which receives image features extracted by the capsule network and text features extracted by Bert. Different weights are assigned to feature map pairs of different scales. A four-layer convolutional structure is used to process the largest-scale feature map pairs, and the number of layers decreases as its scale decreases. Each convolutional layer consists of a 3X3 filter, followed by batch normalization and a maximum pooling layer. Finally, the relationship score between the samples in the query set and the samples in the support set is obtained through two fully connected layers. The power image feature vector is finely matched with the text feature vector to form a power image-text feature matching pair. Construct an image-text matching set.
[0073] CLIP is a multimodal learning model that aims to improve the capabilities of computer vision and natural language processing tasks through pre-training. It was proposed by OpenAI in 2021 and quickly became an important cornerstone in the multimodal field. The main innovation of the CLIP model is that it uses a contrastive learning mechanism to learn the deep association between images and text through massive data. The network structure of the CLIP model mainly includes two modules, Text Encoder and Image Encoder, which extract text and image features respectively, and then let the model learn the text-image matching relationship based on contrastive learning. During the training process, the CLIP model used 400 million sets of image-text pairs, covering most scenes in nature, enabling it to learn more general visual semantic information and provide assistance for downstream tasks. During the training process, the CLIP model matches sentences and pictures, and then finds the closest template sentence during the reasoning process, realizing the zero-shot learning method and breaking the fixed category label paradigm. The CLIP model has a wide range of applications in image classification, image retrieval, content regulation, etc., and can be adapted to different tasks and data sets through fine-tuning. Such as Figure 3 The figure shows a matrix contrastive learning model. The training data of the Clip model is a text-image pair: an image and its corresponding text description. Through contrastive learning, the model can learn the matching relationship between image-text pairs. Here, the extracted text features and image features are contrastively learned. N and I N are the Nth text feature and the Nth image feature, respectively. For a training batch containing N text-image pairs, the N text features and N image features are combined in pairs, and the Clip model will predict the similarities of N2 possible text-image pairs. The similarity here directly calculates the cosine similarity of the text features and the image features, that is, Figure 3 There are N positive samples, that is, the text and image that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 -N text-image pairs are negative samples, so the training goal of the Clip model is to maximize the similarity of N positive samples and minimize N 2 -Similarity of N negative samples.
[0074] In this embodiment, the BERT model is a language representation model, and BERT stands for Bidirectional Encoder Representations from Transformers. BERT is designed to pre-train deep bidirectional representations by jointly adjusting the left and right contexts in all layers. Therefore, only an additional output layer is needed to fine-tune the pre-trained BERT representation to create state-of-the-art models for a wide range of tasks (such as question answering and language inference tasks) without requiring a large number of task-specific model structure modifications.
[0075] The Transformer model is a deep learning model that is mainly used to process sequence data, especially in the field of natural language processing (NLP). It solves the gradient vanishing and gradient exploding problems encountered by traditional recurrent neural networks (RNNs) when processing long sequences by introducing a self-attention mechanism. The core of the Transformer model consists of an encoder and a decoder, each of which contains multiple self-attention sublayers and feedforward neural network layers. The encoder is responsible for encoding the input sequence into a series of vectors, while the decoder decodes these vectors into an output sequence. The training process of the Transformer model can be carried out in parallel, which improves the training efficiency. In addition, the Transformer model has achieved remarkable success in tasks such as machine translation, text generation, and speech recognition, and has derived many variants, such as pre-trained models such as BERT and GPT.
[0076] Example 2
[0077] Based on the same inventive concept, the present invention also provides a device for generating a summary of an image of an electric power equipment, the device comprising:
[0078] A feature extraction module is used to sequentially pass the power equipment image through the Darknet model and the capsule network to extract key structural features corresponding to the power equipment image;
[0079] The analysis module is used to use the key structural features as input of a pre-trained Transformer model to obtain summary information corresponding to the power equipment image output by the pre-trained Transformer model.
[0080] In this embodiment, the mathematical model of the capsule network is as follows:
[0081]
[0082] In the above formula, v jis the squash activation function value corresponding to the output of the j-th digital capsule layer, s j is the output of the j-th digital capsule layer, j∈[1,J], and J is the total number of digital capsule layers.
[0083] In one embodiment, the output of the j-th digital capsule layer is as follows:
[0084]
[0085] In the above formula, is the transformation amount of the j-th digital capsule layer corresponding to the i-th 1×8-dimensional two-dimensional feature, c ij is the coupling coefficient matrix between the i-th 1×8-dimensional two-dimensional feature and the j-th digital capsule layer, i∈[1,I], I is the number of 1×8-dimensional two-dimensional features extracted by the Darknet model.
[0086] In one embodiment, the j-th digital capsule layer transformation amount corresponding to the i-th 1×8-dimensional two-dimensional feature is as follows:
[0087]
[0088] In the above formula, u i Extract the i-th 1×8-dimensional 2D feature of the image for the Darknet model, w ab It is a 8×16 dimensional transformation matrix.
[0089] In this embodiment, the pooling layer in the capsule network is replaced by a convolution with a step size of 2.
[0090] In this embodiment, the training process of the pre-trained Transformer model includes:
[0091] Pass the selected power equipment image through the Darknet model and the capsule network in sequence to extract the key structural features corresponding to the selected power equipment image;
[0092] The summary information corresponding to the selected power equipment image is passed through a BERT model to extract text features corresponding to the selected power equipment image;
[0093] Using the key structural features and text features as inputs of a multi-scale relationship generation network to obtain image-text feature matching pairs;
[0094] Passing the image-text feature matching pair through the CLIP model to generate image-text feature matching pair positive samples and image-text feature matching pair negative samples;
[0095] Constructing training data using the image-text feature matching pair positive samples and the image-text feature matching pair negative samples;
[0096] The Transformer model is trained using the training data to obtain the pre-trained Transformer model.
[0097] Example 3
[0098] Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a method for generating a summary of an electric power equipment image in the above embodiment.
[0099] Example 4
[0100] Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of a method for generating a power equipment image summary in the above embodiment.
[0101] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for generating a summary of an electric power equipment image, characterized in that: The method comprises: The power equipment image is sequentially passed through the Darknet model and the capsule network to extract the key structural features corresponding to the power equipment image; The key structural features are used as input of a pre-trained Transformer model to obtain summary information corresponding to the power equipment image output by the pre-trained Transformer model.
2. The method according to claim 1, characterized in that The mathematical model of the capsule network is as follows: In the above formula, v j is the squash activation function value corresponding to the output of the j-th digital capsule layer, s j is the output of the j-th digital capsule layer, j∈[1,J], and J is the total number of digital capsule layers.
3. The method according to claim 2, characterized in that The output of the j-th digital capsule layer is as follows: In the above formula, is the transformation amount of the j-th digital capsule layer corresponding to the i-th 1×8-dimensional two-dimensional feature, c ij is the coupling coefficient matrix between the i-th 1×8-dimensional two-dimensional feature and the j-th digital capsule layer, i∈[1,I], I is the number of 1×8-dimensional two-dimensional features extracted by the Darknet model.
4. The method according to claim 3, characterized in that The transformation amount of the j-th digital capsule layer corresponding to the i-th 1×8-dimensional two-dimensional feature is as follows: In the above formula, u i Extract the i-th 1×8-dimensional 2D feature of the image for the Darknet model, w ab It is a 8×16 dimensional transformation matrix.
5. The method according to claim 1, characterized in that The pooling layer in the capsule network is replaced by a convolution with a stride of 2.
6. The method according to claim 1, characterized in that The training process of the pre-trained Transformer model includes: Pass the selected power equipment image through the Darknet model and the capsule network in sequence to extract the key structural features corresponding to the selected power equipment image; The summary information corresponding to the selected power equipment image is passed through a BERT model to extract text features corresponding to the selected power equipment image; Using the key structural features and text features as inputs of a multi-scale relationship generation network to obtain image-text feature matching pairs; Passing the image-text feature matching pair through the CLIP model to generate image-text feature matching pair positive samples and image-text feature matching pair negative samples; Constructing training data using the image-text feature matching pair positive samples and the image-text feature matching pair negative samples; The Transformer model is trained using the training data to obtain the pre-trained Transformer model.
7. A device for generating a power equipment image summary based on the method according to any one of claims 1 to 6, characterized in that: The device comprises: A feature extraction module is used to sequentially pass the power equipment image through the Darknet model and the capsule network to extract key structural features corresponding to the power equipment image; The analysis module is used to use the key structural features as input of a pre-trained Transformer model to obtain summary information corresponding to the power equipment image output by the pre-trained Transformer model.
8. A computer device, characterized in that: include: one or more processors; The processor is configured to execute one or more programs; When the one or more programs are executed by the one or more processors, the method for generating a summary of an electric power equipment image according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the method for generating a summary of an electric power equipment image as described in any one of claims 1 to 6 is implemented.
