Transform-based electric power image-text cross-modal retrieval method and related equipment

Abstract

The invention provides an electric power image-text cross-modal retrieval method based on Transform and a related device. The method comprises the steps of obtaining a target text; performing text feature extraction based on the target text to obtain target text features; retrieving in a to-be-processed image data set based on the target text features, and determining a target image conforming to the target text semantics, including: performing image feature extraction on the to-be-processed image in the to-be-processed image data set based on a trained retrieval model to obtain image features; performing matching based on the target text features and the image features to obtain target image features; and determining the target image based on the target image feature.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a Transformer-based cross-modal retrieval method for electric power graphics and text and related equipment. Background Art

[0002] With the rapid development of technologies such as big data, artificial intelligence, and the Internet of Things, the breadth and depth of the analysis and utilization of power sample data are constantly extending. Due to the large amount of power sample data, the non-uniform data format and dimension, and the semantic gap between information expressions of different modalities, the multimodal retrieval capability of power sample data is insufficient. Summary of the invention

[0003] The present disclosure proposes a Transformer-based cross-modal retrieval method for electric power graphics and text and related equipment to solve the technical problem of insufficient multi-modal retrieval capability of electric power sample data to a certain extent.

[0004] In a first aspect, the present disclosure provides a Transformer-based cross-modal retrieval method for electric power graphics and text, comprising:

[0005] Get the target text;

[0006] Performing text feature extraction based on the target text to obtain target text features;

[0007] Searching in the image data set to be processed based on the target text feature to determine the target image that meets the semantics of the target text includes:

[0008] Extracting image features of the image to be processed in the image data set to be processed based on the trained retrieval model to obtain image features;

[0009] Matching the target text feature with the image feature to obtain a target image feature;

[0010] The target image is determined based on the target image feature.

[0011] In a second aspect, the present disclosure provides a Transformer-based power graphic and text cross-modal retrieval device, comprising:

[0012] An acquisition module is used to acquire the target text;

[0013] A feature extraction module, used to extract text features based on the target text to obtain target text features;

[0014] A retrieval module is used to search in the image data set to be processed based on the target text features to determine the target image that meets the semantics of the target text, including:

[0015] Extracting image features of the image to be processed in the image data set to be processed based on the trained retrieval module to obtain image features;

[0016] Matching the target text feature with the image feature to obtain a target image feature;

[0017] The target image is determined based on the target image feature.

[0018] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0020] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0021] From the above, it can be seen that the Transformer-based cross-modal retrieval method and related equipment provided by the present disclosure, by deeply integrating the powerful capabilities of the Transformer model, applies the Transformer's self-attention mechanism to the feature extraction and matching process of images and texts, not only realizing the accurate capture of features, but also strengthening the association learning between modalities. Furthermore, through in-depth semantic reasoning through the Transformer layer, the model can capture more complex structural relationships and semantic information, thereby significantly improving the accuracy of cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 Schematic diagram of a Transformer-based cross-modal retrieval architecture for electric power graphics and text according to an embodiment of the present disclosure.

[0024] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0025] Figure 3 The present invention is a flowchart of a Transformer-based cross-modal retrieval method for electric power graphics and texts according to an embodiment of the present invention.

[0026] Figure 4 It is a schematic diagram of a Transformer-based cross-modal retrieval method for electric power graphics and texts according to an embodiment of the present disclosure.

[0027] Figure 5 These are some category images and text examples of the substation equipment defect dataset in the embodiment of the present disclosure.

[0028] Figure 6 It is a schematic diagram of a Transformer-based cross-modal retrieval device for electric power graphics and texts according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0032] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0033] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0034] Figure 1 FIG. 1 is a schematic diagram of a Transformer-based cross-modal retrieval architecture for power graphics and text in an embodiment of the present disclosure. Figure 1 , the Transformer-based power graphic cross-modal retrieval architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0035] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0036] It should be noted that the Transformer-based cross-modal retrieval method for electric power graphics and text provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0037] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0038] Processor 202 may be a central processing unit (CPU), a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0039] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (for example, program instructions for implementing the Transformer-based cross-modal retrieval method for power graphics and text in the embodiment of the present disclosure) and data to be processed (for example, the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0040] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 206 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0041] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0042] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0043] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0044] In the field of cross-modal retrieval, many innovative results have been achieved in recent years. The current mainstream cross-modal retrieval algorithms all construct a common semantic space to map the data of the two modalities, image and text, to the same semantic or dimensional space in order to evaluate their similarity. According to the numerical type of feature representation, cross-modal retrieval algorithms can be divided into hash-based algorithms and real-number-based algorithms. The hash-based cross-modal retrieval algorithm maps the feature information of the two modalities to a common space, uses binary coding to represent the feature vector, and uses the Hamming distance to measure the similarity. This method has the advantages of low storage overhead and fast calculation speed, but the real value is mapped to binary coding, which will lose a lot of information and lead to poor retrieval accuracy. Therefore, the real-number-based cross-modal retrieval algorithm is the main direction of current academic research.

[0045] Cross-modal retrieval algorithms based on real numbers are mainly divided into three categories: algorithms based on modal overall features, algorithms based on modal features, and algorithms based on Transformer pre-trained models. The Deep Visual-Semantic Embedding method proposed by From et al. is a classic algorithm based on modal overall features. This method uses a convolutional neural network pre-trained on ImageNet to extract image features, uses the Skip-Gram language model to extract text features, and uses cosine similarity to measure the similarity between the two modalities. Finally, the triple loss function is used as the objective function for feature output. This type of method directly extracts overall features and maps them to a common semantic space, ignoring detailed information, resulting in poor retrieval accuracy. Karpathy et al. proposed a method for pairing local features of images and texts, which is a classic feature-based algorithm. This method determines whether the image and text match by finding the similarity relationship between the image region and the text word pair. However, due to the difference between images and texts, there is no intuitive semantic association information between the regions obtained by extracting salient regions, so the semantic order of the image is often ignored. This type of method focuses on detecting and aligning information within the modalities and constructing a semantic knowledge graph, but how to accurately construct the semantic structure graph within the two modalities is a problem that the feature algorithm needs to solve.

[0046] Pre-trained models based on the Transformer structure have achieved significant success in the fields of natural language processing and computer vision. Transformer is a typical pre-trained model that uses the classic "encoder-decoder" structure. It was originally used in the field of natural language processing and receives text sequences as input for machine translation tasks. Subsequently, scholars began to use object detection algorithms such as Faster RCNN to extract salient regions in images and use the extracted salient region sets as inputs to the Transformer network. Therefore, the Transformer structure has become a universal processing tool and a bridge across the gap between text and visual semantics. It has shown good results in various multimodal tasks, such as UNITER

[54] (UNiversal Image-TExt Representation), Vi LBERT

[55] Vision and language BERT), Unicoder-VL

[56] , etc., all of which attempt to design a unified Transformer structure that receives both image regions and text words as inputs, and then performs pre-training on several tasks to obtain the final pre-trained model. However, this method based on pre-trained models requires large datasets, a large amount of computing resources, and a large amount of manual data annotation.

[0047] Therefore, how to connect the features of different modal data, automatically classify power multimodal sample data, establish a sample retrieval engine with multi-level business knowledge granularity, realize the refined matching of high-quality sample multimodal data in power business scenarios and personalized and customized semantic search of sample libraries is the key link in the emergence of power data value and has become a technical problem that needs to be solved urgently.

[0048] In view of this, the embodiment of the present disclosure provides a method and related equipment for cross-modal retrieval of electric power images and texts based on Transformer. A fine-grained image-text retrieval network based on Transformer (FITRNT) aims to solve the heterogeneous gap between the two modalities of image and text and the problem of local similarity and global similarity association between images and texts. The structure has the following improvements: 1) The attention mechanism is introduced to match and aggregate the original visual and text features to generate a unified fusion tensor, thereby strengthening the association learning between modalities; 2) The Transformer layer is used for semantic reasoning to strengthen the feature expression of the inter-modal similarity vector to capture the local structural relationship between features for complex structure matching, thereby improving the accuracy of the inter-modal similarity measurement; 3) Multiple similarity calculations are used for comparative learning, so that the model fully learns the global alignment relationship between the two modalities, and a triple loss function is added to constrain the image-text matching model, and finally realizes the retrieval of cross-modal image text.

[0049] See also Figure 3 , Figure 3 A schematic flow chart of a power image and text cross-modal retrieval method based on Transformer according to an embodiment of the present disclosure is shown. The power image and text cross-modal retrieval method based on Transformer according to an embodiment of the present disclosure can be deployed on a server or terminal. Figure 3 In the embodiment, the Transformer-based power graphic and text cross-modal retrieval method 300 may further include the following steps.

[0050] In step S310, the target text is obtained.

[0051] The target text may refer to a descriptive text obtained from the user, which may be used to define a search condition or criterion. For example, the target text may be a text input by the user: "A black cat is napping in the sun.".

[0052] In step S320, text feature extraction is performed based on the target text to obtain target text features.

[0053] Among them, the target text features can refer to useful features extracted from the target text. These features are usually some keywords or phrases that can represent the main information of the text. In the deep learning scenario, the text can be converted into a one-dimensional or multi-dimensional vector form for processing. For example, for the above target text, "black", "cat", "sunshine", and "nap" may be the extracted keywords. If word embedding technology is used, this text will be converted into a fixed-length vector, which is called the target text feature.

[0054] In step S330, searching in the image data set to be processed based on the target text feature to determine a target image that meets the semantics of the target text includes:

[0055] Extracting image features of the image to be processed in the image data set to be processed based on the trained retrieval model to obtain image features;

[0056] Matching the target text feature with the image feature to obtain a target image feature;

[0057] The target image is determined based on the target image feature.

[0058] Among them, the technology of searching in the image data set to be processed based on the target text features can achieve accurate and efficient cross-modal information matching, significantly improving the relevance and speed of the search. By quickly extracting image features through a trained retrieval model and matching them with the target text features, this method not only reduces the false detection rate, but also greatly optimizes resource utilization. In addition, this technology enhances the correlation between language and images, provides a more user-friendly interaction method, enables users to directly retrieve images through natural language descriptions, and enriches the display form of search results. Its powerful cross-modal retrieval capabilities and wide applicability are not only suitable for multiple fields such as e-commerce and social media, but also support application expansion in multilingual environments, showing a high degree of flexibility and adaptability, thereby significantly improving user experience and service quality.

[0059] In some embodiments, matching the target text feature with the image feature to obtain the target image feature includes:

[0060] Based on the attention mechanism, the target text feature and the image feature are matched locally to obtain local fusion features and global fusion features;

[0061] Performing multiple similarity learning based on the local fusion features and the global fusion features to obtain a similarity result;

[0062] The target image feature among the image features is determined based on the similarity result.

[0063] Among them, the Transformer architecture is used as an inference model to process image and text features, aiming to mine the significant semantic information in images and texts and infer the potential hidden relationship between the two. The Transformer model mainly includes two parts: encoder and decoder. It captures the long-distance dependency between features through a multi-layer multi-head self-attention mechanism and a feed-forward neural network. Specifically, image encoding can be to extract the image feature map from the input image through a convolutional neural network (CNN), and then input it into the Transformer encoder. Text encoding can be to convert the input text into a word vector through an embedding layer, and then input it into the Transformer encoder. The multi-head self-attention mechanism can be processed in parallel through multiple attention heads, so that the model can focus on information at different positions and capture features at different levels. Feature fusion can be to fuse image features and text features through a decoder after the encoder output, so as to infer the potential relationship between the image and text.

[0064] For example, there is a picture of a cat sitting on a sofa, and the text description is "a cat sitting on a blue sofa". The Transformer model first extracts image features through CNN, and then converts the text into word vectors through the embedding layer. Next, the multi-head self-attention mechanism establishes a connection between the image feature map and the text feature vector. For example, the model recognizes the relationship between the word "cat" and the area where the cat is located in the image, and infers the association between "blue" and the color of the sofa.

[0065] In some embodiments, determining the text local features of the target text features and the image local features in the image features based on the attention mechanism includes:

[0066] Determine a local text feature in the target text feature based on an attention mechanism and an attention region of the image feature;

[0067] The local image features in the image features are determined based on the attention mechanism and the attention text of the target text features.

[0068] Among them, through the fine-grained image-text matching formula with attention mechanism, attention scores are reasonably assigned to the image region of each word to achieve more comprehensive semantic alignment. Attention mechanism can be divided into multiple levels. For example, word-level attention can assign a weight to each word to indicate the importance of the word in the image. Region-level attention can divide the image into multiple regions and assign a weight to each region to indicate the importance of the region in the matching. Further, fine-grained matching between images and texts can be achieved by combining word-level attention and region-level attention. For example, in the above example, the attention mechanism will assign higher attention scores to keywords such as "cat", "sit", and "sofa", and match them with the corresponding regions in the image. For example, the word "cat" may correspond to the area where the cat is located in the image, while "sofa" corresponds to the location of the sofa. This fine-grained matching helps the model understand the correspondence between images and texts more accurately.

[0069] In some embodiments, the target text feature and the image feature are matched with local features based on the attention mechanism to obtain local fusion features and global fusion features, including:

[0070] Determine the text local features of the target text features and the image local features of the image features based on an attention mechanism;

[0071] Obtaining the local fusion feature by fusing the local text feature and the local image feature;

[0072] The global fusion feature is obtained by weighted summing of the local fusion features.

[0073] Among them, the image-text feature aggregation is performed using the universal summation method to establish local and global cross-modal relationships. Local feature aggregation can be a weighted sum of the features of each image region and the corresponding text word features to obtain a local feature vector. Global feature aggregation can be a weighted sum of all local feature vectors to obtain a global feature vector. Cross-modal relationship modeling can be to establish a cross-modal relationship between image and text through a global feature vector. For example, in the feature aggregation stage, the model will integrate image and text features to establish local and global cross-modal relationships. For example, the model may combine the features of the cat in the image with the part of the text that describes the cat, and combine the features of the sofa with the part that describes the sofa. At the same time, the model will also consider the relationship between the overall image and text to ensure global consistency. For example, the model will match the overall description of "the cat is sitting on the sofa" with the overall scene in the image.

[0074] In some embodiments, multiple similarity learning is performed based on the local fusion feature and the global fusion feature to obtain a similarity result, including:

[0075] Calculating similarity based on the local fusion feature and the target text to obtain a first similarity;

[0076] Calculate similarity based on the global fusion feature and the target text to obtain a second similarity;

[0077] A weighted sum is performed based on the first similarity and the second similarity to obtain the similarity result.

[0078] Among them, multiple similarity calculations are used for comparison, so that the model can fully learn the global alignment relationship between the two modalities. In the similarity calculation, the feature representation can be to represent the feature vectors of the image and text as points in a high-dimensional space. The similarity measurement can be to use multiple similarity measurement methods (such as cosine similarity, Euclidean distance, etc.) to calculate the similarity score between the image and text features. Contrastive learning can be to use a contrastive learning framework so that the similarity score between the positive sample pairs (correct image-text pairs) is higher than that between the negative sample pairs (wrong image-text pairs). For example, in the similarity calculation stage, the model calculates the similarity score between the image and text features. For example, the model can use cosine similarity to measure the similarity between the image feature vector and the text feature vector. Through multiple calculations and comparisons, the model can gradually optimize the alignment relationship to ensure that the image and text are consistent globally. For example, the model will make the text describing "a cat sitting on a blue sofa" match the corresponding image more closely through repeated adjustments.

[0079] In some embodiments, determining the target image feature among the image features based on the similarity result includes:

[0080] The image feature corresponding to the maximum value in the similarity results is determined as the target image feature.

[0081] Among them, a triplet loss function can also be added to constrain the image-text matching model and optimize the model performance. The triplet loss function forces the model to learn the correct matching relationship by constructing positive sample pairs and negative sample pairs. Among them, the positive sample pair can be a correct image-text pair. The negative sample pair can be an incorrect image-text pair. The loss function can be a triplet loss function so that the distance between the positive sample pairs is less than the distance between the negative sample pairs plus a preset margin. For example, during the training process, the use of the triplet loss function can ensure that the distance between the image and its corresponding positive sample (correct text description) is less than the distance between the image and the negative sample (incorrect text description). For example, for a picture containing a cat, the correct text description should be "a cat sitting on a blue sofa", while the incorrect description may be "a dog lying on a red carpet." By minimizing the distance of the positive sample and maximizing the distance of the negative sample, the model can better learn the correct matching relationship between the image and the text.

[0082] It can be seen that the Transformer reasoning model, attention mechanism, feature aggregation, similarity calculation, and triple loss can effectively mine the significant semantic information between images and texts and establish the potential hidden relationship between the two. This method not only improves the reasoning ability of the model, but also improves the accuracy of image and text matching.

[0083] See also Figure 4 , Figure 4 A schematic diagram of a Transformer-based cross-modal retrieval method for electric power graphics and text according to an embodiment of the present disclosure is shown. Figure 4In the paper, the fine-grained cross-modal image-text retrieval model based on Transformer mainly consists of three parts: image feature extraction module, text feature extraction module and modal interaction module. The image feature extraction module uses the improved Swin Transformer to extract image features, which can capture the key information in the image. The text feature extraction module uses BERT to extract text features. BERT is a pre-trained language model that can effectively extract semantic features in text. The Transformer reasoning model in the modal interaction module uses Transformer as a reasoning model to reason about image and text features, mine the salient semantic information of images and texts, and infer the potential hidden relationship between the two modalities. Through the fine-grained image-text matching formula with attention, the image area of ​​each word is reasonably assigned an attention score to achieve more comprehensive semantic alignment. The universal summation method is used to aggregate image-text features and establish local and global cross-modal relationships. Through multiple similarity calculations for comparison, the model can fully learn the global alignment relationship between the two modalities. The triplet loss function is added to constrain the image-text matching model and optimize the model performance.

[0084] After completing the above-mentioned feature extraction and interaction process, the system can realize fine-grained cross-modal image text retrieval, that is, users can query related images by inputting text, and vice versa. Implementation effect The present invention can realize fine-grained alignment on data sets that do not require manual annotation, thereby improving the accuracy and efficiency of cross-modal retrieval. Through the attention mechanism, the model can focus on key areas in the image, making the semantic alignment more comprehensive and more interpretable. The added triple loss function helps to improve the robustness and generalization ability of the model, thereby improving the effect of cross-modal retrieval.

[0085] Specifically, 14 types of common defects of substation equipment, such as transformers, reactors, circuit breakers, disconnectors, and lightning arresters, can be taken as research objects, and the defects of substation equipment can be divided into three categories: defect identification, state identification, and safety risks. The defect identification category includes 20 subcategories, such as meter damage, insulator damage, and oil leakage. The state identification category includes 8 subcategories, such as abnormal meter readings, oil level status, and silicone discoloration. The safety risk category includes 11 subcategories, such as crossing the line, intrusion, not wearing a safety helmet, and not wearing work clothes. 27,367 image samples containing various types of defects were obtained, and the image samples were all in .jpg format. According to the needs of the construction of the image-text cross-modal retrieval dataset, business experts are used to write corresponding text descriptions for each type of defect image. Figure 5This paper provides some examples of images and texts of some categories of self-built substation defect dataset. Based on the Wikipedia dataset structure commonly used in cross-modal retrieval, a list of substation defect image-text pairs is constructed, and the self-built substation defect dataset is divided into training set and test set in a ratio of 8:2. Experimental results show that the average precision of the proposed method on the self-built substation equipment inspection image and text cross-modal dataset reaches 0.458. Compared with the existing methods, the average precision is improved by 0.011 to 0.195, which is better than all the comparison methods and can provide technical support and algorithm reference for cross-modal retrieval of electric power.

[0086] It can be seen that according to the method of the embodiment of the present disclosure, by deeply integrating the powerful capabilities of the Transformer model, the self-attention mechanism of the Transformer is applied to the feature extraction and matching process of the image and text, which not only realizes the accurate capture of the features, but also strengthens the association learning between the modalities. Furthermore, through the in-depth semantic reasoning of the Transformer layer, the model can capture more complex structural relationships and semantic information, thereby significantly improving the accuracy of cross-modal retrieval. In addition, multiple similarity calculation methods and contrastive learning mechanisms are introduced to more comprehensively evaluate the similarity between images and texts, and to enhance the global alignment capabilities of the model, aiming to solve the complex image and text data retrieval needs in fields such as electricity. In view of the problem that there are many types of power data modalities and insufficient multi-modal retrieval capabilities, the embodiment of the present disclosure proposes a power image and text cross-modal retrieval method based on Transformer, introduces an attention mechanism to match and aggregate the original visual and text features, and uses Transformer to infer feature similarity information to achieve feature matching of different data types of power samples.

[0087] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0088] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a Transformer-based power graphic cross-modal retrieval device, see Figure 6 , the Transformer-based power graphic and text cross-modal retrieval device, the device comprising:

[0090] An acquisition module is used to acquire the target text;

[0091] A feature extraction module, used to extract text features based on the target text to obtain target text features;

[0092] A retrieval module is used to search in the image data set to be processed based on the target text features to determine the target image that meets the semantics of the target text, including:

[0093] Extracting image features of the image to be processed in the image data set to be processed based on the trained retrieval module to obtain image features;

[0094] Matching the target text feature with the image feature to obtain a target image feature;

[0095] The target image is determined based on the target image feature.

[0096] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0097] The device of the above embodiment is used to implement the corresponding Transformer-based cross-modal retrieval method for electric power graphics and text in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0098] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the Transformer-based cross-modal retrieval method for power graphics and text as described in any of the above embodiments.

[0099] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0100] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the Transformer-based cross-modal retrieval method for power graphics and text as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0101] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0102] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0103] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0104] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A Transformer-based cross-modal retrieval method for electric power images and texts, comprising: Get the target text; Performing text feature extraction based on the target text to obtain target text features; Searching in the image data set to be processed based on the target text feature to determine the target image that meets the semantics of the target text includes: Extracting image features of the image to be processed in the image data set to be processed based on the trained retrieval model to obtain image features; Matching the target text feature with the image feature to obtain a target image feature; The target image is determined based on the target image feature.

2. The method according to claim 1, wherein: The target image feature is obtained by matching the target text feature with the image feature, including: Based on the attention mechanism, the target text feature and the image feature are matched locally to obtain local fusion features and global fusion features; Performing multiple similarity learning based on the local fusion features and the global fusion features to obtain a similarity result; The target image feature among the image features is determined based on the similarity result.

3. The method according to claim 2, wherein: Based on the attention mechanism, the target text features and the image features are matched with local features to obtain local fusion features and global fusion features, including: Determine the text local features of the target text features and the image local features of the image features based on an attention mechanism; Obtaining the local fusion feature by fusing the local text feature and the local image feature; The global fusion feature is obtained by weighted summing of the local fusion features.

4. The method according to claim 3, wherein: Determining the text local features of the target text features and the image local features in the image features based on the attention mechanism includes: Determine a local text feature in the target text feature based on an attention mechanism and an attention region of the image feature; The local image features in the image features are determined based on the attention mechanism and the attention text of the target text features.

5. The method according to claim 3, wherein: Multiple similarity learning is performed based on the local fusion feature and the global fusion feature to obtain a similarity result, including: Calculating similarity based on the local fusion feature and the target text to obtain a first similarity; Calculate similarity based on the global fusion feature and the target text to obtain a second similarity; A weighted sum is performed based on the first similarity and the second similarity to obtain the similarity result.

6. The method according to claim 1, wherein: Determining the target image feature among the image features based on the similarity result includes: The image feature corresponding to the maximum value in the similarity results is determined as the target image feature.

7. The method according to claim 5, further comprising: Calculate the triplet loss function based on the similarity results of the training positive samples and the training negative samples; The triplet loss function of the training positive samples is minimized, and the triplet loss function of the training negative samples is maximized to obtain the trained retrieval model.

8. A Transformer-based cross-modal retrieval device for electric power graphics and text, comprising: An acquisition module is used to acquire the target text; A feature extraction module, used to extract text features based on the target text to obtain target text features; A retrieval module is used to search in the image data set to be processed based on the target text features to determine the target image that meets the semantics of the target text, including: Extracting image features of the image to be processed in the image data set to be processed based on the trained retrieval module to obtain image features; Matching the target text feature with the image feature to obtain a target image feature; The target image is determined based on the target image feature.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images