A multimodal power data retrieval method, device, and medium

Abstract

The application relates to a multi-modal power data retrieval method, device and medium, belonging to the technical field of data retrieval, and comprising the following steps: acquiring power data and preprocessing, inputting text and images into a multi-layer perception machine to obtain initial feature vectors, and then outputting text and image feature vectors through a routing capsule algorithm; adopting linear transformation to map the text and image feature vectors to a common space to obtain projection vectors; inputting the projection vectors into an additive attention model to output fusion feature vectors. A hash function is constructed, the power data index is calculated based on the fusion feature vectors, and data with the same index is stored in corresponding hash table barrels. A user query is acquired, query fusion feature vectors and indexes are calculated, the similarity of the query fusion feature vectors with the fusion feature vectors of each power data in the corresponding barrels is calculated, and the R power data with the highest similarity score are returned as the retrieval result. The application comprehensively utilizes multi-modal information, adjusts the importance through an attention mechanism, and effectively improves the retrieval effect.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-modal power data retrieval method, device and medium, belonging to the technical field of data retrieval. BACKGROUND

[0002] With the rapid development of the power industry and the in-depth promotion of informationization construction, the types and scales of power system data are showing an explosive growth, and how to efficiently retrieve and utilize these data has become a problem to be solved. Power data covers multiple modal information such as text (such as equipment operation parameters, fault reports, maintenance records, etc.) and images (such as equipment photos, fault scene images, power line patrol images, etc.). These data have extremely important value for fault diagnosis, maintenance decision, operation optimization, etc. of power equipment.

[0003] However, the traditional power data retrieval method is often limited to single modal data processing, and cannot effectively integrate multi-modal information such as text and images, resulting in limited accuracy and comprehensiveness of the retrieval results. For example, early text retrieval methods are mainly based on keyword matching, while image retrieval methods rely on hand-designed features, which makes it difficult to capture deep associations and semantic information between data when facing complex power data environments. In addition, with the continuous expansion of the power system scale and the improvement of intelligence and automation level, the way of simply relying on manual screening and sorting of data has been unable to meet the actual demand, and an intelligent and efficient retrieval method is urgently needed to integrate multi-modal power data. SUMMARY

[0004] In order to solve the problems existing in the prior art, the present application proposes a multi-modal power data retrieval method, device and medium.

[0005] The technical scheme of the present application is as follows:

[0006] On the one hand, the present application proposes a multi-modal power data retrieval method, comprising the following steps:

[0007] Obtaining power data and preprocessing; wherein the power data includes text and its corresponding images;

[0008] Inputting the text and images into a multi-layer perception machine respectively to obtain initial text feature vectors and initial image feature vectors, taking the initial text feature vectors and initial image feature vectors as inputs of a routing capsule algorithm, and outputting text feature vectors and image feature vectors;

[0009] Mapping the text feature vectors and image feature vectors to a common space respectively by using a linear transformation method to obtain text projection vectors and image projection vectors;

[0010] The text projection vector and the image projection vector are taken as inputs of an additive attention model, and an integrated feature vector is outputted;

[0011] The additive attention model integrates a multi-head gated attention mechanism;

[0012] A hash function is constructed, the index of the power data is calculated based on the integrated feature vector and the hash function, and the power data with the same index are stored in the bucket corresponding to the index in the hash table;

[0013] A user query is obtained, an integrated query feature vector is obtained based on the user query, and the index of the user query is calculated according to the integrated query feature vector;

[0014] The similarity between the integrated query feature vector and the integrated feature vector of each power data in the bucket corresponding to the index of the user query is calculated, a similarity score is obtained, and the R power data corresponding to the highest similarity score are returned to the user as the retrieval result.

[0015] Preferably, the preprocessing of the text is specifically:

[0016] The text is taken as the input of a BERT encoder, and a text vector is outputted, and the text vector is normalized, which is expressed by the following formula:

[0017] ;

[0018] In the formula, denotes the normalized text vector of the i-th power data, denotes a BERT encoder processing function, denotes the text of the i-th power data, denotes the L2 norm of the text vector of the i-th power data

[0019] The preprocessing of the image is specifically:

[0020] A pixel vector is constructed by extracting the pixel value of the image pixel, and the pixel vector is normalized, which is expressed by the following formula:

[0021] ;

[0022] In the formula, denotes the pixel vector of the i-th power data, denotes the normalized pixel vector of the i-th power data.

[0023] ​​​​​​Preferably, the normalized text vector and pixel vector are input into a multilayer perceptron, and the initial text feature vector and initial image feature vector are obtained based on the hidden layer of the multilayer perceptron, as expressed by the formula:

[0024] ;

[0025] ;

[0026] In the formula, Indicates the first The initial text feature vector of each power data point Represents the ReLU activation function. This represents the text weight matrix of the hidden layer. This represents the text bias vector of the hidden layer. Indicates the first The initial image feature vector of the power data, This represents the image weight matrix of the hidden layer. This represents the CNN encoder processing function. This represents the image bias vector of the hidden layer;

[0027] The initial text feature vector is divided into equal-length segments. The initial text feature vectors of the group are used as initial text capsules;

[0028] The initial image feature vector is divided into equal-length segments. The initial image feature vectors of the group are used, and each initial image feature vector is regarded as an initial image capsule;

[0029] Obtain the prediction vector of the high-level text capsule based on the initial text capsule;

[0030] The prediction vectors of high-level image capsules are obtained based on the initial image capsule;

[0031] Based on the prediction vectors of the high-level image capsule and the high-level text capsule, a dynamic routing mechanism is used to obtain the high-level image capsule and the high-level text capsule. The specific steps are as follows:

[0032] S1. Initialize the number of image route pairs and text route pairs;

[0033] S2, Set the number of iterations to... ;

[0034] S3. Calculate the image coupling coefficient and text coupling coefficient based on the image routing logarithm and text routing logarithm;

[0035] S4. Obtain the high-level image capsule and the high-level text capsule based on the image coupling coefficient, the text coupling coefficient, the prediction vector of the high-level image capsule, and the prediction vector of the high-level text capsule.

[0036] S5, if If the value is 0, proceed to step S6; otherwise... = -1, update the image route logarithm and text route logarithm based on the high-level text capsule and high-level image capsule, and perform step S3, expressed as a formula:

[0037] ;

[0038] ;

[0039] In the formula, Indicates the first The first electricity data A high-level text capsule, Indicates the first The first electricity data A high-level image capsule, Indicates the first The first electricity data The initial text capsule for the first The number of text routes in a high-level text capsule Indicates the first The first electricity data The initial image capsule for the first Image routing logarithm of a high-level image capsule Indicates the first The first electricity data The initial text capsule for the first The prediction vector of a high-level text capsule. Indicates the first The first electricity data The initial image capsule for the first Predicted vectors of a high-level image capsule;

[0040] S6. Construct text feature vectors and image feature vectors based on high-level text capsules and high-level image capsules, expressed by the following formula:

[0041] ;

[0042] ;

[0043] In the formula, Indicates the number of high-level capsules. Indicates the first Image feature vectors of power data, a text feature vector representing the i-th power data.

[0044] Preferably, a linear transformation method is used to map the text feature vector to a common space, which is expressed as:

[0045] ;

[0046] wherein, a text projection vector representing the i-th power data, represents a text mapping weight matrix, represents a text mapping bias vector.

[0047] A linear transformation method is used to map the image feature vector to a common space, which is expressed as:

[0048] ;

[0049] wherein, an image projection vector representing the i-th power data, represents an image mapping weight matrix, represents an image mapping bias vector. Preferably, the text projection vector and the image projection vector are respectively subjected to multi-head projection to obtain a multi-head text projection vector and a multi-head image projection vector, which is expressed as:

[0050]

[0051] ;

[0052] ;

[0053] wherein, an i-th head text projection vector representing the i-th power data, an i-th head image projection vector representing the i-th power data, represents an i-th head projection matrix. Based on the multi-head text projection vector and the multi-head image projection vector, a multi-head text attention score and an image attention score are obtained, which is expressed as: ;

[0054]

[0055] ;

[0056] ;

[0057] ​​​​​​​wherein, denotes the th head text attention score of the th power data, denotes the th head image attention score of the th power data, denotes the transpose of the additive attention context vector of the th head, denotes the hyperbolic tangent activation function, denotes the additive attention weight matrix of the th head, denotes the additive attention bias vector of the th head.

[0058] Based on the multi-head text attention scores and image attention scores, multi-head text attention weights and multi-head image attention weights are obtained, which is expressed in the formula as:

[0059] ;

[0060] ;

[0061] wherein, denotes the th head text attention weight of the th power data, denotes the th head image attention weight of the th power data.

[0062] Based on the multi-head text attention weights, multi-head image attention weights, multi-head text projection vectors and multi-head image projection vectors, a multi-head fusion feature vector is obtained, which is expressed in the formula as:

[0063] ;

[0064] wherein, denotes the th head fusion feature vector of the th power data, denotes the Sigmoid function, denotes the gating vector of the th head, denotes the gating projection matrix of the th head, denotes the element-wise multiplication operator.

[0065] The multi-head fusion feature vectors are spliced to obtain a fusion feature vector, which is expressed in the formula as: ​

[0066] ;

[0067] In the formula, Indicates the first A fusion feature vector of power data This indicates a splicing operation. Indicates the number of long positions. This indicates the output projection matrix.

[0068] Preferably, construct A number of different hash functions are expressed by the following formula:

[0069] ;

[0070] ;

[0071] In the formula, Indicates the first A hash function, This indicates that the vector follows the mean vector. And the covariance matrix is ​​the identity matrix. normal distribution ;

[0072] The hash value of the fused feature vector is obtained based on the fused feature vector and the hash function, expressed by the formula:

[0073] ;

[0074] In the formula, Indicates the first The fusion feature vector of the power data at the th ... Hash value under each hash function Represents a symbolic function. Hash function Transpose of;

[0075] based on A hash sequence is constructed from the hash values, and the hash sequence is used as an index for the electricity data.

[0076] Preferably, the method further includes extracting common words, common phrases, and power terms from each power data text in the bucket to construct a local dictionary;

[0077] Replace the text of each power data with an integer sequence, where each element of the integer sequence is the corresponding index of the text content in the local dictionary;

[0078] Entropy coding is applied to integer sequences to obtain compressed text data of each power data point;

[0079] selecting one of the images of the power data as a reference image, calculating a residual error of each image relative to the reference image, and compressing the residual error using an image compression algorithm to obtain image compression data of the power data;

[0080] storing the local dictionary, the reference image, the text compression data, and the image compression data.

[0081] Preferably, a similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index of the user query is calculated, and the similarity is expressed by a formula as follows:

[0082]

[0083] wherein, represents the query fusion feature vector, represents a similarity score of the fusion feature vector of the i-th power data in the bucket corresponding to the index of the user query, represents the L2 norm of the query fusion feature vector, represents the L2 norm of the fusion feature vector.

[0084] In another aspect, the present application also provides an electronic device having a computer program stored thereon, wherein the computer program is executed by a processor to implement the multi-modal power data retrieval method according to any one of the embodiments of the present application.

[0085] In another aspect, the present application also provides a computer readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors, cause the one or more processors to implement the multi-modal power data retrieval method according to any one of the embodiments of the present application.

[0086] The present application has the following advantages:

[0087] 1. Traditional power data retrieval is mostly dependent on single modal processing, such as using only text keywords or image features, which is difficult to capture deep semantic associations between data. The present application fuses text and image dual modal data, and utilizes the complementary nature of structured information (such as device parameters and fault description) of text and unstructured information (such as device appearance and state) of image to achieve more comprehensive semantic understanding.

[0088] ​​​​​2、With the explosive growth of power data, traditional high-dimensional space nearest neighbor search is inefficient. The invention maps high-dimensional fusion features to low-dimensional hash space through hash function, and builds hash table to realize fast retrieval. Similar data is mapped to the same hash bucket, and only needs to be searched in the corresponding bucket when retrieving, reducing the time complexity from linear to approximately constant, significantly improving the retrieval efficiency of large-scale data.

[0089] 3、Power data contains a large amount of text and high-resolution images, and direct storage costs are high. The invention extracts public vocabulary, phrases and terms from text to build a local dictionary, and then compresses the text into an integer sequence after entropy coding; the image uses reference image residual compression technology. Using the characteristics of power data field (similar data points text / image highly similar), differential compression is realized, which significantly reduces the storage cost while ensuring data availability, and adapts to the long-term storage needs of historical data.

[0090] 4、In multi-modal fusion, different modalities and features have different contributions to the task. The invention uses additive attention model and multi-head gated attention mechanism to dynamically adjust the weights of modalities and features. For example, when processing power equipment images containing background noise, the attention mechanism can automatically focus on the main body of the equipment and reduce background interference. This mechanism makes the fusion features more suitable for retrieval requirements, enhances the feature representation ability, and improves the retrieval effect. BRIEF DESCRIPTION OF DRAWINGS

[0091] Figure 1 The flowchart of the embodiment method of the invention is implemented. DETAILED DESCRIPTION

[0092] The technical solutions in the embodiments of the invention will be described clearly and completely below with reference to the drawings in the embodiments of the invention. Obviously, the described embodiments are only part of the embodiments of the invention, not all. Based on the embodiments in the invention, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the invention.

[0093] It should be understood that the step numbers used herein are only for the convenience of description, and are not limited to the execution sequence of the steps.

[0094] It should be understood that the terms used in the specification of the invention are only for the purpose of describing specific embodiments and are not intended to limit the invention. As used in the specification and the appended claims of the invention, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0095] The terms "comprise" and "contain" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0096] The term "and / or" means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0097] Embodiment I:

[0098] Referring to Figure 1 The present application provides a multi-modal power data retrieval method, comprising the following steps:

[0099] Obtain a plurality of power data and preprocess; wherein the power data includes text and its corresponding image;

[0100] Input the text and image into a multi-layer perception machine respectively to obtain an initial text feature vector and an initial image feature vector, and input the initial text feature vector and the initial image feature vector as inputs of a routing capsule algorithm to output a text feature vector and an image feature vector;

[0101] Map the text feature vector and the image feature vector to a common space respectively by using a linear transformation method to obtain a text projection vector and an image projection vector;

[0102] Input the text projection vector and the image projection vector as inputs of an additive attention model to output a fusion feature vector;

[0103] The additive attention model fuses a multi-head gated attention mechanism;

[0104] Construct a hash function, calculate the index of the power data based on the fusion feature vector and the hash function, and store the power data with the same index in the same index corresponding hash table bucket (hash bucket);

[0105] Obtain a user query, obtain a query fusion feature vector based on the user query, and calculate the index of the user query according to the query fusion feature vector;

[0106] Calculate the similarity of the fusion feature vector of each power data in the bucket corresponding to the index of the user query with the query fusion feature vector, obtain a similarity score, and return the power data corresponding to the highest similarity score R (R is 5 in this embodiment) to the user as a retrieval result.

[0107] The method for obtaining the query fusion feature vector can refer to the method for obtaining the power data fusion feature vector disclosed in the embodiment. The difference is that there are four cases such as "only complete text description, only partial text description, only picture, and partial text description + picture". If there is no text description in the user query, the user query does not need to generate a text feature vector through a multi-layer perception and a routing capsule algorithm, and the text projection vector is a 0 vector, so the text attention weight obtained is also 0, and the fusion feature vector only contains the image projection vector. If there is no image input in the user query, the user query does not need to generate an image feature vector through a multi-layer perception and a routing capsule algorithm, and the image projection vector is a 0 vector, so the image attention weight obtained is also 0, and the fusion feature vector only contains the text projection vector.

[0108] The power data contains professional text description and high-resolution images, and direct processing faces the problems of dimension disaster and semantic gap. The numerical information in the text (such as "20kV") needs to be understood in the context, and the device components in the image need to be accurately positioned. The pre-trained BERT encoder can convert the text into a vector representation rich in semantic information, but the vector length output by the BERT encoder is not fixed, which will affect the accuracy of subsequent similarity calculation. L2 norm normalization converts the vector into a unit vector, ensuring that subsequent calculations are completely based on the direction of the vector (semantic similarity) rather than the length. In particular, the context understanding ability of the BERT encoder can capture key semantics, so it can still complete accurate retrieval when the input text of the user query and the text description of the power data to be retrieved are different.

[0109] The image pixel value range is 0-255, the value is large, and direct input into the model will cause gradient explosion or unstable calculation, and normalization to the [0, 1] interval can greatly improve the numerical stability.

[0110] Preferably, the preprocessing for the text is specifically:

[0111] The text is input into the BERT encoder, and a text vector is output, and the text vector is normalized, which is expressed by the formula:

[0112] ;

[0113] In the formula, denotes the normalized text vector of the i-th power data, denotes a BERT encoder processing function, denotes the text of the i-th power data, denotes the L2 norm of the text vector of the i-th power data .​​​

[0114] The specific preprocessing for the image is as follows:

[0115] Extract the pixel values ​​of image pixels to construct a pixel vector, and then normalize the pixel vector, as expressed by the formula:

[0116] ;

[0117] In the formula, Indicates the first A pixel vector of power data, Indicates the first A pixel vector obtained after normalizing power data.

[0118] CNN encoders are neural networks specifically designed for processing image data, possessing powerful feature extraction capabilities. Images of power equipment contain rich spatial information, such as local features like edges, textures, and shapes. CNNs effectively capture these features through convolutional kernels and pooling operations. Directly using raw pixels as input to the hidden layers of a multilayer perceptron not only results in high dimensionality but also contains a large amount of redundant information. Instead, CNNs, through multiple convolutions and nonlinear transformations, can extract meaningful semantic features from pixels, compressing high-dimensional image data into low-dimensional but information-rich feature vectors, which are then input into the hidden layers of the multilayer perceptron to extract the initial image feature vector. Leveraging the powerful fitting capabilities of the multilayer perceptron's nonlinear transformations, it can learn how to extract the most effective high-order abstract features for power data retrieval tasks from general features.

[0119] Preferably, the normalized text vector and pixel vector are input into a multilayer perceptron, and the initial text feature vector and initial image feature vector are obtained based on the hidden layer of the multilayer perceptron, as expressed by the formula:

[0120] ;

[0121] ;

[0122] In the formula, Indicates the first The initial text feature vector of each power data point Represents the ReLU activation function. This represents the text weight matrix of the hidden layer. This represents the text bias vector of the hidden layer. Indicates the first The initial image feature vector of the power data, This represents the image weight matrix of the hidden layer. This represents the CNN encoder processing function. This represents the image bias vector of the hidden layer;

[0123] Multilayer perceptron (MLP) feature extraction cannot capture the structural relationships between components of power equipment. The routing capsule algorithm explicitly models part-whole relationships through dynamic routing. Power data exhibits a clear hierarchical structure: conceptual hierarchies in text descriptions (e.g., "insulator-crack") and spatial relationships in images (e.g., "transformer-oil conservator") both require structured representation. The dynamic routing algorithm iteratively optimizes coupling coefficients, causing higher-level capsules to aggregate relevant lower-level features (initial capsules), with the length of the output vector representing the probability of feature presence. This representation better reflects the compositional characteristics of power equipment, providing rich structural information. Separately processing text and image features ensures that the characteristics of each modality are fully extracted, providing high-quality input for subsequent fusion.

[0124] The initial text feature vector is divided into equal-length segments. The initial text feature vectors of the group are considered as initial text capsules, expressed by the formula:

[0125] ;

[0126] In the formula, Indicates the first The first electricity data An initial text capsule, Indicates the initial number of capsules;

[0127] The initial image feature vector is divided into equal-length segments. The initial image feature vectors of the group are considered as initial image capsules, expressed by the formula:

[0128] ;

[0129] In the formula, Indicates the first The first electricity data An initial image capsule;

[0130] The prediction vectors for higher-level text capsules are obtained based on the initial text capsules, expressed by the formula:

[0131] ;

[0132] In the formula, Indicates the first The first electricity data The initial text capsule for the first The prediction vector of a high-level text capsule. Indicates the first The text transformation matrix of the first power data is used to transform the first... the initial text capsule of the i-th power data is mapped to the space of the j-th high-level text capsule;

[0133] The prediction vector of the high-level image capsule based on the initial image capsule is obtained, and is expressed by a formula as follows:

[0134] ;

[0135] In the formula, the prediction vector of the j-th high-level image capsule based on the i-th initial image capsule of the j-th power data is represented as The image transformation matrix of the j-th power data is represented as

[0136] The high-level image capsule and the high-level text capsule are obtained based on the prediction vector of the high-level image capsule and the prediction vector of the high-level text capsule by using a dynamic routing mechanism, and the specific steps are as follows:

[0137] S1, the number of image routing pairs and the number of text routing pairs are initialized, for example, the number of image routing pairs and the number of text routing pairs are set to 0;

[0138] S2, the number of iterations is set to 5, and the number of image routing pairs is set to 5;

[0139] S3, the image coupling coefficient and the text coupling coefficient are calculated based on the number of image routing pairs and the number of text routing pairs, and are expressed by a formula as follows:

[0140] ;

[0141] ;

[0142] In the formula, the text coupling coefficient of the j-th high-level text capsule based on the i-th initial text capsule of the j-th power data is represented as ​​​​​​​​​​​​​​​​​​​​​​​a number of image routing pairs of the high-level image capsules, representing the th high-level text capsule of the th power data, a number of image routing pairs of the high-level image capsules, representing a Softmax activation function;

[0143] S4, obtaining the high-level image capsules and the high-level text capsules based on the image coupling coefficient, the text coupling coefficient, the predicted vector of the high-level image capsules and the predicted vector of the high-level text capsules, which is expressed as:

[0144] ;

[0145] ;

[0146] wherein, representing the th high-level text capsule of the th power data, representing a Squash function, representing the th high-level image capsule of the th power data;

[0147] S5, if = 0, executing step S6, otherwise, = -1, updating the image routing pairs and the text routing pairs based on the high-level text capsules and the high-level image capsules, and executing step S3, which is expressed as:

[0148] ;

[0149] ;

[0150] S6, constructing the image feature vector and the text feature vector based on the high-level text capsules and the high-level image capsules, which is expressed as

[0151] ;

[0152] ;

[0153] wherein, representing the number of high-level capsules, representing the image feature vector of the th power data, representing the text feature vector of the th power data.

[0154] Preferably, a linear transformation method is used to map the text feature vectors to a common space, expressed by the formula:

[0155] ;

[0156] In the formula, Indicates the first Text projection vector of electricity data, Represents the text mapping weight matrix. Represents the text mapping bias vector;

[0157] The image feature vectors are mapped to a common space (i.e., dimensionally consistent) using a linear transformation method, as expressed by the formula:

[0158] ;

[0159] In the formula, Indicates the first Image projection vector of each power data point, Represents the image mapping weight matrix. This represents the image mapping bias vector.

[0160] Simple weighted fusion cannot fully utilize the complementarity of multimodal information; the importance of different modalities in power data dynamically changes with the content. Text is more important in fault descriptions, while images are more crucial for visual inspections. A multi-head mechanism allows the additive attention model to focus on different feature subspaces in parallel, while a gating mechanism dynamically controls the information flow. The calculation of attention weights considers the correlation between features, and the gating vector further filters out noise. This design enhances the expressive power of the additive attention model, improves the quality of fused features, and prepares the data for index construction.

[0161] Preferably, multi-head projection is performed on the text projection vector and the image projection vector respectively to obtain multi-head text projection vector and multi-head image projection vector, which are expressed by the following formula:

[0162] ;

[0163] ;

[0164] In the formula, Indicates the first The first electricity data Individual text projection vectors Indicates the first The first electricity data Individual image projection vector, Indicates the first Individual projection matrix;

[0165] The multi-head text attention score and the image attention score are obtained based on the multi-head text projection vector and the multi-head image projection vector, and are expressed in a formula as follows:

[0166] ;

[0167] ;

[0168] In the formula, the first head text attention score of the i-th power data is represented as The i-th head image attention score of the i-th power data is represented as The i-th head text attention score of the i-th power data is represented as The i-th head image attention score of the i-th power data is represented as The i-th head text attention score of the i-th power data is represented as The i-th head image attention score of the i-th power data is represented as The i-th head text attention score of the i-th power data is represented as The i-th head image attention score of the i-th power data is represented as The i-th head text attention score of the i-th power data is represented as The i-th head image attention score of the i-th power data is represented as The i-th head text attention score of the i-th power data is represented as The i-th head image attention score of the i-th power data is represented as The i-th head text attention score of the i-th power data is represented as The i-th head image attention score of the i-th power data is represented as

[0169] The conventional additive attention needs to dynamically select the most relevant part in multiple keys to the current query; since the role of the additive attention in the embodiment is to evaluate the global importance weight of a single modal feature vector, there is no alignment concept of "key" and "query", , The "key" is not the feature to be evaluated, but the feature itself, The "query" is not the learned importance evaluation standard, but the learned importance evaluation standard, and The feature transformation network is composed of and, which is prepared for importance evaluation, so the network structure of the additive attention in the embodiment is fine-tuned.

[0170] The multi-head text attention weight and the multi-head image attention weight are obtained based on the multi-head text attention score and the image attention score, and are expressed in a formula as follows:

[0171] ;

[0172] ;

[0173] In the formula, the first head text attention weight of the i-th power data is represented as The i-th head image attention weight of the i-th power data is represented as The i-th head text attention weight of the i-th power data is represented as The i-th head image attention weight of the i-th power data is represented as The i-th head text attention weight of the i-th power data is represented as The i-th head image attention weight of the i-th power data is represented as The i-th head text attention weight of the i-th power data is represented as

[0174] The multi-head fusion feature vector is obtained based on the multi-head text attention weights, multi-head image attention weights, multi-head text projection vectors, and multi-head image projection vectors, expressed by the formula:

[0175] ;

[0176] In the formula, Indicates the first The first electricity data Individual fusion feature vectors This represents the Sigmoid function. Indicates the first The gating vector of the size, Indicates the first Individual gated projection matrix This represents the element-wise multiplication operator;

[0177] The multi-head fusion feature vectors are concatenated to obtain the fusion feature vector, which can be expressed by the formula:

[0178] ;

[0179] In the formula, Indicates the first A fusion feature vector of power data This indicates a splicing operation. Indicates the number of long positions. This indicates the output projection matrix.

[0180] The fused feature vectors are usually in a high-dimensional space. Directly performing nearest neighbor search in such a high-dimensional space faces the problem that the computational complexity increases exponentially with the dimension, making brute-force search infeasible on large-scale datasets.

[0181] The idea behind this step is to... hash functions Mapping high-dimensional vectors to a low-dimensional hash signature space while preserving similarity relationships. Hash function. Sampling from the standard normal distribution ensures the randomness and uniformity of the projection, giving the hash function family ideal local sensitivity. The sign function discretizes the continuous projection values ​​into binary codes (+1 or -1). This binarization not only compresses storage space, but more importantly, it defines clear boundaries, dividing the feature space into... Different hash buckets.

[0182] If two vectors are similar (with a small angle) in a high-dimensional space, then their projections onto random directions are more likely to have the same sign. An independent hash function We can guarantee with high probability that similar vectors will be mapped to the same hash bucket. This reduces the retrieval time complexity from brute-force search. Reduced to approximate Bucket search plus The search within the bucket, among which It is the average size of the bucket. It refers to the quantity of electricity data. For large-scale electricity data, this acceleration effect is particularly significant.

[0183] Preferably, construct A number of different hash functions are expressed by the following formula:

[0184] ;

[0185] ;

[0186] In the formula, Indicates the first A hash function, This indicates that the vector follows the mean vector. And the covariance matrix is ​​the identity matrix. normal distribution ;

[0187] The hash value of the fused feature vector is obtained based on the fused feature vector and the hash function, expressed by the formula:

[0188] ;

[0189] In the formula, Indicates the first The fusion feature vector of the power data at the th ... Hash value under each hash function This represents the sign function, used to output +1 or -1, if... If the result of the dot product is greater than or equal to zero, then Output +1, otherwise output -1. Hash function Transpose of;

[0190] based on A hash sequence is constructed from hash values ​​(for example, a hash sequence might be [+1,-1,-1,+1,...,-1], and the sequence length depends on...). The value can be adjusted according to the size of the power data (this embodiment does not impose specific limitations), and the hash sequence is used as the index of the power data.

[0191] For example, electricity data with indices [+1,-1,-1,+1,+1] will be placed in the first hash bucket.

[0192] Electricity data with indices [+1,-1,+1,+1,+1] will be placed in a different hash bucket.

[0193] Electricity data with index [-1,-1,-1,-1,-1] will be placed into the third hash bucket.

[0194] Power data includes text descriptions and high-resolution images, and storing them directly would consume a huge amount of storage space. For example, a 1024×1024 image of power equipment may require several MB of storage space, while text descriptions, although relatively small, can accumulate to a considerable amount in large-scale scenarios.

[0195] Preferably, the method further includes extracting common words, common phrases, and power terms from each power data text in the bucket to construct a local dictionary;

[0196] The common vocabulary consists of words that appear multiple times in the hash bucket, such as: check, exception, run, etc.

[0197] Common phrases are phrases that appear multiple times in the hash bucket, such as: insulation resistance, load switch, three-phase imbalance, etc.

[0198] Electrical terminology includes: current, voltage, load, power, inrush current, flashover, ferroresonance, etc.

[0199] Replace the text of each power data with an integer sequence (if a word or phrase that cannot be found in the local dictionary appears when replacing the text, add the word or phrase to the local dictionary and set an index value before replacing it), where each element of the integer sequence is the corresponding index of the text content in the local dictionary;

[0200] Entropy coding (such as Huffman coding or arithmetic coding) is applied to integer sequences to obtain compressed text data of each power data.

[0201] Select an image from the images of each power data set as a reference image (e.g., determined by cluster centers, average images, or manual settings), and calculate the residual between each image and the reference image, expressed by the formula:

[0202] ;

[0203] In the formula, Indicates the number of hash buckets The residuals of the power data Indicates the number of hash buckets An image of electricity data, Indicates a reference image;

[0204] The residuals are compressed using an image compression algorithm (such as compressing to WebP, AVIF, or JPEG-XL formats) to obtain image compressed data of each power data.

[0205] The local dictionary, reference images, compressed text data, and compressed image data are stored.

[0206] When returning the search results, the compressed text data is decompressed, and the integer sequence is used to search the local dictionary to replace the integer sequence back with the text of the power data.

[0207] The compressed image data is decompressed and stitched with a reference image to obtain an image of the power data;

[0208] Images and text based on power data are combined to create a complete power data report, which is then returned to the user.

[0209] Data storage compression employs a differentiation strategy based on intra-bucket similarity. Electricity data text is highly domain-specific, containing a large amount of repetitive technical terms and descriptions. Traditional general-purpose compression algorithms cannot fully utilize this domain knowledge. By constructing a local dictionary for each hash bucket, we are essentially leveraging the local relevance of text within the bucket; similar data points not only have similar feature vectors, but their text descriptions also often share a large number of words and phrase structures. Entropy coding further utilizes the probability distribution of character occurrences, assigning short codes to high-frequency characters to achieve optimal compression.

[0210] Images of power equipment possess unique visual characteristics: images within the same bucket often depict similar equipment, from similar angles, or exhibiting the same type of fault. The core insight of residual coding is that storing the differences between images is more efficient than storing the complete image. The reference image captures the common features of images within the bucket, while the residual encodes only individual differences, which typically contain a large number of zero-values ​​or low-frequency components, making them highly compressible.

[0211] Preferably, the similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index queried by the user is calculated, expressed by the formula:

[0212] ;

[0213] In the formula, This indicates a query to fuse feature vectors. This indicates the index number in the bucket corresponding to the index queried by the user. Fusion feature vector of power data Similarity score, Represents the query fusion feature vector L2 norm, Represents the fused feature vector L2 norm of the L2 norm of the difference between the two vectors.

[0214] Embodiment Two:

[0215] The embodiment provides an electronic device, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal power data retrieval method according to any one of the embodiments of the application.

[0216] Embodiment Three:

[0217] The embodiment provides a computer readable storage medium, which is used for storing one or more programs, and when the one or more programs are executed by one or more processors, the one or more processors implement the multi-modal power data retrieval method according to any one of the embodiments of the application.

[0218] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the cases of A alone, A and B together, and B alone. Wherein A and B can be singular or plural. The character " / " generally represents that the associated objects before and after it are in an "or" relationship. "At least one of the following" and the like expressions mean any combination of these items, including any combination of single or multiple items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, wherein a, b, and c can be single or multiple.

[0219] Those skilled in the art can realize that the units and algorithm steps described in the embodiments disclosed in the present application can be realized by electronic hardware, computer software and combination of electronic hardware and computer software. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0220] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0221] In several embodiments provided in the present application, any function, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0222] The above description is only some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, based on the content of the specification and drawings of the present application, are also included in the patent protection scope of the present application.

Claims

1. A multi-modal electric power data retrieval method, characterized by, The method comprises the following steps: acquiring power data and preprocessing to obtain normalized text vectors and pixel vectors; wherein the power data comprises text and corresponding images; inputting the normalized text vectors and pixel vectors into a multi-layer perception machine, and obtaining initial text feature vectors and initial image feature vectors based on the hidden layers of the multi-layer perception machine, which is expressed by a formula as follows: ； ； In the formula, represents the initial text feature vector of the first power data, represents the ReLU activation function, represents the text weight matrix of the hidden layer, represents the text bias vector of the hidden layer, represents the initial image feature vector of the first power data, represents the image weight matrix of the hidden layer, represents the CNN encoder processing function, represents the image bias vector of the hidden layer, represents the normalized text vector of the first power data, represents the normalized pixel vector of the first power data. dividing the initial text feature vector into initial text feature sub-vectors of the group, and regarding each initial text feature sub-vector as an initial text capsule; dividing the initial image feature vector into initial image feature sub-vectors of the group, and regarding each initial image feature sub-vector as an initial image capsule; obtaining a prediction vector of a high-level text capsule based on the initial text capsule; obtaining a prediction vector of a high-level image capsule based on the initial image capsule; obtaining the high-level image capsule and the high-level text capsule based on the prediction vector of the high-level image capsule and the prediction vector of the high-level text capsule by using a dynamic routing mechanism, and the specific steps are as follows: S1, initializing the image routing logarithm and the text routing logarithm; S2, set the iteration number to ; S3, calculating the image coupling coefficient and the text coupling coefficient based on the image routing logarithm and the text routing logarithm; S4, obtaining the high-level image capsule and the high-level text capsule based on the image coupling coefficient, the text coupling coefficient, the prediction vector of the high-level image capsule and the prediction vector of the high-level text capsule; S5, if = 0, then step S6 is performed, otherwise, = 0, then step S6 is performed, otherwise, - 1, the image routing log and the text routing log are updated based on the high-level text capsule and the high-level image capsule, and step S3 is performed, which is expressed in a formula as follows: ； ； In the formula, represents the number of high-level text capsules of the th electric power data, represents the number of high-level image capsules of the th electric power data, represents the number of high-level image capsules of the th electric power data, represents the number of high-level text capsules of the th electric power data, represents the number of high-level image capsules of the th electric power data, represents the number of high-level image capsules of the th electric power data, represents the number of high-level text capsules of the th electric power data, represents the number of high-level text capsules of the th electric power data, represents the number of high-level image capsules of the th electric power data, represents the number of high-level text capsules of the th electric power data, represents the number of high-level image capsules of the th electric power data. S6, constructing the text feature vector and the image feature vector based on the high-level text capsule and the high-level image capsule, which is expressed by a formula as follows: ； ； In the formula, represents the number of high-layer capsules, represents the image feature vector of the first power data, represents the text feature vector of the first power data; mapping the text feature vector and the image feature vector to a common space by using a linear transformation method to obtain a text projection vector and an image projection vector; taking the text projection vector and the image projection vector as inputs of an additive attention model to output a fusion feature vector; wherein the additive attention model fuses a multi-head gated attention mechanism; constructing a hash function, calculating the index of the power data based on the fusion feature vector and the hash function, and storing the power data with the same index in the bucket corresponding to the index in the hash table; acquiring a user query, obtaining a query fusion feature vector based on the user query, and calculating the index of the user query according to the query fusion feature vector; calculating the similarity of the fusion feature vectors of each power data in the bucket corresponding to the index of the user query, obtaining a similarity score, and returning the R corresponding power data with the highest similarity score to the user as a retrieval result.

2. The multi-modal electric power data retrieval method of claim 1, wherein, The preprocessing of the text is specifically: taking the text as an input of a BERT encoder to output a text vector, and normalizing the text vector, which is expressed by a formula as follows: ； wherein denotes a BERT encoder processing function, denotes the text of the th power data, denotes the text vector of the th power data L2 norm of the text vector The preprocessing of the image is specifically: extracting the pixel values of the image pixels to construct a pixel vector, and normalizing the pixel vector, which is expressed by a formula as follows: ； In the formula, represents the pixel vector of the i th power data.

3. The multi-modal electric power data retrieval method of claim 1, wherein, mapping the text feature vector to a common space by using a linear transformation method, which is expressed by a formula as follows: ； In the formula, represents the text projection vector of the first power data, represents the text mapping weight matrix, represents the text mapping bias vector; mapping the image feature vector to a common space by using a linear transformation method, which is expressed by a formula as follows: ； wherein represents a vector of image projections of the represents an image mapping weight matrix, represents an image mapping bias vector.​ 4. The multi-modal electric power data retrieval method of claim 3, wherein, performing multi-head projection on the text projection vector and the image projection vector to obtain multi-head text projection vectors and multi-head image projection vectors, which are expressed by a formula as follows: ； ； In the formula, represents the first head text projection vector of the first power data, represents the first head image projection vector of the first power data, represents the first head projection matrix; obtaining multi-head text attention scores and image attention scores based on the multi-head text projection vectors and the multi-head image projection vectors, which are expressed by a formula as follows: ； ； In the formula, Indicates the first The first electricity data Individual text attention score, Indicates the first The first electricity data Individual image attention score, Indicates the first The additive attention context vector of the head transpose, This represents the hyperbolic tangent activation function. Indicates the first The additive attention weight matrix for the size, Indicates the first Additive attention bias vector for size; The multi-head text attention weight and the multi-head image attention weight are obtained based on the multi-head text attention score and the image attention score, and are expressed by a formula as follows: ； ； In the formula, denotes the first head text attention weight of the first power data, denotes the first head image attention weight of the first power data; The multi-head fusion feature vector is obtained based on the multi-head text attention weight, the multi-head image attention weight, the multi-head text projection vector and the multi-head image projection vector, and is expressed by a formula as follows: ； wherein, represents the i-th head fusion feature vector of the i-th power data, represents a Sigmoid function, represents the i-th head gating vector of the i-th power data, represents an element-wise multiplication operator;​​​​​ The fusion feature vector is obtained by splicing the multi-head fusion feature vector, and is expressed by a formula as follows: ； In the formula, represents the fusion feature vector of the first power data, represents the splicing operation, represents the number of multiple heads, represents the output projection matrix.

5. The multi-modal electric power data retrieval method of claim 4, wherein, Constructing a different hash function, expressed in formula as: ； ； wherein represents the i-th hash function, represents a normal distribution with mean vector and covariance matrix ;​​ The hash value of the fusion feature vector is obtained based on the fusion feature vector and a hash function, and is expressed by a formula as follows: ； In the formula, denotes the hash value of the fusion feature vector of the first electric power data under the first hash function, denotes the sign function, denotes the transpose of the hash function . Based on a hash value to construct a hash sequence, and the hash sequence is taken as an index of the power data.

6. The multi-modal electric power data retrieval method of claim 1, wherein, The method further comprises extracting public words, public phrases and power terms in each power data text in the bucket to construct a local dictionary; Each power data text is replaced by an integer sequence, and each element of the integer sequence is the corresponding index of the content of the text in the local dictionary; The integer sequence is applied to entropy coding to obtain text compression data of each power data; An image is selected as a reference image in each power data image, and a residual error between each image and the reference image is calculated, and the residual error is compressed by using an image compression algorithm to obtain image compression data of each power data; The local dictionary, the reference image, the text compression data and the image compression data are stored.

7. The multi-modal electric power data retrieval method of claim 1, wherein, The similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index of the user query is calculated, and is expressed by a formula as follows: ； wherein, denotes a similarity score of the query fusion feature vector, denotes a similarity score of the query fusion feature vector, denotes a similarity score of the query fusion feature vector, denotes a similarity score of the query fusion feature vector, denotes a L2 norm of the query fusion feature vector, denotes a L2 norm of the query fusion feature vector, denotes a L2 norm of the query fusion feature vector, denotes a L2 norm of the query fusion feature vector.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the multi-modal power data retrieval method of any one of claims 1 to 7.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the multi-modal power data retrieval method of any one of claims 1 to 7. The program is executed by the processor to implement the multi-modal power data retrieval method of any one of claims 1 to 7.

Images