Multi-modal power data retrieval method and device and medium

Abstract

The invention relates to a multi-modal power data retrieval method and device and a medium, and belongs to the technical field of data retrieval, and the method comprises the following steps: obtaining and preprocessing power data, inputting a text and an image into a multi-layer perceptron to obtain an initial feature vector, and outputting a text feature vector and an image feature vector through a routing capsule algorithm; mapping to a public space by adopting linear transformation to obtain a projection vector; and inputting the projection vector into the additive attention model, and outputting a fusion feature vector. And constructing a hash function, calculating an electric power data index based on the fused feature vector, and storing data with the same index into a corresponding hash table bucket. And obtaining a user query, calculating a query fusion feature vector and an index, calculating the similarity of the query fusion feature vector and the index with each power data fusion feature vector in the corresponding bucket, and returning R power data with the highest similarity score as a retrieval result. According to the method, multi-modal information is integrated, the importance is adjusted by using an attention mechanism, and the retrieval effect is effectively improved.

Description

Technical Field

[0001] This invention relates to a multimodal power data retrieval method, device, and medium, belonging to the field of data retrieval technology. Background Technology

[0002] With the rapid development of the power industry and the deepening of information technology construction, the types and scale of power system data have exploded. How to efficiently retrieve and utilize this data has become an urgent problem to be solved. Power data encompasses multiple modalities of information, including text (such as equipment operating parameters, fault reports, maintenance records, etc.) and images (such as equipment photos, fault site images, power line inspection images, etc.). This data is of paramount value for power equipment fault diagnosis, maintenance decisions, and operational optimization.

[0003] However, traditional power data retrieval methods are often limited to single-modal data processing, failing to effectively integrate multimodal information such as text and images, thus limiting the accuracy and comprehensiveness of retrieval results. For example, early text retrieval methods mainly relied on keyword matching, while image retrieval methods depended on manually designed features. This made it difficult to capture deep-seated relationships and semantic information between data in complex power data environments. Furthermore, with the continuous expansion of power system scale and the improvement of intelligence and automation levels, simply relying on manual data screening and organization can no longer meet practical needs. There is an urgent need for an intelligent and efficient retrieval method to integrate multimodal power data. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a multimodal power data retrieval method, device, and medium.

[0005] The technical solution of the present invention is as follows: On the one hand, this invention proposes a multimodal power data retrieval method, comprising the following steps: Acquire power data and preprocess it; wherein, the power data includes text and its corresponding images; The text and image are input into a multilayer perceptron to obtain initial text feature vectors and initial image feature vectors. The initial text feature vectors and initial image feature vectors are then used as inputs to a routing capsule algorithm to output text feature vectors and image feature vectors. A linear transformation method is used to map the text feature vector and image feature vector to a common space to obtain the text projection vector and image projection vector, respectively. The text projection vector and image projection vector are used as inputs to the additive attention model, and the output is a fused feature vector. Among them, the additive attention model incorporates a multi-head gating attention mechanism; Construct a hash function, calculate the index of the power data based on the fused feature vector and the hash function, and store the power data with the same index in the bucket of the hash table corresponding to the index; Obtain the user query, obtain the query fusion feature vector based on the user query, and calculate the index of the user query based on the query fusion feature vector; Calculate the similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index queried by the user, obtain a similarity score, and return the power data corresponding to the R with the highest similarity scores as the search results to the user.

[0006] Preferably, the preprocessing for text includes: The text is used as input to the BERT encoder, and the output is a text vector. The text vector is then normalized, as expressed by the formula: ; In the formula, Indicates the first A text vector after normalizing power data. This represents the BERT encoder processing function. Indicates the first Text of electricity data, Indicates the first Text vector of electricity data The L2 norm; The specific preprocessing for the image is as follows: Extract the pixel values ​​of image pixels to construct a pixel vector, and then normalize the pixel vector, as expressed by the formula: ; In the formula, Indicates the first A pixel vector of power data Indicates the first A pixel vector obtained after normalizing power data.

[0007] Preferably, the normalized text vector and pixel vector are input into a multilayer perceptron, and the initial text feature vector and initial image feature vector are obtained based on the hidden layer of the multilayer perceptron, as expressed by the formula: ; ; In the formula, Indicates the first The initial text feature vector of the power data, Represents the ReLU activation function. This represents the text weight matrix of the hidden layer. This represents the text bias vector of the hidden layer. Indicates the first The initial image feature vector of the power data, This represents the image weight matrix of the hidden layer. This represents the CNN encoder processing function. This represents the image bias vector of the hidden layer; The initial text feature vector is divided into equal-length segments. The initial text feature vectors of the group are used as initial text capsules; The initial image feature vector is divided into equal-length segments. The initial image feature vectors of the group are used, and each initial image feature vector is regarded as an initial image capsule; Obtain the prediction vector of the high-level text capsule based on the initial text capsule; The prediction vectors of high-level image capsules are obtained based on the initial image capsule; Based on the prediction vectors of the high-level image capsule and the high-level text capsule, a dynamic routing mechanism is used to obtain the high-level image capsule and the high-level text capsule. The specific steps are as follows: S1. Initialize the number of image route pairs and text route pairs; S2, Set the number of iterations to... ; S3. Calculate the image coupling coefficient and text coupling coefficient based on the image routing logarithm and text routing logarithm; S4. Obtain the high-level image capsule and the high-level text capsule based on the image coupling coefficient, the text coupling coefficient, the prediction vector of the high-level image capsule, and the prediction vector of the high-level text capsule. S5, if If the value is 0, proceed to step S6; otherwise... = -1, update the image route logarithm and text route logarithm based on the high-level text capsule and high-level image capsule, and perform step S3, expressed as a formula: ; ; In the formula, Indicates the first The first electricity data A high-level text capsule, Indicates the first The first electricity data A high-level image capsule, Indicates the first The first electricity data The initial text capsule for the first The number of text routes in a high-level text capsule Indicates the first The first electricity data The initial image capsule for the first Image routing logarithm of a high-level image capsule Indicates the first The first electricity data The initial text capsule for the first The prediction vector of a high-level text capsule. Indicates the first The first electricity data The initial image capsule for the first Predicted vectors of a high-level image capsule; S6. Construct text feature vectors and image feature vectors based on high-level text capsules and high-level image capsules, expressed by the following formula: ; ; In the formula, Indicates the number of high-level capsules. Indicates the first Image feature vectors of power data, Indicates the first Textual feature vectors of electricity data.

[0008] Preferably, a linear transformation method is used to map the text feature vectors to a common space, expressed by the formula: ; In the formula, Indicates the first Text projection vector of electricity data, Represents the text mapping weight matrix. Represents the text mapping bias vector; The image feature vectors are mapped to a common space using a linear transformation method, as expressed by the formula: ; In the formula, Indicates the first Image projection vector of each power data point, Represents the image mapping weight matrix. This represents the image mapping bias vector.

[0009] Preferably, multi-head projection is performed on the text projection vector and the image projection vector respectively to obtain multi-head text projection vector and multi-head image projection vector, which are expressed by the following formula: ; ; In the formula, Indicates the first The first electricity data Individual text projection vectors Indicates the first The first electricity data Individual image projection vector, Indicates the first Individual projection matrix; The multi-head text attention score and image attention score are obtained based on the multi-head text projection vector and multi-head image projection vector, expressed by the formula: ; ; In the formula, Indicates the first The first electricity data Individual text attention score, Indicates the first The first electricity data Individual image attention score, Indicates the first The additive attention context vector of the head transpose, This represents the hyperbolic tangent activation function. Indicates the first The additive attention weight matrix for the size, Indicates the first Additive attention bias vector for size; The multi-head text attention weight and multi-head image attention weight are obtained based on the multi-head text attention score and image attention score, expressed by the formula: ; ; In the formula, Indicates the first The first electricity data Individual text attention weights, Indicates the first The first electricity data Individual image attention weights; The multi-head fusion feature vector is obtained based on the multi-head text attention weight, multi-head image attention weight, multi-head text projection vector, and multi-head image projection vector, expressed by the formula: ; In the formula, Indicates the first The first electricity data Individual fusion feature vectors This represents the Sigmoid function. Indicates the first The gating vector of the size, Indicates the first Individual gated projection matrix This represents the element-wise multiplication operator; The multi-head fusion feature vectors are concatenated to obtain the fusion feature vector, which can be expressed by the formula: ; In the formula, Indicates the first A fusion feature vector of power data This indicates a splicing operation. Indicates the number of long positions. This indicates the output projection matrix.

[0010] Preferably, construct A number of different hash functions are expressed by the following formula: ; ; In the formula, Indicates the first A hash function, This indicates that the vector follows the mean vector. And the covariance matrix is ​​the identity matrix. normal distribution ; The hash value of the fused feature vector is obtained based on the fused feature vector and the hash function, expressed by the formula: ; In the formula, Indicates the first The fusion feature vector of the power data at the th ... Hash value under each hash function Represents a symbolic function. Hash function Transpose of; based on A hash sequence is constructed from the hash values, and the hash sequence is used as an index for the electricity data.

[0011] Preferably, the method further includes extracting common words, common phrases, and power terms from each power data text in the bucket to construct a local dictionary; Replace the text of each power data with an integer sequence, where each element of the integer sequence is the corresponding index of the text content in the local dictionary; Entropy coding is applied to integer sequences to obtain compressed text data of each power data point; Select an image from the images of each power data as a reference image, calculate the residual between each image and the reference image, and compress the residual using an image compression algorithm to obtain the image compressed data of each power data. The local dictionary, reference images, compressed text data, and compressed image data are stored.

[0012] Preferably, the similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index queried by the user is calculated, expressed by the formula: ; In the formula, This indicates a query to fuse feature vectors. This indicates the index number in the bucket corresponding to the index queried by the user. Fusion feature vector of power data Similarity score, Represents the query fusion feature vector L2 norm, Represents the fused feature vector The L2 norm.

[0013] In another aspect, the present invention also provides an electronic device having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal power data retrieval method as described in any embodiment of the present invention.

[0014] In another aspect, the present invention also provides a computer-readable storage medium for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the multimodal power data retrieval method as described in any embodiment of the present invention.

[0015] The present invention has the following beneficial effects: 1. Traditional power data retrieval often relies on single-modal processing, such as using only text keywords or image features, making it difficult to capture deep semantic relationships between data. This invention achieves a more comprehensive semantic understanding by fusing text and image dual-modal data, leveraging the complementarity between structured information in text (such as equipment parameters and fault descriptions) and unstructured information in images (such as equipment appearance and status).

[0016] 2. With the surge in electricity data volume, traditional high-dimensional space nearest neighbor search is inefficient. This invention maps high-dimensional fused features to a low-dimensional hash space using a hash function, constructing a hash table for fast retrieval. Similar data are mapped to the same hash bucket, and retrieval only requires searching within the corresponding bucket, reducing the time complexity from linear to near constant, significantly improving retrieval efficiency for large-scale data.

[0017] 3. Power data contains a large amount of text and high-resolution images, making direct storage costly. This invention constructs a local dictionary by extracting common words, phrases, and terms from the text, converts the text into an integer sequence, and then uses entropy encoding compression; for images, reference image residual compression technology is employed. Leveraging the characteristics of the power data domain (high similarity between text and images for similar data points), differentiated compression is achieved, significantly reducing storage costs while ensuring data availability, and adapting to the needs of long-term historical data storage.

[0018] 4. In multimodal fusion, different modalities and features contribute differently to the task. This invention employs an additive attention model and a multi-head gating attention mechanism to dynamically adjust the weights of modalities and features. For example, when processing images of power equipment with background noise, the attention mechanism can automatically focus on the main body of the equipment, reducing background interference. This mechanism makes the fused features more aligned with retrieval needs, enhances feature representation capabilities, and improves retrieval performance. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the implementation of the method in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0022] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0024] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0025] Example 1: See Figure 1 This invention provides a multimodal power data retrieval method, comprising the following steps: Multiple power data points are acquired and preprocessed; wherein, the power data includes text and its corresponding images; The text and image are input into a multilayer perceptron to obtain initial text feature vectors and initial image feature vectors. The initial text feature vectors and initial image feature vectors are then used as inputs to a routing capsule algorithm to output text feature vectors and image feature vectors. A linear transformation method is used to map the text feature vector and image feature vector to a common space to obtain the text projection vector and image projection vector, respectively. The text projection vector and image projection vector are used as inputs to the additive attention model, and the output is a fused feature vector. Among them, the additive attention model incorporates a multi-head gating attention mechanism; Construct a hash function, calculate the index of the power data based on the fused feature vector and the hash function, and store power data with the same index in the bucket (hash bucket) of the hash table corresponding to the same index; Obtain the user query, obtain the query fusion feature vector based on the user query, and calculate the index of the user query based on the query fusion feature vector; Calculate the similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index queried by the user, obtain a similarity score, and return the power data with the highest similarity score R (R is 5 in this embodiment) as the search results to the user.

[0026] The method for obtaining the fused feature vector can refer to the steps for obtaining the fused feature vector of power data disclosed in this embodiment. The difference lies in the fact that user queries can fall into four categories: "only complete text description," "only partial text description," "only image," and "partial text description + image." If the user query contains no text description, then the user query does not need to generate a text feature vector through the multilayer perceptron and routing capsule algorithm, and the text projection vector is 0. Therefore, the text attention weight is also 0, and the fused feature vector only contains the image projection vector. If the user query does not contain an image input, then the user query does not need to generate an image feature vector through the multilayer perceptron and routing capsule algorithm, and the image projection vector is 0. Therefore, the image attention weight is also 0, and the fused feature vector only contains the text projection vector.

[0027] Electricity data contains specialized text descriptions and high-resolution images, posing significant challenges to direct processing due to the curse of dimensionality and the semantic gap. Numerical information in the text (such as "20kV") requires contextual understanding, while equipment components in images need precise localization. Pre-trained BERT encoders can convert text into semantically rich vector representations, but the variable length of the output vectors can affect the accuracy of subsequent similarity calculations. L2 norm normalization converts the vectors into unit vectors, ensuring that subsequent calculations are based entirely on vector direction (semantic similarity) rather than length. In particular, the contextual understanding capabilities of the BERT encoder capture key semantics, enabling accurate retrieval even when the user's input text differs from the textual description of the electricity data being retrieved.

[0028] Image pixel values ​​range from 0 to 255, which are relatively large. Directly inputting these values ​​into the model can lead to gradient explosion or computational instability. Normalizing them to the [0,1] interval can significantly improve numerical stability.

[0029] Preferably, the preprocessing for text includes: The text is used as input to the BERT encoder, and the output is a text vector. The text vector is then normalized, as expressed by the formula: ; In the formula, Indicates the first A text vector after normalizing power data. This represents the BERT encoder processing function. Indicates the first Text of electricity data, Indicates the first Text vector of electricity data The L2 norm; The specific preprocessing for the image is as follows: Extract the pixel values ​​of image pixels to construct a pixel vector, and then normalize the pixel vector, as expressed by the formula: ; In the formula, Indicates the first A pixel vector of power data Indicates the first A pixel vector obtained after normalizing power data.

[0030] CNN encoders are neural networks specifically designed for processing image data, possessing powerful feature extraction capabilities. Images of power equipment contain rich spatial information, such as local features like edges, textures, and shapes. CNNs effectively capture these features through convolutional kernels and pooling operations. Directly using raw pixels as input to the hidden layers of a multilayer perceptron not only results in high dimensionality but also contains a large amount of redundant information. Instead, CNNs, through multiple convolutions and nonlinear transformations, can extract meaningful semantic features from pixels, compressing high-dimensional image data into low-dimensional but information-rich feature vectors, which are then input into the hidden layers of the multilayer perceptron to extract the initial image feature vector. Leveraging the powerful fitting capabilities of the multilayer perceptron's nonlinear transformations, it can learn how to extract the most effective high-order abstract features for power data retrieval tasks from general features.

[0031] Preferably, the normalized text vector and pixel vector are input into a multilayer perceptron, and the initial text feature vector and initial image feature vector are obtained based on the hidden layer of the multilayer perceptron, as expressed by the formula: ; ; In the formula, Indicates the first The initial text feature vector of the power data, Represents the ReLU activation function. This represents the text weight matrix of the hidden layer. This represents the text bias vector of the hidden layer. Indicates the first The initial image feature vector of the power data, This represents the image weight matrix of the hidden layer. This represents the CNN encoder processing function. This represents the image bias vector of the hidden layer; Multilayer perceptron (MLP) feature extraction cannot capture the structural relationships between components of power equipment. The routing capsule algorithm explicitly models part-whole relationships through dynamic routing. Power data exhibits a clear hierarchical structure: conceptual hierarchies in text descriptions (e.g., "insulator-crack") and spatial relationships in images (e.g., "transformer-oil conservator") both require structured representation. The dynamic routing algorithm iteratively optimizes coupling coefficients, causing higher-level capsules to aggregate relevant lower-level features (initial capsules), with the length of the output vector representing the probability of feature presence. This representation better reflects the compositional characteristics of power equipment, providing rich structural information. Separately processing text and image features ensures that the characteristics of each modality are fully extracted, providing high-quality input for subsequent fusion.

[0032] The initial text feature vector is divided into equal-length segments. The initial text feature vectors of the group are considered as initial text capsules, expressed by the formula: ; In the formula, Indicates the first The first electricity data An initial text capsule, Indicates the initial number of capsules; The initial image feature vector is divided into equal-length segments. The initial image feature vectors of the group are considered as initial image capsules, expressed by the formula: ; In the formula, Indicates the first The first electricity data An initial image capsule; The prediction vectors for higher-level text capsules are obtained based on the initial text capsules, expressed by the formula: ; In the formula, Indicates the first The first electricity data The initial text capsule for the first The prediction vector of a high-level text capsule. Indicates the first The text transformation matrix of the first power data is used to transform the first... The initial text capsule is mapped to the first... A space for a high-level text capsule; The prediction vectors of higher-level image capsules are obtained based on the initial image capsule, expressed by the formula: ; In the formula, Indicates the first The first electricity data The initial image capsule for the first The prediction vector of a high-level image capsule, Indicates the first The image transformation matrix of the first power data is used to transform the first power data into an image transformation matrix. The initial image capsule is mapped to the first... The space of a high-level image capsule; Based on the prediction vectors of the high-level image capsule and the high-level text capsule, a dynamic routing mechanism is used to obtain the high-level image capsule and the high-level text capsule. The specific steps are as follows: S1. Initialize the number of image route pairs and text route pairs, for example, by... , Set to 0; S2, Set the number of iterations to... ,Will Set to 5; S3. Calculate the image coupling coefficient and text coupling coefficient based on the image routing logarithm and text routing logarithm, expressed by the formula: ; ; In the formula, Indicates the first The first electricity data The initial text capsule for the first The text coupling coefficient of a high-level text capsule. Indicates the first The first electricity data The initial image capsule for the first Image coupling coefficients of high-level image capsules, Indicates the first The first electricity data The initial text capsule for the first The number of text routes in a high-level text capsule Indicates the first The first electricity data The initial image capsule for the first Image routing logarithm of a high-level image capsule This represents the Softmax activation function; S4. Based on the image coupling coefficient, text coupling coefficient, prediction vector of the high-level image capsule, and prediction vector of the high-level text capsule, obtain the high-level image capsule and the high-level text capsule, expressed by the formula: ; ; In the formula, Indicates the first The first electricity data A high-level text capsule, This refers to the Squash function. Indicates the first The first electricity data A high-level image capsule; S5, if If the value is 0, proceed to step S6; otherwise... = -1, update the image route logarithm and text route logarithm based on the high-level text capsule and high-level image capsule, and perform step S3, expressed as a formula: ; ; S6. Construct text feature vectors and image feature vectors based on high-level text capsules and high-level image capsules, expressed by the following formula: ; ; In the formula, Indicates the number of high-level capsules. Indicates the first Image feature vectors of power data, Indicates the first Textual feature vectors of electricity data.

[0033] Preferably, a linear transformation method is used to map the text feature vectors to a common space, expressed by the formula: ; In the formula, Indicates the first Text projection vector of electricity data, Represents the text mapping weight matrix. Represents the text mapping bias vector; The image feature vectors are mapped to a common space (i.e., dimensionally consistent) using a linear transformation method, as expressed by the formula: ; In the formula, Indicates the first Image projection vector of each power data point, Represents the image mapping weight matrix. This represents the image mapping bias vector.

[0034] Simple weighted fusion cannot fully utilize the complementarity of multimodal information; the importance of different modalities in power data dynamically changes with the content. Text is more important in fault descriptions, while images are more crucial for visual inspections. A multi-head mechanism allows the additive attention model to focus on different feature subspaces in parallel, while a gating mechanism dynamically controls the information flow. The calculation of attention weights considers the correlation between features, and the gating vector further filters out noise. This design enhances the expressive power of the additive attention model, improves the quality of fused features, and prepares the data for index construction.

[0035] Preferably, multi-head projection is performed on the text projection vector and the image projection vector respectively to obtain multi-head text projection vector and multi-head image projection vector, which are expressed by the following formula: ; ; In the formula, Indicates the first The first electricity data Individual text projection vectors Indicates the first The first electricity data Individual image projection vector, Indicates the first Individual projection matrix; The multi-head text attention score and image attention score are obtained based on the multi-head text projection vector and multi-head image projection vector, expressed by the formula: ; ; In the formula, Indicates the first The first electricity data Individual text attention score, Indicates the first The first electricity data Individual image attention score, Indicates the first Transpose of the additive attention context vector of the size This represents the hyperbolic tangent activation function. Indicates the first The additive attention weight matrix for the size, Indicates the first Additive attention bias vector for size; Traditional additive attention requires dynamically selecting the most relevant part to the current query from multiple keys; however, since additive attention in this embodiment evaluates the global importance weight of a single modality feature vector, it lacks the concept of alignment between "key" and "query". , It's not the "key," but the feature itself that is being evaluated. It's not about "querying," but rather about the importance assessment criteria learned. and To prepare for importance assessment, a feature transformation network is constructed. Therefore, the network structure of additive attention is fine-tuned in this embodiment.

[0036] The multi-head text attention weight and multi-head image attention weight are obtained based on the multi-head text attention score and image attention score, expressed by the formula: ; ; In the formula, Indicates the first The first electricity data Individual text attention weights, Indicates the first The first electricity data Individual image attention weights; The multi-head fusion feature vector is obtained based on the multi-head text attention weight, multi-head image attention weight, multi-head text projection vector, and multi-head image projection vector, expressed by the formula: ; In the formula, Indicates the first The first electricity data Individual fusion feature vectors This represents the Sigmoid function. Indicates the first The gating vector of the size, Indicates the first Individual gated projection matrix This represents the element-wise multiplication operator; The multi-head fusion feature vectors are concatenated to obtain the fusion feature vector, which can be expressed by the formula: ; In the formula, Indicates the first A fusion feature vector of power data This indicates a splicing operation. Indicates the number of long positions. This indicates the output projection matrix.

[0037] The fused feature vectors are usually in a high-dimensional space. Directly performing nearest neighbor search in such a high-dimensional space faces the problem that the computational complexity increases exponentially with the dimension, making brute-force search infeasible on large-scale datasets. The idea behind this step is to... hash functions Mapping high-dimensional vectors to a low-dimensional hash signature space while preserving similarity relationships. Hash function. Sampling from the standard normal distribution ensures the randomness and uniformity of the projection, giving the hash function family ideal local sensitivity. The sign function discretizes the continuous projection values ​​into binary codes (+1 or -1). This binarization not only compresses storage space, but more importantly, it defines clear boundaries, dividing the feature space into... Different hash buckets.

[0038] If two vectors are similar (with a small angle) in a high-dimensional space, then their projections onto random directions are more likely to have the same sign. An independent hash function We can guarantee with high probability that similar vectors will be mapped to the same hash bucket. This reduces the retrieval time complexity from brute-force search. Reduced to approximate Bucket search plus The search within the bucket, among which It is the average size of the bucket. It refers to the quantity of electricity data. For large-scale electricity data, this acceleration effect is particularly significant.

[0039] Preferably, construct A number of different hash functions are expressed by the following formula: ; ; In the formula, Indicates the first A hash function, This indicates that the vector follows the mean vector. And the covariance matrix is ​​the identity matrix. normal distribution ; The hash value of the fused feature vector is obtained based on the fused feature vector and the hash function, expressed by the formula: ; In the formula, Indicates the first The fusion feature vector of the power data at the th ... Hash value under each hash function This represents the sign function, used to output +1 or -1, if... If the result of the dot product is greater than or equal to zero, then Output +1 otherwise output -1. Hash function Transpose of; based on A hash sequence is constructed from hash values ​​(for example, a hash sequence might be [+1,-1,-1,+1,...,-1], and the sequence length depends on...). The value can be adjusted according to the size of the power data (this embodiment does not impose specific limitations), and the hash sequence is used as the index of the power data.

[0040] For example, electricity data with indices [+1,-1,-1,+1,+1] will be placed in the first hash bucket.

[0041] Electricity data with indices [+1,-1,+1,+1,+1] will be placed in a different hash bucket.

[0042] Electricity data with index [-1,-1,-1,-1,-1] will be placed into the third hash bucket.

[0043] Power data includes text descriptions and high-resolution images, and storing them directly would consume a huge amount of storage space. For example, a 1024×1024 image of power equipment may require several MB of storage space, while text descriptions, although relatively small, can accumulate to a considerable amount in large-scale scenarios.

[0044] Preferably, the method further includes extracting common words, common phrases, and power terms from each power data text in the bucket to construct a local dictionary; The common vocabulary consists of words that appear multiple times in the hash bucket, such as: check, exception, run, etc. Common phrases are phrases that appear multiple times in the hash bucket, such as: insulation resistance, load switch, three-phase imbalance, etc. Electrical terminology includes: current, voltage, load, power, inrush current, flashover, ferroresonance, etc.

[0045] Replace the text of each power data with an integer sequence (if a word or phrase that cannot be found in the local dictionary appears when replacing the text, add the word or phrase to the local dictionary and set an index value before replacing it), where each element of the integer sequence is the corresponding index of the text content in the local dictionary; Entropy coding (such as Huffman coding or arithmetic coding) is applied to integer sequences to obtain compressed text data of each power data. Select an image from the images of each power data set as a reference image (e.g., determined by cluster centers, average images, or manual settings), and calculate the residual between each image and the reference image, expressed by the formula: ; In the formula, Indicates the number of hash buckets The residuals of the power data Indicates the number of hash buckets An image of electricity data, Indicates a reference image; The residuals are compressed using an image compression algorithm (such as compressing to WebP, AVIF, or JPEG-XL formats) to obtain image compressed data of each power data. The local dictionary, reference images, compressed text data, and compressed image data are stored.

[0046] When returning the search results, the compressed text data is decompressed, and the integer sequence is used to search the local dictionary to replace the integer sequence back with the text of the power data. The compressed image data is decompressed and stitched with a reference image to obtain an image of the power data; Images and text based on power data are combined to create a complete power data report, which is then returned to the user.

[0047] Data storage compression employs a differentiation strategy based on intra-bucket similarity. Electricity data text is highly domain-specific, containing a large amount of repetitive technical terms and descriptions. Traditional general-purpose compression algorithms cannot fully utilize this domain knowledge. By constructing a local dictionary for each hash bucket, we are essentially leveraging the local relevance of text within the bucket; similar data points not only have similar feature vectors, but their text descriptions also often share a large number of words and phrase structures. Entropy coding further utilizes the probability distribution of character occurrences, assigning short codes to high-frequency characters to achieve optimal compression.

[0048] Images of power equipment possess unique visual characteristics: images within the same bucket often depict similar equipment, from similar angles, or exhibiting the same type of fault. The core insight of residual coding is that storing the differences between images is more efficient than storing the complete image. The reference image captures the common features of images within the bucket, while the residual encodes only individual differences, which typically contain a large number of zero-values ​​or low-frequency components, making them highly compressible.

[0049] Preferably, the similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index queried by the user is calculated, expressed by the formula: ; In the formula, This indicates a query to fuse feature vectors. This indicates the index number in the bucket corresponding to the index queried by the user. Fusion feature vector of power data Similarity score, Represents the query fusion feature vector L2 norm, Represents the fused feature vector The L2 norm.

[0050] Example 2: This embodiment provides an electronic device that stores a computer program, which, when executed by a processor, implements the multimodal power data retrieval method as described in any embodiment of the present invention.

[0051] Example 3: This embodiment provides a computer-readable storage medium for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the multimodal power data retrieval method as described in any embodiment of the present invention.

[0052] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0053] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0054] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0055] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0056] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal power data retrieval method, characterized in that, Includes the following steps: The process involves acquiring and preprocessing power data, which includes text and its corresponding images. The text and images are input into a multilayer perceptron to obtain initial text feature vectors and initial image feature vectors. These initial text and image feature vectors are then used as input to a routing capsule algorithm, outputting text and image feature vectors. A linear transformation method is used to map the text and image feature vectors to a common space, yielding text and image projection vectors. These projection vectors are then used as input to an additive attention model, outputting a fused feature vector. This additive attention model incorporates a multi-head gating attention mechanism. A hash function is constructed, and an index for the power data is calculated based on the fused feature vector and the hash function. Power data with the same index are stored in buckets corresponding to the hash table. A user query is retrieved, and a query fused feature vector is obtained based on the query. The index of the user query is calculated based on the query fused feature vector. The similarity between the query fused feature vector and the fused feature vectors of each power data in the bucket corresponding to the index of the user query is calculated to obtain a similarity score. The R corresponding power data with the highest similarity scores are returned to the user as search results.

2. The multimodal power data retrieval method according to claim 1, characterized in that, The specific preprocessing for the text is as follows: The text is used as input to the BERT encoder, and the output is a text vector. The text vector is then normalized, as expressed by the formula: ； In the formula, Indicates the first A text vector after normalizing power data. This represents the BERT encoder processing function. Indicates the first Text of electricity data, Indicates the first Text vector of electricity data The L2 norm; The specific preprocessing for the image is as follows: Extract the pixel values ​​of image pixels to construct a pixel vector, and then normalize the pixel vector, as expressed by the formula: ； In the formula, Indicates the first A pixel vector of power data Indicates the first A pixel vector obtained after normalizing power data.

3. The multimodal power data retrieval method according to claim 2, characterized in that, The normalized text vector and pixel vector are input into a multilayer perceptron, and the initial text feature vector and initial image feature vector are obtained based on the hidden layer of the multilayer perceptron, as expressed by the formula: ； ； In the formula, Indicates the first The initial text feature vector of the power data, Represents the ReLU activation function. This represents the text weight matrix of the hidden layer. This represents the text bias vector of the hidden layer. Indicates the first The initial image feature vector of the power data, This represents the image weight matrix of the hidden layer. This represents the CNN encoder processing function. This represents the image bias vector of the hidden layer; The initial text feature vector is divided into equal-length segments. The initial text feature vectors of the group are used as initial text capsules; The initial image feature vector is divided into equal-length segments. The initial image feature vectors of the group are used, and each initial image feature vector is regarded as an initial image capsule; Obtain the prediction vector of the high-level text capsule based on the initial text capsule; The prediction vectors of high-level image capsules are obtained based on the initial image capsule; Based on the prediction vectors of the high-level image capsule and the high-level text capsule, a dynamic routing mechanism is used to obtain the high-level image capsule and the high-level text capsule. The specific steps are as follows: S1. Initialize the number of image route pairs and text route pairs; S2, Set the number of iterations to... ; S3. Calculate the image coupling coefficient and text coupling coefficient based on the image routing logarithm and text routing logarithm; S4. Obtain the high-level image capsule and the high-level text capsule based on the image coupling coefficient, the text coupling coefficient, the prediction vector of the high-level image capsule, and the prediction vector of the high-level text capsule. S5, if If the value is 0, proceed to step S6; otherwise... = -1, update the image route logarithm and text route logarithm based on the high-level text capsule and high-level image capsule, and perform step S3, expressed as a formula: ； ； In the formula, Indicates the first The first electricity data A high-level text capsule, Indicates the first The first electricity data A high-level image capsule, Indicates the first The first electricity data The initial text capsule for the first... The number of text routes in a high-level text capsule Indicates the first The first electricity data The initial image capsule for the first Image routing logarithm of a high-level image capsule Indicates the first The first electricity data The initial text capsule for the first The prediction vector of a high-level text capsule. Indicates the first The first electricity data The initial image capsule for the first Predicted vectors of a high-level image capsule; S6. Construct text feature vectors and image feature vectors based on high-level text capsules and high-level image capsules, expressed by the following formula: ； ； In the formula, Indicates the number of high-level capsules. Indicates the first Image feature vectors of power data, Indicates the first Textual feature vectors of electricity data.

4. The multimodal power data retrieval method according to claim 3, characterized in that, The text feature vectors are mapped to a common space using a linear transformation method, as expressed by the formula: ； In the formula, Indicates the first Text projection vector of electricity data, Represents the text mapping weight matrix. Represents the text mapping bias vector; The image feature vectors are mapped to a common space using a linear transformation method, as expressed by the formula: ； In the formula, Indicates the first Image projection vector of each power data point, Represents the image mapping weight matrix. This represents the image mapping bias vector.

5. The multimodal power data retrieval method according to claim 4, characterized in that, Perform multi-head projection on the text projection vector and the image projection vector respectively to obtain the multi-head text projection vector and the multi-head image projection vector, which can be expressed by the formula: ； ； In the formula, Indicates the first The first electricity data Individual text projection vectors Indicates the first The first electricity data Individual image projection vector, Indicates the first Individual projection matrix; The multi-head text attention score and image attention score are obtained based on the multi-head text projection vector and multi-head image projection vector, expressed by the formula: ； ； In the formula, Indicates the first The first electricity data Individual text attention score, Indicates the first The first electricity data Individual image attention score, Indicates the first The additive attention context vector of the head transpose, This represents the hyperbolic tangent activation function. Indicates the first The additive attention weight matrix for the size, Indicates the first Additive attention bias vector for size; The multi-head text attention weight and multi-head image attention weight are obtained based on the multi-head text attention score and image attention score, expressed by the formula: ； ； In the formula, Indicates the first The first electricity data Individual text attention weights Indicates the first The first electricity data Individual image attention weights; The multi-head fusion feature vector is obtained based on the multi-head text attention weight, multi-head image attention weight, multi-head text projection vector, and multi-head image projection vector, expressed by the formula: ； In the formula, Indicates the first The first electricity data Individual fusion feature vectors This represents the Sigmoid function. Indicates the first The gating vector of the size, Indicates the first Individual gated projection matrix This represents the element-wise multiplication operator; The multi-head fusion feature vectors are concatenated to obtain the fusion feature vector, which can be expressed by the formula: ； In the formula, Indicates the first A fusion feature vector of power data This indicates a splicing operation. Indicates the number of long positions. This indicates the output projection matrix.

6. The multimodal power data retrieval method according to claim 5, characterized in that, Build A number of different hash functions are expressed by the following formula: ； ； In the formula, Indicates the first A hash function, This indicates that the vector follows the mean vector. And the covariance matrix is ​​the identity matrix. normal distribution ; The hash value of the fused feature vector is obtained based on the fused feature vector and the hash function, expressed by the formula: ； In the formula, Indicates the first The fusion feature vector of the power data at the th ... Hash value under each hash function Represents a symbolic function. Hash function Transpose of; based on A hash sequence is constructed from the hash values, and the hash sequence is used as an index for the electricity data.

7. The multimodal power data retrieval method according to claim 1, characterized in that, The method also includes extracting common words, common phrases, and power terms from each power data text in the bucket to construct a local dictionary; Replace the text of each power data with an integer sequence, where each element of the integer sequence is the corresponding index of the text content in the local dictionary; Entropy coding is applied to integer sequences to obtain compressed text data of each power data point; Select an image from the images of each power data as a reference image, calculate the residual between each image and the reference image, and compress the residual using an image compression algorithm to obtain the image compressed data of each power data. The local dictionary, reference images, compressed text data, and compressed image data are stored.

8. The multimodal power data retrieval method according to claim 1, characterized in that, The similarity between the query fusion feature vector and the fusion feature vector of each power data in the bucket corresponding to the index queried by the user is calculated, expressed by the formula: ； In the formula, This indicates a query to fuse feature vectors. This indicates the index number in the bucket corresponding to the index queried by the user. Fusion feature vector of power data Similarity score, Represents the query fusion feature vector L2 norm, Represents the fused feature vector The L2 norm.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal power data retrieval method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the multimodal power data retrieval method as described in any one of claims 1 to 8.

Images